LLM Cost Metrics, Latency Budgets, and Fallback Logic: Before Implementing Model Routing

Model Routing looks straightforward on paper: send cheap requests to a cheap model, expensive requests to a capable one, handle failures by retrying on a backup. Teams build this mental model, add a routing layer, and watch the results diverge from the prediction.

The divergence isn’t random. It traces to three distinct variables — token cost structure, latency budget geometry, and fallback state mechanics — that interact in ways the simple routing mental model doesn’t account for. Miss any one of them and the logic will seem to work until it doesn’t; the failures will arrive in production, where diagnosing them is considerably more expensive.

The Pricing Asymmetry That Reshapes Every Cost Estimate

LLM APIs charge in a unit that looks symmetric but isn’t. Both input and output are priced per million tokens, billed at different rates for a structural reason: each output token requires a full forward pass through the model, while input tokens are processed in parallel during prefill.

Not a billing quirk. A hardware cost.

Output tokens cost two to five times more than input tokens — which means a prompt-heavy workflow (large system prompt, short responses) has an entirely different cost profile than a generation-heavy workflow, even at identical total token volume (Silicon Data).

What concepts do you need to understand before setting up model routing?

Three foundations. Two are pricing concepts, one is a timing concept — and they interact in ways that a cost-only model misses entirely.

Token cost structure. The spread across providers is wide: as of 2026, input pricing spans from roughly $0.10/MTok on the budget end to $30/MTok for frontier reasoning models, with output rates carrying their multiplier across that entire range. Before routing can optimize cost, it needs to know the expected output/input ratio per workload. A routing rule that minimizes input cost on a document generation task may accidentally maximize output cost.

Two mechanisms exist at the gateway level for reducing effective per-call cost. Prompt caching can cut costs up to 90% for workloads with stable system prompts; batch processing can cut them up to 50% for non-streaming tasks — together, they can reduce effective cost to roughly a quarter of standard rates (PEC Collective). Neither applies universally: streaming responses can’t be batched, and caching requires prompt stability that many dynamic applications don’t have.

Latency SLOs by workload. Latency in LLM inference is not a single number. Three metrics describe different phases:

• TTFT ( Time To First Token): wall-clock time from request arrival to the first output token. This is perceived responsiveness — what the user feels as “the model is starting.”
• ITL (Inter-Token Latency): the gap between consecutive tokens during streaming decode. This determines whether streamed output feels smooth or stuttered.
• TPOT (Time Per Output Token): average decode speed across the full response. A throughput metric for capacity planning; not a direct user-facing SLO target.

The appropriate TTFT target depends on the application type. Per P99 SLO targets documented in 2026 inference engineering guidance (Spheron Blog):

Voice agents need the tightest budget: text-to-speech pipelines add another 100–200ms after the first token arrives, so the LLM must respond faster than text-only delivery requires. RAG-augmented chat gets more slack because retrieval latency already compounds the prefill — the user expects a slightly longer wait, and that expectation is built into the SLO.

A LLM Gateway that routes a voice request to a cheaper-but-slower provider can satisfy the cost budget while violating the user experience contract. The routing rule needs the SLO table before it can determine which providers are even eligible for a given workload.

Rate limiting primitives. Rate Limiting in LLM APIs operates on two dimensions simultaneously. RPM (Requests Per Minute) controls concurrency and protects providers from request floods; TPM (Tokens Per Minute) maps to GPU compute consumption and is the primary driver of 429 errors at scale. A request that passes the RPM check can still fail on TPM if the prompt is unexpectedly large — which means rate limit errors in production are often prompt-size errors, not request-count errors.

Virtual Keys are gateway-managed credentials that map to underlying provider keys stored in a vault. They separate rate limit visibility from credential management: teams get per-key RPM and TPM caps adjustable without touching provider configuration or changing application code.

The API Gateway pattern handles routing at the HTTP level; the LLM gateway extends this with token-aware semantics. An API gateway that doesn’t understand TPM will forward every request until the provider rate-limits the entire application — at which point the error surfaces at the application layer, not the routing layer, and is harder to trace.

The State Machine Inside Every Fallback

Fallback logic is where routing gets formally interesting. The naive model — “if primary fails, try secondary” — sounds complete. It isn’t. A provider can fail partially: returning errors only on large requests, degrading latency without dropping connections, or failing for specific model versions while others remain healthy. A binary primary/secondary model doesn’t handle gradual degradation.

The mechanism that does is the circuit breaker — a pattern borrowed from electrical engineering, where the name is genuinely literal: the behavior is identical.

What are the technical failure modes of model routing in production?

Circuit breaker state transitions. A circuit breaker operates in three states (Maxim AI):

• CLOSED: Normal operation. Requests pass through; the breaker tracks error rate over a rolling window.
• OPEN: Error threshold exceeded. All requests fast-fail or route to fallback immediately — no retry, no wait. The circuit is interrupted to protect downstream systems.
• HALF-OPEN: After a cooldown period, the breaker allows one probe request through. Success closes the circuit; failure reopens it.

Community-calibrated thresholds: 5 consecutive failures to trip to OPEN; 60-second cooldown before HALF-OPEN; alert threshold at a greater-than-5% error rate over the rolling window; critical escalation at greater-than-15% (Maxim AI). A threshold set too low causes constant tripping under normal variance (thrashing); too high leaves a failing provider in rotation longer than the outage warrants. December 2025 incident data from IsDown recorded 47 incidents across major AI systems in a single month — with multiple providers experiencing partial outages within overlapping windows (Maxim AI). A routing layer without circuit breakers distributes requests evenly to a partially-failed provider; one with circuit breakers detects degradation and stops.

Fallback type taxonomy. The Fallback Strategy itself has distinct failure modes depending on what caused the primary to fail. LiteLLM exposes three separate fallback parameters that distinguish between them (LiteLLM Docs):

fallbacks                  # general provider/model failure
context_window_fallbacks   # prompt exceeds model context limit
content_policy_fallbacks   # content policy rejection

The distinction matters for routing correctness. A context window overflow needs a different model or a shorter prompt — routing it to a secondary provider with the same context size will fail again. A content policy rejection needs a model with different policies or a prompt rewrite — retrying the same model class will also fail again. Routing all three failure types through a single generic fallback chain means each will silently retry in ways that cannot succeed.

Openrouter handles fallbacks through allow_fallbacks: true (the default), progressing from primary to secondary to tertiary providers on failure and billing only for the successful run (OpenRouter Docs). The preferred_max_latency parameter accepts a number or a percentile object (p50/p75/p90/p99), computed over a rolling 5-minute window, to deprioritize slow providers without excluding them entirely. Routing is price-based by default via inverse-square weighting that heavily favors cheaper providers; latency constraints layer on top as a filter.

Portkey (acquired by Palo Alto Networks, completed May 29, 2026; integrating into Prisma AIRS for enterprise security) routes to over 1,600 LLMs (Portkey). Its standalone commercial pricing tiers remain available as of mid-2026, but the enterprise integration path will likely change as the acquisition completes — verify current pricing and positioning at the source (Palo Alto Networks).

The shared-outage cascade. Fallback chains assume providers fail independently. When the primary and secondary share upstream infrastructure, belong to the same cloud provider’s GPU cluster, or both respond to the same regional capacity event, the fallback chain fails serially — one provider at a time — until the chain is exhausted. Circuit breakers protect against single-provider degradation; they offer no protection when the failure is structural and shared.

LiteLLM, a widely used open-source routing layer, had two significant security incidents in early 2026: a supply chain compromise that allowed credential theft, and a critical SQL injection vulnerability exploited within hours of public disclosure. Both were patched, but the incidents illustrate that the security posture of the routing layer itself is part of the threat model — particularly for self-hosted deployments.

Security & compatibility notes:

• LiteLLM supply chain compromise (March 24, 2026): Versions v1.82.7 and v1.82.8 contained a credential stealer that exfiltrated environment variables, SSH keys, and cloud credentials. Fix: Pin to ≥v1.83.0. The Docker image ghcr.io/berriai/litellm was not affected (LiteLLM Blog).
• LiteLLM CVE-2026-42208 (CVSS 9.3): Critical SQL injection, actively exploited within 26 hours of disclosure. Fixed in v1.83.7-stable (April 19, 2026). Fix: Pin to ≥v1.83.7-stable (The Hacker News).

What the Three Variables Predict About Your Configuration

Token cost asymmetry, SLO budgets, and circuit breaker state transitions aren’t independent concepts — each one predicts a specific category of routing failure.

If your output/input token ratio is higher than modeled, routing to the cheapest model by input price will consistently undershoot cost targets when output volume rises. The error doesn’t appear during development (where prompts are short and outputs are limited) — it appears in production under realistic load.

If your TTFT P99 exceeds the SLO budget at the tail, routing to a model with a lower average latency doesn’t solve the problem. P99 is a tail behavior, not an average; a model with a lower median latency can have a worse P99 under concurrent load. The SLO needs to be specified at the percentile level before a routing rule can be validated against it.

If the circuit breaker cooldown is shorter than the provider’s typical recovery time, the HALF-OPEN probe will fail repeatedly — extending the outage in the routing layer beyond the outage at the provider.

Rule of thumb: establish TTFT P99 SLO targets by workload type before selecting model tiers — the SLO narrows the eligible provider set, cost preference selects within that constraint. Reversing the order produces routing rules that are technically cheaper and behaviorally wrong.

When it breaks: fallback chains fail silently when all providers in the chain share the same infrastructure failure. Circuit breakers prevent request floods against a single degraded provider; they do not protect against correlated outages that propagate through every tier of the fallback sequence simultaneously.

The Data Says

Token pricing asymmetry (output 2–5x input cost), workload-specific TTFT SLOs (100ms to 400ms at P99), and circuit breaker state machines (CLOSED → OPEN → HALF-OPEN) are three independent variables that determine whether routing logic behaves predictably in production. Understanding them separately is necessary. Seeing how they constrain each other — SLO narrows providers, cost selects within them, circuit breakers protect the chain at runtime — is what makes routing decisions defensible rather than just plausible.