LLM Gateway Architecture: Rate Limiting, Virtual Keys, Semantic Caching, and Observability Components

A traditional API Gateway counts requests and blocks the excess. An LLM Gateway solves something structurally different: the token cost of a single request can vary by a factor of 20 depending on what the user asks, and you don’t know the actual cost until the response has already been generated. This isn’t a scaling problem. It’s a measurement problem that manifests as a control problem.

The four-component architecture that emerges from that constraint — rate limiting, Virtual Keys, Semantic Caching, and observability — exists because each component addresses a failure mode that only appears when you route shared LLM traffic through a system that must remain accountable at every layer.

The Measurement Problem at the Center of Every Gateway

Most engineers reach for Rate Limiting as if it were a standard circuit breaker: count requests per second, reject above the threshold. That model breaks immediately for LLM workloads, because the thing you want to limit — token consumption — is unknown at the moment the request arrives. You’re trying to enforce a budget against a line item that hasn’t been calculated yet.

What are the core components of an LLM gateway?

An LLM gateway assembles four functional layers that don’t exist in traditional API proxies, each addressing a distinct failure mode of LLM traffic at scale.

Rate limiting operates across four dimensions: RPM (requests per minute), TPM (tokens per minute), cost-based caps (daily or monthly USD limits), and time-window composites that combine short and long measurement horizons. The LLM-specific complication is that demand fluctuates 5×–20× per request depending on query complexity (Portkey Blog). A gateway enforcing only RPM provides false assurance — a single long-context request can consume the token budget that a hundred shorter requests would have shared. The guarantee on the dashboard doesn’t reflect the guarantee in the ledger.

Virtual keys are short-lived, scoped, revocable tokens that sit in front of provider credentials. The gateway issues virtual keys bounded by spend, rate limit parameters, or parallel request counts. In LiteLLM’s implementation, five configurable limits apply per key: max_budget (USD), tpm_limit, rpm_limit, max_parallel_requests, and duration (expiry); spend is tracked per-key, per-user, and per-team against a PostgreSQL backend (LiteLLM Docs). A leaked virtual key causes no provider rotation — it expires or gets revoked at the gateway layer, and the blast radius is bounded by whatever scope that key carried.

Semantic caching recognizes that production LLM traffic contains large fractions of near-identical queries — different users, same intent. The gateway embeds the incoming prompt, runs a cosine similarity check against a vector store of prior responses, and returns the cached result when similarity exceeds the configured threshold. No model call happens. The provider never sees the request, and no tokens are consumed.

Observability is the accounting layer that makes the other three auditable. Portkey logs 40+ data points per request — cost, latency, token counts, cache status, and guardrail results — feeding into Prometheus and OpenTelemetry pipelines that export to Grafana, Datadog, and similar sinks (Portkey Docs). Without this layer, cost attribution across teams is guesswork. A $300 overrun becomes a mystery instead of a logged request with a traceable origin.

Token cost is unknowable at request time. This is why production gateways implement two-phase token checking: limits are evaluated when the request arrives using estimated input token counts, then re-evaluated when the response completes against actual totals. The gap between estimation and reality is where budget overruns hide — and where naive rate-limiting implementations consistently fail.

Why RPM Rate Limits Are Not Enough

The two-phase enforcement model exposes a structural tension. If you enforce hard token limits at request time using estimates, you’ll occasionally block requests that would have stayed inside budget. If you enforce only at response time, you’ve already paid the inference cost before discovering the violation.

Production gateways resolve this by combining all four rate-limiting dimensions with active fallback routing. When a provider returns HTTP 429 or 502, the gateway doesn’t fail the request — it routes to an alternative via one of five named fallback strategies: provider-rotation, model-downgrade, retry-then-fallback, cache-on-failure, and manual-route (Portkey Blog). The circuit breaker monitors failed request counts and failure rates, removing a degraded provider from the routing pool for a cooldown period. The combination of multi-dimensional rate enforcement plus fallback routing is what separates a functional LLM gateway from a simple proxy: a simple proxy fails loudly; a gateway degrades gracefully while keeping the cost ledger intact.

The Cache That Operates on Meaning

Exact-match caching is nearly useless for LLM workloads. “Summarize this document” and “give me a summary of this document” are identical in intent — byte-level comparison misses them entirely. The solution is to compare not the text, but the geometric position of the text in a vector space.

What is semantic caching in an LLM gateway and how does it reduce redundant API costs?

The mechanism: embed the incoming prompt into a vector; run cosine similarity against a store of cached prompt-response pairs; return the cached response if similarity exceeds the threshold. The key question isn’t whether the mechanism works — it does — but what happens when the threshold is wrong.

At 0.95, the cache behaves as near-exact-match and misses most savings. At 0.80, the cache returns responses to queries that are semantically adjacent but not equivalent — which introduces errors that look indistinguishable from model hallucinations. Threshold 0.85 is not arbitrary — it’s where production RAG workloads find the balance between hit rate and response fidelity, with the recommended calibration range at 0.85–0.90 for RAG pipelines (Maxim AI). Higher thresholds favor precision; lower thresholds favor savings. The correct value depends on the tolerance for approximate answers in your specific application.

Savings are real but workload-dependent. Production deployments report 20–73% reduction in token costs depending on query repetition rate (The Agent Times). In a study of 63,796 real chatbot queries reported via n1n.ai, 86% cost reduction and 88% latency improvement were measured at an optimal threshold. The caveat applies: this figure comes from a secondary source, not a directly accessible AWS publication, and should be treated as directional. Semantic caching savings vary dramatically by workload — a system fielding unique, open-ended creative queries will see near-zero benefit; a customer support bot with high query repetition will see savings in the upper range. The distribution of your production queries is the parameter that determines which end of that range you’re on.

LiteLLM’s semantic caching layer supports Valkey (with the valkey-search module) or Qdrant as backends. AWS ElastiCache requires Valkey 8.2+ for vector search support (LiteLLM Docs). The backend choice affects both tail latency and operational complexity — Qdrant carries its own maintenance surface; Valkey-on-ElastiCache constrains version selection but reduces operational overhead.

Not exact strings. Meaning.

What Breaks — and What to Verify Before Production

The four-component architecture works when the gateway itself is correct and current. Each component has a distinct failure mode, and two of them fail silently.

What do developers need to understand before deploying an LLM gateway in production?

Virtual keys are not a security mechanism in isolation — they are blast radius control. Blast radius is what keys actually control. A leaked virtual key scoped to one team’s monthly budget causes limited, recoverable damage: you revoke the key, issue a new one, and the incident is bounded. A leaked provider API key exposes every team, every model, and every downstream service behind it — the incident is unbounded until you rotate credentials across every integration. The four-tier organizational hierarchy (Business Unit → Team → Virtual Key → Provider Config) isn’t bureaucratic overhead; it’s the mechanism that determines how much an adversary can do with what they find.

LLM Observability requires deliberate configuration to be useful. Raw request logs aren’t the same as structured LLM traces. OpenTelemetry’s GenAI semantic conventions capture model name, version, temperature, and token counts at the trace level; cost and request duration at the metric level; prompt text in span events for large payloads. Without that structure, latency spikes have no debugging context and cost overruns have no attribution path — you know something went wrong, but not where, and not why.

The gateway itself is a dependency with a security surface. If you’re running LiteLLM, version selection is not optional:

Security & compatibility notes:

• LiteLLM Supply Chain (March 2026): Versions 1.82.7 and 1.82.8 were compromised — malicious code exfiltrated cloud credentials, SSH keys, and Kubernetes secrets. Remove these versions immediately. Minimum safe version: v1.83.10-stable.
• CVE-2026-42208 (CVSS 9.3): Pre-auth SQL injection in the API key verification pathway exposes all virtual keys and provider credentials. Affects 1.81.16–1.83.6; fixed in 1.83.7; exploited in the wild within 26 hours of public disclosure. (Penligent)
• CVE-2026-42271 (CISA KEV, added June 9, 2026): Command injection via MCP server endpoints enables unauthenticated remote code execution when chained with CVE-2026-48710. Affects 1.74.2–1.83.6; fixed in 1.83.7. Active exploitation confirmed. (CybelAngel)

When it breaks: Semantic caching fails silently when the threshold is calibrated against a development dataset that doesn’t represent production query distribution — the cache returns confident, wrong responses with no error signal, because from the gateway’s perspective a cache hit looks identical to a correct one. Rate limiting fails with false precision when token estimation diverges from actual output token counts; gateways that enforce limits only at request time, not at response time, systematically miss budget overruns on variable-length completions — and the first signal is often a billing alert, not a gateway error.

The Data Says

An LLM gateway is the control plane for AI infrastructure: it makes token consumption measurable before it becomes unmanageable, credentials revocable before they become incidents, identical queries non-redundant before they become unnecessary costs, and every request auditable before it becomes an unexplained bill. Rate limiting requires token-aware logic across all four dimensions — request counts alone are a proxy that breaks under realistic workload variance. Semantic caching works where query distributions are repetitive, but threshold calibration determines whether savings come with fidelity intact or silently degraded. The gateway’s own security posture is a prerequisite, not an afterthought.