This theme is curated by our AI council — see how it works.
LLMOps is the discipline of running large language models in production: routing every request to the right model, containing token costs, surviving provider outages, and proving — with traces, logs, and controlled experiments — that the system still behaves the way it did yesterday. The theme spans the full operational stack, from the gateway a request enters to the audit trail it leaves behind. This page maps that stack: what to read first, what each layer is for, and where the layers get confused with each other.
A prototype that calls one model endpoint is a weekend project; the same feature in production is a distributed system with a nondeterministic dependency you don’t control, billed by the token. Providers rate-limit you, deprecate models under you, and fail in ways your existing retry logic makes worse. For a developer, LLMOps reframes “AI in production” into familiar engineering territory — traffic management, resilience patterns, cost budgets, and telemetry — with one crucial difference: the failure you must catch most often is not an error but a silent change in output quality.
Two concepts carry everything else in this theme, and they mirror a pattern every backend engineer already knows: instrument what you run, and put a control point in front of what you don’t.
Instrumentation comes first, because in an LLM system you cannot debug what you didn’t record. LLM observability adapts distributed tracing to prompt chains — how span-based tracing captures every step of a prompt chain is the best first read in the theme, and the MLOps prerequisites no observability tool can fix is its honest companion on what tooling cannot save you from. When you are ready to pick a platform, the Langfuse vs LangSmith vs Arize Phoenix comparison shows how production teams actually split the market.
The control point is the LLM gateway: a proxy layer between your applications and every model provider, owning routing, rate limits, authentication, and unified logging in one place. How request routing, fallback, and unified auth work under the hood explains the architecture; the counterweight read, gateway latency and single points of failure, tells you the price of putting a chokepoint in front of all your AI traffic — worth reading before you commit to one, not after.
With these two in place — a gateway you control and traces you can query — every layer in the next tier has somewhere to live and a way to be measured.
This tier is where the day-to-day production decisions live, and each of its four layers plugs into the foundations: the gateway enforces them, observability verifies them.
Model routing decides which model serves which request — by cost, latency, or required quality — so simple queries stop paying frontier-model prices. What model routing is and how gateways direct requests is the orientation read; cost metrics, latency budgets, and fallback logic covers what you must measure before writing your first routing rule. Routing is also the biggest lever in LLM cost management, the discipline of keeping token spend proportional to value — start with how token-based pricing scales in production, then move to the hands-on guide to cutting costs with routing, prompt caching, and batch APIs.
Resilience is its own layer because provider APIs fail in correlated, rate-limited ways that generic retry libraries handle badly. LLM fallback and retry patterns cover exponential backoff, circuit breakers, and provider failover — read how backoff prevents API cascade failures first, then what makes LLM retry logic break for the thundering-herd and timeout traps; the multi-provider failover guide with LiteLLM, Portkey, and Tenacity turns the patterns into working code.
The quietest layer is the model registry — the versioned catalog recording which model artifact, weights, and configuration are actually deployed, so rollbacks and audits have something to point at. How MLOps artifact stores work is the entry point, and the MLflow vs W&B vs SageMaker vs DVC comparison is the decision read when your team picks one.
Run these four and you have a system that routes sensibly, spends predictably, survives outages, and knows what version of itself is running. What it doesn’t yet have is proof — that it holds under load, that changes actually improve it, and that its records would satisfy an auditor. That is the next tier.
This tier separates a system that works from a system you can defend — to an SRE, a compliance officer, or your own postmortem.
LLM load testing replaces classic requests-per-second thinking with token-aware metrics: time-to-first-token, tokens per second, p99 latency under concurrent streams. How TTFT, tokens-per-second, and p99 latency are measured is the orientation read; the vLLM benchmark suite and GenAI-Perf guide gets you a reproducible benchmark harness.
Load testing tells you the system is fast; A/B testing for LLMs tells you a change made it better. It is harder than testing a button color, because output quality has no click-through rate — sample size, statistical power, and why measuring LLM quality is hard explains the statistical trap, and LLM-as-judge vs human raters weighs the two ways to score variants at scale.
LLM logging and auditing extends observability’s traces into compliance territory: prompt/response capture, PII redaction, cost attribution, and audit trails that survive a regulator’s questions. How production systems capture prompts, costs, and traces is the entry point; PII redaction, trace sampling, and the limits of logging at scale covers what breaks when volume grows.
The last topic operates inside the request itself. Context window management decides what fits into the model’s token budget — conversation summaries, sliding windows, priority-based packing — and it is where latency, cost, and quality collide in a single decision. The token limits that shape every LLM interaction explains the constraint, and how real products handle context limits shows how teams resolve it now that 10M-token windows compete with compression.
The three traffic-control concepts in this theme — gateway, routing, fallback — get used interchangeably, and the confusion produces real architecture mistakes. They answer different questions:
| LLM gateway | Model routing | Fallback and retry | |
|---|---|---|---|
| Question it answers | Where do all requests pass through | Which model should serve this request | What happens when the call fails |
| When it acts | Every request, always | Per request, before the call | Only on failure or timeout |
| Optimises for | Control, auth, unified logging | Cost, latency, quality fit | Availability |
| Failure it prevents | Key sprawl, unmanaged providers | Overpaying frontier models for trivial queries | One provider outage taking your feature down |
Three more pairs blur just as often:
Q: Where should I start with LLMOps as a backend developer? A: Instrument before you optimize: without traces you cannot see which layer needs work. Start with span-based tracing for prompt chains, then put a gateway in front of your providers — the rest of the stack plugs into those two.
Q: Do I need a full gateway, or is a retry library enough? A: A retry library saves a single service from transient errors; a gateway pays off once multiple apps, providers, or teams share LLM traffic and need one place for auth, limits, and failover. The LiteLLM and Portkey deployment guide shows what a gateway adds beyond retries.
Q: Should I cut costs with model routing or with caching? A: They attack different waste: routing stops you overpaying for simple queries, caching stops you paying twice for repeated context. Route first if traffic is varied, cache first if prompts repeat. Model tiering vs prompt caching gives the decision rules.
Q: Why did output quality drop when nothing was deployed? A: Usually a change you didn’t make: a provider updated a model, a fallback silently switched routes, or accumulated context crossed a truncation threshold. Silent model switching and opaque fallbacks covers the accountability gap; regression alerts on your traces are how you catch it early.
Q: What do I need in place before running LLM A/B tests? A: Stable observability, cost attribution per variant, and an evaluation method you trust — statistical power over LLM outputs is far harder to reach than over click-through rates. How controlled experiments evaluate prompt and model variants lists the prerequisites.
Q: How do I plan capacity for an LLM feature before launch? A: Load test with token-aware metrics, not requests per second — streaming makes TTFT and tokens-per-second the numbers users actually feel. Load testing architecture: key components and metrics covers what to measure and which bottlenecks appear only under concurrency.
Coming from software engineering? Bridge articles map this theme onto what you already know — which of your instincts still apply, which quietly break, and where to dive deeper once you're oriented.
A/B testing for LLMs runs controlled experiments that compare two or more prompt versions, model configurations, or …
Context window management encompasses the techniques used to fit relevant information within an LLM's fixed token limit …
LLM Cost Management covers the strategies and tooling used to control operational expenses in LLM-powered systems. It …
LLM fallback and retry patterns are resilience strategies that keep AI-powered applications running when a model …
An LLM Gateway is an API management layer that sits between your application and one or more LLM providers. It handles …
LLM load testing measures how an AI system performs under realistic traffic — tracking tokens-per-second output, …
LLM Logging and Auditing covers production practices for capturing, storing, and analyzing prompt/response pairs in LLM …
LLM Observability is the practice of monitoring, tracing, and debugging large language model applications in production. …
A model registry is the often-overlooked bridge between training and production: it enforces that every deployed model …
Model routing is the practice of dynamically directing each LLM request to the most appropriate model based on query …
MONA's articles build your mental model — how things work, why they work that way, and what intuition to develop.
Updated Jul 15, 2026
Concepts covered
LLM observability applies distributed tracing to AI chains. Five span types (LLM, Task, Workflow, Tool, Agent) give each step a measurable, queryable identity.
LLM gateway: middleware routing app requests across AI providers, with fallback, unified auth, and token tracking. Semantic caching cuts costs 20–40%.
LLM gateways add 2–240ms overhead. Latency is rarely the bottleneck — single points of failure and security vulnerabilities are the real engineering tradeoffs.
LLM gateways govern AI traffic via four components: rate limiting (RPM/TPM/budget), scoped virtual keys, semantic caching, and per-request observability.
LLM observability needs baselines before the first trace. No tool prevents hallucinations or non-deterministic output — only detects them after the fact.
Model routing is a decision layer between your app and multiple LLMs that directs each request to the optimal model by cost, latency, or task complexity.
LLM cost management controls token-based API costs at production scale. Output tokens cost 2–6× more than input; doubling context quadruples attention compute.
A model registry tracks ML model versions, weights, and deployment configs. MLflow 3 replaced fixed stages with named aliases such as champion.
LLM fallback and retry patterns reroute failed calls to backup models. Exponential backoff with jitter stops retry storms from cascading into outages.
A model registry tracks versions. It does not enforce input contracts, cross-team isolation, or output schema. At scale, those gaps compound.
LLM retry logic breaks in a specific way: pure exponential backoff synchronizes all clients, creating a thundering herd that sustains the overload.
Model routing rests on three prerequisites: token cost asymmetry, P99 latency SLOs by workload, and circuit breaker fallback logic for provider failures.
LLM API cost optimization requires understanding KV cache mechanics, input/output token asymmetry, and prompt caching discounts — not just switching models.
LLM load testing measures TTFT, TPS, and p99 latency under concurrent load. KV-cache GPU memory—not CPU threads—is the bottleneck REST API benchmarks miss.
LLM logging captures prompts, tokens, latency, and costs per call. OTel GenAI trace hierarchies let teams reconstruct what a production system did — and why.
Context window management controls what an LLM sees at inference time. Past 15K words, latency can spike sevenfold — how production systems handle it.
A/B testing for LLMs evaluates prompt and model variants by splitting production traffic and measuring quality, latency, and token cost in parallel.
LLM load tests fail silently on streaming gaps, KV cold starts, and tool metric inconsistencies—three flaws that make staging latency numbers unreliable.
LLM output quality has up to 34% within-model variance — enough to invalidate a standard A/B test. The sample size math and eval design that holds.
Production LLMs use sliding windows, KV caching, and summarization to manage context. GPU memory — not token limits — defines the real ceiling in 2026.
LLM logging at scale pits capture fidelity against PII risk and cost. Tail sampling, PII redaction, and immutable audit logs are the tradeoffs teams navigate.
LLM load testing requires metrics standard tools miss. TTFT, inter-token latency, and goodput determine whether an endpoint handles real concurrent load.
