Model Routing

Every request that reaches your LLM stack passes through one decision before a token gets generated: which model actually answers it. Get that decision right and simple queries stop paying frontier prices while hard ones still get the capability they need; get it wrong and you either overspend by default or quietly degrade answers to save money. Model routing is the request-by-request layer of the LLMOps production stack that makes that call, which is why it sits at the center of the theme’s core control loop rather than at either edge.

Reading model routing in the right order

Start with what model routing is and how LLM gateways direct requests by cost, latency, and quality — it lays out the decision layer the rest of this topic assumes you already understand. Read the cost metrics, latency budgets, and fallback logic to know before implementing model routing next: it is the prerequisite check most teams skip, and skipping it is exactly what turns a routing project into a production incident.

Once the concepts are set, the guide to cost optimization, fallback, and latency control with OpenRouter and Portkey gets a routing policy live on managed infrastructure fastest; the step-by-step guide to a self-hosted router with LiteLLM, Bifrost, and Braintrust is the deeper build for teams that want the tiering policy under their own control. For the state of the market, how real teams are routing LLM traffic in 2026 with Braintrust, OpenRouter, and LiteLLM separates what’s shipped from what’s still a claim, and the case for treating routing as an accountability question, not only an infrastructure one closes the ladder with the question the other four never ask: who is responsible when the router decides.

How model routing differs from cost management, registries, and A/B tests

Three neighbouring topics get folded into model routing, and each folding hides a different mistake.

• Model routing is not the whole of LLM cost management. Routing is the single biggest lever inside that discipline, but caching, batching, and prompt-length budgets are separate levers that solve different waste — a team that only routes still overpays on repeated context.
• Model routing is not a model registry. Routing decides which model answers this request, in milliseconds, based on a live policy; the registry answers a slower question — which model artifact and version is actually deployed, so a rollback has something to point at. A router with no registry behind it can fail over to a model nobody approved.
• Model routing is not A/B testing for LLMs. A router optimizes every request against a fixed policy; an A/B test deliberately serves two policies to different slices of traffic to find out which one should become the fixed policy. Routing runs the decision, testing produces it.

Questions about routing models in production

Q: Do I need model routing if I only ever call one frontier model? A: Only if every request is equally hard — most workloads aren’t. Routing lets simple queries hit a cheaper model while complex ones still reach the frontier tier, the request-by-request framework the OpenRouter and Portkey guide builds instead of a single model-wide choice.

Q: Can a routing decision get made without anyone knowing which model actually answered? A: Yes — most gateways don’t log or surface the winning model by default, so a downgrade made to save cost is invisible unless you build the check in yourself. Black-Box Routing examines what that invisibility costs once a wrong answer reaches a user.

Q: Should I build my own router or use a managed one like OpenRouter? A: A managed router gets fallback and multi-provider coverage live fastest; a self-hosted stack like LiteLLM or Bifrost costs more setup time but keeps the tiering policy entirely under your control. The self-hosted router guide walks the trade-off in build order.

Q: Why does a routing setup sometimes cost more than sending everything to one model? A: Usually because the tiers were never mapped before the config was written — routing without a cost and latency budget per tier just adds infrastructure on top of the same spend. The prerequisites read covers the budgeting step teams skip.

Q: Does model routing stop mattering once providers converge on price? A: No — routing optimizes latency and quality fit as much as cost, and what real teams are doing with Braintrust, OpenRouter, and LiteLLM in 2026 shows routing has become load-bearing production infrastructure rather than a pure cost hack, even as price gaps close.

Part of the LLMOps production theme · closest neighbour: LLM cost management.