Model Routing

What model routing is

Token-based LLM pricing means every request hits your bill. A customer support auto-reply and a multi-step code analysis both consume tokens, but they don’t require the same model. Model routing solves this mismatch: it acts as a decision layer between your application and the model pool, selecting the right model for each request instead of sending everything to the most capable — and most expensive — option by default.

Think of it as call routing for AI. Just as a phone system forwards simple questions to an automated menu and escalates complex issues to a human specialist, model routing forwards straightforward requests to a fast, affordable model and escalates demanding tasks to a more capable one.

The decision can be based on multiple signals. Common routing criteria include estimated token count (shorter inputs often indicate simpler tasks), task category (classification, summarization, and extraction tend to need less reasoning than code generation or multi-step analysis), latency tolerance (some tasks can wait for a slower but cheaper model), and the presence of structured output requirements. The routing layer evaluates these signals and picks the appropriate model before dispatching the request.

Two broad patterns exist. Rule-based routing maps conditions to models using static logic — for example, “if the input is under 500 tokens and the task type is ‘summarization’, use the small model.” This is predictable, auditable, and requires no training data. Learned routing uses a lightweight classifier that predicts which model will produce acceptable output for a given input. It can adapt to traffic patterns but requires labeled examples and ongoing calibration.

In LLM cost management, model routing is one of the most cost-effective techniques available because it attacks the cost structure at the request level, not at the infrastructure level. You don’t need to provision different hardware or rewrite your application logic. The routing layer intercepts calls, applies a routing policy, and your application code remains unchanged. The result is a lower average cost per request across the full traffic mix — not by degrading quality, but by right-sizing the model to the task.

How model routing is used in practice

The most common scenario: a team building a customer-facing AI assistant realizes that a large portion of incoming queries are simple — rephrasing a sentence, answering a FAQ, or extracting a field from a short document. These tasks don’t need a flagship model. By routing them to a smaller, faster model, the team keeps response times low and reduces token costs significantly for the bulk of traffic.

A second use case is cost-aware fallback chains. If a premium model is rate-limited or returns an error, the router automatically retries with the next-best model rather than failing the request. This adds resilience without changing the calling code.

Routing logic typically lives in an LLM gateway or proxy layer — tools like LiteLLM are common choices — which sits between the application and model providers. The application calls a single endpoint; the gateway handles the routing policy.

Start with a simple rule-based router before adding a learned classifier. Map task categories to models manually (e.g., classification → small model, long-form generation → large model). Measure the quality delta before investing in a more complex routing strategy. Most teams find that three or four static rules cover the majority of their traffic.

Reading model routing in the right order

Start with what model routing is and how LLM gateways direct requests by cost, latency, and quality — it lays out the decision layer the rest of this topic assumes you already understand. Read the cost metrics, latency budgets, and fallback logic to know before implementing model routing next: it is the prerequisite check most teams skip, and skipping it is exactly what turns a routing project into a production incident.

Once the concepts are set, the guide to cost optimization, fallback, and latency control with OpenRouter and Portkey gets a routing policy live on managed infrastructure fastest; the step-by-step guide to a self-hosted router with LiteLLM, Bifrost, and Braintrust is the deeper build for teams that want the tiering policy under their own control. For the state of the market, how real teams are routing LLM traffic in 2026 with Braintrust, OpenRouter, and LiteLLM separates what’s shipped from what’s still a claim, and the case for treating routing as an accountability question, not only an infrastructure one closes the ladder with the question the other four never ask: who is responsible when the router decides.

How model routing differs from cost management, registries, and A/B tests

Three neighbouring topics get folded into model routing, and each folding hides a different mistake.

• Model routing is not the whole of LLM cost management. Routing is the single biggest lever inside that discipline, but caching, batching, and prompt-length budgets are separate levers that solve different waste — a team that only routes still overpays on repeated context.
• Model routing is not a model registry. Routing decides which model answers this request, in milliseconds, based on a live policy; the registry answers a slower question — which model artifact and version is actually deployed, so a rollback has something to point at. A router with no registry behind it can fail over to a model nobody approved.
• Model routing is not A/B testing for LLMs. A router optimizes every request against a fixed policy; an A/B test deliberately serves two policies to different slices of traffic to find out which one should become the fixed policy. Routing runs the decision, testing produces it.

Questions about routing models in production

Q: Do I need model routing if I only ever call one frontier model? A: Only if every request is equally hard — most workloads aren’t. Routing lets simple queries hit a cheaper model while complex ones still reach the frontier tier, the request-by-request framework the OpenRouter and Portkey guide builds instead of a single model-wide choice.

Q: Can a routing decision get made without anyone knowing which model actually answered? A: Yes — most gateways don’t log or surface the winning model by default, so a downgrade made to save cost is invisible unless you build the check in yourself. Black-Box Routing examines what that invisibility costs once a wrong answer reaches a user.

Q: Should I build my own router or use a managed one like OpenRouter? A: A managed router gets fallback and multi-provider coverage live fastest; a self-hosted stack like LiteLLM or Bifrost costs more setup time but keeps the tiering policy entirely under your control. The self-hosted router guide walks the trade-off in build order.

Q: Why does a routing setup sometimes cost more than sending everything to one model? A: Usually because the tiers were never mapped before the config was written — routing without a cost and latency budget per tier just adds infrastructure on top of the same spend. The prerequisites read covers the budgeting step teams skip.

Q: Does model routing stop mattering once providers converge on price? A: No — routing optimizes latency and quality fit as much as cost, and what real teams are doing with Braintrust, OpenRouter, and LiteLLM in 2026 shows routing has become load-bearing production infrastructure rather than a pure cost hack, even as price gaps close.

Part of the LLMOps production theme · closest neighbour: LLM cost management.

When model routing fits — and when it doesn't