What Is Model Routing and How LLM Gateways Direct Requests by Cost, Latency, and Quality

Most developers assume their API call travels directly from application to model. In production systems that have thought carefully about this, it doesn’t. A classification step intercepts each request first — and the model your query actually reaches is the output of a decision function, not the default frontier endpoint you configured on day one.

The instinct is to compare this to load balancing. Load balancers distribute identical requests across identical instances. Model routing operates on a different axis entirely: the decision isn’t about capacity, it’s about fit. Sending a factual lookup to a frontier model with hundreds of billions of parameters is not a resource-sharing question — it’s a category error with a billing consequence attached.

The Decision Layer the Request Never Sees

Think of a model router as a traffic inspector positioned before the API call executes. It reads the request, evaluates it against cost, quality, and latency constraints, and dispatches it to a specific model — all before any tokens are generated. The application layer never changes; the dispatch target does.

What is model routing in AI systems?

Model routing is a decision layer between an application and multiple LLM providers that directs each request to the optimal model based on cost, latency, quality, or task type (Survey paper, Moslem & Kelleher 2026). The decision runs at inference time — not at configuration time. You are not choosing a model when you build; the router is choosing one per request, based on signals that exist only at runtime.

The key distinction lives at the semantic layer. Traditional API Gateway patterns handle authentication, traffic shaping, and protocol enforcement. Model routing adds something the gateway cannot infer from HTTP headers: whether this particular request warrants a frontier model at all. The two concerns stack — you need both — but they answer different questions.

Not a load balancer. A classifier.

The practical consequence: a well-configured router makes your application appear to always use the best model, while actually using the cheapest model that meets quality thresholds for each request.

How does a model router decide which LLM handles each request?

The decision has three independent dimensions (Survey paper, Moslem & Kelleher 2026):

When the decision runs: Pre-generation (before any tokens are produced), post-generation (after a cheap model has already responded, then deciding whether to accept the output or escalate), or multi-stage (cascading through a sequence of models in order of cost).

What signals the router reads: Query features — length, detected complexity, inferred topic — combined with model metadata (cost per token, context window size, known capability profiles) and, in post-generation variants, actual response quality scores evaluated against a rubric.

How the decision is made: Rule-based systems use keyword triggers and cost thresholds. Classifier-based routers train a small neural network to predict the optimal model from request features. Bandit algorithms trade exploration for exploitation across providers. Reinforcement learning trains on accumulated outcome feedback from production traffic.

RouteLLM, an open-source framework from lm-sys, demonstrates what the classifier approach achieves when trained on human preference data: more than twice the cost reduction compared to always using the strong model, with no measurable quality loss — and the learned routing policy generalizes across different model pairs (RouteLLM paper, Ong et al.). The intuition is that a router doesn’t need to be large to be useful. A small, fast classifier that routes confidently on the easy majority of cases is the mechanism that makes the economics work.

The Architecture of a Routing System

A model routing system is not a single component. It is a pipeline with at least three distinct stages, each with a different job and a different failure mode. Understanding where the stage boundaries fall matters because the savings — and the errors — tend to concentrate at specific stages.

What are the main components of an LLM model routing system?

The three-stage pipeline (Survey paper, Moslem & Kelleher 2026) describes how production routing systems are structured:

Pre-router: Intercepts the request before any generation begins. Applies rules, classifiers, or scoring functions to classify the request by predicted complexity or cost tier. Simple requests go to a cheaper model; complex ones go to a stronger one.

Post-generation verifier: After the cheap model responds, evaluates output quality against a defined threshold. The verifier asks a binary question: is this answer good enough to return? If yes, the pipeline terminates. If not, the request escalates.

Escalation policy: Specifies what happens on a failed verification — accept with low confidence, request refinement, reject and re-route to a stronger model, or defer to a human review queue.

This structure defines the cascade pattern. When the cascade is calibrated correctly, it can outperform a single frontier model on both cost and quality simultaneously — because the cheaper model absorbs the bulk of volume while the stronger model concentrates only on the requests that genuinely require it. The quality ceiling rises because the expensive model never wastes capacity on simple queries.

In production systems, an LLM Gateway orchestrates these stages while handling the infrastructure layer: Fallback Strategy for provider outages, Rate Limiting per model endpoint, and Virtual Keys for credential isolation across teams or tenants. Observability hooks like Helicone instrument the gateway to log which model ran for which request and at what cost — without that signal, you cannot tune the routing policy on real traffic.

Portkey’s open-source gateway (v1.15.2) illustrates the full feature envelope that mature routing infrastructure requires: support for 1,600+ models across 45+ providers, automatic retries, circuit breaker for degraded providers, load balancing with weighted distribution, and canary testing for routing policy changes (Portkey GitHub). As of May 29, 2026, Portkey was acquired by Palo Alto Networks — the gateway remains open-source, but future product direction and enterprise pricing post-acquisition have not been publicly confirmed (Palo Alto Networks Press Release).

OpenRouter exposes routing intent at the model-slug level: appending :nitro prioritizes throughput, :floor selects the lowest-cost provider for a given model, and :exacto targets tool-calling quality — the latter auto-enabled for requests that include tool definitions (OpenRouter Docs). As of June 2026, the platform covers 300+ models across 60+ providers. One operational note: hard-coded model slugs can break when providers deprecate or rename models; the resilience pattern is to use the platform’s presets feature rather than pinning to a specific slug.

API compatibility note: Both Portkey and OpenRouter expose OpenAI-compatible API endpoints — existing applications require no SDK changes to switch routing infrastructure (OpenRouter Docs).

What the Router Reveals About Your Request Distribution

The routing math is more interesting than the routing promises. Cost savings of 30–85% depending on workload and model mix are achievable (TrueFoundry Blog), but the range is wide for a reason: the ceiling is set by distribution, not infrastructure. A concrete benchmark: 97% of GPT-4 Turbo quality at 24% of the cost, achieved by routing only 14% of queries to the strong model (TrueFoundry Blog). That 14% is the crux. It implies 86% of requests were answerable by a cheaper model — and your savings ceiling depends on whether your workload has a similar shape.

If your users ask factual questions, generate structured output, or perform retrieval over a fixed corpus, a high fraction of requests belongs on cheaper models. If your workload concentrates in complex reasoning, multi-turn analysis, or code generation across long contexts, the routing gain compresses — because most requests legitimately belong on the frontier, and the cascade has little left to intercept.

The classification step is fast enough to never become the bottleneck. Sub-10ms routing latency is achievable in practice; one reference implementation measures 3–4ms per routing decision at more than 350 requests per second on a single compute node (TrueFoundry Blog). LLM generation latency is measured in seconds; routing overhead is measured in milliseconds.

Practical implications that follow from the mechanism:

• If your application serves more than one request type, model routing pays — even a simple rule-based pre-router on request length captures meaningful savings.
• If your routing policy was configured months ago and your user behavior has since drifted, the savings estimate you modeled at launch no longer applies; the classifier’s confidence distribution is now wrong.
• If you are running a cascade, the effective quality ceiling is determined by your verifier’s precision, not your strong model’s capability. A verifier that passes bad outputs degrades quality silently — and produces no visible signal until users surface it.

Rule of thumb: Before tuning routing weights, characterize your request distribution. The potential savings are already in your logs; the router captures them, it does not create them.

When it breaks: Classifier-based routers degrade silently when the incoming request distribution drifts from the distribution they were trained on. A router trained on general Q&A will misclassify novel domain-specific requests — often routing them to cheap models with high confidence. Without monitoring on model-selection decisions, a misconfigured routing policy is indistinguishable from model underperformance.

The Data Says

Model routing is a precision instrument applied to a distribution problem. A cascade that knows your workload can outperform a frontier model on both cost and quality; a cascade trained on the wrong distribution introduces silent error without visible cost. The savings are real and reproducible — but they are a function of your request distribution, not of the routing infrastructure you chose.