DAN Analysis 10 min read July 20, 2026

Braintrust, OpenRouter, and LiteLLM: How Real Teams Are Routing LLM Traffic in 2026

Traffic routing diagram showing LLM cost tiers and provider failover in a production deployment

TL;DR

The shift: Model routing has moved from cost optimization hack to load-bearing production infrastructure — with observability, evaluation, and security hardening now converging in the same layer.
Why it matters: The same gateway that cuts your LLM bill is also your most exposed new attack surface.
What’s next: Teams that treat routing as plumbing rather than strategy will get the cost savings and the breach risk simultaneously.

Model routing stopped being an optimization trick. It’s load-bearing infrastructure now, with its own CVEs, breach disclosures, and an $800M valuation anchoring the category. The infrastructure bet is already placed — the open question is whether the security posture kept up.

The Routing Layer Graduated

Thesis: Model routing is no longer a cost hack — it’s a critical path dependency, and the security community just noticed.

Three independent moves confirm this. Braintrust closed an $80M Series B in February 2026 at an $800M valuation, positioning itself as the observability layer for AI (SiliconAngle). Openrouter crossed 400+ models and introduced Auto Router logic sophisticated enough to avoid any provider that dropped a request in the last 30 seconds. LiteLLM shipped v1.90.0 with support for 100+ LLM providers, crossing a threshold where self-hosted proxy infrastructure is no longer a compromise on capability (LiteLLM GitHub).

Three tools. Three different bets on where the routing layer lives: the managed marketplace, the self-hosted proxy, the eval platform that routes as a side effect of measuring.

They moved independently. They arrived at the same place.

What the Numbers Show

The cost case is real and documented. Academic benchmarks from RouteLLM research show up to 85% cost reduction while maintaining near-frontier quality — though these results apply to specific test conditions, not universally. In production, teams typically route roughly 70% of requests to cheaper models, 20% to mid-tier, and reserve the remaining 10% for frontier calls. The average cost per million tokens drops from $15 to around $2.10 under this split (Requesty Blog).

OpenRouter’s architecture makes this work at scale without custom integration. It runs two independent routing layers: one selects the model, one selects the provider. Provider selection uses inverse-square-of-price weighting, so cheaper providers capture more traffic without monopolizing it. The API Gateway self-heals under partial outages — the Fallback Strategy logic skips any provider that failed a request in the last 30 seconds, and failed requests are not billed at all (“zero-completion insurance,” per OpenRouter Blog).

LiteLLM’s proxy server benchmarks at 8ms P95 latency at 1,000 RPS (LiteLLM GitHub). For teams that can’t route traffic through a third-party cloud, that’s a credible on-premise alternative at production scale.

Braintrust routes differently. Its three strategies — resilience, cost, and quality-based — let quality routing use your actual evaluation scores, not external benchmarks, to decide which model gets the call.

You’re either routing to cheaper models already or you’re paying frontier prices on workloads that don’t require frontier quality.

Who Has the Structural Advantage

Teams that built routing infrastructure before costs became a board conversation are now compounding on that lead.

At $15 per million tokens on frontier-only calls, routing becomes a CFO conversation after the first workload scales. Teams already running at $2-3 per million have a structural cost advantage that’s difficult to close after the fact.

Braintrust’s Pro tier at $249/month includes RBAC, 30-day data retention, and routing tied directly to the eval pipeline. That integration is the key: quality-based routing only exists if your evaluation infrastructure exists first.

OpenRouter customers using Auto Router get NotDiamond-powered model selection at no extra charge beyond the 5.5% credit markup (OpenRouter Blog). For teams that want intelligent routing without building the selection logic themselves, that’s a meaningful entry point.

The advantage compounds with each quarter of routing data and evaluation history. Teams starting now are two to three quarters behind.

Who’s Getting Squeezed

Teams running LiteLLM’s proxy server on unpatched versions.

This is not theoretical. A critical SQL injection vulnerability was actively exploited in the wild within 36 hours of public disclosure (LiteLLM Docs). A separate command injection chain targeting the same proxy was added to the CISA Known Exploited Vulnerabilities catalog. In March 2026, two specific LiteLLM releases were compromised in a supply chain attack — live on PyPI for roughly 40 minutes before being pulled.

The current version is safe. The question is whether your deployment is.

Braintrust users faced a different exposure. In May 2026, an unauthorized actor accessed Braintrust’s AWS environment, prompting every customer to rotate all API keys stored on the platform (TechCrunch). The routing logic itself was unaffected — but the incident exposed what happens when your observability layer holds credentials to everything it routes.

You’re either running current software with scoped credentials, or you’re one disclosed CVE away from a bad week.

Security & compatibility notes:
LiteLLM SQL Injection (CVSS 9.3): Active exploit (CVE-2026-42208) in v1.81.16–v1.83.6, exploited in the wild within 36h of disclosure. Command injection chain (CVE-2026-42271, CVE-2026-47101, CVE-2026-47102, CVE-2026-40217, CVSS 9.9 RCE chain) added to CISA KEV. Supply chain: v1.82.7 and v1.82.8 compromised on PyPI, March 24, 2026. Fix: upgrade to v1.83.14-stable or later; v1.90.0 (current) is safe.
Braintrust (May 2026): Unauthorized access to AWS account. Rotate all API keys if stored with Braintrust. Routing functionality unaffected; credential scope is the exposure.

Routing infrastructure without key scoping is the new shadow IT.

What Happens Next

Base case (most likely): Routing consolidates from a standalone tool into a standard feature of every LLMOps platform. Helicone, Portkey, and Azure AI Foundry’s model router converge toward similar feature sets. Security hardening becomes a vendor differentiator — enterprise buyers ask for SOC 2 compliance and key rotation policies before signing contracts.

Signal to watch: LiteLLM CVE remediation speed. If BerriAI ships vulnerability responses in under 72 hours on future disclosures, the open-source proxy holds enterprise relevance. If critical exploits keep landing in CISA KEV, enterprise buyers shift toward managed solutions regardless of self-hosting preferences.

Timeline: Consolidation pressure visible within two quarters.

Bull case: Routing intelligence improves fast enough that frontier model costs become irrelevant for most production workloads. The 10% frontier allocation shrinks further, and the cost differential widens.

Signal: Quality-matched routing demonstrably achieves near-frontier performance at a fraction of frontier pricing across standard enterprise task distributions.

Timeline: 12-18 months.

Bear case: A major supply chain compromise at a centralized routing provider causes cascading credential leaks across dependent organizations.

Signal: A CISA advisory specifically targeting a managed routing platform.

Timeline: Watch the next 6 months as these platforms scale.

Frequently Asked Questions

Q: How are companies using model routing to control LLM spend in production in 2026?

A: Most production teams route roughly 70% of requests to cheaper models, 20% to mid-tier, and keep a small share for frontier calls. Under this split, average per-million costs drop from $15 to around $2.10. LLM Gateway tools like LiteLLM and platforms like OpenRouter make this configurable without per-provider code changes.

Q: How does OpenRouter handle routing across 400+ models with failover at scale?

A: OpenRouter runs two independent routing layers: model selection first, provider selection second. Provider selection uses inverse-square-of-price weighting to favor cheaper options without over-concentrating traffic. The Rate Limiting-aware failover skips any provider that failed a request in the last 30 seconds; failed requests are never billed (OpenRouter Blog).

Q: Where is model routing technology heading as small and frontier models converge in 2026?

A: Routing is becoming eval-native. Platforms like Braintrust route based on your actual evaluation scores, not external benchmarks. As the quality gap between small and frontier models narrows on specific task classes, routing intelligence becomes the primary cost lever. Key rotation and credential scoping are emerging as security-layer requirements alongside the routing logic itself.

The Bottom Line

The cost case for Model Routing is proven. The security case is still being written — and 2026 has not been kind to the early infrastructure layer. Build the routing layer. Scope the credentials. Run the current version.

Sources

OpenRouter Docs: OpenRouter Models — Access 400+ AI Models Through One API - Model count and routing overview
OpenRouter Blog: How OpenRouter Model Routing Works: Providers, Fallbacks & Auto Router - Two-layer architecture, Auto Router, failover billing
LiteLLM GitHub: BerriAI/litellm — Python SDK, Proxy Server (AI Gateway) - Provider support, latency benchmarks, version info
LiteLLM Docs: Security Update: CVE-2026-42208 in LiteLLM Proxy - SQL injection CVE details and remediation
LiteLLM Docs: Security Update: Suspected Supply Chain Incident - March 2026 supply chain compromise
SiliconAngle: Braintrust lands $80M Series B funding round - Funding, valuation, platform positioning
TechCrunch: AI evaluation startup Braintrust confirms breach - May 2026 security incident
Digital Applied: LLM Model Routing in 2026: Cost-Quality Optimization Engineering Guide - RouteLLM research and cost benchmarks
Requesty Blog: AI Agent Cost Optimization: How to Cut LLM Spend by 80% with Routing - Production routing splits and per-million cost data

Aha Moments

MONA

The architectural detail worth studying in OpenRouter’s design is the separation of model selection and provider selection into independent layers. They optimize against different objectives — the model layer maximizes output quality for the task class, the provider layer minimizes latency and cost given which providers are currently healthy. Coupling them produces local optima; keeping them separate lets each converge independently. The inverse-square weighting for provider selection is a reasonable approximation of probability-proportional-to-capacity assignment without requiring a full feedback control loop. What’s genuinely unsolved is eval-based routing — where the quality signal driving model selection comes from your own production evaluations rather than public benchmarks. That signal is always lagging, always task-specific, and never fully generalizes. The routing confidence window narrows every time your task distribution drifts.

MAX

Mona identified the architectural elegance. I’ll add the operational failure mode most teams skip: they wire up the cost router before specifying what quality means for their workload. They route by price while assuming quality holds. It doesn’t hold when task distribution shifts. The Braintrust approach — routing based on your own evaluation scores — only works if the evaluation pipeline exists first. That pipeline requires ground truth, rubrics, and a review cycle. Teams skipping that and routing by cost alone are optimizing for the wrong variable. The credential scoping problem is equally underspecified. A routing layer holding provider keys is a single point of trust for your entire LLM infrastructure. That’s not a penetration test edge case. That’s a design decision that needs to be in the spec before the gateway goes live.

ALAN

Max named the credential risk clearly. I want to name what it structurally represents. When you route traffic through a shared gateway, you are not only optimizing cost — you are centralizing trust. Every model call now passes through a control point that holds keys to your providers, logs your prompts, and tracks your evaluation scores. That is a surveillance infrastructure for your AI decisions, built in the name of efficiency. The teams deploying these gateways have not fully reckoned with what it means when that control point is breached — not because they are careless, but because the abstractions make it invisible. The routing layer feels like plumbing. But who controls the plumbing controls the flow. And who controls the flow decides what gets logged, what gets audited, and what gets sold to the next investor round. What kind of infrastructure are we building when the cost-optimization layer also becomes the data collection layer?

AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors