Model Tiering vs. Prompt Caching: When to Route to Cheaper LLMs and When Caching Pays Off

The monthly bill arrived and the number was wrong. Not a billing error — wrong like you ran 50,000 document extraction calls on Claude Fable 5 because that was the model you tested with. The system prompt: 8,000 tokens, identical on every call. You never added a cache breakpoint. You never checked whether 80% of those calls needed the flagship at all.

That specific mistake is fixable. Here is the spec to fix it.

Before You Start

You’ll need:

• A cost baseline from your LLM Cost Management tooling — at minimum a rough breakdown of monthly token spend by request type
• A routing proxy or gateway: LiteLLM v1.89.4 or equivalent that supports multiple model backends and routing rules
• An LLM Observability tool for tracking cache hit rates and per-route cost after deployment

Concepts you should know:

• Model Routing — how requests get classified and dispatched to different model endpoints
• Prompt Caching — how providers store and reuse prefix token states across calls
• Model Tiering — how to map task complexity to model capability and price tier

This guide teaches you: A decision framework for choosing model tiering, prompt caching, or both — based on your traffic pattern and prefix structure, not intuition.

The Expensive Default

Most teams treat routing and caching as alternatives. Switch models OR add a cache breakpoint. That framing leaves significant savings untouched.

These two techniques solve different problems. Model tiering reduces per-request cost by sending simpler tasks to a cheaper inference endpoint. Prompt caching reduces the token cost of each call by reusing a shared prefix — on whatever endpoint you use. You can apply both to the same request pipeline simultaneously.

Here is the pattern I see every week. A team adds caching to their flagship pipeline and gets a meaningful cost reduction. Then they analyze their traffic and realize 60–80% of those requests were simple enough that a smaller model handles them without quality regression (DigitalApplied). They cached the right prefix on the wrong model.

Start with routing. Add caching second. Routing lowers your cost floor across all traffic. Caching compounds the savings on the high-volume routes that remain.

Step 1: Classify Your Traffic Before You Route

You cannot route what you have not classified. Before writing a single routing rule, understand three dimensions of your request mix.

Task complexity:

• Simple — extraction, classification, format conversion, template fill: small model candidates
• Moderate — summarization with reasoning, multi-step Q&A, light code completion: mid-tier
• Complex — code generation, agentic steps, multi-hop reasoning, nuanced judgment: flagship

Context reuse pattern:

• Same system prompt across most requests → strong caching candidate
• Diverse contexts — different documents, different users, different tasks each time → caching adds little
• Sub-pipelines where a shared reference document precedes every call → structural caching opportunity

Volume and prefix length:

• High request volume (100+ per hour) with a long shared prefix (2,000+ tokens) → caching delivers measurable savings
• Low volume, short prompts, or fully diverse contexts → caching overhead may not recover its write cost

The Architect’s Rule: You need at least two reads within the TTL window to recover the write overhead. At fewer than two reads per write, you are paying for storage you do not consume.

Step 2: Lock Down the Routing and Caching Spec

Classification gives you the picture. Now define the rules before you implement anything.

Routing constraints — specify each:

• Model assignment per task class, with an explicit fallback (“if classifier confidence is below threshold, escalate to mid-tier”)
• Quality floor per tier — what does acceptable output look like for each task class? Write the success criteria; this is your routing spec
• Latency budget — rule-based routing adds approximately zero overhead; classifier-based routing adds 50–150ms (DigitalApplied). Choose your approach before you build it
• Failure handling — what happens when the small model produces out-of-spec output? Define the retry and escalation path explicitly

Caching constraints — verify each before enabling:

• Identify your static prefix — the portion of the prompt that is identical across calls: system prompt, instruction block, reference document. The static prefix is your cache anchor.
• Verify your Token Budget against the model’s minimum threshold. Haiku 4.5 requires a minimum of 4,096 tokens to cache. Sonnet 4.6 and Opus 4.8 require 1,024 tokens. Fable 5 requires 512 tokens (Anthropic Docs). If your system prompt falls below the model’s threshold, caching is not available for that route.
• Choose your TTL: Anthropic offers a 5-minute TTL at 1.25× the write cost, or a 1-hour TTL at 2× the write cost. Cache reads always cost 0.1× the base input price (Anthropic Docs). Choose based on your actual request cadence — a 5-minute TTL expires before an hourly batch run starts.
• Platform mode: automatic caching is available only on the Anthropic API directly. On AWS Bedrock or Google Cloud, explicit cache breakpoints are required (Anthropic Docs).

The Spec Test: If your static prefix is 3,500 tokens and you are routing to Haiku 4.5, caching is unavailable — the minimum is 4,096 tokens. Either pad the static prefix to meet the threshold or route to a model with a lower minimum.

Step 3: Implement in the Right Order

The sequence matters. Routing lowers your baseline cost first. Caching then compounds savings on each individual route. Get the order wrong and you optimize the wrong layer.

Implementation order:

1. Rule-based routing — classify by token count, output type, and task structure using deterministic heuristics. Zero latency overhead. Catches the majority of obvious routing decisions before you touch a classifier.

2. Cache the system prompt per route — once routing is stable, add cache breakpoints to the static portion of each model’s prompt. Every routing path benefits from prefix reuse independently.

3. Classifier-based routing for ambiguous requests — add an embedding or lightweight model classifier for requests your rules cannot reliably categorize. Accept the 50–150ms overhead only after measuring whether your latency budget supports it.

4. Semantic caching as a third layer — Semantic Caching matches similar queries at the application level using embedding similarity and returns a cached response, bypassing model inference altogether. Prompt caching makes each inference call cheaper; semantic caching eliminates calls outright (Redis Blog). Stack them when request volume justifies the infrastructure investment.

For the routing proxy: LiteLLM v1.89.4 supports routing strategies including least-busy, latency-based, cost-based, and complexity-based routing (LiteLLM Docs). Configure model groups, routing weights, and fallback order.

A verified supply chain security concern exists for LiteLLM: an incident was disclosed in March 2026 that could allow malicious packages into your dependency chain. Use it in production only after verifying package integrity and pinning to a specific version hash. Details below.

Security and compatibility notes:

• LiteLLM Supply Chain (March 2026): A supply chain security incident was disclosed. Pin to a specific version hash and verify package integrity before deploying in production (LiteLLM Blog). Latest stable: v1.89.4.
• LiteLLM Docker tag: The main-stable Docker tag stops publishing June 30, 2026 — migrate to :latest (LiteLLM Blog).
• Claude Opus 4.7+ tokenizer: The new tokenizer uses up to 35% more tokens for the same fixed text compared to pre-4.7 Claude models. Cost estimates built on older model pricing may understate actual token consumption (Anthropic Docs).

For batch workloads that can tolerate latency: Anthropic’s Batch API processes up to 100,000 requests per batch with typical turnaround under one hour (Anthropic Docs). Stack it with prompt caching for maximum compression — Sonnet 4.6 at batch pricing plus cache read pricing combines to $0.15/MTok effective, 95% off list (Anthropic Docs).

Step 4: Validate Cost and Quality Together

You have routing rules and cache breakpoints. Now measure whether they are working — and whether quality held across routing tiers.

Validation checklist:

• Cache hit rate — failure looks like: hit rate below 50% on routes you designed for caching. Root cause: prefix is not truly static across calls, TTL is shorter than request cadence, or request volume is lower than projected.
• Quality pass rate per tier — failure looks like: task success rate drops on small-model routes. Root cause: routing boundary is too aggressive; moderate-complexity tasks are landing on Haiku instead of the mid-tier.
• Cost per request by route — failure looks like: per-request cost unchanged after adding caching. Root cause: fewer than two reads per write within the TTL window — you are paying for writes you do not consume.
• Latency by route — failure looks like: p95 latency spikes on complex-task routes. Root cause: classifier overhead or cold-start on the flagship endpoint under load.

Use your LLM Observability tooling — OpenTelemetry traces or a dedicated LLMOps platform — to collect these metrics per route. Distributed Tracing lets you see where token spend and latency concentrate across multi-step pipelines.

Recalibrate after 10,000 requests. Your initial classification is an estimate. Real traffic distribution refines the routing boundary. Run the numbers before treating any boundary as permanent.

Common Pitfalls

Pro Tip

Routing boundary is your specification. Document which task types route where, what the quality floor is for each tier, and what the escalation condition triggers. When the boundary shifts — and it will, as models improve and traffic patterns evolve — the spec tells you exactly what changed and why. A routing config without a spec is a magic number in production: nobody knows what it was optimizing for, and nobody will touch it when something breaks.

Frequently Asked Questions

Q: When is it better to route a task to a smaller, cheaper LLM versus staying on a flagship model for cost savings?

A: Route when the task is mechanical: classification, extraction, format conversion, template fill. Stay on flagship when the task requires multi-step reasoning, code generation, or nuanced judgment. The boundary shifts by domain — legal document summarization often requires more reasoning than it looks. Start conservative: only route task classes where you have verified quality from the smaller model on representative production samples. Watch for drift — a task class that routes cleanly today may grow in complexity as users push the system harder.

Q: Is prompt caching worth the engineering overhead compared to simply switching to a cheaper model?

A: Yes — and they are not competing choices. Routing changes which model runs inference; prompt caching reduces what each inference call costs on whichever model you use. The implementation overhead on Anthropic’s API is minimal: add a cache_control parameter on the static content block, or enable automatic mode on the direct API. The breakeven condition is two reads per write within the TTL window (Anthropic Docs). For Gemini, factor in the separate per-hour storage fee before comparing ROI across providers — the math works out differently from Anthropic and OpenAI.

Your Spec Artifact

By the end of this guide, you should have:

• A request classification map: task types grouped by complexity tier, with a model assignment and explicit quality floor per tier
• A caching spec: static prefix identified, minimum token threshold verified against your chosen model, TTL selected based on actual request cadence, platform caching mode confirmed
• A validation checklist: cache hit rate target per route, quality pass rate per tier, latency budget per model, and a recalibration schedule after the first significant traffic batch

Your Implementation Prompt

Paste this into Claude Code, Cursor, or Codex. Replace each bracketed placeholder with your own constraints before running.

I need to build an LLM routing and caching layer for my production pipeline.

PIPELINE CONTEXT:
- Current model: [flagship model and MTok pricing — e.g., Fable 5 at $10 input / $50 output]
- Pipeline type: [document processing / Q&A / code generation / describe yours]
- Daily request volume: [N requests per day, N requests per hour on peak routes]

ROUTING SPEC:

Simple task class:
  Definition: [extraction / classification / format conversion / template fill / describe yours]
  Route to: [cheap model and MTok pricing — e.g., Haiku 4.5 at $1/$5]
  Quality floor: [define what acceptable output looks like — format, accuracy requirement]
  Escalation condition: [when should a failed small-model response retry on mid-tier?]

Moderate task class:
  Definition: [summarization / multi-step Q&A / light code / describe yours]
  Route to: [mid-tier model and MTok pricing — e.g., Sonnet 4.6 at $3/$15]
  Quality floor: [define success criteria for this tier]
  Escalation condition: [threshold for flagship escalation]

Complex task class:
  Keep on: [flagship model]
  No automatic escalation — route here deliberately

CACHING SPEC:
  Static prefix token count: [N tokens]
  Minimum threshold check:
    Haiku 4.5: 4,096 minimum | Sonnet 4.6 / Opus 4.8: 1,024 | Fable 5: 512
  Request cadence on this route: [N requests per hour]
  TTL selection: 5-minute if >2 requests per 5-min window; 1-hour if >2 per hour; skip if below breakeven
  Platform caching mode: [Anthropic API direct = automatic | Bedrock or GCP = explicit breakpoints required]

VALIDATION TARGETS:
  Cache hit rate: [>50% on designed cache routes]
  Quality gate per tier: [define pass/fail criteria per task class]
  Latency budget: [p95 in ms per tier]
  Observability tool: [LangFuse / OpenTelemetry / other]

Build in this order: rule-based routing → system prompt caching per route →
classifier-based routing for ambiguous requests → semantic caching as third layer.

Using LiteLLM v1.89.4: configure model groups for each tier with explicit fallback order.
Add cache breakpoints on the static system prompt for each routing path.
Log cache hit rate and cost per route from the first request.

Validate in sequence: cache hit rate → quality per route → cost per request → latency p95.

Ship It

You now have two levers, not one. Routing lowers the cost floor across all traffic by matching task complexity to model capability. Caching compounds those savings on high-volume routes with shared context. Neither technique alone is optimal — the real compression comes from applying both to the same pipeline in the right sequence.

Run the spec against 10,000 real requests, then recalibrate the routing boundary.