Calling a Model Means Inheriting Its Runtime Cost Contract

A cached prefix processes almost for free. Send the opening of a conversation twice and the second pass returns fast — the model doesn’t reread what it already read, the way your reverse proxy doesn’t refetch a cached asset. Then you extend that same conversation by a few thousand tokens, and the response time doesn’t creep up with it. It roughly quadruples, and the invoice for that one call follows the same curve, for what looks, token for token, like a linear increase in payload.

The request didn’t change shape. The arithmetic running behind it was never linear to begin with — and neither is the architecture producing that arithmetic, whichever one happens to be running behind the endpoint you called.

Calling a model enrolls you in its runtime contract: a cost curve, a latency profile, a caching behavior, a memory footprint — set by an architecture you rarely see and never chose. None of those four move the way a payload-size mental model predicts, and which one breaks first depends on which architecture family sits behind the endpoint. This is not a guide to how it’s built. It is a guide to what you inherit when you call it.

The Bill That Ignores Your Payload Math

Most hosted chat endpoints run a decoder-only Transformer Architecture — as of March 2026, nearly every frontier system, from GPT to Claude to Gemini to DeepSeek, is built this way: one direction, no separate encoding pass. It generates one token, appends it, and reads the whole sequence again to generate the next. To avoid recomputing everything from scratch at every step, it keeps a running cache of the key and value vectors it already computed — the Attention Mechanism equivalent of a memoization table. That’s the part of your instinct that transfers cleanly: a repeated prefix really is cheap, because its keys and values already sit in that cache, computed once, reused freely.

The part that doesn’t transfer is what happens to the new tokens. Every fresh token still has to compute its relevance against the entire cached history — not against a fixed-size window, against everything the model has read so far in that sequence. As the conversation grows, that per-token cost grows with it. Push the sequence length up by a modest multiple and the underlying FLOPs count doesn’t rise by the same multiple — it can climb into the thousands, not the dozens, for the same proportional growth in length. The full arithmetic, and the point where GPU memory becomes the binding constraint, is in the memory-bottleneck explainer.

Mental Model Map: Attention Cost Behind One Endpoint From: token count is a linear payload size Shift: attention plus the KV cache turn sequence length into a quadratic cost driver To: budget by sequence length first, request count second Key insight: doubling your prompt doesn’t double the bill — the architecture makes it multiply.

There’s a second wrinkle worth knowing. That cache of keys and values doesn’t stay small. It grows with every token you add, across every layer and every attention head, and past a few tens of thousands of tokens its memory footprint can rival the size of the model’s own weights. Your GPU runs out of room to hold conversation history before it runs out of room to hold the model itself.

Picture a support chatbot that stays open for an hour instead of five minutes. The first exchange is cheap and instant, like every exchange before it. By the fortieth turn, nothing about the request has changed shape — same short question, same short answer — but the model is now re-deriving relevance against everything the conversation has ever said, and the cache holding that history has grown into a real line item on its own. The session didn’t get more complex. It got longer, and longer is the variable that was quadratic the entire time.

In practice, this means two separate budgets, not one: a compute budget that scales quadratically with how long the conversation gets, and a memory budget for the cache that grows right alongside it — both invisible in a token-count estimate that only looks at request size.

The Router That Doesn’t Shrink Your Memory Bill

Provisioning a Mixture Of Experts endpoint means budgeting for parameters you don’t use on any single request, not only the ones that fire. A meaningful share of frontier models behind hosted APIs now split the feed-forward layer into anywhere from a handful to a few hundred smaller expert networks, with a trained gate that wakes a subset of them per token — the full mechanism, gate math and all, lives in a separate explainer. What matters here is the behavior: a unit of work arrives, gets examined, gets routed to a subset of available workers — the shape your load-balancer instinct already reasons about daily.

What that instinct doesn’t prepare you for is that this router cannot be stepped through. There is no branch to set a breakpoint on. The routing decision is a matrix of learned weights, evaluated per token, and it can drift toward favorites over training the same way a cache with a poor eviction policy drifts toward thrashing — a specific, named failure mode that explains why two prompts that look nearly identical to you can take wildly different paths through the same model, at wildly different latencies, with nothing in your request predicting which path either one takes.

Two support tickets, worded almost identically, can enter the same endpoint and take entirely different paths through it — one routed to a pair of experts trained on billing language, the other to a pair trained on something adjacent — and come back at meaningfully different speeds. Nothing in either ticket predicted which one the gate would treat as harder. That unpredictability is not a bug report waiting to be filed. It is the router working as designed, on a distribution of traffic it was never guaranteed to treat evenly.

Here is the misconception that costs teams the most in production. A model advertising a small active-parameter count sounds like it should behave, cost-wise, like a small model. It doesn’t. Every expert, active or not, still has to sit in GPU memory, because the gate’s decision is made per token and any expert might be needed on the next one. Sparse activation is a compute discount, not a memory discount — budgeting it as one is how teams get the latency they expected and the hosting bill they didn’t.

Once that mental model holds, actually deploying or fine-tuning one of these models is a distinct skill — Max’s guide to running open-weight MoE models covers the hardware mapping this bridge deliberately leaves out. In practice, this means your capacity plan has to read the total-parameter number on the model card, not the active-parameter number the marketing leads with.

A Fixed Memory Budget With No Overflow Alarm

Some providers now run a third family behind the same kind of endpoint: State Space Model architectures, Mamba-style hybrids built to sidestep the quadratic tax the first section described entirely. Instead of keeping every past token explicitly accessible the way attention does, an SSM compresses the entire conversation so far into one fixed-size hidden state, updated at every step. That’s a genuine trade: a flat cost curve, in place of the one the previous section spent its length teaching you to fear.

Fixed-size is the word your capacity-planning instinct should catch on. A cache or a database that fills up tells you about it — a rejected write, an eviction metric, a capacity alert you can wire a page to. The hidden state inside a state-space model has no such mechanism. It doesn’t reject anything. It keeps writing new information into a container that was never going to grow, which means older information gets compressed past the point of recovery. Not an error. A property of the architecture.

That’s why hybrid designs — a mostly-SSM backbone with a thin fraction of attention layers woven back in — are the default for long-context systems in the 256K-token-and-up range, and not a research curiosity. AI21’s Jamba runs one attention layer for every seven Mamba layers; NVIDIA’s Nemotron-H pairs 54 Mamba-2 layers and 54 MLP layers with only 10 attention layers — different labs, different ratios, but the same bet: a small attention budget recovers most of the exact-recall capability the fixed state gives up, without surrendering the flat cost curve entirely. Pure SSMs still measurably lag on tasks that need exact recall from an arbitrary earlier position — a documented, benchmarked gap, not a hunch — because there is no mechanism to pull one specific past token back out of a state that already folded it into everything else.

Picture the same hour-long support session from the earlier section, but running on a hybrid model instead of a dense one. The customer states their account number in the first five minutes. Three hours in, a dense transformer can still look that number up directly — at a cost that kept climbing the whole time. A hybrid model carries the session at close to flat cost instead, but the account number now survives only if the thin attention layer happened to flag it as worth keeping when it first appeared. Two different failure surfaces, for two different reasons, and neither one shows up as an error.

In practice, this means a long conversation with a hosted model that starts confidently misremembering something from early on may not be the model forgetting. It may be the hidden state doing exactly what it was built to do: making room by overwriting what it decided you no longer needed.

A Second Model Hiding Inside Your Endpoint

You’ve been reasoning about “the model” as a single component this whole time — one architecture, one cost curve, one behavior to predict. That holds until the same endpoint accepts an image alongside your text, and the cost pattern stops matching anything the sections above predict.

Multimodal Architecture bolts a separate stage onto the front of the same backbone: a specialist encoder compresses the image or audio into feature vectors, a connector translates them into something the decoder’s token stream can read, and then the same generation loop as before takes over. The full three-part split, and what each stage actually does, is covered in the multimodal explainer. What matters for your budget is simpler: an image doesn’t cost like a few thousand extra text tokens. It costs like running a separate model first. How expensive that first step is depends entirely on which connector design the vendor chose — a detail no request parameter exposes to you, and one the frontier labs mostly keep private.

The same holds for audio. A minute of speech becomes a long sequence of encoder-produced vectors before the backbone ever starts drafting a reply, and that encoding step has its own latency floor — independent of how short the question buried inside the recording actually was. A ten-second question wrapped in a two-minute voice memo pays for the memo, not the question.

In practice, this means the same request that returned in half a second yesterday can return in three seconds today, for reasons that have nothing to do with your prompt and everything to do with an encoder pass you never see a line item for.

Four architecture families, four different runtime contracts. What your classical instincts get right about each, and where they stop predicting correctly:

Shift Diagram: Request Cost Behind One Endpoint Classic: Estimate payload size → Route request to a worker → Cache fills, then errors on overflow AI: Estimate token count → Route each token through a learned gate → Hidden state compresses, then quietly overwrites

Before You Scale This Endpoint

None of the four contracts above are visible from the request you send. The questions that surface them live in your own stack, not in the vendor’s documentation:

For the full prerequisite map, and the more literal questions this bridge deliberately left for a reference page — what a transformer is, how the families differ, which one fits which task — see the topic hub.

The architecture behind your endpoint decides how token count turns into latency, memory, and cost — quadratically for dense attention, unevenly for a learned router, flatly but silently for a compressed state, and invisibly for anything that isn’t text. Before the next scale-up, find out which of the four contracts you actually signed.