Context Costs, Wrong Parameters, and the Hard Limits of LLM Tool Calling

The weather lookup demo always works. It doesn’t show you the 497 tokens consumed before the model reads your user’s message, the schema constraints that differ between providers, or the quiet confidence with which location was filled as "New York, NY" — when the user was asking about Berlin.

Tool calling is not “attaching functions to a chat.” It’s a three-party contract between your function definitions, the model’s probability distribution, and a runtime executor that assumes the call is valid. Each party fails differently, and each failure is nearly invisible until it propagates downstream.

The Hidden Grammar of Function Signatures

Before you touch an API client, you need to know which schema rules your provider enforces. This sounds like a compatibility footnote. It is, in practice, one of the most common sources of silent failures in production tool integrations — because the model will generate a call that looks valid to your parser while violating the semantic intent of your definition.

What JSON and API knowledge do you need before implementing LLM tool calling?

Every Tool Use in Prompts implementation rests on three layers of prerequisite knowledge, and skipping any one of them shifts the failure to a later, harder-to-diagnose stage.

The first layer is JSON Schema fundamentals: how type, properties, required, enum, and description interact. The Tool Calling Schema you send to the model is a JSON Schema document. Its description fields are not documentation — they are the model’s primary signal for how to fill each parameter. A vague description is a missing constraint.

The second layer is provider-specific schema subsets. Each provider enforces a different schema dialect. OpenAI’s strict mode requires additionalProperties: false for every nested object, and optional parameters must appear in the required array with a union type like "type": ["string", "null"] (OpenAI Docs). Anthropic’s interpreter supports allOf composition but handles recursive schema structures differently. The Model Context Protocol standardized on JSON Schema 2020-12 since its launch in November 2024 (Sourcemeta Blog) — but MCP tools still flow through each provider’s own schema interpreter, so that version standardization does not eliminate provider-level dialect differences.

The third layer is what happens when you enable Structured Output Prompting modes. Constrained generation significantly raises schema compliance; the tradeoff is additional latency and the narrower set of schema features each mode allows. Understanding which mode you’re using — and what it excludes — is a prerequisite to reasoning about why a schema that validates locally might not behave as expected at inference time.

When the Model Guesses

The wrong-parameter problem has two shapes, and they are not the same failure. Research on tool hallucinations identifies them separately: tool selection hallucination, where the model invokes the wrong function entirely, and tool usage hallucination, where the right function is called with invented or incorrectly typed arguments (arXiv (Relign)). Both emerge from the same root cause — the model generates the most statistically probable token sequence, weighted by its training distribution and the precision of your schema description.

Why do LLMs call tools with wrong parameter types or missing required fields?

For missing required parameters, the failure is particularly subtle. Claude Sonnet may infer and fill a required field rather than requesting clarification — so location becomes "New York, NY" because that value appeared frequently in training contexts where a city was required but unspecified (Anthropic Docs). Claude Opus is more likely to surface the ambiguity as a clarifying question. Neither behavior represents an error in the model’s probability distribution; both are predictable consequences of what that distribution was trained to do. What’s not predictable is which behavior your integration is implicitly relying on.

Constrained Decoding — where providers enforce schema conformance through token-level constraints — raises structural compliance dramatically. As of June 2026, OpenAI Structured Outputs reached approximately 99.9% schema compliance, Anthropic tool use approximately 99.8%, and Gemini schema mode approximately 99.7% (Glukhov.org). Schema compliance is not semantic correctness. A model can produce a perfectly valid JSON call with the correct types and still pass a semantically wrong value. The Berkeley Function Calling Leaderboard (BFCL v3) measures something harder: whether the model completes the actual task end-to-end. As of June 2026, the top-ranked model (GLM 4.5) scored 76.7%, with Qwen3 32B close behind at 75.7% (BFCL Leaderboard). The gap between 99%+ schema conformance and 76-77% task accuracy is not a measurement inconsistency — those benchmarks are testing different surfaces of the problem.

Not a broken protocol.

A reasoning gap.

Parallel Tool Calling introduces its own failure mode. When multiple tools run concurrently, some models fail to track which tools have already been dispatched within the same inference step. OpenAI’s gpt-4.1-nano has a documented instance of this: it can issue duplicate concurrent calls; the recommended workaround is setting parallel_tool_calls: false (OpenAI Docs). The mechanism is predictable — parallel dispatch requires the model to maintain a call inventory within a single completion, and that tracking breaks under certain conditions with smaller models.

The Token Bill for Every Schema

Schema definitions are not transmitted alongside the conversation; they are injected into the system prompt and billed as input tokens. On OpenAI’s API, tool definitions are explicitly documented as being injected into the system message (OpenAI Docs). On Anthropic’s API, the cost appears as a system prompt overhead before any user content reaches the Context Window.

How much context window do tool calling schemas consume in practice?

Anthropic publishes per-model overhead numbers for tool use (Anthropic Docs). The variation across model generations is significant:

These figures represent system prompt overhead before your schema definitions or user message — the baseline cost before a single parameter description is read. System prompt overhead precedes every user message. A single get_weather tool call with one user message totals approximately 403 tokens for the complete request (Anthropic Docs). Add more tools, longer description fields, or nested object schemas, and the input token count scales accordingly; the table values are a floor, not a ceiling.

Parallel tool results compound this in a different direction. If ten tools each return five hundred lines of output, you’ve added roughly five thousand lines of content to the context window for the next inference step (TianPan.co). Platforms typically cap concurrent tool dispatches — around forty tools maximum — partly to prevent context exhaustion from response accumulation (Zylos Research).

Prompt Testing And Evaluation baselines established on one model generation may not transfer forward. The Claude Fable 5 / Mythos 5 tokenizer produces approximately 30% more tokens than pre-Opus-4.7 models for the same text (Anthropic Docs). If your token budget assumptions were calibrated on an older Claude model, the same schema definition will cost measurably more against the new tokenizer.

Security & compatibility notes:

• OpenAI gpt-4.1-nano parallel calls: Known duplication bug where gpt-4.1-nano can issue identical concurrent tool calls in the same turn. Workaround: set parallel_tool_calls: false (OpenAI Docs).
• Claude Fable 5 / Mythos 5 tokenizer: Produces approximately 30% more tokens than pre-Opus-4.7 models for the same text (Anthropic Docs). Token count baselines from older Claude models do not transfer directly.

What the Schema Contract Predicts

The schema contract has predictable failure modes once the mechanism is clear.

If your description fields are vague about edge cases, the model will infer missing or ambiguous parameters. If those inferences look structurally valid — and with constrained generation, they usually will — no error propagates. The downstream system receives wrong data with no signal that anything failed.

If parallel tool results are large, context fills proportionally with each inference step. A multi-tool design that performs well at two concurrent calls may exhaust the window at ten. Batching results and pruning what returns to the model is not premature optimization; it’s basic context accounting.

If you’re switching between providers — or between model versions within the same provider — schema compatibility is not guaranteed. A definition that conforms on one interpreter may behave differently on another. Treat each migration as requiring explicit schema validation, not assumption transfer.

Prompt Optimization for tool calling operates at the schema level, not just the system prompt: trimming verbose description fields, eliminating redundant nested properties, and reducing enum lists to the values the model genuinely needs to distinguish. Token budget and parameter accuracy are coupled — over-specified schemas with long, ambiguous descriptions increase both cost and the surface area for inference errors.

Structure and semantics fail at different layers. The structural failure — invalid JSON, wrong type, missing required key — is caught at parse time. The semantic failure — correct structure, wrong value — propagates downstream unless you validate tool call outputs independently of the schema layer. Prompt Injection through tool results occupies this same gap: valid content, malicious intent. Schema enforcement does not protect against either semantic failure class.

Rule of thumb: Design the description field for the model, not the developer. It is the model’s instruction set for that parameter — make it prescriptive, not descriptive.

When it breaks: The most dangerous production failure is a semantically wrong value inside a structurally valid call. The call passes schema validation, passes your orchestrator’s type checks, and propagates into downstream systems without a visible error. The only systematic defense is explicit output validation at the boundary between the tool call result and the function that acts on it.

The Data Says

Schema conformance and task accuracy measure different surfaces of the same problem. Conformance above 99% shows the protocol works; task accuracy near 76-77% on BFCL v3 (as of June 2026) shows that reasoning gaps remain. The token costs are not theoretical overhead — they are billed on every inference step, whether or not the model chose the right tool with the right parameters. Understanding where the contract is enforced and where it isn’t is the prerequisite knowledge that separates a working demo from a production implementation.