What Is Tool Use in Prompts and How LLMs Parse Function Calling Schemas

The model never touches your database. It has no network socket, no open file descriptor. When an LLM “calls a tool,” it emits a structured text string that your application reads, interprets as a function invocation, executes the actual call, and hands the result back as another message in the conversation.

Not execution. Serialized intent.

That distinction is not cosmetic. Misunderstanding it is why most tool-use failures get diagnosed in the wrong layer — not in the function logic, but in the instructions that told the model when and how to reach for it.

The Execution Loop the Model Never Sees

The idea that LLMs could invoke external tools was formalized in the Toolformer paper (Schick et al., February 2023), which demonstrated that models could learn to insert API calls into generated text through self-supervised fine-tuning. What Toolformer proposed as a learned behavior has since become a first-class API feature at every major provider. The underlying execution loop is architecturally identical across all of them.

When you define tools in an API request, the provider serializes those tool definitions into the model’s context — injected as additional tokens, positioned early in the prompt structure. The model reads a description of available capabilities alongside your query. When the query maps to a described capability, the model outputs a structured object rather than prose. The runtime intercepts that output, recognizes it as a tool invocation, executes the function on your side, and appends the result to the conversation. The model reads the result as another message and continues generation.

Not magic. A state machine with JSON edges.

What is tool use in LLM prompts?

Tool use in LLM prompts — also called function calling — is the mechanism by which a language model signals that it wants your application to execute a specific function before continuing its response. The model does not execute anything. It produces a Tool Calling Schema-compliant output object that specifies: which function to invoke, which arguments to pass, with what values.

The user’s question triggers the model to generate text. That text, under the right conditions, takes the form of a structured JSON object rather than natural language. Your orchestration layer catches it, dispatches the function call, and returns the output as a tool_result turn. The model generates its next output — this time informed by what the function returned.

The loop can iterate. A single user query can trigger three sequential tool calls if the first result surfaces the need for a second, and the second for a third. In agentic contexts, that loop runs without user intervention, which is precisely why the reliability of the model’s routing decision matters far more than the implementation of the function itself.

How does function calling work in large language models?

The mechanics differ slightly by provider, but the structure is stable enough to generalize.

When you include a tools array in an OpenAI API request, each tool is an object with a type of "function", a name, a human-readable description, and a parameters field. Note that the older functions and function_call parameters were deprecated in December 2023 — any article or codebase using them is built on a removed interface (OpenAI Docs). The model produces a tool_calls array in its response when it decides to invoke a function, where each entry names the function and supplies arguments as a JSON object.

On the Anthropic side, Claude returns a tool_use content block when it invokes a client-defined tool — the stop_reason is literally "tool_use" — and your code returns a tool_result block to continue the conversation (Anthropic Docs). Claude Opus 4.8, the current flagship as of May 28, 2026, supports this across a range of model variants down to Haiku 4.5 (Anthropic Docs).

Google’s Gemini API follows the same conceptual pattern: a FunctionDeclaration object in a tools array, a function call object in the model’s response, and a function response appended for the follow-up turn (Google AI Docs). Gemini-2.5-flash and gemini-3-flash-preview both support this interface.

The common shape: inject definitions → model outputs intent → you execute → model reasons over result. Everything the model needs is in the accumulated conversation. The model has no persistent state between calls; the conversation history is its memory.

What the Model Is Actually Reading in Your Schema

Three fields do most of the work in any tool definition. The fourth is where reliability lives or dies.

Schema definitions are not configuration files the model loads — they are tokens, injected into the prompt and subject to the same attention mechanics as the user message, the system prompt, and all prior conversation turns. The model doesn’t parse your JSON Schema the way a JSON validator does. It reads the description as natural language, uses the schema structure to understand the expected output shape, and adjusts its token sampling to satisfy both.

This means the description field is not documentation. It is the mechanism.

What are the components of a function calling schema for LLMs?

Across providers, a tool definition has three required components:

name — An identifier. OpenAI accepts most strings; Anthropic enforces a stricter regex (^[a-zA-Z0-9_-]{1,64}$, Anthropic Docs). The name appears in the model’s output when it invokes the tool, so it contributes a small semantic signal — get_current_weather is less ambiguous than fn_007.

description — The primary semantic attractor. This is the field the model reads to decide whether a query maps to this tool. A vague description produces erratic routing. A precise description — scoped to the function’s actual domain, explicitly excluding adjacent uses the model might conflate with this tool — produces consistent behavior. This is where Prompt Optimization delivers the highest return in tool-use systems.

parameters — A JSON Schema object describing the function’s input structure: properties, type annotations, required fields, and optionally additionalProperties: false. This defines the space of argument values the model is permitted to generate. With strict: true (supported by both OpenAI and Anthropic), the model is constrained to produce arguments that exactly satisfy the schema — no additional keys, no missing required fields.

A fourth element, strict: true, enables Constrained Decoding-adjacent enforcement at the API level. Libraries like Instructor, Outlines, XGrammar, and BAML implement more granular schema enforcement at the decoding layer — forcing valid JSON token-by-token rather than relying on the model to self-correct after the fact. Structured Output Prompting techniques compose naturally with tool use here: the same principles that make response_format: json_object reliable also apply to tool argument generation.

One concrete operational figure: adding tools to a Claude request increases the prompt overhead — 290 tokens when tool_choice is auto or none, 410 tokens when it is any or tool (Anthropic Docs). That cost scales with the number and verbosity of tool definitions. Schema hygiene is a token efficiency concern, not just a correctness one.

How do LLMs decide when to call a tool versus answer directly?

The model’s routing decision is a probability judgment, not a rule lookup.

When tool_choice is auto (the default across all major providers), the model evaluates whether the user query maps to a described tool capability and whether invoking that tool would produce a better answer than generating from its parametric knowledge alone. Anthropic’s documentation offers the clearest articulation of the heuristic: Claude calls a tool “when the request maps to that tool’s described capability and the answer isn’t already in context” (Anthropic Docs). For stable knowledge — mathematical facts, language questions, conversational responses — it answers directly. For live data, user-specific state, or operations outside its training distribution, it reaches for a tool.

The tool_choice parameter controls how much discretion the model retains:

Forcing a tool call (required or named function) bypasses the routing judgment — appropriate in structured pipelines where the tool invocation is architectural, not optional. In open-ended agentic systems, auto preserves the model’s ability to short-circuit when a direct answer is cheaper and sufficient. Choosing incorrectly between them is the second most common cause of brittle tool-use systems after poor description quality.

What the Schema Predicts — and Where It Fails

If you understand the mechanism, the failure modes follow directly.

If two tool descriptions overlap semantically, the model will assign probability mass to both and call the wrong one under ambiguous queries. Description collision is the leading cause of erratic routing in systems with more than three tools. The fix is not longer descriptions — it is mutually exclusive boundary conditions. Each description should include at least one explicit scope exclusion: “use this for X, not for Y.” Treat the description as a contract between your intent and the model’s probability distribution.

If the parameters schema is underspecified, the model fills in ambiguous fields with plausible-sounding values from its training distribution. A field typed as string with no description will receive whatever token best completes the pattern — which may bear no relationship to what your function expects. Add descriptions to properties entries the same way you would write them for a junior engineer: precise, with the expected value range and a concrete example in the description text.

If you use Parallel Tool Calling, the model may invoke multiple tools simultaneously when their arguments can be determined independently — all three major providers support this behavior (Zylos Research). The Salesforce AI Research W&D framework (February 2026) measured the speedup from parallel scaling at approximately four times that of sequential invocation in agentic search tasks, though real-world gains vary substantially by task structure (arXiv W&D paper). Parallel calls introduce a correctness risk: if Tool B’s correct arguments depend on Tool A’s result, forcing them to run simultaneously produces garbage inputs to B. Disable with parallel_tool_calls: false when the tools have data dependencies.

The security implication deserves its own sentence. Tool results are returned as messages in the conversation — which means an attacker who controls the content of a tool result can embed instructions into the model’s context. Prompt Injection via tool results is structurally identical to indirect injection in retrieval pipelines: the model cannot reliably distinguish legitimate data from embedded commands if both appear in the same context turn. Prompt Testing And Evaluation frameworks increasingly include tool-result injection as a required test category; treat every tool result as untrusted input until explicitly validated.

Rule of thumb: If you can’t articulate what query types should and shouldn’t trigger a tool in two sentences, the model won’t route it reliably either. The description is your specification; the schema is your contract. Only one of those can be enforced automatically.

When it breaks: Tool use fails predictably when two tools share overlapping semantic domains in their descriptions, when the parameters schema uses anyOf or recursive types that exceed the model’s effective schema comprehension, or when the conversation accumulates enough tool results that the routing signal is diluted by context noise. Strict mode prevents argument format violations; it cannot prevent routing confusion caused by ambiguous descriptions.

The Protocol Layer Taking Shape Beneath the Tools

One structural development worth noting as of mid-2026: the Model Context Protocol (MCP) — originally from Anthropic, now under vendor-neutral oversight (Zylos Research) — is converging on a shared wire format for tool definitions across providers and runtimes. A tool defined once in an MCP server can be consumed by any compliant client without per-provider schema translation. What was an integration exercise is becoming a portability problem with an emerging standard.

The Berkeley Function Calling Leaderboard (BFCL), now in V4, reflects the same trajectory: it scores agentic behavior — multi-step planning, correct tool sequencing, loop termination — rather than single-call schema adherence alone (Zylos Research). The benchmark is now measuring what actually fails in production, which is rarely “did the model produce valid JSON.”

The Data Says

The model’s decision to call a tool is a probabilistic routing judgment guided primarily by the description field — not the function name, not the parameter types. Schema strictness (strict: true) enforces argument format compliance after the routing decision is made; it cannot compensate for a description that fails to distinguish the function from adjacent capabilities. Every systematic failure in a tool-use system traces back to either a description collision, an underdescribed parameter, or a tool_choice setting that conflicts with the pipeline’s actual requirements.