Visual Hallucination, Resolution Caps, and the Unsolved Technical Limits of Multimodal Prompting in 2026

Send a Multimodal Prompting model an image of an empty kitchen countertop, and in a fraction of a second it will describe — with calm, measured confidence — a coffee mug that doesn’t exist. The model’s training corpus established the correlation: countertop → kitchen → morning scene → coffee; generating the mug is lower-loss than expressing uncertainty about its absence. That is the defining anomaly: sight is not perception, it is inference — and inference built on language priors will produce language-shaped errors.

The Geometry of What the Model Never Actually Sees

Before a Visual Question Answering system can reason about an image, the image must become tokens — because the transformer architecture processes nothing else. A vision encoder (typically CLIP-based) maps the image into a series of patch embeddings: small spatial regions, each compressed into a vector that the language model’s attention layers can attend to.

The critical detail is what that compression loses. CLIP encoders are trained on paired image-text data and optimized for coarse semantic alignment: does this image match this caption? That objective produces encoders that are excellent at recognizing dominant objects and scenes, and systematically poor at fine-grained spatial detail. Salient objects — the refrigerator, the table, the central figure — register with high confidence. A specific count of cables on a desk, or the relative position of UI elements in a dense interface, falls into an attention shadow where the patch embedding carries only probabilistic information about what might be there.

why do multimodal AI models hallucinate image details that are not in the image

The mechanism operates at three separate layers, each compounding the previous.

The first layer is the vision encoder itself. CLIP-based encoders exhibit salient-object bias — they encode prominent visual features at the expense of peripheral context. When a visual feature maps ambiguously — when the encoder isn’t confident which linguistic concept corresponds to what it’s representing — the language model must resolve that ambiguity using its language priors. The resolution tends to favor statistically dominant associations rather than expressing uncertainty.

The second layer is object co-occurrence bias. The language model has internalized co-occurrence statistics from training at scale. Refrigerator and microwave appear together in descriptions so reliably that the model treats their co-occurrence as near-certain. If the vision encoder registers “refrigerator” with high confidence, the language model assigns elevated probability to “microwave” — whether or not the image contains one. This is fabrication in its purest architectural form: not a misperception of what’s there, but an inference about what must be there given the statistical neighborhood.

The third layer is the training objective itself. Next-token prediction rewards confident generation. The loss function doesn’t penalize false certainty — it penalizes incorrect token predictions. The practical consequence is that models learn to produce confident descriptions rather than hedged ones when visual evidence is ambiguous. Expressing “I am not certain there is a coffee mug” is higher-loss than asserting the mug’s presence, because training data contains far more confident descriptions than epistemic hedges.

Researchers have now formalized what practitioners were observing empirically. Si et al. 2025 identify two distinct Hallucination types in multimodal models: omission — failing to identify objects that are present, caused by insufficient confidence when mapping visual features to language — and fabrication — inventing absent objects through spurious cross-modal associations from statistical biases in the training corpus. The distinction matters because the two failure modes require different mitigations. Conflating them into a generic “hallucination” label obscures which architectural layer is causing the problem.

Not random noise. Structured bias with a clear source.

Pixels, Patches, and Token Budgets

The second class of technical limits in multimodal prompting is less anthropomorphizable but equally concrete: images consume Context Window tokens, and those tokens compete with text. This is the domain of Context Engineering for visual inputs — an arithmetic problem with architectural constraints baked in.

Modern multimodal APIs handle images through tiling: the image is divided into fixed-size patches (28×28 pixels for Claude; 512×512 tiles for GPT-4o, which was retired in February 2026), and each patch is encoded as a fixed number of visual tokens. Higher resolution means more patches, more tokens, and a shrinking text budget in the same request. Resolution is not a free variable; it is a cost.

image resolution and context window limits in multimodal prompting 2026

The token arithmetic varies substantially across models in 2026, and the ceiling has risen considerably from where it was even eighteen months ago.

Claude Fable 5 and Mythos 5 (released June 9, 2026) access a high-resolution tier where the maximum long edge is 2,576 pixels, capped at 4,784 visual tokens per image. The token formula is ⌈width/28⌉ × ⌈height/28⌉ — so a standard 1920×1080 image costs approximately 2,691 tokens in the high-res tier, or roughly 1,560 tokens for models in the standard tier (maximum long edge: 1,568 pixels). Claude’s documentation explicitly flags that counting is approximate and inaccurate for large numbers of small objects, that coordinate and localization outputs are approximate, and that images under 200 pixels risk hallucination — “Claude might hallucinate or make mistakes” at that scale (Anthropic Docs).

GPT-5.5, the current OpenAI flagship, extends to 6,000 pixels per dimension in “original mode,” with up to 10,000 patches available (OpenAI Developer Docs). This is a substantially wider ceiling than most multimodal tutorials assume — because the majority of those tutorials were written against GPT-4o, which was retired from the OpenAI API on February 16–17, 2026. Any prompt code or pipeline referencing gpt-4o in a vision context is now broken (VentureBeat).

Compatibility note:

• GPT-4o (vision API): API shutdown completed February 16–17, 2026. Migrate to GPT-5.x — gpt-4o calls fail at the endpoint level.

Gemini 3.1 Pro uses a 768×768 px tile system at 258 tokens per tile. A 960×540 image generates 6 tiles, totaling 1,548 tokens, per the tile formula in Google AI Docs. The media_resolution parameter in Gemini 3 allows explicit token budget control per image — a useful escape valve when context budget is the constraint.

Qwen-VL — specifically Qwen3-VL, the current recommended version from Alibaba — supports images above 1 million pixels with a 32× spatial compression ratio, yielding approximately 1,280 visual tokens per image and flexible min_pixels/max_pixels budget control (Qwen3-VL GitHub). Its context window is 256K tokens natively, expandable to 1M via YaRN. For open-weight deployments where context budget matters, this is the current ceiling.

The advertised 1M context windows across the frontier don’t, in practice, deliver 1M tokens of reliable retrieval. Effective usable context across leading models drops to approximately 600–700K for reliable retrieval, per elvex context comparison. When a high-resolution image costs 2,000–4,000 tokens, and a Multi-Turn Prompt Design session involves multiple images across several turns, that arithmetic becomes a design constraint rather than a theoretical limit.

Dynamic tiling — which splits high-resolution images into independently-encoded patches — solves the detail-loss problem while introducing a new one: each patch is encoded without spatial awareness of adjacent patches, fragmenting cross-tile spatial context entirely. An architectural diagram that spans four tiles is never processed as a single visual unit; each tile is a local encoding with no cross-tile attention during vision encoding. Research into efficient visual encoding (LLaVA-UHD v4) formalizes this trade-off: tiling preserves local detail while multiplying visual token count and fracturing cross-tile spatial continuity.

What the Geometry Predicts in Practice

These two constraint classes — hallucination bias and token budgets — produce failure modes that are predictable once you understand the mechanism. That predictability is useful; it shifts you from discovering errors empirically in production to anticipating them at design time.

If you send a model an image containing many small instances of the same object — bolt counts on a machine part, pedestrians in a crowd, cells in a microscopy slide — expect systematic inaccuracy. The vision encoder’s salient-object bias, combined with language priors toward plausible-sounding round numbers, means the model will generate a count that sounds right more often than one that is right. Claude’s documentation acknowledges this directly: counting is approximate and degrades for large numbers of small objects.

If you need to reason about fine-grained spatial relationships across a large image — the relative positions of UI elements, the configuration of a circuit board, the labeling within a dense information graphic — the tiling architecture creates seams where spatial context is severed. The model may accurately describe each tile in isolation while getting cross-tile relationships wrong; the representation never holds the full spatial structure simultaneously.

Rule of thumb: semantic salience, not resolution, determines sight. A low-resolution image of a prominent object often parses more reliably than a high-resolution image of many peripheral ones. Increasing resolution helps when the relevant feature is large enough to survive tile-level encoding with high confidence. It does not help when the failure mode is language-prior fabrication — the coffee mug problem has nothing to do with image resolution.

Using a System Prompts instruction to request explicit uncertainty expression does shift model behavior at the margin. Instructing the model to hedge when visual confidence is low, or adopting a Role Prompting approach toward an “honest-uncertainty” analyst persona, creates measurable differences compared to an unconstrained prompt. Prompt Engineering for visual accuracy is a distinct discipline from prompt engineering for text; chain-of-thought and few-shot techniques that reliably reduce language hallucination have a more complicated and less predictable relationship with visual tasks, because the failure is embedded in the encoder layer that precedes the language model entirely.

When it breaks: Visual hallucination resists mitigation because the bias is architectural — embedded in training-corpus co-occurrence statistics, reinforced by the next-token objective, and operating in the vision encoder before any prompt instruction is processed. No prompt structure can fully override the geometry of the latent space; it can only shift the probability distribution at the margin. For tasks where false confidence is genuinely dangerous — medical imaging, infrastructure inspection, legal document analysis — the correct design decision is not better prompting. It is human verification.

The Data Says

Multimodal models in 2026 treat images as probabilistic language problems, not as ground-truth visual inputs. The two hallucination types — omission (insufficient visual confidence) and fabrication (language-prior co-occurrence) — have distinct causes and require distinct mitigations. Token budget arithmetic from tiling adds a third, independent failure mode that scales with image resolution and multi-image session length. As of June 2026, GPT-5.4 Pro held the top position on MMMU-Pro at approximately 94%, with Claude Mythos 5 close behind at 92.7% — though scores vary slightly by source (pricepertoken.com) — but benchmark performance on curated test sets systematically underestimates failure rates on the fine-grained, compositionally complex images found in real workloads. The gap between benchmark score and production reliability is, itself, a kind of hallucination about model capability.