Vision Transformer

Authors 6 articles 70 min total read
Definition
A deep-learning architecture that treats an image as a sequence of small fixed-size patches and processes them with the same Transformer encoder used for language, replacing convolutions with self-attention across all patches at every layer.
Also called: ViT, Image Transformer, Transformer for Images

What a Vision Transformer is

If you’ve used visual search in Google Lens or a multimodal chatbot that can “see” screenshots, there’s a good chance a Vision Transformer is doing the looking. Before 2020, almost every serious computer vision system was built on convolutional neural networks (CNNs) — architectures that scan images with small filters, much like a magnifying glass moved across a photograph. ViT threw that playbook out. Instead of scanning, it cuts the image into a grid of patches and lets every patch attend to every other patch through self-attention — the same mechanism that powers ChatGPT and Claude.

Take a 224×224 image and chop it into a 14×14 grid of 16×16-pixel patches. Flatten each patch into a vector, run it through a linear layer to get a patch embedding, and add a learned position embedding so the model knows which square came from where. According to Dosovitskiy et al., that’s the entire input preparation — an image becomes a sequence of tokens, and from that point the model is identical to a language Transformer.

Stacked on top of the patch embeddings is a standard Transformer encoder: alternating blocks of multi-head self-attention and feed-forward layers wrapped in residual connections and layer normalization. A special [CLS] token is prepended to the sequence, and its final representation is used for classification. According to Hugging Face ViT Docs, the canonical setup produces 197 tokens (196 patches plus [CLS]) for a 224×224 image, and model sizes range from ViT-Base at around 86M parameters to ViT-Huge at around 630M. The original paper’s headline result: given enough pretraining data, this plain transformer matches or beats the best CNNs on image classification — no convolutions required.

Self-attention over patches drops the built-in assumptions of convolution, so data or a pretrained backbone has to supply them. · More AI cheat sheets
  • Attention replaces convolution’s built-in spatial assumptions, which is why a Vision Transformer needs far more training data or a strong pretrained backbone to match a well-tuned CNN.
  • Backbone choice now splits by job: SigLIP 2 for image-text alignment, DINOv2/DINOv3 for dense features, ConvNeXt hybrids where convolution still earns its place.
  • Fine-tuning a modern ViT backbone is PEFT LoRA on the attention projections plus a trainable head, not a full retrain.
  • Patch-level attacks and uncurated training bias are real production risks once a ViT backbone reaches a high-stakes system.

How a Vision Transformer is used in practice

Today, most people encounter ViT without knowing it. When you drag a screenshot into Claude, ChatGPT, or Gemini and ask what an error means, a ViT-style encoder is almost certainly processing the image before the language half generates an answer. The same is true when you search photos by description, auto-crop with a design tool, or run medical imaging software that highlights suspicious regions.

The second place ViT shows up is as a pretrained feature extractor for teams building their own products. Instead of training from scratch — which demands millions of labeled images and weeks of GPU time — a team can download a ViT pretrained on a huge image collection with self-supervised learning and fine-tune it on their task. According to Meta AI Blog, the DINOv3 family of self-supervised ViTs scales to 7 billion parameters trained on roughly 1.7 billion unlabeled images, and these checkpoints are now a default starting point for work from satellite imagery analysis to retail product recognition.

If you need strong vision features but have a small labeled dataset, skip training from scratch. Start with a self-supervised ViT like DINOv3 or SigLIP 2 as a frozen backbone and only fine-tune the task head — you’ll usually beat a from-scratch CNN with a fraction of the data.

Reading the vision transformer stack: patches before backbones

Start with how image patches replaced convolutions in computer vision — it establishes the patch-as-token substitution everything else here assumes. Patch embeddings, class tokens, and 2D positional encoding opens the hood on how those patches become a sequence a transformer can actually read, and the CNN-to-ViT prerequisites and hard limits is the counterweight: what a convolutional background does and doesn’t transfer, and why the architecture is data-hungry without it.

Once the mechanism is settled, the SigLIP 2, DINOv2, and ViT fine-tuning guide turns backbone choice into a build decision instead of a leaderboard screenshot. For the market context behind that choice, the 2026 vision-backbone landscape tracks how the field split into three specialized tracks rather than converging on one winner. Close with the ethical risks of biased training data and patch-level attacks before any ViT backbone reaches a hospital, a surveillance system, or another high-stakes deployment.

MAX asks: 'I pointed my Vision Transformer at high-resolution photos and training time exploded — why?' MONA answers: 'Attention compares every patch to every other patch, so doubling resolution roughly quadruples the comparisons. Convolution never billed you that way.' — comic dialog.
More patches capture more detail, and attention charges for every comparison.

Where the vision transformer stops and its neighbours begin

Three neighbours get folded into “vision transformer” in casual conversation, and each folding hides a different decision.

A vision transformer is not a convolutional network wearing a new name. A CNN slides a fixed filter across the image, so nearby pixels influence each other for free; a ViT patch carries no such assumption, which is exactly why the CNN-to-ViT prerequisites piece treats data volume as a real cost, not a footnote. That the industry hasn’t abandoned convolution either — ConvNeXt hybrids keep reappearing inside production stacks — is the tell that neither design fully replaced the other.

Patch embedding is also not tokenization. Tokenizer architecture maps discrete text units to vocabulary IDs; a ViT patch embedding linearly projects continuous pixel values into a vector, with no vocabulary involved. Calling both “tokenizing” hides that a ViT has no fixed vocabulary to run out of — but it does have a fixed patch size that decides how much detail survives the cut.

And a vision transformer is usually just an encoder. Classification and dense-feature backbones stop at the encoder stack; generative tasks like image captioning bolt on a decoder, at which point the design becomes an instance of encoder-decoder architecture rather than a plain ViT. Confusing the two blames the vision backbone for failures that actually belong to the decoder generating the language.

When a Vision Transformer fits — and when it doesn't

ScenarioUseAvoid
Image classification with hundreds of millions of pretraining images✅
Tiny dataset with no pretrained backbone available❌
Multimodal model where text and images share one encoder stack✅
Real-time inference on older phone CPUs with tight latency budgets❌
Segmentation or depth estimation using pretrained ViT features✅
Ultra-low-latency video pipelines where convolution-friendly hardware still wins❌

Common questions about vision transformers

Q: Which Vision Transformer article should I read first if I already know convolutional networks? A: Start with how image patches replaced convolutions — it maps the patch-as-token substitution onto what a CNN already taught you, so the components read next (class tokens, positional encoding) lands as elaboration, not a fresh start.

Q: Why does my fine-tuned Vision Transformer underperform the numbers on its model card? A: Almost always a mismatched recipe, not a bad backbone — loading a naflex checkpoint with the wrong model class, or full-fine-tuning a giant backbone on a small labeled set, both quietly erase accuracy. The SigLIP 2 and DINOv2 fine-tuning guide lists the exact failure patterns.

Q: Do I need to understand class tokens and positional encoding just to use a pretrained Vision Transformer? A: Not to run inference — a checkpoint works behind one function call. The moment fine-tuning results look wrong, the components walkthrough is what tells you whether the class token, the positional encoding, or your own labels are the problem.

Q: Is patch-level attack risk a reason to avoid Vision Transformers in security-sensitive systems? A: It’s a reason to audit, not to avoid — the same patch-based processing that makes ViT data-hungry also gives an attacker a narrow, corruptible surface, on top of training data no one curated. The ethical risks piece frames what to check before deployment, not after.

Part of the transformer and attention internals theme · closest neighbour: encoder-decoder architecture. New to this from a software background? Start with the story: Transformer Internals for Developers: What Maps, What Breaks.

Sources

1

Understand the Fundamentals

Vision transformers challenge the idea that images need convolutional priors to be understood. Explore how treating image patches as tokens unlocks global attention and what that means for how machines learn to see.

2

Build with Vision Transformer

These guides walk you through fine-tuning modern vision transformer backbones for real classification, detection, and multimodal tasks. Expect practical trade-offs between compute budgets, data requirements, and the pre-trained backbone you choose.

4

Risks and Considerations

Vision transformers inherit biases from massive training datasets and can be fooled by carefully crafted patch-level attacks. Consider these risks before deploying them in medical imaging, surveillance, or other high-stakes visual systems.