
What Is a Vision Transformer and How Image Patches Replaced Convolutions in Computer Vision
Vision Transformers treat images as token sequences, not pixel grids. Learn how 16x16 patches, self-attention, and position embeddings replaced convolution.
If you’ve used visual search in Google Lens or a multimodal chatbot that can “see” screenshots, there’s a good chance a Vision Transformer is doing the looking. Before 2020, almost every serious computer vision system was built on convolutional neural networks (CNNs) — architectures that scan images with small filters, much like a magnifying glass moved across a photograph. ViT threw that playbook out. Instead of scanning, it cuts the image into a grid of patches and lets every patch attend to every other patch through self-attention — the same mechanism that powers ChatGPT and Claude.
Take a 224×224 image and chop it into a 14×14 grid of 16×16-pixel patches. Flatten each patch into a vector, run it through a linear layer to get a patch embedding, and add a learned position embedding so the model knows which square came from where. According to Dosovitskiy et al., that’s the entire input preparation — an image becomes a sequence of tokens, and from that point the model is identical to a language Transformer.
Stacked on top of the patch embeddings is a standard Transformer encoder: alternating blocks of multi-head self-attention and feed-forward layers wrapped in residual connections and layer normalization. A special [CLS] token is prepended to the sequence, and its final representation is used for classification. According to Hugging Face ViT Docs, the canonical setup produces 197 tokens (196 patches plus [CLS]) for a 224×224 image, and model sizes range from ViT-Base at around 86M parameters to ViT-Huge at around 630M. The original paper’s headline result: given enough pretraining data, this plain transformer matches or beats the best CNNs on image classification — no convolutions required.
Today, most people encounter ViT without knowing it. When you drag a screenshot into Claude, ChatGPT, or Gemini and ask what an error means, a ViT-style encoder is almost certainly processing the image before the language half generates an answer. The same is true when you search photos by description, auto-crop with a design tool, or run medical imaging software that highlights suspicious regions.
The second place ViT shows up is as a pretrained feature extractor for teams building their own products. Instead of training from scratch — which demands millions of labeled images and weeks of GPU time — a team can download a ViT pretrained on a huge image collection with self-supervised learning and fine-tune it on their task. According to Meta AI Blog, the DINOv3 family of self-supervised ViTs scales to 7 billion parameters trained on roughly 1.7 billion unlabeled images, and these checkpoints are now a default starting point for work from satellite imagery analysis to retail product recognition.
If you need strong vision features but have a small labeled dataset, skip training from scratch. Start with a self-supervised ViT like DINOv3 or SigLIP 2 as a frozen backbone and only fine-tune the task head — you’ll usually beat a from-scratch CNN with a fraction of the data.
Start with how image patches replaced convolutions in computer vision — it establishes the patch-as-token substitution everything else here assumes. Patch embeddings, class tokens, and 2D positional encoding opens the hood on how those patches become a sequence a transformer can actually read, and the CNN-to-ViT prerequisites and hard limits is the counterweight: what a convolutional background does and doesn’t transfer, and why the architecture is data-hungry without it.
Once the mechanism is settled, the SigLIP 2, DINOv2, and ViT fine-tuning guide turns backbone choice into a build decision instead of a leaderboard screenshot. For the market context behind that choice, the 2026 vision-backbone landscape tracks how the field split into three specialized tracks rather than converging on one winner. Close with the ethical risks of biased training data and patch-level attacks before any ViT backbone reaches a hospital, a surveillance system, or another high-stakes deployment.

Three neighbours get folded into “vision transformer” in casual conversation, and each folding hides a different decision.
A vision transformer is not a convolutional network wearing a new name. A CNN slides a fixed filter across the image, so nearby pixels influence each other for free; a ViT patch carries no such assumption, which is exactly why the CNN-to-ViT prerequisites piece treats data volume as a real cost, not a footnote. That the industry hasn’t abandoned convolution either — ConvNeXt hybrids keep reappearing inside production stacks — is the tell that neither design fully replaced the other.
Patch embedding is also not tokenization. Tokenizer architecture maps discrete text units to vocabulary IDs; a ViT patch embedding linearly projects continuous pixel values into a vector, with no vocabulary involved. Calling both “tokenizing” hides that a ViT has no fixed vocabulary to run out of — but it does have a fixed patch size that decides how much detail survives the cut.
And a vision transformer is usually just an encoder. Classification and dense-feature backbones stop at the encoder stack; generative tasks like image captioning bolt on a decoder, at which point the design becomes an instance of encoder-decoder architecture rather than a plain ViT. Confusing the two blames the vision backbone for failures that actually belong to the decoder generating the language.
| Scenario | Use | Avoid |
|---|---|---|
| Image classification with hundreds of millions of pretraining images | ✅ | |
| Tiny dataset with no pretrained backbone available | ❌ | |
| Multimodal model where text and images share one encoder stack | ✅ | |
| Real-time inference on older phone CPUs with tight latency budgets | ❌ | |
| Segmentation or depth estimation using pretrained ViT features | ✅ | |
| Ultra-low-latency video pipelines where convolution-friendly hardware still wins | ❌ |
Q: Which Vision Transformer article should I read first if I already know convolutional networks? A: Start with how image patches replaced convolutions — it maps the patch-as-token substitution onto what a CNN already taught you, so the components read next (class tokens, positional encoding) lands as elaboration, not a fresh start.
Q: Why does my fine-tuned Vision Transformer underperform the numbers on its model card? A: Almost always a mismatched recipe, not a bad backbone — loading a naflex checkpoint with the wrong model class, or full-fine-tuning a giant backbone on a small labeled set, both quietly erase accuracy. The SigLIP 2 and DINOv2 fine-tuning guide lists the exact failure patterns.
Q: Do I need to understand class tokens and positional encoding just to use a pretrained Vision Transformer? A: Not to run inference — a checkpoint works behind one function call. The moment fine-tuning results look wrong, the components walkthrough is what tells you whether the class token, the positional encoding, or your own labels are the problem.
Q: Is patch-level attack risk a reason to avoid Vision Transformers in security-sensitive systems? A: It’s a reason to audit, not to avoid — the same patch-based processing that makes ViT data-hungry also gives an attacker a narrow, corruptible surface, on top of training data no one curated. The ethical risks piece frames what to check before deployment, not after.
Part of the transformer and attention internals theme · closest neighbour: encoder-decoder architecture. New to this from a software background? Start with the story: Transformer Internals for Developers: What Maps, What Breaks.
Vision transformers challenge the idea that images need convolutional priors to be understood. Explore how treating image patches as tokens unlocks global attention and what that means for how machines learn to see.
Concepts covered

Vision Transformers treat images as token sequences, not pixel grids. Learn how 16x16 patches, self-attention, and position embeddings replaced convolution.

Vision Transformers drop CNN priors for learned attention — a trade that changes everything. Learn the prerequisites, CNN mappings, and hard limits of ViT.

How Vision Transformers turn images into token sequences — inside patch embeddings, the CLS token, and the shift from 1D to modern 2D positional encoding.
These guides walk you through fine-tuning modern vision transformer backbones for real classification, detection, and multimodal tasks. Expect practical trade-offs between compute budgets, data requirements, and the pre-trained backbone you choose.
Tools & techniques

Pick the right Vision Transformer backbone for 2026. Spec-first guide to fine-tuning SigLIP 2, DINOv2, and ViT with Hugging Face, PyTorch, and PEFT LoRA.
New self-supervised and contrastive approaches are changing what multimodal systems can see. Track which architectures are winning benchmarks and why convolutional networks keep returning.
Models & benchmarks
Updated April 2026

The vision backbone race split into three tracks. Why SigLIP 2, DINOv3, and ConvNeXt hybrids now power every major multimodal AI stack in 2026.
Vision transformers inherit biases from massive training datasets and can be fooled by carefully crafted patch-level attacks. Consider these risks before deploying them in medical imaging, surveillance, or other high-stakes visual systems.
Risks & metrics

Vision Transformers deployed in healthcare and surveillance inherit bias from web-scraped datasets. From LAION to CheXzero — who bears the cost of scale?