Vision Transformer: Images become patch sequences

The event and the narrow claim

On 22 October 2020, a group of researchers at Google Research submitted a paper with a deliberately literal title: An Image is Worth 16x16 Words. Its claim was precise and, at the time, provocative. A standard Transformer encoder, the same class of model built for language in Attention Is All You Need, could be applied almost unchanged to image classification if the image was first cut into a grid of fixed-size patches and fed in as a sequence of tokens. The authors, Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov and nine colleagues, reported that this "Vision Transformer" (ViT) matched or beat leading convolutional networks on several benchmarks while using substantially less compute to train, provided it was pre-trained on a large enough dataset.

The narrow documented result is worth separating from the broad historical judgment. The paper did not prove that convolution was finished, nor that one architecture would suit every visual task. It made a conditional, evidence-backed claim: that image-specific architectural machinery was not strictly necessary for strong classification, and that data scale could substitute for built-in visual priors. The wider significance, that attention-centered models would spread across computer vision, is a judgment that comes from what happened next, not from the 2020 experiment itself.

Why convolution had held the field

For most of the deep-learning decade, image recognition was synonymous with the convolutional neural network. The architecture encodes two assumptions that match natural images well: locality, meaning nearby pixels are more related than distant ones, and translation equivariance, meaning a feature detector that finds an edge in one corner finds it everywhere. These are inductive biases, structural assumptions baked into the model before it sees any data. They let convolutional networks learn efficiently from moderate amounts of labeled images, which is why they dominated from AlexNet onward and why highly tuned families such as EfficientNet remained the benchmark to beat.

Those same biases were also a constraint. Convolution composes information locally and only aggregates long-range relationships by stacking many layers, so a pixel in one corner influences a pixel in the opposite corner only after the receptive field has grown deep enough to span the image. A Transformer, by contrast, lets any token attend to any other in a single layer. The open question ViT confronted was whether removing convolution's helpful priors would cost more than the flexibility of global attention was worth.

Attention had already been creeping into vision

ViT did not arrive in a field innocent of attention. Several lines of work had spent years grafting attention onto convolutional backbones. Non-local Neural Networks added a self-attention-like operation to capture long-range dependencies in video and images while keeping convolution as the core. Attention Augmented Convolutional Networks concatenated attention feature maps with convolutional ones, and Stand-Alone Self-Attention in Vision Models went further, replacing convolutions with local self-attention entirely, though it required specialized implementations that were hard to run efficiently on accelerators.

The closest predecessor conceptually was the analysis in On the Relationship between Self-Attention and Convolutional Layers, which extracted 2x2 patches and applied full self-attention, showing the approach could work on small images. Separately, OpenAI's Generative Pretraining from Pixels, known as Image GPT, applied a Transformer to sequences of pixels for generative pretraining, but it operated on drastically downscaled images because a token per pixel is prohibitively expensive. ViT's contribution was not the idea of attention on images. It was the specific, scalable recipe: use large patches rather than pixels, keep the Transformer standard, and lean on scale rather than clever local operators.

How the Vision Transformer actually works

The mechanism is almost aggressively simple, which is part of the paper's rhetorical point. Take a 224 by 224 image and split it into a grid of non-overlapping 16 by 16 patches. That yields 196 patches. Flatten each patch into a vector and pass it through a single learned linear projection to produce a patch embedding, the visual analogue of a word embedding. Prepend a learnable classification token whose final state serves as the image representation, and add learned position embeddings so the model knows where each patch sat in the grid. The resulting sequence of 197 tokens flows through a conventional Transformer encoder of stacked multi-head self-attention and feed-forward blocks, exactly as in language models.

Almost nothing about images is hardwired beyond the initial patch cut and the position embeddings. There is no convolution in the body, no pooling pyramid, no hand-designed multi-scale structure. The model must learn spatial relationships, locality and any notion of scale from data. The authors also tested a "hybrid" variant that feeds convolutional feature maps into the Transformer instead of raw patches, which helped smaller models but offered little advantage at large scale. Three sizes were defined, ViT-Base, ViT-Large and ViT-Huge, with the patch size noted in the name, so ViT-H/14 uses 14 by 14 pixel patches and therefore a longer, more expensive token sequence than the 16-patch variants. Two further design choices are telling. The position embeddings are learned one-dimensional vectors, and the authors reported that supplying explicitly two-dimensional coordinates gave no meaningful improvement, evidence that the grid structure could be recovered from data. And the classification decision reads out the single prepended token rather than pooling over all patch tokens, mirroring the sentence-level tokens used in language models and keeping the architecture faithful to its text-domain origin.

The evidence: numbers and the data-scale crossover

The empirical heart of the paper is a story about data volume. Trained only on the roughly 1.3 million images of ImageNet, ViT underperformed comparably sized Big Transfer ResNets: without convolution's priors, it overfit. Pre-trained on the 14-million-image ImageNet-21k, it drew level. Pre-trained on Google's internal 303-million-image JFT-300M dataset, it pulled ahead. This crossover, where larger models with weaker inductive biases win only once the dataset is large enough, is the paper's central and most carefully argued finding.

The headline numbers came from ViT-H/14 pre-trained on JFT-300M. It reached 88.55 percent top-1 accuracy on ImageNet, 90.72 percent on the cleaned ImageNet-ReaL labels, 94.55 percent on CIFAR-100, 97.56 percent on Oxford-IIIT Pets, and 77.63 percent averaged across the 19-task VTAB suite. These are task-specific evaluation results, not any measure of general ability. The efficiency argument was as important as the accuracy: ViT-H/14 required roughly 2.5 thousand TPUv3-core-days of pre-training, against about 9.9 thousand for the BiT-L ResNet and about 12.3 thousand for the Noisy Student EfficientNet it was compared with. Matching state-of-the-art accuracy at a fraction of the training cost was the result that made practitioners pay attention. For reference, the BiT-L ResNet it was measured against reached 87.54 percent on ImageNet and the Noisy Student EfficientNet about 88.4 percent, so ViT's gain was real but modest in accuracy and large in efficiency. One practical detail underlies the transfer numbers: models were pre-trained at lower resolution and fine-tuned at higher resolution, which changes the number of patches, so the learned position embeddings were interpolated to the new grid, a small trick that let a single pre-trained backbone serve many downstream image sizes.

The inductive-bias tradeoff the paper made explicit

The authors were unusually candid about what they were trading away. They wrote plainly that ViT has much less image-specific inductive bias than convolutional networks, so notions of locality and two-dimensional neighborhood structure have to be learned from scratch rather than assumed. That honesty is the paper's intellectual signature. Rather than claiming a free lunch, it framed the result as a bargain: give up the priors, pay for it with data, and gain a simpler, more scalable, more general architecture.

This is why the event resists the "convolution is obsolete" reading that circulated afterward. On mid-sized datasets, the priors still won. The paper's own results showed ResNets ahead in the low-data regime, and it explicitly positioned scale, not architecture alone, as the enabling condition. The correct summary is conditional: when very large pre-training data is available, the flexibility of attention can outweigh the efficiency of convolution's assumptions. Strip away the condition and the claim no longer holds. The same logic explains why the hybrid variant helped at small scale and faded at large scale: convolutional features are a useful prior when data is limited, and a redundant crutch when the Transformer has enough examples to learn the equivalent structure itself.

The competing approaches of its moment

ViT did not enter an empty arena. Its direct rivals were the strongest convolutional pipelines of 2019 and 2020: EfficientNet trained with the Noisy Student self-training procedure, and the Big Transfer recipe that transferred large pre-trained ResNets to many downstream tasks. These were not obsolete baselines; they held the ImageNet records ViT was measured against, and on equal mid-sized data they remained competitive.

A pointed rebuttal arrived within two months. Facebook's Data-efficient Image Transformers, or DeiT, showed that a Vision Transformer could be trained to strong ImageNet accuracy using only ImageNet-1k, no giant proprietary dataset required, by combining aggressive data augmentation, careful regularization and a distillation token that learns from a convolutional teacher. DeiT reframed the debate. ViT had argued that scale substitutes for inductive bias; DeiT argued that better training recipes could partly substitute for scale, widening access to a method that had otherwise depended on data few labs possessed. The two papers together clarified that the real levers were data, regularization and architecture jointly, not any single one.

The downstream lineage

The strongest evidence for ViT's significance is not its 2020 benchmark table but the research it enabled. Because a plain ViT produces a single-scale sequence, it was awkward as a backbone for dense tasks like detection and segmentation. The Swin Transformer answered this by computing attention within shifted local windows and building a hierarchical, multi-scale feature pyramid, making Transformers practical general-purpose vision backbones and reintroducing a measured dose of locality. Object detection had actually been reframed around Transformers slightly earlier by End-to-End Object Detection with Transformers, or DETR, though it used a convolutional feature extractor feeding a Transformer rather than a pure patch model.

Self-supervised learning proved a natural fit. DINO showed that self-distillation with no labels produced ViT features whose attention maps segmented objects without ever being trained to, an emergent property that drew wide interest. Masked Autoencoders then masked a large fraction of image patches and trained a ViT to reconstruct them, an efficient pretraining scheme directly enabled by the patch-token formulation. Perhaps most consequentially, CLIP adopted a ViT image encoder to align images and text at scale, and that class of vision-language encoder became a standard component of multimodal systems and text-to-image generators. The patch-sequence view of images became the connective tissue that let vision plug into the same architectural ecosystem as language. It also carried into generation: the diffusion models behind much later image synthesis increasingly adopted Transformer backbones operating on latent patches, and video and audio models reused the same tokenize-then-attend template. Each of these is a distinct contribution with its own ideas, but each reused ViT's core move of treating a signal as a sequence of embedded patches.

What the paper did not claim, and its honest limits

Careful placement requires noting what ViT left unsettled. Self-attention over patches is quadratic in the number of tokens, so higher-resolution images, which mean more patches, become expensive quickly, a constraint that motivated much of the windowed and hierarchical work that followed. The dependence on large-scale pre-training, whether the proprietary JFT-300M or a strong distillation recipe, meant the vanilla method was not obviously reproducible by every lab, a limitation DeiT partly addressed but did not erase. The paper reported classification results; it did not establish that a bare ViT was the best structure for detection, segmentation or video, where multi-scale reasoning matters and where hybrid or hierarchical designs often won.

There is also a subtler caveat about interpretation. That attention maps sometimes align with salient objects, as DINO vividly showed, is an appealing observation, but attention weights are not by themselves faithful explanations of a model's decisions, a caution established in the broader literature on attention. Historical importance and mechanistic transparency are separate questions. ViT reorganized what was easy to build in vision; it did not make vision models self-explaining.

How to place ViT in AI history

The defensible reading is that Vision Transformer was a unifying architectural pivot, not a lone invention. Its narrow contribution was to demonstrate, with concrete benchmarks and a clear compute accounting, that a standard Transformer applied to image patches could compete with the best convolutional networks once pre-training data was large enough. Its broader importance is that it collapsed a long-standing divide: after ViT, images and text could be modeled with the same core machinery, so advances in one modality could transfer to the other, and multimodal systems became architecturally natural rather than a bolted-together compromise.

A useful way to navigate the timeline around this event is to ask three questions of it. Which earlier idea did it depend on? The Transformer encoder and years of attention-in-vision experiments. What did it actually change? It removed the assumption that image models must be built from convolution, replacing image-specific priors with data scale. And which of its limits required later discoveries? Efficient high-resolution attention, label-free pretraining, hierarchical backbones and data-efficient training were all follow-on problems. Answered that way, ViT reads as the moment computer vision joined the Transformer era, with the honest qualification that it did so under conditions, and that convolution's assumptions kept earning their keep wherever data was scarce or structure was cheap.

References

• Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
• Vaswani et al. (2017). Attention Is All You Need.
• Chen et al. (2020). Generative Pretraining from Pixels.
• Wang et al. (2018). Non-local Neural Networks.
• Bello et al. (2019). Attention Augmented Convolutional Networks.
• Ramachandran et al. (2019). Stand-Alone Self-Attention in Vision Models.
• Cordonnier, Loukas and Jaggi (2019). On the Relationship between Self-Attention and Convolutional Layers.
• Kolesnikov et al. (2019). Big Transfer: General Visual Representation Learning.
• Sun et al. (2017). Revisiting Unreasonable Effectiveness of Data in Deep Learning Era.
• Touvron et al. (2020). Training data-efficient image transformers and distillation through attention.
• Carion et al. (2020). End-to-End Object Detection with Transformers.
• Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.
• Caron et al. (2021). Emerging Properties in Self-Supervised Vision Transformers.
• He et al. (2021). Masked Autoencoders Are Scalable Vision Learners.
• Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision.