Vision Transformers: From ViT to DINOv2

ISRO's Bhuvan portal serves Sentinel-2 and Resourcesat tiles at 10-metre resolution to agricultural monitoring teams tracking crop stress across Punjab and Haryana. A wilting signature in the northeast corner of a tile is often caused not by anything local — a pest, a soil defect — but by a blocked irrigation channel two kilometres away, dozens of pixels distant in the same image. A convolutional network reads that tile through kernels that see 3x3 or 5x5 neighbourhoods at a time; a signal has to pass through many stacked layers before the receptive field of any single output unit grows large enough to span both the blockage and the stressed field. Depth-4 layers of 3x3 convolutions gives a receptive field of roughly 9 pixels across; you need on the order of ten to fifteen stacked layers before two points 20+ pixels apart can influence the same output unit at all, and even then the influence is diluted by everything the kernel swept over in between. A Vision Transformer sidesteps this entirely: from the very first layer, every patch attends to every other patch in the image with an unweighted, un-decayed connection — the irrigation blockage and the wilting field are one dot product apart, at layer one, not layer twelve. This is the actual mechanical reason ViTs matter for exactly this kind of spatially non-local pattern, and it is the thread this chapter follows from the original ViT architecture through to DINOv2, the self-supervised training regime that makes ViT features usable in domains — like satellite agriculture — where labelled data barely exists.

Patch embedding: turning an image into a sequence

A transformer encoder, as covered in the G11 transformer-internals chapter, consumes a sequence of vectors. An image is a 2D grid of pixels, not a sequence, so the first job of a Vision Transformer (Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," ICLR 2021) is to tokenize it. The image is cut into non-overlapping square patches — the canonical ViT-Base/16 configuration uses 16x16 pixel patches on a 224x224 input, giving 224/16 = 14 patches per side, 14 x 14 = 196 patches total. Each patch, originally a 16x16x3 tensor (three RGB channels), is flattened into a single vector of length 16 x 16 x 3 = 768 and passed through one shared linear projection E to produce a 768-dimensional patch embedding. Because that projection is shared across all 196 patches, it is implemented in practice as a single Conv2d layer with kernel size and stride both set to the patch size — the convolution's "sliding window" ends up visiting each patch exactly once with no overlap, which is mathematically identical to flattening each patch and multiplying by a shared weight matrix.

Self-attention has no built-in notion of position — the attention operation is a weighted average over values, weighted by content similarity between queries and keys, and that weighting is exactly the same whether patch 7 sits at the top-left or bottom-right of the image. So ViT adds a learned position embedding, one 768-dimensional vector per patch slot, added elementwise to each patch embedding before the first encoder block. A learnable classification token ([CLS]) is also prepended to the sequence — its final-layer output vector is what the classification head reads, rather than any individual patch's output, so that the model has one designated summary slot instead of having to pick a patch arbitrarily. With the CLS token, ViT-Base/16 processes a sequence of 197 tokens, each 768-dimensional.

Worked example: hand-tracing patch embedding and attention on a 4x4 toy image

Real ViT-Base numbers (196 patches, 768 dimensions) are too large to trace by hand, so here is the identical mechanism on a 4x4 grayscale image with patch size 2x2, giving 4 patches, projected down to embedding dimension 2. Every step below is arithmetic you can re-derive.

Image pixels (row-major):
 1  2  5  6
 3  4  7  8
 9 10 13 14
11 12 15 16

Cutting this into 2x2 non-overlapping patches in raster order (top-left, top-right, bottom-left, bottom-right), and flattening each patch row-major, gives:

patch1 = [1, 2, 3, 4]      (top-left)
patch2 = [5, 6, 7, 8]      (top-right)
patch3 = [9, 10, 11, 12]   (bottom-left)
patch4 = [13, 14, 15, 16]  (bottom-right)

Project each 4-vector to a 2-dimensional embedding using the shared weight matrix W where column 1 sums positions 1 and 3 of the flattened patch, and column 2 sums positions 2 and 4 — i.e. e = [x1+x3, x2+x4]:

e1 = [1+3, 2+4]     = [4, 6]
e2 = [5+7, 6+8]     = [12, 14]
e3 = [9+11, 10+12]  = [20, 22]
e4 = [13+15, 14+16] = [28, 30]

Now compute single-head self-attention with Q = K = V = identity (each token's query, key, and value equal its own embedding, to keep the arithmetic visible) for patch 1 as the query, scaled by 1/sqrt(d_k) with d_k = 2, so the scale factor is 1/sqrt(2) ≈ 0.7071:

score(1,1) = q1·k1 = 4x4 + 6x6   = 16+36 = 52   -> 52/1.4142   = 36.77
score(1,2) = q1·k2 = 4x12 + 6x14 = 48+84 = 132  -> 132/1.4142  = 93.34
score(1,3) = q1·k3 = 4x20 + 6x22 = 80+132 = 212 -> 212/1.4142  = 149.9
score(1,4) = q1·k4 = 4x28 + 6x30 = 112+180 = 292 -> 292/1.4142 = 206.5

Softmax over [36.77, 93.34, 149.9, 206.5] is, for all practical purposes, one-hot: the gap between the largest score and the rest is on the order of 50-100 in the exponent, so exp(206.5) dominates exp(149.9) by a factor of roughly e^56.6 — an astronomically large ratio. Patch 1's attention output collapses to essentially just v4 = e4 = [28, 30], ignoring every other patch including itself. This is not a toy artifact to wave away — it is a real failure mode. If one patch in a satellite tile happens to have unusually bright raw pixel values (a sun-glint off water, a sensor saturation artifact, a cloud edge), its raw dot-product scores with every query will be inflated purely by magnitude, and attention will fixate on that one patch regardless of whether it is semantically relevant.

This is exactly why every ViT (and every modern transformer) block applies LayerNorm to each token before computing Q, K, and V — a "pre-norm" architecture. LayerNorm normalizes a single token's own feature vector to zero mean and unit variance, independent of every other token and every other example in the batch: for a vector x with mean m and standard deviation s (computed across the embedding dimension only), the output is (x - m) / s. Applying this to e1 = [4, 6]: mean m = 5, variance = ((4-5)^2 + (6-5)^2)/2 = 1, so s = 1, giving normalized output [-1, 1]. Because every one of the four toy patches here is a constant-offset shift of the pattern [c, c+1, c+2, c+3] (patch2's pixels are exactly patch1's plus 4, and so on), and because the specific W chosen makes each e a two-vector of the form [a, a+2], LayerNorm collapses all four patches to the identical [-1, 1] — a genuine mathematical fact about 2-dimensional LayerNorm with no learned gain or bias: for any two-element vector [p, q] with p ≠ q, the normalized output is always exactly [sign(p-q), -sign(p-q)], because the two-element mean and standard deviation reduce the normalization to a pure sign extraction. This is a dimensionality artifact of using D=2 for hand-tracing, not a property of real ViT-Base, where D=768 gives LayerNorm 768 independent directions to preserve relative structure in — collapsing to a near-binary code only happens when the embedding dimension itself is pathologically small. The operation is identical at D=768; there is just far more room for it to preserve information.

A common point of confusion here: students who have seen BatchNorm in CNN courses assume LayerNorm works the same way, just "in a transformer instead." It does not. BatchNorm normalizes each channel using statistics computed across the batch dimension (all images in the minibatch, that channel, every spatial position) — which is why BatchNorm behaves differently at training time (batch statistics) versus inference time (a running average, frozen), and why it degrades with small batch sizes. LayerNorm normalizes each token independently, using only that token's own feature vector — no dependency on batch size, no discrepancy between train and inference, no running statistics to track. This is essential for ViT specifically because self-attention already mixes information across tokens within a sequence; adding a normalization that also mixes statistics across the batch would create a second, uncontrolled channel of cross-example information leakage on top of it. It is also why ViT tolerates small-batch fine-tuning and single-image inference far more gracefully than a BatchNorm-based CNN does.

The encoder block, and the real parameter count

Each of ViT-Base's 12 encoder blocks runs: LayerNorm, then multi-head self-attention across all 197 tokens (12 heads, 64 dimensions per head, since 768/12 = 64), added back to the input as a residual; then a second LayerNorm, then a two-layer MLP that expands 768 to 3072 and projects back to 768 (a 4x expansion, GELU activation between the layers), again added as a residual. Every block preserves the (197, 768) shape — depth adds representational power, not sequence length or width. Self-attention's compute cost is O(N^2 x D) per layer, dominated by the N x N score matrix; at N = 197 this is manageable, but it is the term that makes very high-resolution ViTs expensive, since doubling image side length quadruples patch count.

It is worth sizing where the parameters actually live. The patch-embedding convolution has weight shape (768 output channels, 3 input channels, 16, 16), i.e. 768 x 3 x 16 x 16 = 768 x 768 = 589,824 weights plus 768 biases = 590,592 parameters — the flattened-patch dimension (3 x 16 x 16 = 768) happens to equal ViT-Base's embedding dimension, which is a deliberate but not universal choice (ViT-Large uses D=1024 with the same 16x16x3=768 flattened patch, so its projection genuinely changes dimensionality). The learned position embedding table is (197, 768) = 151,296 parameters, and the CLS token itself is 768 parameters. Summed, tokenization and position information account for roughly 742,656 parameters — under 1% of ViT-Base/16's approximately 86 million total parameters. The other 99%+ lives in the 12 stacked attention and MLP blocks, confirming that depth of processing, not the input tokenizer, is where a ViT's capacity sits.

Code: a minimal ViT forward pass

import torch
import torch.nn as nn

class MinimalViT(nn.Module):
    def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768,
                 depth=12, num_heads=12, mlp_ratio=4.0, num_classes=1000):
        super().__init__()
        self.patch_embed = nn.Conv2d(in_chans, embed_dim,
                                      kernel_size=patch_size, stride=patch_size)
        num_patches = (img_size // patch_size) ** 2
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=embed_dim, nhead=num_heads,
            dim_feedforward=int(embed_dim * mlp_ratio),
            activation="gelu", batch_first=True, norm_first=True)
        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=depth)
        self.norm = nn.LayerNorm(embed_dim)
        self.head = nn.Linear(embed_dim, num_classes)

    def forward(self, x):
        B = x.shape[0]
        x = self.patch_embed(x)                 # (B, 768, 14, 14)
        x = x.flatten(2).transpose(1, 2)         # (B, 196, 768)
        cls = self.cls_token.expand(B, -1, -1)   # (B, 1, 768)
        x = torch.cat((cls, x), dim=1)           # (B, 197, 768)
        x = x + self.pos_embed                   # (B, 197, 768)
        x = self.encoder(x)                      # (B, 197, 768)
        x = self.norm(x)
        return self.head(x[:, 0])                # (B, 1000), CLS token only

model = MinimalViT()
x = torch.randn(2, 3, 224, 224)
logits = model(x)
print(logits.shape)   # torch.Size([2, 1000])

Tracing the shapes: the Conv2d with kernel 16 and stride 16 on a 224-pixel input produces floor((224-16)/16)+1 = 14 output positions per side, so patch_embed(x) has shape (B, 768, 14, 14). flatten(2) merges the two spatial dimensions into 196, giving (B, 768, 196); transpose(1, 2) swaps channel and sequence axes to (B, 196, 768), the (tokens, features) layout every subsequent operation expects. Concatenating the CLS token along the sequence dimension gives (B, 197, 768), and the position embedding — already shaped (1, 197, 768) — broadcasts and adds elementwise without changing shape. nn.TransformerEncoder with norm_first=True (pre-LN, matching ViT's actual architecture rather than the original 2017 post-LN transformer) preserves (B, 197, 768) through all 12 layers. Only the final line, indexing x[:, 0] to keep just the CLS token, drops the sequence dimension, producing (B, 768) before the linear head maps to (B, 1000). With B=2, the print statement correctly reports torch.Size([2, 1000]).

Misconception: "no inductive bias" does not mean "always better"

A convolutional kernel bakes in two assumptions before it ever sees data: locality (nearby pixels are more likely to be related) and translation equivariance (a pattern learned in one location works the same way shifted to another). Self-attention imposes neither — a patch can attend fully to any other patch regardless of distance, and there is nothing analogous to weight-sharing across positions beyond the shared patch-embedding projection. It is tempting to read this as strictly better: more flexibility, fewer restrictive assumptions, so ViT should dominate. The original ViT paper's own results say otherwise. Trained from scratch on ImageNet-1k's 1.28 million images, ViT underperforms similarly-sized ResNets, because those CNN inductive biases function as a built-in regularizer — a helpful shortcut that lets a CNN generalize from comparatively little data, which a ViT must instead discover purely from examples. ViT's advantage only appears at larger pretraining scale: pretrained on JFT-300M (300 million images, a Google-internal dataset) and fine-tuned, ViT-Large/16 and ViT-Huge/14 reach roughly high-80s percent top-1 accuracy on ImageNet, matching or exceeding comparable CNNs of the era — because with enough data, unconstrained attention discovers spatial relationships the CNN's fixed local-then-global schema cannot express as efficiently. Touvron et al.'s DeiT ("Training data-efficient image transformers & distillation through attention," ICML 2021) later closed much of this gap without hundreds of millions of images, using aggressive augmentation (RandAugment, Mixup, CutMix) and a distillation token trained to match a CNN teacher's predictions, reaching roughly 83% top-1 on ImageNet-1k alone. The correct framing is a trade, not a free upgrade: ViT exchanges built-in spatial priors for raw capacity, and that capacity only pays off once the model is given either enough data or a teacher to distill priors from.

Self-supervised pretraining: DINO's self-distillation

Labelled ImageNet-scale datasets are the exception, not the rule, in most real deployments — nobody has hand-labelled millions of satellite tiles for "irrigation-stressed" versus "healthy." DINO (Caron et al., "Emerging Properties in Self-Supervised Vision Transformers," ICCV 2021) trains a ViT with zero labels using self-distillation. Two networks with identical architecture — a student and a teacher — both process augmented crops of the same unlabeled image. The teacher only ever sees "global" crops covering most of the image; the student sees both global crops and several smaller "local" crops covering less than half of it. Each network's output passes through a small MLP head and a softmax over a large number of "prototype" dimensions (65,536 in the original paper), producing a probability distribution. The student is trained to match the teacher's output distribution via cross-entropy, across every local-global crop pairing — forcing the student to infer, from a small cropped fragment, what the same underlying image looks like when the teacher sees the whole thing. This "local-to-global" correspondence, learned with no label ever stating that the two crops come from the same photo, is the entire self-supervised signal.

DINO uses no negative pairs — unlike contrastive methods such as SimCLR (Chen et al., 2020) or MoCo (He et al., 2020), which explicitly push apart representations of different images, DINO never compares against a different image at all. That immediately raises the obvious collapse risk: if the loss only ever asks the student to match the teacher, the trivial solution — both networks output the same constant vector for every input — achieves zero loss without learning anything. DINO avoids this with two mechanisms applied only on the teacher side. Centering subtracts a running mean of recent teacher outputs from the teacher's logits before the softmax, preventing any single prototype dimension from dominating every image (the easiest collapse for the student to copy). Sharpening uses a lower softmax temperature on the teacher than on the student, producing a more confidently peaked distribution, which blocks the second collapse mode — smearing probability uniformly across all prototypes. Additionally, the teacher's weights are never updated by backpropagation at all; they are an exponential moving average (EMA) of the student's weights, θ_teacher ← m·θ_teacher + (1-m)·θ_student, with momentum m following a cosine schedule from 0.996 toward 1 over training, and gradients are explicitly stopped from flowing into the teacher branch. A slowly-moving, never-directly-trained teacher combined with centering and sharpening is what replaces the role negative pairs play elsewhere. The paper's headline empirical finding is that the resulting [CLS] token's attention maps, with no segmentation labels anywhere in training, correlate strongly with actual object boundaries — self-supervised attention that looks like unsupervised segmentation.

DINOv2: a general-purpose frozen vision backbone

DINOv2 (Oquab et al., 2023, Meta AI Research) scales this recipe into a backbone meant to be used frozen, across tasks, without fine-tuning. Three changes matter most. First, the training data: rather than a hand-curated academic dataset, DINOv2 builds LVD-142M, 142 million images automatically selected from a much larger uncurated web-image pool via retrieval-based deduplication and clustering against curated seed datasets — curation without manual labeling. Second, the objective is no longer only image-level: DINOv2 combines the original DINO [CLS]-token distillation loss with a patch-level loss borrowed from iBOT (Zhou et al., "iBOT: Image BERT Pre-Training with Online Tokenizer," ICLR 2022), where some patch tokens are masked and the student must predict the teacher's representation of the masked patches — a self-distillation analogue of masked-language-model pretraining, but for individual image patches rather than whole images. A KoLeo regularizer (based on the Kozachenko-Leonenko differential entropy estimator) is added to explicitly encourage the feature vectors within a batch to spread out uniformly rather than clump together, improving nearest-neighbour retrieval quality. DINOv2 also explores Sinkhorn-Knopp normalization of teacher outputs, a technique borrowed from SwAV (Caron et al., 2020), as an alternative or complement to simple centering for stability at scale. Third, engineering for scale: the largest DINOv2 teacher, ViT-g/14, has roughly 1.1 billion parameters, trained with techniques like stochastic depth and fused attention kernels for memory efficiency, then distilled down into smaller, openly released students — ViT-S (~21M params), ViT-B (~86M), and ViT-L (~300M) — that retain most of the larger model's feature quality.

The practical payoff is exactly the frozen-backbone use case a data-scarce Indian agricultural-monitoring team needs. A DINOv2 ViT-L/14 backbone, pretrained on 142 million natural images with zero manual labels, already carries general-purpose visual structure — edges, textures, repeating patterns, boundaries — that transfers surprisingly well even into a domain as different as 10-metre satellite tiles, because that low- and mid-level structure is not specific to natural photographs. Freezing the backbone and training only a linear classifier (or even doing k-nearest-neighbour classification with no training at all) on a few hundred labelled tiles avoids the fundamental problem of training a randomly-initialized CNN like ResNet-50 on the same handful of examples: a from-scratch network has tens of millions of parameters that must learn both low-level filters and the task's decision boundary from almost no data, while a frozen DINOv2 backbone plus linear head has only a few thousand trainable weights doing the actual task-specific learning, which is a vastly better-conditioned problem given 200 labels.

Diagram: ViT tokenization and encoder alongside DINO/DINOv2 self-distillation

Active recall

Attempt these before reading the answers below.

1. Why does ViT need a learnable position embedding, when an RNN or a CNN does not need one added explicitly?
2. For the toy 4x4 image with patch size 2x2, if the image were instead 6x6 with the same patch size, how many patches result, what is the new sequence length including the CLS token, and how does the raw self-attention score matrix's compute cost scale relative to the 4x4 case?
3. A classmate says "ViT has no convolutions, so it must always beat a CNN of similar size since it has no restrictive assumptions." Is this correct? Use the ImageNet-1k versus JFT-300M results to justify your answer.
4. Why doesn't DINO collapse to a trivial constant output for every image, given it uses no negative pairs and no labels?
5. A team has 50,000 unlabeled satellite tiles and only 200 labeled tiles marked "irrigation stress" vs "healthy." Why might a frozen DINOv2 ViT-L/14 backbone with a small trained linear head outperform training a ResNet-50 from scratch on the 200 labeled tiles?
6. If patch 4 in the original toy image had every pixel value multiplied by 10 due to a sensor saturation artifact (x4 = [130, 140, 150, 160] instead of [13, 14, 15, 16]), how would this change (a) the raw pre-LayerNorm attention scores, and (b) the post-LayerNorm attention scores computed earlier?

Worked answers

1. Self-attention is permutation-invariant: it computes a weighted sum over all tokens' values based purely on content similarity between queries and keys, with no notion of where a token sits in a sequence or grid unless that information is explicitly injected. A CNN encodes position implicitly through the fixed spatial arrangement of its sliding kernels and pooling operations; an RNN encodes order through sequential recurrence, one step at a time. ViT flattens its 14x14 patch grid into an unordered sequence of 196 tokens for attention, so without a position embedding, patch (0,0) and patch (13,13) look positionally identical to the attention mechanism — only their content would differ. ViT adds a learned 1D position embedding, one vector per patch slot (shape 197x768 for ViT-Base, including the CLS slot), added to each patch embedding before the first encoder block, letting the model learn whatever spatial structure it needs directly from data.

2. 6/2 = 3 patches per side, so 3 x 3 = 9 patches; with the CLS token, sequence length = 10, versus 5 in the original 4x4 case (4 patches + 1 CLS). Self-attention's raw score matrix has N^2 entries, so going from N=5 to N=10 is (10/5)^2 = 4x more score entries (25 pairwise scores versus 100). The Q/K/V linear projections, by contrast, scale only linearly with N (5xD versus 10xD multiply-adds), so the quadratic term is specifically the attention score computation, not the whole block. The same ripple, at real ViT-Base scale: doubling image side length from 224 to 448 (same 16-pixel patches) doubles patches per side from 14 to 28, so N goes from 196 to 784 — a 4x increase in token count — and the attention score matrix, being N^2, grows 16x. The position-embedding table also grows, but only linearly with N: from 197x768 to 785x768, roughly 4x more parameters, not 16x — a distinction many students miss because they assume every part of the model scales the same way as attention does.

3. No. Lacking built-in locality and translation-equivariance means ViT must learn these spatial priors purely from data; with limited data this is a real handicap, not a neutral absence of restriction. Trained from scratch on ImageNet-1k's 1.28 million images, ViT underperforms similarly-sized ResNets, because the CNN's inductive biases act as a helpful regularizer that lets it generalize from comparatively little data. ViT's advantage only shows up at larger pretraining scale: pretrained on JFT-300M (300 million images) and fine-tuned, ViT-Large/16 and ViT-Huge/14 reach roughly high-80s percent top-1 accuracy on ImageNet, matching or exceeding comparable CNNs — because with enough data, attention's flexibility finds spatial relationships a fixed local-then-global CNN schema cannot express as efficiently. DeiT later showed much of this data-efficiency gap can be closed without massive pretraining, using strong augmentation and a distillation token learning from a CNN teacher, reaching around 83% top-1 on ImageNet-1k alone. The correct framing: ViT trades inductive bias for capacity, and that trade only pays off with either enough scale or a source of distilled priors.

4. Two mechanisms on the teacher side prevent both collapse modes. Centering subtracts a running mean of recent teacher outputs from the teacher's logits before the softmax, stopping any single prototype dimension from dominating for every image — the easiest collapse for the student to imitate. Sharpening uses a lower softmax temperature on the teacher than the student, producing a more confidently peaked distribution, which blocks the other collapse mode of smearing probability uniformly across all prototypes. Combined with the teacher being a stop-gradient exponential moving average of the student — never directly trained by backpropagation — these substitute for the role negative pairs play in contrastive methods like SimCLR or MoCo, without requiring any explicit "push these apart" comparison against a different image.

5. DINOv2's ViT-L/14 backbone was pretrained self-supervised on 142 million curated images with zero labels, learning general-purpose visual structure — texture, edges, boundaries, spatial relationships. Freezing that backbone and training only a linear classifier on the 200 labeled tiles means the number of trainable parameters is tiny relative to the labeled set, sharply limiting overfitting. A ResNet-50 trained entirely from scratch has roughly 25 million parameters that must learn both low-level filters and the task-specific decision boundary from just 200 examples — a hopelessly underdetermined problem, likely to memorize rather than generalize. The 50,000 unlabeled tiles are not wasted either: they could be used for further self-supervised adaptation of the backbone to satellite-domain statistics before the linear-probe step, though even the off-the-shelf natural-image backbone typically transfers well, since low- and mid-level visual structure is largely domain-independent even when high-level semantics differ.

6. (a) Raw scores change drastically, because e4 = W^T x4 scales linearly with x4: e4 becomes 10 x [13+15, 14+16] = [280, 300] instead of [28, 30]. The dot product q1·k4 (with q1 = e1 = [4, 6]) becomes 4x280 + 6x300 = 2920, exactly 10x the original 292, so the pre-softmax score jumps from 206.5 to 2065 after the 1/sqrt(2) scaling — an even more extreme one-hot collapse onto patch 4. This is precisely the sensor-saturation pathology real remote-sensing systems face: one anomalously bright patch dominating raw attention regardless of relevance. (b) Post-LayerNorm scores are completely unaffected: LayerNorm's output is (x - mean(x)) / std(x), and both the mean and standard deviation of a token's own vector scale by the same multiplicative factor as the vector itself, so the ratio — and hence the normalized output — is invariant to any positive scalar multiplication applied uniformly to one token. Patch 4 still normalizes to the same [-1, 1] (or [1, -1], depending on sign convention) as before the artifact. This is exactly why pre-LN transformer blocks are robust to per-patch intensity anomalies like sun-glint, cloud shadow, or sensor saturation — a real and common condition in satellite and low-light imagery.

Think About It

Think about this: How would you explain vision transformers: from vit to dinov2 to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.