The story of modern computer vision is best told backwards from a strange fact: the strongest general-purpose image features available today were trained without a single label, and most of the systems built on them never update the backbone at all. Getting there took a decade of removing bottlenecks one at a time — an information bottleneck between encoder and decoder, a sequential-computation bottleneck, a labeled-data bottleneck, a taxonomy bottleneck, and finally the stability bottlenecks of scale itself. This narrative walks that chain, and ends where it currently pays rent: industrial anomaly detection, where every foundation idea in the story gets stress-tested against the pixel.
The substrate
Three ideas from before 2018 supply everything that follows. Attention began as a fix for a translation problem — a decoder that could look back at any encoder state instead of squeezing a sentence through a fixed vector — and turned out to be something much more general: a differentiable, content-addressed lookup. The transformer then made the radical subtraction: remove recurrence entirely and let attention alone carry information between positions, in parallel, at every layer. What had been a patch on RNNs became the substrate on which nearly every model in this story is built.
The third idea looks unrelated at first. Knowledge distillation — training a small student on a large teacher's softened output distribution — was published as a model-compression trick in 2015. Keep it in view as this story unfolds: it reappears as DeiT's distillation token, as DINO's label-free teacher, as the mechanism that turns one 7B flagship into a family of deployable backbones, and — reversed — as an anomaly sensor on the factory floor. Few ideas in this atlas resurface as many times in as many costumes.
Images become tokens
For a decade, convolutional inductive bias — locality, weight sharing, hierarchy — was considered the price of admission for vision. ViT called that bluff: cut the image into 16×16 patches, embed each patch as a token, and hand the sequence to a standard transformer encoder. The catch was data. Trained on ImageNet-1k alone, ViT lost to ResNets; it needed the 300M-image JFT corpus to justify discarding the convolutional prior.
DeiT closed that gap without touching the architecture. A carefully engineered recipe of augmentation and regularization — and a distillation token, a learnable slot in the sequence supervised by a convnet teacher's decisions through every layer of self-attention — made the same ViT-B competitive using ImageNet-1k only, trained on one node in three days. The lesson was not "inductive biases are unnecessary"; it was that biases can be bought back through data augmentation and a teacher, at training time, without changing the model. The field learned which biases are worth reintroducing and where they should live.
From classifier to representation
The next bottleneck was the label itself. Between 2019 and 2021, self-supervised learning converged on a deceptively simple recipe: create two augmented views of an image and demand the network represent them the same way — while somehow avoiding the trivial solution where everything maps to the same point. SimCLR held collapse off with in-batch negatives, MoCo with a momentum-updated queue of negatives, BYOL with an asymmetric predictor and a slow-moving target network. Each answer traded a different resource: batch size, memory, architectural asymmetry.
Two 2021 papers then split the road. MAE went the reconstruction way — mask 75% of the patches, rebuild the pixels — buying cheap pretraining and superb fine-tuned accuracy, at the cost of frozen features that lag the alternatives. DINO went the distillation way: a student matching an EMA teacher of itself on multi-crop views, collapse held off by nothing more than centering and sharpening of the teacher's softmax. Two properties emerged that nobody explicitly asked for — k-NN classification on frozen features that nearly matches a linear probe, and attention maps that segment objects without ever seeing a mask. That second property is the first hint of the frozen-backbone era: the representation itself, not the classifier on top, had become the product.
Distillation expands
Scaling DINO's recipe was not a matter of turning up the dials. DINOv2 needed a curated 142M-image corpus assembled without human annotation, an added patch-level masked objective borrowed from iBOT, Sinkhorn-Knopp centering for stability, and — completing the circle back to Hinton — distillation from the expensive ViT-g flagship down to the S/B/L variants people actually deploy. The result was the first self-supervised backbone whose frozen features beat text-supervised rivals on dense tasks like segmentation and depth.
Scale then exposed two failure modes worth their own stops. In large ViTs, low-information patches get quietly recycled as computation scratch space, producing high-norm artifact tokens that corrupt dense feature maps — the fix, dedicated register tokens that give that computation somewhere legitimate to live, is now standard in the family. And over very long schedules, patch features slowly lose locality even as image-level accuracy keeps climbing. DINOv3 diagnosed and repaired that drift with gram anchoring — pinning the pairwise similarity structure of patch features to an early, dense-consistent teacher checkpoint while letting the features themselves keep moving. With a 7B teacher, 1.7B curated images, and a multi-student distillation into ViTs and ConvNeXts, one frozen encoder now competes with specialized fine-tuned pipelines on detection, segmentation, depth and 3D correspondence.
Vision meets language
A parallel line attacked a different bottleneck: the closed taxonomy. CLIP trained an image encoder and a text encoder contrastively on 400M web image–text pairs, so that classification became retrieval — embed the image, embed a sentence describing each candidate class, take the nearest. Language became the interface to the visual representation, and "zero-shot" stopped being a parlor trick. SigLIP 2 is the same idea grown up: a sigmoid loss, multilingual data, and dense-feature objectives that pull the recipe toward the localization quality the self-supervised family had claimed as its own turf.
SAM generalized the interface move to a task rather than a taxonomy. Built on an MAE-pretrained encoder and a billion masks, it turned segmentation into a promptable service: a point, a box, or a mask sketch in, a mask out, for anything. The pattern deserves attention because it recurs: a mature supervised task, given a foundation-scale encoder, tends to stop being a task and become an interface.
Three bridges into anomaly detection
Industrial anomaly detection is the perfect stress test for foundation features, because its central constraint is that anomalies cannot be enumerated in advance — you have normal samples, a pixel-level accuracy bar, and often a real-time budget. Three separate bridges carried foundation ideas into this domain. The oldest is distillation, inverted: Uninformed Students trained students to regress a frozen teacher's patch descriptors on normal data only, so that the failure of imitation becomes the anomaly score. EfficientAD engineered that idea to the millisecond regime — a small distilled teacher, a loss that keeps the student from generalizing too well, industrial frame rates.
The second bridge is the frozen embedding itself: if pretrained patch features are good enough, normality can simply be remembered rather than learned — the insight PatchCore pushed to near-saturation on MVTec with a coreset memory bank and nearest-neighbour distances, with no training on the target class at all. As backbones improved from ImageNet-supervised to DINOv2-class features, this bridge widened: better features move anomaly detection forward without a single change to the detector. The third bridge is language: WinCLIP matched windowed CLIP features against textual descriptions of normal and damaged states, making anomaly detection zero-shot — no defect examples, just words.
The anomaly-detection design space
Seen together — the survey page maps this in detail — the modern design space sorts by what you have and what you can afford. Plenty of normal images and a tight latency budget favors the student–teacher line (EfficientAD). A handful of normal images and no training budget favors memory banks over frozen features (PatchCore). SimpleNet marks a middle path: replace the memory bank with a learned feature adaptor and a discriminator trained on synthetic feature-space anomalies — simpler and faster, and evidence that the field keeps re-litigating the store-versus-learn question. And when even normal samples are scarce or the object family is open-ended, the language bridge takes over: WinCLIP for zero-shot state descriptions, AnomalyCLIP for learned object-agnostic prompts that transfer across objects and even from industrial parts to medical scans.
The punchline of the whole narrative sits here. None of these systems trains a backbone. The decade-long chain — attention to transformer to ViT, labels traded for self-supervision, distillation compressing flagships into deployable families, language attached as an interface — converges on a single operational fact: on the factory floor, the model that finds the scratch on the casting is a frozen inheritance from every stop on this map.