Bias in the Filter
Canny frames edge detection as an optimization problem, not a guess at a plausible filter shape. Three explicit criteria pin down the solution: a detection signal-to-noise ratio, a localisation precision, and a constraint on the spacing between adjacent noise-driven responses that keeps a single edge from producing multiple detections. Under an additive white Gaussian noise model for a step edge, these three criteria fix one filter shape up to a spatial scale factor. Changing the assumption about the noise or the edge profile means rederiving the optimum, not adjusting a threshold.
The first derivative of a Gaussian is adopted as the working filter, chosen for its separability into two one-dimensional convolutions rather than for exact optimality: it reaches roughly 20% below the criteria's optimal detection-localisation product. Detection and localisation trade off through the single smoothing parameter sigma, and the tradeoff is provably unbeatable within this model: no choice of sigma improves both at once. Canny Edge Detector states the bias plainly, inside the filter's own derivation, for one edge type under one noise model. It says nothing about what happens when the same scene is viewed at a different scale or rotation, the two invariances the next generation of hand-designed methods has to add explicitly.
Bias in the Descriptor
SIFT inherits Canny's reliance on gradients but adds two invariances by explicit construction rather than derivation from a noise model. Scale invariance comes from searching for extrema across a Difference-of-Gaussian pyramid built over multiple octaves; rotation invariance comes from assigning each keypoint the dominant peak of a local gradient-orientation histogram and sampling the descriptor in that rotated frame. Every one of these choices, the pyramid, the histogram peak, the 128-dimensional gradient layout, is fixed by the paper's authors before a single weight is trained.
HOG keeps the local gradient-orientation histogram and removes both invariances SIFT built. There is no scale-space search and no per-window rotation normalisation, just a dense grid of 8×8-pixel cells with unsigned orientation bins on a single fixed detection window. The choice is not stylistic: prior Gaussian smoothing before the gradient step, which SIFT effectively performs through its scale space, cuts pedestrian-detection recall from 89% to 80% at a fixed false-positive rate, and removing block normalisation costs a comparable margin. Density and speed on a fixed window matter more to HOG's target than the generality SIFT was built for. Both descriptors still commit their invariances, or the deliberate absence of them, at design time, separate from the classifier that consumes the vector. The next generation of methods drops that separation and folds the assumption into the layer that produces the features.
Bias Becomes Architecture
Convolution encodes the same assumption SIFT and HOG hand-built: a pattern useful in one part of an image is useful everywhere. Weight sharing convolves a single learned kernel across every spatial position, and local connectivity restricts each unit's input to a small neighbourhood rather than the whole feature map; the two constraints, not any single filter, are what the designer now specifies, and training fills in the numbers left free. Convolutional Neural Network states this explicitly: the priors cut free parameters by orders of magnitude relative to a fully connected network and encode the translation symmetry that SIFT and HOG each encoded by hand.
AlexNet is the proof the bet pays off at scale. Trained end to end on two GPUs across roughly 1.2 million labelled images, its eight learned layers reach a top-5 error of 15.3% on ILSVRC-2012, against 26.2% for the second-place entry, and 37.5%/17.0% top-1/top-5 on ILSVRC-2010 against a prior best of 45.7%/25.7% from a SIFT-and-Fisher-Vector pipeline. That margin displaces the hand-engineered pipelines the previous chapter described. The architecture is not decorative: removing any single middle convolutional layer costs about 2% top-1 accuracy, so every constraint it imposes is load-bearing. What it does not yet answer is how far the same convolutional bet can be pushed before its benefits stop compounding with depth.
Scaling One Prior
VGG answers with a controlled experiment: hold the convolutional assumption fixed, change only depth. Every configuration uses exclusively 3×3 convolutions; the family spans eleven to nineteen weight layers, and top-5 error decreases systematically across that range before saturating at the deepest configuration. No new assumption enters the design, which is exactly what makes the depth ablation informative: the gain is attributable to depth alone.
Depth eventually breaks the recipe on its own terms. A 34-layer plain convolutional network reaches a higher validation error than an 18-layer one, 28.54% against 27.94% top-1, a degradation distinct from vanishing gradients since batch-normalised plain networks still show healthy gradients while getting worse. ResNet removes that ceiling with an identity shortcut, , that lets a block default to passing its input through unchanged rather than having to learn the identity mapping from nonlinear layers. The shortcut does not touch what the network assumes about images; it only changes what depth is learnable given that assumption, enabling systematic scaling from AlexNet's eight layers and VGG's nineteen to a hundred and fifty-two. A decade of headline progress on ImageNet ran on one architectural bet made steadily more trainable. The bet itself, that locality and shared weights are the right prior for any image, was never the thing under test.
Prior Moves to Objective
ViT is the test. It deletes the convolutional prior outright, replacing stacked local kernels with fixed-size patches fed to a standard transformer encoder with no built-in locality and no weight sharing across space. The bias does not vanish; it becomes a bet on pretraining scale: on ImageNet-1k alone the pure transformer loses to convolutional backbones with their built-in spatial inductive bias, and only wins decisively once pretraining data crosses roughly a hundred million images. Below that crossover the missing assumption is a handicap; above it, the absence of a built-in constraint becomes room to keep improving.
MAE and DINOv2 relocate the remaining bias again, out of the architecture and into the training objective. MAE masks a large majority of an image's patches, roughly three-quarters, chosen by uniform random sampling, and trains the network to predict them from the small visible remainder; the masking ratio turns out to be the single largest lever on how useful the resulting features are, not a knob tuned only for speed. DINOv2 chooses a different objective again: a student network's output is trained to match a slower-moving teacher's, with no pixel reconstruction at all, and the resulting features work directly, frozen, on dense tasks that MAE's reconstruction objective supports only after full fine-tuning. Two networks with near-identical patch-based architectures now differ almost entirely in what they were trained to predict, not in how they are wired.
Task Systems Inherit
SAM and VGGT build working systems on top of that choice rather than making it themselves. SAM's image encoder is an MAE-pretrained ViT-H, resized so the longest side is 1024 pixels and patchified into a 64×64 token grid; SAM's own contribution is a lightweight prompt encoder and a fast two-way cross-attention decoder trained on 1.1 billion masks, not a new assumption about what an image is. Almost nothing in SAM's own architecture encodes a prior about images; the assumption that makes the encoder useful was already fixed during someone else's pretraining.
VGGT goes further still. Each input image is patchified by a frozen DINOv2 encoder, and a transformer backbone that alternates frame-wise and cross-frame self-attention processes every view at once, replacing the pairwise reconstruction and global-alignment optimisation earlier feed-forward geometry pipelines needed. There is no per-scene optimisation loop left, and little architectural assumption left either. What the model treats as a plausible surface or camera motion was decided upstream, during DINOv2's training objective, not VGGT's own design.
The two systems make concrete the migration this chain has traced: the assumption that once sat in a hand-derived filter, then a descriptor, then a convolutional layer, now sits inside a training objective chosen once and reused downstream. Once the inductive bias lives in training data and objective choice instead of a filter you can read, who actually chooses it, and how would anyone recognize the choice as wrong?