Table Makes Exhaustive Search Cheap
Integral Image is Crow's 1984 summed-area table: an entry at every pixel holds the running sum of everything above and to its left, so the pixel sum of any rectangle reduces to four table reads and three additions, independent of the rectangle's size. Viola and Jones renamed the identical structure in 2001 and built Viola-Jones Object Detector directly on top of it: a fixed 24 × 24 pixel window scans every position and scale of an image, and each window is scored against a cascade drawn from more than 180,000 candidate rectangle features. AdaBoost selects which of those features enter each stage and in what order; the first stage alone reaches close to 100% detection rate at roughly 40% false-positive rate, and on the MIT+CMU benchmark an average of 10 features out of a total of 6,061 are evaluated per sub-window.
What AdaBoost trains is which rectangle-intensity test to run and in what order, not whether to test a given window at all. The 24 × 24 window and the pyramid of scales it slides across stay a fixed, exhaustive enumeration, and the vocabulary of tests available to score each window is still limited to sums and differences of rectangle intensities.
Better Templates, Same Window
HOG: Histograms of Oriented Gradients keeps the sliding window and replaces that vocabulary with gradient-orientation histograms: each cell casts votes into orientation bins, and overlapping blocks of cells are locally normalised before the result is fed to a linear classifier. A pedestrian's silhouette contrast, not local pixel-intensity contrast, is the reliable cue, and on that cue HOG-based detection outperforms Haar-wavelet-based detectors by more than an order of magnitude in false-positives-per-window on INRIA.
Deformable Part Models keeps the same HOG feature pyramid and the same sliding window but stops treating the object as one rigid template. A coarse root filter covers the whole object, and the number of parts is fixed at six per component, each part filter sitting at twice the root's spatial resolution and free to shift under a quadratic deformation penalty. Both detectors still slide the identical window across the identical pyramid of scales that Viola-Jones used: nothing yet decides where to look before every position is tested.
Window Learns Fewer Places
Faster R-CNN removes the exhaustive scan itself. Generating candidate object locations had meant roughly 1.5 s of hand-tuned Selective Search computation on a CPU; Faster R-CNN's Region Proposal Network shares convolutional features with the detection head and turns candidate generation into a 10 ms learned forward pass on the same GPU-resident features. This is the first removal of the search, not merely of the feature computed inside each window.
The image pyramid goes next, still inside the two-stage family. Feature Pyramid Networks build the multi-scale representation from the network's own feedforward hierarchy: a top-down pathway upsamples the coarsest map and adds each finer backbone stage through a lateral connection, so every level from stride 4 to stride 32 carries strong semantics without the input ever being rescaled. The region-proposal and detection heads are reattached unchanged to each level, and proposal recall rises from 48.3 to 56.3 over the single-scale baseline. The pyramid that Viola-Jones, HOG and the deformable parts model enumerated over the image is now computed once, inside the network.
What either network proposes from is still a fixed, hand-chosen prior — FPN only redistributes it, one anchor scale per level: anchors are placed at every spatial location, three scales crossed with three aspect ratios, and only boxes near one of those nine templates get refined into a detection. The pipeline also keeps a two-stage split: one network proposes, a second classifies and refines. Both the anchor grid and the propose-then-classify division are the next machinery to go.
One Pass, Then No Proposals
YOLOv1 discards the propose-then-classify split entirely. A single forward pass over the whole image regresses boxes and class probabilities directly from a 7×7 spatial grid, each cell predicting two bounding boxes and responsible for objects whose center falls inside it. The grid and its two boxes per cell are still a fixed, hand-set structure, and NMS adds 2–3% mAP by cleaning up the duplicate detections that structure produces around the same object.
DETR removes the anchor boxes and that non-maximum-suppression step together. A transformer decoder takes learned object queries and decodes them in parallel into class-and-box pairs; a bipartite match, not a hand-tuned overlap threshold, decides which query is responsible for which ground-truth object, via the Hungarian algorithm, during training rather than at inference. What this removal does not touch is the cost of reaching it: DETR requires 300–500 epochs against 12–36 for Faster R-CNN, 10–25× more wall-clock training compute to reach equal accuracy, before its query-based decoder converges at all.
Where Machinery Actually Went
RF-DETR shows that a DETR-style set predictor no longer needs to be a slow, from-scratch research architecture. A DINOv2 self-supervised ViT backbone supplies pretrained features, and a single training run performs weight-sharing neural architecture search, training thousands of sub-networks jointly without separate retraining per configuration and then grid-searching 6,468 sub-network configurations on the validation set with no additional training. The result, RF-DETR (2x-large), reaches 60.1 AP on COCO, the first real-time detector to surpass 60 AP.
The query-based decoder survives from DETR largely unchanged; what changed is treating the backbone and its size as another dial to search, not something to redesign from scratch per method. Non-maximum suppression is still absent at inference. But the assignment problem it used to solve at test time is still being solved: the Hungarian algorithm now runs inside every training step instead of after every inference pass. Did the transformer detector really delete non-maximum suppression, or did it just move the same box-matching decision from a test-time overlap threshold into the Hungarian loss used during training?