One Equation, Two Unknowns
Optical Flow assigns every pixel a two-dimensional velocity from the assumption that brightness is conserved as a point moves between frames. That assumption collapses to a single linear constraint, , relating the two unknown velocity components at every pixel. The constraint is one linear equation in two unknowns per pixel, and no amount of algebra recovers a second one from it.
Geometrically the equation fixes a line in the plane, not a point. A single edge, viewed through any finite aperture, determines only the gradient-aligned component of its motion. Recovering the full 2-D velocity requires either pooling pixels whose gradient directions span two dimensions, or imposing a prior over the velocity field. Every method below accepts this equation as given, then makes a choice the equation itself cannot make: where the missing second constraint, the one pinning down the component along the edge, comes from. Two incompatible answers to that choice appear in the same year.
Two 1981 Answers
In 1981, Lucas-Kanade and Horn-Schunck supply the two founding answers, and they disagree about where locality belongs. Lucas-Kanade borrows the missing constraint from a small neighbourhood: under the assumption that all pixels in a window share a single velocity, the brightness-constancy equations over the window form an over-determined system solvable in the least-squares sense. The system is solvable only where the window's gradient directions span two independent dimensions; on a blank patch or a single edge the coefficient matrix is rank-deficient, and the method produces no estimate there rather than a wrong one.
Horn-Schunck borrows the constraint from the opposite end of the scale. The smoothness prior resolves the aperture problem, one brightness-constancy equation, two unknowns per pixel, by propagating boundary velocity information across uniform regions through iterated local averaging, filling in a dense field everywhere, including where Lucas-Kanade declines to commit. Both papers appeared in 1981 and together founded the optical flow field; neither supersedes the other; they are the two foundational threads of optical flow. The same averaging that fills blank interiors also blurs real motion boundaries, and the errors it leaves behind sit exactly where two motions meet.
Making the Prior Robust
Black-Anandan Robust Optical Flow keeps the machinery of both 1981 answers, the windowed regression and the global smoothness energy, and changes only the penalty each one places on disagreement. Both outputs are obtained by replacing the quadratic (L2) penalty in the data conservation and spatial smoothness terms with a redescending robust M-estimator, solved via Iteratively Reweighted Least Squares within a Graduated Non-Convexity continuation schedule. A quadratic penalty treats every residual as equally informative, so a second motion crossing a window or a neighbourhood drags the least-squares estimate toward an average of both; an estimator whose influence function is redescending equals zero once the residual passes a fixed threshold, so a residual large enough to belong to a different motion stops pulling on the fit entirely.
Because the resulting energy is non-convex, the scale is initialised large enough that the objective is convex, then reduced in stages, so the solver commits to which residuals are outliers only gradually. Both the data term and the smoothness term must use robust penalties; robustifying the smoothness term alone leaves the data term quadratic and can increase flow error relative to the fully quadratic baseline. What survives untouched is the requirement that a person hand-derive the penalty, the scale schedule, and the smoothness structure.
The Prior Becomes Learned
RAFT keeps the single fixed-resolution flow field and the iterative refinement of the classical methods, but hands the update itself to a trained network. In place of a hand-derived data term, the model builds an all-pairs 4D correlation volume, computed once as a single matrix multiplication and queried repeatedly at the current flow estimate, together with a weight-tied recurrent update operator that plays the role of an optimization step at every iteration. Every pixel's similarity to every pixel in the other frame is available from the start, instead of being reconstructed one Taylor expansion at a time.
The update operator replaces the closed-form gradient-descent step of the classical methods with a learned one: rather than a subpixel Taylor approximation of the data term, the operator learns to propose the descent direction. Trained only on synthetic data, the resulting network still generalizes to real footage, and its accuracy keeps improving over iterations from 1 to 200, far beyond what it ever saw during training. What has disappeared by name is the explicit smoothness term that Horn-Schunck wrote down and Black-Anandan made robust. Is a smoothness prior still needed once the update rule is learned, or has it simply moved into the network's weights?