Definition

Bundle adjustment is the joint nonlinear least-squares refinement of all camera parameters — intrinsics, lens-distortion coefficients, and per-view extrinsics — and, in the general structure-from-motion case, all 3-D point positions, that minimises the total reprojection error across every observed image point.

In camera calibration the 3-D point positions $M_j$ are fixed by the target geometry and only the camera parameters are free; in structure-from-motion the points are unknown and refined jointly.

Mathematical Description

Reprojection-error objective

Under i.i.d. zero-mean Gaussian image noise of equal variance in both pixel coordinates, the maximum-likelihood estimate of all parameters is exactly the minimiser of the sum of squared Euclidean reprojection errors,

$\min \sum_i \sum_j \bigl\| m_{ij} - \hat{m}(K, k, R_i, t_i, M_j) \bigr\|^2.$

Zhang's final calibration stage minimises this jointly over the five intrinsic parameters, the two radial-distortion coefficients, and the per-view extrinsics — a full bundle adjustment restricted to the camera variables. Weng et al. minimise the identical pixel-residual sum over the non-distortion intrinsics and the five distortion coefficients.

Levenberg-Marquardt solver

The standard solver is Levenberg-Marquardt, a damped Gauss-Newton method. With residual vector $r(\theta)$ and Jacobian $J = \partial r/\partial\theta$, one iteration solves the damped normal equations

$\bigl(J^\top J + \lambda D\bigr)\,\delta = -J^\top r,$

where $\lambda > 0$ is the damping parameter and $D$ is typically $\mathrm{diag}(J^\top J)$. Large $\lambda$ makes $\delta$ a scaled steepest-descent step; small $\lambda$ approaches the Gauss-Newton step. The algorithm raises $\lambda$ when a step fails to decrease the cost and lowers it when a step is accepted, interpolating between robust gradient descent far from the minimum and fast quadratic convergence near it. Zhang reports convergence in three to five iterations from a closed-form linear initialisation; Tsai's stage-2 refinement is a two-iteration solve restricted to three unknowns.

Sparse block structure

In the full structure-from-motion case each scalar residual depends only on the parameters of one camera and one point. The resulting $J^\top J$ is block-sparse — camera-camera and point-point diagonal blocks with camera-point off-diagonal blocks. The Schur complement eliminates the numerous small point blocks, leaving a reduced camera-only system; this is what keeps bundle adjustment tractable for thousands of views and millions of points. In the calibration-only case the 3-D points are fixed, there are no point blocks, and the normal equations are dense in the camera parameters.

Initialisation from linear estimates

The objective is non-convex and Levenberg-Marquardt converges only to a local minimum, so a sufficiently accurate closed-form initialisation is required to place the solver in the basin of the global minimum. Zhang's linear stage solves a homogeneous system for the image of the absolute conic, extracts the intrinsics in closed form, and recovers per-view extrinsics from the homographies — the seed for the nonlinear refinement. Tsai's radial-alignment-constraint linear stage and Weng's central-point linear solve play the same role. In every case the final nonlinear refinement is a bundle adjustment seeded by a linear estimate.

Numerical Concerns

Initialisation dependence. The reprojection-error objective is non-convex; solution quality depends critically on the initial estimate. Zhang notes that the closed-form linear estimate of the first distortion coefficient can carry the wrong sign — a local-minimum risk that joint refinement resolves only because the other parameters are already well initialised.

Rotation parameterisation. Rotations lie on the three-dimensional manifold $SO(3)$; adding an unconstrained perturbation to a rotation matrix breaks orthogonality. The Rodrigues 3-vector keeps the nine matrix entries consistent and the Jacobian unconstrained; a unit quaternion is the alternative, with a unit-norm constraint enforced by normalisation after each update.

Gauge freedom. In unconstrained structure-from-motion the absolute scale is unobservable — rescaling all points and translations leaves every reprojection error unchanged — so $J^\top J$ is rank-deficient and a gauge-fixing convention is required. With a known calibration target the point positions are fixed and gauge freedom does not arise.

Jacobian conditioning. Parameters of very different scale — focal lengths in pixels ($\sim 10^3$) versus distortion coefficients ($\sim 10^{-3}$) — produce $J$ columns of very different norm and a poorly scaled $J^\top J$. Marquardt damping with $D = \mathrm{diag}(J^\top J)$ normalises each parameter direction to its own curvature; double precision is required for the small intermediate quantities.

Schur-complement cost. Eliminating point blocks introduces fill-in in the reduced camera matrix whenever two cameras share a point; for large problems the reduced system's sparsity must be analysed before choosing a direct Cholesky or a preconditioned-conjugate-gradient solver.

Outlier sensitivity. The least-squares objective penalises all residuals quadratically, so a single mismatch can dominate the Jacobian. Robust loss functions — Huber, Cauchy, truncated quadratic — grow more slowly for large residuals, approximating maximum likelihood under a heavy-tailed error distribution; standard practice alternates robust bundle adjustment with outlier rejection.

Where it appears

Bundle adjustment — in the restricted form of joint camera-parameter refinement over a fixed 3-D pattern — is the final and most accurate step of every classical camera-calibration pipeline in the atlas.

• zhang-planar-calibration — the nonlinear Levenberg-Marquardt refinement is the bundle-adjustment stage: all intrinsics, both distortion coefficients, and all per-view extrinsics refined jointly by minimising the total reprojection error, seeded by the closed-form IAC solve.
• tsai-versatile-calibration — stage 2 is a restricted bundle adjustment: with rotation and the lateral translation fixed from the linear stage, only focal length, depth translation, and one distortion coefficient are refined.
• scaramuzza-omni-calibration — the omnidirectional model's final nonlinear refinement minimises the same reprojection-error objective with the same solver, over the polynomial coefficients and per-view extrinsics.

Bundle adjustment is the numerical procedure that recovers the parameters of the pinhole-camera-model to maximum-likelihood accuracy from image observations.

References

1. B. Triggs, P. McLauchlan, R. Hartley, A. Fitzgibbon. Bundle Adjustment — A Modern Synthesis. In Vision Algorithms: Theory and Practice, LNCS 1883, Springer, 2000.
2. Z. Zhang. A Flexible New Technique for Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11)
   –1334, 2000.
3. R. Y. Tsai. A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology. IEEE Journal on Robotics and Automation, 3(4)
   –344, 1987.
4. J. Weng, P. Cohen, M. Herniou. Camera Calibration with Distortion Models and Accuracy Evaluation. IEEE TPAMI, 14(10)
   –980, 1992.
5. R. Hartley, A. Zisserman. Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press, 2004.