Energy as Objective
Energy Minimization poses segmentation as a single scalar objective: a data term measuring how well a label fits observed evidence at each pixel, plus a pairwise smoothness term penalising label disagreement between neighbours. Graph-Cut Interactive Segmentation instantiates this directly. A user marks object and background seed pixels; those seeds become hard constraints on an s-t graph built over every pixel, with n-links carrying the boundary term and t-links carrying the region term learned from the seed histograms. A single minimum cut on that graph then returns the global optimum of the combined energy, so a labelling failure traces to the cost function a human wrote down, not to a local minimum the solver settled for.
The boundary term is not free of human input either. Its bias toward short cuts through the graph means the weight balancing region against boundary evidence needs tuning per image, since too small a value produces small segments and too large a value fragments the result through region competition. The practitioner supplies both a set of labelled pixels and a free parameter of the objective, in exchange for a global guarantee, but that guarantee comes at the price of a labour-intensive interaction: dense seed strokes across the whole image, not a single click.
Coordinate Descent Priors
GrabCut Iterative Segmentation keeps graph-cut's energy but stops asking the human to supply an appearance model directly. A colour-mixture model is fit and re-fit at every pass: assign each unlabelled pixel to its nearest Gaussian component, re-estimate the components from their assigned pixels, run a global min-cut over the resulting energy, and repeat. Because each full pass minimises the joint objective over one group of variables, the energy falls monotonically across the loop, though only to a local minimum that depends on where the rectangle was drawn. What the human now supplies is a single rectangle around the object, not the dense foreground and background strokes graph-cut required.
Felzenszwalb–Huttenlocher Graph-Based Image Segmentation runs on the same dissimilarity-weighted pixel graph as graph-cut but answers a different question. Rather than minimising any global energy, it sorts every edge by weight and greedily merges two components whenever their cheapest cross-boundary edge does not exceed a size-adaptive internal-variation threshold. No seeds, no rectangle, no supervision of any kind: the algorithm partitions the whole image on its own. What it gives up in exchange is graph-cut's global guarantee, since the output is not the minimiser of any scalar energy, only a fixed merge rule a human designed, not one a network learned from data.
Dense Prediction Takes Over
FCN: Fully Convolutional Networks removes the hand-written energy altogether. It reinterprets a classifier's fully-connected layers as 1x1 convolutions, so a network already trained on ImageNet becomes a dense predictor with no architectural change: the boundary between foreground and background is now whatever the training set and backbone learned. Earlier pooling stages are fused into the final prediction by element-wise sum, recovering some spatial detail that repeated pooling destroys, but the fusion stays coarse.
U-Net keeps the fully-convolutional idea but disagrees with that fusion choice. Its decoder is symmetric to the encoder and concatenates the full, cropped encoder feature map at every resolution, carrying much more spatial detail forward than a summed score map does. The trade for that richer decoder is data: U-Net is trained from scratch on tens of images via heavy elastic-deformation augmentation, where FCN leans on ImageNet pretraining it cannot use in the same small-data regime.
DeepLab disagrees with both on preserving resolution in the first place. Atrous convolution enlarges the receptive field without discarding spatial resolution or adding parameters, replacing strided downsampling outright, and a multi-scale pooling head gathers context from several dilation rates in one pass. Dense CRF post-processing then tightens boundaries that the CNN over-smooths. Three networks, three answers, yet each still emits one class label per pixel across the whole image, unable to tell two touching instances of the same class apart.
Instance and Token Mixing
Mask R-CNN answers the instance problem directly. It extends a two-stage detector with a third parallel mask branch, a small FCN producing independent per-class binary masks at each region of interest, so the network outputs per detected instance: a class label, a score, a box, and its own mask. Detection now precedes segmentation, so two touching objects of the same class arrive as two separate regions, each earning a mask of its own.
SegFormer instead questions the backbone. It eliminates positional encodings entirely, letting a small convolution inside the feed-forward block supply positional information via zero-padding, and pairs a hierarchical Transformer encoder with a lightweight all-MLP decoder that requires no ASPP, OCR, or other context modules. The convolutional inductive bias that shaped every network up to this point is no longer load-bearing.
Mask2Former changes the output itself. Instead of a class label per pixel or a mask per detected box, the model predicts a fixed set of masks paired with class labels, one per candidate segment, supervised by bipartite matching rather than per-pixel cross-entropy. The same architecture and loss train on semantic, instance, or panoptic supervision without changing the head structure. None of these three networks, though, accepts guidance from the user at inference time; each commits to its output the moment the forward pass finishes.
Interactive Goes Feedforward
RITM returns interactivity to the pipeline, but not by seeding a graph. Its input is an accumulated positive-click map, a negative-click map, and a previous-mask channel, forming a five-channel tensor alongside the RGB image, and a single forward pass per user interaction produces the next mask, in contrast to inference-time-optimisation methods that run backward passes at test time to refine predictions. Training a model this way demands a loss that keeps the gradient bounded as accuracy improves click by click; a normalized focal loss renormalizes by its own total weight so that the aggregate gradient magnitude remains comparable to plain cross-entropy regardless of how well the model is doing.
FocalClick keeps the click-as-input-channel idea but changes where the forward pass runs. Each click triggers inference on two small local crops only, a Target Crop fed to a lightweight Segmentor and a Focus Crop fed to a Refiner, reaching sub-300 ms per click on a CPU. Its Progressive Merge step writes only the largest connected update region containing the new click back into the global mask, leaving the rest of a preexisting mask untouched, something RITM's whole-image forward pass does not target directly. Both networks, though, are trained end to end on one dataset's object distribution; a genuinely open vocabulary of objects means training something bigger, once, and reusing it everywhere.
Prompt Conditions a Decoder
SAM is that bigger, once-trained thing. Its heavy image encoder runs once per image, and a lightweight prompt encoder together with a fast mask decoder then answer any point, box, or text prompt in about 50 ms on CPU with a precomputed embedding. What the decoder answers against is a mask vocabulary learned from SA-1B, 1.1 billion masks collected across 11 million licensed images, not from one dataset's category list. The click that once seeded a min-cut graph, or fed a trained network as an extra channel, now conditions a decoder that already encodes a general notion of what an object's boundary looks like.
MobileSAM shows how much of that system is really the encoder. Distilling the image encoder from 632 million parameters down to 5.78 million, in less than one GPU-day on a fraction of SA-1B, while SAM's own prompt encoder and mask decoder remain frozen at their published weights, is enough to keep the same interface at a fraction of the cost. The prior itself does not move; only the size of the network holding it shrinks.
Once the prior lives inside a pretrained decoder, what does the user's click still specify?