Segment Anything: A foundation model for segmentation

The event and the specific claim

On 5 April 2023, researchers at Meta AI's Fundamental AI Research (FAIR) lab released "Segment Anything," a project that bundled three things at once: a new task, a new model, and a new dataset. The paper, led by Alexander Kirillov and eleven colleagues, introduced promptable segmentation, the Segment Anything Model (SAM), and the Segment Anything 1-Billion dataset (SA-1B), which the authors called "the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images." Meta's accompanying announcement put the count at more than 1.1 billion masks across roughly 11 million images, about four hundred times more masks than any earlier segmentation dataset.

The narrow, documented claim is worth separating from the surrounding excitement. Segment Anything did not claim to understand scenes, name objects, or reason about them. It claimed that one model, prompted with a point, a box, or a rough mask, could output a valid segmentation mask for almost any object in almost any image, including categories and image distributions it had never been trained on. The broader historical judgment, that this was an early and influential attempt to build a foundation model for a vision task, rests on what happened afterward, not on the paper alone.

What segmentation looked like before promptability

Segmentation, deciding which pixels belong to an object or region, was one of the oldest problems in computer vision, and by 2023 it was fragmented into subfields with their own models and datasets. Interactive segmentation tools let a user click or draw to extract a foreground object; the classic example is GrabCut, which used iterated graph cuts to separate foreground from background given a bounding box. Fully automatic systems took a different route: Mask R-CNN extended Faster R-CNN to predict an instance mask for each detected object, and panoptic segmentation unified the "stuff" of semantic segmentation with the "things" of instance segmentation into a single labeling.

The shared constraint across all of these was specialization. A Mask R-CNN trained on the eighty categories of COCO segmented those categories; pointed at satellite imagery, microscopy, or an unfamiliar object, it had no vocabulary for the result. Each new domain meant new labeled data and, usually, a retrained model. Interactive tools generalized better across objects but were procedures, not reusable models with a transferable representation. Segment Anything's ambition was to remove that per-domain retraining step for the specific act of "select this thing," the way a single pretrained language model had begun to serve many downstream text tasks.

The other constraint was data. High-quality segmentation masks are among the most expensive labels in vision, because a good mask traces an object's exact silhouette pixel by pixel rather than drawing a box around it. Existing datasets reflected that cost: COCO and its successors offered on the order of hundreds of thousands to a few million masks, enough to train a specialist but far short of the diversity a general model would need. Any project aiming at generality therefore had to solve the labeling economics first, which is why the dataset and the model in Segment Anything were designed together rather than in sequence.

Reframing segmentation as a promptable task

The conceptual move that organizes the whole project is the definition of a promptable segmentation task. Borrowing the framing that had reshaped natural language processing, the authors asked for a model that, given any prompt indicating what to segment, returns a valid mask, and that does so well enough to be composed into larger systems by prompt engineering rather than retraining. A prompt could be a set of foreground and background points, a bounding box, a coarse mask, or, as a proof of concept, free-form text.

Crucially, the task builds in ambiguity. A single click on a person's shirt could reasonably mean the shirt, the torso, or the whole person, and there is no way to know which from the click alone. Rather than force a single answer, SAM predicts three masks per prompt, corresponding roughly to a subpart, a part, and a whole object, and returns a confidence estimate (a predicted intersection-over-union) for each. This design choice is what lets a genuinely ambiguous prompt still produce useful output, and it distinguishes the task from earlier interactive segmentation, which typically returned one mask and expected further clicks to refine it.

How the Segment Anything Model actually works

SAM is deliberately split into three parts so that interactivity is cheap. A heavyweight image encoder runs once per image to produce an embedding; a lightweight prompt encoder turns points, boxes, and masks into embeddings; and a lightweight mask decoder combines the two to produce masks. The image encoder is a Vision Transformer, the architecture from Dosovitskiy and colleagues, specifically a ViT-H/16 backbone pretrained with the self-supervised objective of masked autoencoders, in which most image patches are hidden and the network learns to reconstruct them. That encoder holds nearly all of the model's parameters and is the expensive component.

The payoff of this asymmetry is real-time use. Once an image embedding is computed, Meta reported that the prompt encoder and mask decoder produce a mask in about 50 milliseconds in a web browser, so a user can move a cursor and watch masks update live. For the text prompting experiment, the authors aligned SAM with the image and text embedding space of CLIP, which let a typed phrase stand in for a spatial prompt. Training combined interactive simulation (sampling prompts from ground-truth masks over many rounds) with the automatic mask generation used to build the dataset, optimizing a mix of focal and dice losses against the best of the three predicted masks.

The data engine and the SA-1B dataset

SA-1B is arguably the project's most consequential artifact, and it was built by a "data engine" that co-develops the model and the data in three stages. In the assisted-manual stage, annotators clicked with an early version of SAM helping them, labeling masks in about fourteen seconds each. In the semi-automatic stage, SAM proposed confident masks automatically and annotators filled in what it missed, increasing mask diversity. In the fully automatic stage, a well-trained SAM was prompted with a dense grid of points and its outputs were filtered for confidence and stability, which generated the overwhelming majority of the dataset: the paper reports that roughly ninety-nine percent of the more than one billion masks were produced fully automatically, averaging around one hundred masks per image.

The images themselves were licensed from a provider of high-resolution photography, and Meta took privacy steps that matter for reuse: faces and license plates were blurred, and images were released downsampled so the shorter side was about 1500 pixels. The licensing split is important: SAM's model weights and code were released under the permissive Apache 2.0 license, while SA-1B was released under a research-only license. That combination let practitioners freely deploy the model while keeping the dataset's photographs governed by narrower terms.

The evidence: zero-shot transfer across many tasks

The empirical case rested on zero-shot transfer, running SAM on tasks and datasets it was never trained for. The centerpiece was zero-shot single-point segmentation across a suite of twenty-three diverse datasets spanning egocentric images, microscopy, underwater scenes, aerial views, and more. Given a single foreground point, SAM's automatically chosen mask beat a strong interactive baseline (RITM) on a majority of those datasets by mean intersection-over-union, and when the best of its three masks was scored, it was ahead on all twenty-three.

The authors also ran a human study, because a single click is ambiguous and the automatic metric can penalize a valid but "wrong" interpretation. Annotators rated mask quality on a scale from one to ten, and SAM's single-point masks consistently drew ratings in the high single digits, often above RITM's even where the automatic metric disagreed, evidence that the numeric gap understated SAM's usefulness. Beyond points, the paper demonstrated four more transfers: zero-shot edge detection, object proposal generation, instance segmentation when fed the boxes of an object detector, and text-to-mask from a typed phrase. These are task-specific results on segmentation benchmarks, not general intelligence scores, and the authors were careful to present them as such.

The composability point deserves emphasis, because it is the part that distinguishes a foundation model from a strong specialist. In the instance-segmentation experiment, SAM was not retrained; it was simply handed the output boxes of an existing detector and asked to fill in masks, and the combined pipeline approached fully supervised systems on some benchmarks. That pattern, "prompt SAM with whatever another system already knows," is exactly how the model was meant to be used, and it foreshadowed the ecosystem of downstream combinations that followed. The evidence, in other words, was as much about a workable interface as about any single accuracy number.

The competing approaches of its moment

Segment Anything did not arrive in a vacuum, and reading it against its contemporaries clarifies what was and was not novel. Interactive segmentation had strong learned baselines such as RITM and FocalClick that produced high-quality masks from clicks, but these were single-purpose click-refinement models, not promptable foundation models with a billion-mask training set. Within days of SAM, a group from the University of Wisconsin-Madison and Microsoft released Segment Everything Everywhere All at Once (SEEM), which pursued a richer prompt vocabulary (points, boxes, scribbles, text, and even referring regions from another image) with a joint visual-semantic space, but at far smaller data scale.

The sharpest contrast is semantic. SAM segments regions without naming them; it will happily mask an object it cannot identify. Open-vocabulary detectors and grounding models occupied the complementary space of connecting language to boxes and labels. That gap was so obvious that the community quickly stitched the pieces together, pairing a language-grounded detector with SAM to turn a text query into a labeled mask. Understanding SAM correctly means seeing it as the "where" module that deliberately left "what" to other systems, a division of labor that its promptable interface was designed to enable.

The downstream influence it actually had

The evidence of influence is concrete and fast. Because SAM shipped with permissive weights, a demo, and a public dataset, an ecosystem formed around it within months. Efficiency-focused variants (MobileSAM, FastSAM) shrank or accelerated the heavy encoder for edge and real-time use. Quality-focused work such as HQ-SAM added a learnable output token to sharpen masks on intricate structures while keeping SAM's zero-shot generality. Domain adaptations carried SAM into medical imaging, remote sensing, and object tracking, where "prompt on frame one, propagate" pipelines used SAM as the segmentation primitive.

Meta then extended the line itself. In 2024 the team released SAM 2, which handles promptable segmentation in both images and video by adding a streaming memory to the transformer so a masked object can be tracked across frames. The authors reported that SAM 2 was about six times faster than the original SAM on images and more accurate on video with roughly three times fewer interactions, and they released a new large video segmentation dataset alongside it. The through-line from SA-1B to SA-V is the same data-engine method applied to a harder modality, which is the clearest signal that the original contribution was a reusable recipe, not a one-off model.

The honest limits and caveats

The paper is unusually candid about what SAM cannot do, and those caveats have held up. SAM can miss fine structures, produce small disconnected artifacts, and generate boundaries that are less crisp than dedicated interactive methods that spend more compute per edge. Its text-to-mask capability was an early proof of concept, not a robust feature, and the authors declined to release it as a finished tool. On specialized domains far from natural photography, zero-shot masks can degrade, so a valid-looking mask may still select the wrong object or the wrong boundary, exactly the failure mode flagged in the project's own framing.

Two structural limits matter for placing the event. First, the heavy ViT-H image encoder makes SAM expensive to run at scale, which is precisely why so much follow-up work targeted efficiency. Second, and more fundamental, segmentation is not scene understanding. SAM answers "which pixels belong to the thing I pointed at," not "what is this, why is it here, or how does it relate to everything else." Treating a strong segmentation result as evidence of visual comprehension is a category error the authors were careful not to make.

How to place Segment Anything in AI history

The most defensible reading is that Segment Anything imported the foundation-model playbook (a general pretrained model, a promptable interface, and transfer by prompting rather than retraining) into a specific vision task and showed it worked well enough to be useful. Its genuine novelty was less any single component (Vision Transformers, masked-autoencoder pretraining, and interactive segmentation all predate it) than the composition: a promptable task definition, an interactivity-friendly architecture, and a data engine that manufactured a billion-mask dataset by bootstrapping the model against annotators.

A useful way to navigate outward from this node is to ask the same three questions the rest of this timeline rewards. Which earlier idea supplied a necessary component (the ViT backbone and MAE pretraining)? Which later project reused the model while changing an ingredient ( HQ-SAM's quality token, or SAM 2's memory for video)? Which limitation demanded new work (efficiency, crisp boundaries, and the missing "what" that grounding models supply)? Answered that way, Segment Anything is a well-documented turning point for interactive and promptable vision, valuable because it opened a design space, not because it closed one.

References

• Kirillov et al. (2023). Segment Anything.
• Meta AI (2023). Introducing Segment Anything: Working toward the first foundation model for image segmentation.
• Rother, Kolmogorov and Blake (2004). GrabCut: interactive foreground extraction using iterated graph cuts.
• He et al. (2017). Mask R-CNN.
• Kirillov et al. (2018). Panoptic Segmentation.
• He et al. (2022). Masked Autoencoders Are Scalable Vision Learners.
• Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.
• Radford et al. (2021). Learning Transferable Visual Models From Natural Language Supervision.
• Zou et al. (2023). Segment Everything Everywhere All at Once.
• Ke et al. (2023). Segment Anything in High Quality.
• Ravi et al. (2024). SAM 2: Segment Anything in Images and Videos.