How Does Facial Recognition Work?
Facial recognition converts a face into a fixed list of numbers called an embedding, then measures the distance between one embedding and another. Faces that produce nearby numbers are reported as the same person. Nothing in the process stores or compares the picture itself.
Last reviewed
In short
- A face becomes a vector of numbers; comparison is arithmetic on those numbers, not on pixels.
- A match is a distance below a threshold, and the threshold is a choice, not a fact.
- Lowering the threshold finds more true matches and more wrong ones, always both.
- Searching a database of millions is a different task from checking two photos, and is measured separately.
- The technology reads geometry. It does not know a name, and it cannot confirm one.
How does facial recognition work, step by step?
Facial recognition runs the same four steps regardless of which system performs it. Only the third step involves anything learned from data; the rest is preparation and arithmetic.
- Detection: the system finds the region of the image that contains a face, and rejects images where it finds none.
- Alignment: the face is rotated and scaled to a standard position so that two photos taken at different angles become comparable.
- Embedding: a trained model converts the aligned face into a fixed list of numbers, typically between 128 and 512 of them.
- Comparison: the distance between two embeddings is measured, and a distance below a chosen threshold is reported as a match.
What is a face embedding, and what does it contain?
An embedding is a list of numbers describing the geometry the model learned to treat as identifying — the relative proportions and structure of a face, not its pixels. It is not a compressed photograph, and a face cannot be read back out of it in any useful form. It is also not human-interpretable: no single number in the list corresponds to a nose or an eye. What makes it useful is only that two photos of the same person tend to produce embeddings that sit close together, and two different people tend not to.
What decides whether two faces are called a match?
A threshold does, and it is set by whoever operates the system rather than discovered in the data. Every threshold trades the two kinds of error against each other, and there is no setting that removes both.
- A stricter threshold reports fewer matches and misses more real ones.
- A looser threshold catches more real matches and returns more strangers.
- The right setting depends on the cost of each mistake, which differs between unlocking a phone and searching the open web.
Why is searching millions of faces harder than comparing two?
They are different tasks and are measured separately. Comparing two photos is verification, written 1:1. Searching a large collection for a face is identification, written 1:N, and NIST evaluates the two in separate tracks for that reason. Every additional face in the collection is another opportunity for a stranger to fall inside the threshold, so an error rate that is negligible on one comparison stops being negligible when the comparison is repeated millions of times.
What does facial recognition need from a photo?
The model was trained on faces it could align, so the recoverable signal depends almost entirely on the input. Quality problems are not a small penalty; they usually decide the outcome.
- The face should be large enough in frame that its structure survives compression.
- It should face roughly toward the camera, because extreme angles hide the geometry the model relies on.
- Lighting should be even, since a hard shadow across one side changes what the model measures.
- Anything covering the eyes and the bridge of the nose removes the most informative region.
How is facial recognition search different from facial recognition surveillance?
They share the same mathematics and almost nothing else. A search compares one photo you supply against pages that are already public, and returns links to those pages. Surveillance systems match faces against an enrolled watchlist in a live camera feed, on a continuous basis, usually without the subject supplying anything. FaceSearch does the first and not the second: it takes a single uploaded image, searches public sources, and returns the pages where a similar face appears. It is not connected to any camera, and it holds no watchlist.
What can facial recognition not tell you?
It cannot tell you who someone is. It reports that two images are geometrically similar, and everything past that — the name, the account, the claim that both pictures show the same living person — comes from the page the match was found on, not from the model. A confident match on a page with the wrong caption is still the wrong answer, which is why a result is a starting point for checking rather than a conclusion.
Sources
- Face Recognition Vendor Test (FRVT) 1:1 Verification, National Institute of Standards and Technology
- Face Recognition Technology Evaluation (FRTE) 1:N Identification, National Institute of Standards and Technology