Face recognition (People)

Face recognition finds faces in photos and videos and groups them into people. Users can name people, merge them, and browse every photo and video of one person. It runs in the server process with ONNX Runtime. No data leaves the server, except the one-time model download.

Face recognition is off by default. An owner turns on the face_recognition feature flag in Admin → Features. People also needs the albums flag.

Face recognition covers photo and video items in albums. It does not cover cinema, music, podcasts, or books.

Privacy and biometric data#

A face vector is biometric data. Laws such as the GDPR (European Union) and the Illinois BIPA treat it as sensitive. This is the reason why the feature is off by default. The server owner decides whether the feature is lawful and appropriate for the people in the library, and whether they must give consent.

What Gallery stores for each face:

The position of the face in the item (a box) and five landmark points.
A vector of 128 numbers (built-in recognizer) that describes the face. Gallery compares vectors to group faces.
For a face found in a video frame: a small 384 px JPEG crop of the face.
Names that users give to people, and portraits that users upload.

Gallery does not store faces in any other place and does not send them to any other service.

Who sees a person. A user sees a person only if the user can see at least one photo or video of that person. Faces in items that a user cannot see never appear for that user, not even as a crop.

Names. A person has one shared name. Everyone who can see the person sees it. If a user names a person in a private album, another user who sees the same person in a different album sees that name. The web app and the mobile app warn the user when they set the first name. A user can also set a private nickname that nobody else sees.

Hide a person. Each user can hide a person, for example a person on a poster. The person stays visible for other users.

Anonymous viewers and share links do not get People. The feature needs a signed-in user.

Delete face data#

Turn the feature off. When an owner turns off face_recognition in Admin → Features, Gallery asks whether to delete all face data too. The default answer keeps the data. Every other way to close the question also keeps it.
Delete at any time. Admin → Face Recognition → Delete all face data deletes every person, face, scan record, nickname, hidden choice, cover choice, and uploaded portrait. It works while the feature is off. Models and settings stay. The action is written to the audit log (face_data_delete).
Deleted items. When an item is deleted, its faces are deleted with it. A person with no faces left is deleted too.
Deleted users. When a user is deleted, the user’s nicknames, hidden choices, cover choices, and portraits are deleted.

Enable the feature#

Make sure that ONNX Runtime loads. The release image includes it. The face models use the same library and the same [SmartSearch] onnxRuntimeLibrary setting as smart search (see Smart Search). If the library does not load, Admin → Face Recognition shows the load error and the rest of the server keeps running.
Turn on the face_recognition flag in Admin → Features.
New photos and videos are scanned after their thumbnails exist. To scan the existing library, open Admin → Face Recognition and choose Generate missing. The two models download on first use.
Gallery groups faces into people a few seconds after each batch of scans. Run clustering now does it at once.

Regenerate all scans every item again. It keeps the people, names, and every correction that users made: Gallery matches the new faces to the old ones by position (photos) or by position and similarity (videos).

Models#

Gallery downloads the models on first use into <BasePath>/data/models/faces/<model id>/ and checks the SHA256 hash of each file. A file put there by hand with the right hash is used as is, so an offline server can be prepared in advance.

Role	Model	License	Size
Detector (default)	YuNet 2023mar	MIT	0.2 MB
Recognizer (default)	SFace 2021dec	Apache 2.0	39 MB

Only models with a permissive license are built in. Models load when needed and unload after they are idle (5 minutes by default). Measured memory: about 55 MB for the detector and 77 MB for the recognizer.

Gallery detects faces on the 1920 px thumbnail, not on the original file. Faces smaller than 40 px (the shorter side of the box) are dropped.

Speed#

Measured on an M1 Pro with 8 threads (CPU only):

	Detect per photo	Embed per face	100k photos (about 4 faces each)
YuNet + SFace (default)	46 ms	18 ms	about 3.5 hours on one worker
SCRFD + ArcFace (see below)	344 ms	162 ms	about 28 hours

Decoding the thumbnail takes extra time.

Settings#

These are stored in the database, not in the config file. Change them in Admin → Face Recognition (owners only):

Setting	Default	Notes
Detector, Recognizer	YuNet, SFace	A built-in or custom model. A different model needs a new scan; the next Generate missing or Regenerate all does it.
Threshold	0 (the recognizer’s own value)	Cosine similarity for the same person. SFace uses 0.55. The threshold belongs to the recognizer. When you change the recognizer, check this value.
Minimum face size	40 px	Smallest face kept, as the shorter box side.
Minimum faces per person	3	Faces a group needs before it becomes a person. Fewer faces stay in Unassigned.
Cluster minimum score	0.7	Detector score a face needs before clustering uses it. Backs of heads and blurred faces score lower and stay unassigned.
Video frames	16	Maximum number of frames scanned per video. Gallery takes one frame every 10 seconds.
Video frame size	1280 px	Longest side of a video frame.
Threads	0 (half the CPUs)	ONNX Runtime threads per model.
Idle minutes	5	Minutes without use before the models unload.
Name grace period	24 hours	The user who set a person’s first name can still change it for this long.

Do not set the threshold lower than the recognizer’s best value. Below it, a model groups many different people into a few large clusters. With SFace the usable window is narrow (0.55–0.60).

How people are formed#

Assign. A new face joins the person who has at least minimum faces per person faces close to it (above the threshold). Otherwise it stays unassigned.
Group. The unassigned faces are grouped with DBSCAN. Each group of at least minimum faces per person faces becomes a new unnamed person.
Users win. Clustering never moves a face that a user assigned, unassigned, or marked as “not this person”.
Videos. Gallery samples frames and groups the faces of one video into tracks, one for each person in that video. A video counts once for each person, not once for each frame. The person page opens the video at the moment where the person first appears.
Cartoon faces. The default detector also finds drawn and cartoon faces, for example on posters. They form their own people. Hide those people.

Names, merges, and corrections#

Name a person. While a person has no shared name, any user who can edit at least one item of that person can set it. The user who set the first name can fix a typo for the grace period (24 hours by default). After that, only users with the people:manage permission (the Owner and Administrator groups by default) can change the shared name. Everyone else sets a private nickname, which only that user sees. Reset name removes the nickname.
Correct a face. A user who can edit an item can move its faces to another person, create a new person from a face, or say “not this person”. The same user can link a person to an item without a face (for example a person seen from behind).
Merge. A merge moves the faces whose items the user can edit. The old person is deleted when no face is left. Owners and Administrators can edit every item, so their merge always moves every face.
Covers. Each user has their own cover for each person: an uploaded portrait, or a chosen face. Without a choice, the cover is the best face that the user can see.
Audit log. Setting or changing a shared name, merging people, and deleting a person are written to the audit log. Nicknames, hidden people, covers, and portraits are private and are not.

Clients#

Client	People
Web (gallery-ui)	Full: browse, name, merge, hide, portraits and covers, unassigned faces, face boxes over photos in the lightbox, admin panel.
Mobile app (gallery-app)	The same as the web, except the admin panel. Face boxes over photos work from the lightbox menu (“Show faces”). A tap on a box opens the same actions as the People card.
TV app (gallery-tv, unreleased)	Browse only: a People screen, a person’s photos and videos, and a Home row.

No client draws face boxes over a video, because a box is valid for one sampled frame only. The People list shows the time where a face was seen (“at 1:12”).

Custom models#

In Admin → Face Recognition → Custom models, an owner can add a detector or a recognizer by URL, for example a mirror on their own network or a model with a license that fits their needs.

Gallery does not check the license of a custom model. The owner who adds it is responsible for the license for their own installation. The form asks the owner to confirm this.

The URL must point to one .onnx file (http or https). The file is stored as <BasePath>/data/models/faces/custom-<id>/model.onnx. Nothing downloads until the model is selected.
A SHA256 per model is optional. Without one, Gallery accepts the file of the first download, stores its SHA256 and size, and rejects any different file later.
Detector. Choose the family of the model, because each architecture needs its own decoder: yunet or scrfd. The input size defaults to 640×640 and must be a multiple of 32 (160 to 1920). The score threshold and the NMS threshold default to the values of the family (YuNet 0.6 and 0.3, SCRFD 0.5 and 0.4). YuNet exports usually have a fixed input size, so set the size that the file was built for. SCRFD exports with a dynamic size accept other sizes.
Recognizer. The model takes an aligned face of 112×112 px and returns one vector. Gallery supports only the 112 px input. Set the mean, the standard deviation, and the channel order. The defaults are the ArcFace values (mean 127.5, std 127.5, RGB). The recognizer needs its own threshold.
Only the name and the description can change later, because stored vectors depend on the file. To use another file, delete the model and add it again. A selected model cannot be deleted.
Change of recognizer. Gallery keeps all boxes, landmarks, and people. It only computes the vectors again from the stored landmarks. Until a face has a vector of the current recognizer, clustering ignores it.
Change of detector. Gallery detects again, and copies each person to the new box that overlaps the old box (IoU 0.5 or more). Manual links stay.

Better models: SCRFD + ArcFace#

During development, the InsightFace buffalo_l models gave the best results of all the models that were tested: the SCRFD-10G detector and the ArcFace R50 recognizer. Gallery does not include them, because their license is non-commercial and does not allow us to ship them. An owner who wants them adds them as custom models. By doing so, the owner makes the license decision for their own installation.

Test results#

Test data: the Labeled Faces in the Wild set (158 people with 10 or more images, 2169 faces, labelled) and 194 real wedding photos judged by eye.

Recognizer	Best threshold	Cluster precision	Cluster recall
ArcFace R50 (non-commercial)	0.45	0.996	0.981
SFace (built in)	0.55	0.981	0.974
AuraFace (Apache 2.0, not built in)	0.50	0.970	0.947

ArcFace is the most forgiving model. It works well from a threshold of 0.35 to 0.55. SFace works only from 0.55 to 0.60.
On the wedding photos, SCRFD + ArcFace gave the cleanest groups. Its largest groups (the couple) were pure, also for side views. SFace with SCRFD gave almost the same group sizes, with a few wrong faces in the largest group.
SCRFD found fewer false faces. YuNet found about 90 more faces, almost all of them drawn faces on decorations.
The default pair is about 8 times faster (see Speed).

Choose SCRFD + ArcFace when the quality of the groups matters more than the scan time, and the license suits your use.

Add the models#

Get the two files from the InsightFace buffalo_l package: det_10g.onnx (detector, 17 MB) and w600k_r50.onnx (recognizer, 174 MB). The package is a zip file, and Gallery downloads one .onnx file for each model. Extract the two files, and host them where the server can download them (for example a web server on your own network). Gallery does not add the download addresses of these files to its code or its seed data.
In Custom models, add the detector:
- Kind: detector, family: scrfd
- Input size: 640×640 (use 1280×1280 to find smaller faces, at the cost of speed)
- Score threshold 0.5, NMS threshold 0.4 (the family defaults)
- SHA256: 5838f7fe053675b1c7a08b633df49e7af5495cee0493c7dcf6697200b85b5b91
Add the recognizer:
- Kind: recognizer, input size 112
- Mean 127.5, std 127.5, channel order RGB
- Threshold: 0.45
- SHA256: 4c06341c33c2ca1f86781dab0e829f88ad5b64be9fba56e56bc9ebdefc619e43
The hashes are for the files of the buffalo_l v0.7 package. If your files differ, leave the SHA256 empty and Gallery pins the first download.
Select both models in Settings, and set the threshold to 0 (the recognizer’s own 0.45). Then choose Regenerate all. People, names, and corrections stay.

To go back, select YuNet and SFace again. The threshold returns to 0.55 when the setting is 0.

Limits#

Face recognition uses the CPU only. Large libraries take hours to scan the first time.
Faces smaller than the minimum face size, and faces below the detector score, are not found.
A person with a large change of pose or glasses can end up as two people. Merge them by hand.
Clustering reads at most 20,000 unassigned faces in one run. Measure large libraries before you depend on it.
Gallery never writes face data into the source files (no XMP face regions).
Search by person name in the command palette and in smart search is not built yet.