Auto-Tagging a Photo Library When a Wrong Tag Costs More Than a Missing One

The brief was simple to say and hard to build. Marketing wanted to type "outdoor kitchen" or "frameless shower glass" and get back real photos of finished work. The one rule I set at the start was that a skipped tag is fine, and a wrong tag is not. If marketing pulls up a "freestanding bathtub" search and the third result is an alcove tub, they stop trusting the tags and go back to scrolling. That rule shaped every decision after it.

Zero-shot gets you a ranking, not an answer

I used SigLIP 2 (Google's image-text embedding model, the base 256px variant) to embed every photo once and cache the vectors. Each tag gets a few short text descriptions. Its score for a photo is how close the photo's embedding sits to those descriptions.

SigLIP fits this job better than a CLIP-style model for one specific reason. It was trained with a sigmoid loss that scores every image-text pair on its own, instead of a softmax that makes tags compete. A photo can strongly match "covered patio", "outdoor kitchen cabinets" and "paver patio" all at once, or match nothing. Multi-label tagging needs exactly that.

Caching matters more than it sounds. Changing a description or a threshold only re-encodes a few lines of text, never the image library, so a tuning loop takes seconds instead of a full pass over every photo. I fingerprint the cached vectors with the model revision and the preprocessing settings, so a changed setting can't silently mix old vectors with new ones.

What zero-shot gives you, though, is a ranking. The raw scores aren't calibrated probabilities, and "0.4" means different things for different tags. A fixed global cutoff was useless. Every tag needed its own threshold, and the threshold had to be earned.

Negative descriptions fix most of the misses

The first version tagged plenty of photos confidently and wrongly, and the misses had a pattern. "Front entry door" matched French doors. "Pendant lighting" matched chandeliers. "Retaining wall" matched raised planters. "Built-in spa" matched pools. The model saw something close and said yes.

The fix was giving each tag negative descriptions written against its actual misses. A tag's score became its mean match against the positive descriptions, minus however far its best-matching negative beat that mean. If a photo looks more like "a French door to a patio" than "a front entry door", the front-door score drops. If the negatives match worse, nothing changes.

I also added a helper class for marketing graphics: collages, flyers, anything with text overlaid on a photo. Those images match everything weakly and had been showing up across dozens of tags. Now they're never auto-tagged and never shown in spot checks.

The workflow that came out of this is plain: look at what a tag got wrong, describe the look-alike in words, re-score. A lot of tags went from unusable to clean this way.

A tag has to pass a gate before it applies itself

A good-looking top 20 doesn't prove a tag is safe. I built a gate, and a tag only moves from "suggested" to "auto-apply" when it passes one of two tests:

• Census: when only a few dozen photos sit above the threshold, every one of them gets checked. Any photo marked wrong is excluded individually, so the tag applies only to photos a check confirmed.
• Sample: when more photos sit above the threshold, a random draw of 40 unchecked photos above it has to come back with zero misses.

Two details made the gate honest. First, the sample is random from the photos the tag would actually apply to. It isn't the top matches, which are always the easiest. Second, a new unchecked photo above a census tag's threshold breaks the census, so the tag pauses until someone checks it. The gate can't drift open as the library grows.

Each check went fast on contact sheets: a grid of thumbnails with the existing verdict printed on each one, marked yes or no. Claude did most of the first-pass checking. A person's verdict always overrides Claude's, and I reviewed about two hundred auto-applied tags by hand without finding a wrong one.

The spot checks were training data the whole time

The first full pass had a problem I didn't see coming. Precision held up, but out of tens of thousands of photos only a small fraction got any tag at all. Being strict had made the system timid. Zero-shot can't tell a tiled shower from bare backer board, or a finished feature wall from a TV on a plain wall. To stay clean, thresholds had to sit high enough to skip a lot of real matches.

But every spot check was a labeled example from this library. So each tag with enough checks (at least 12 yes and 3 no) got a small logistic regression over the frozen image embeddings. Its training set was the checked photos plus a couple thousand random library photos as weak, down-weighted negatives. Its output is blended with the zero-shot score instead of replacing it.

The safeguard is what made me comfortable shipping it. A tag only uses the blend when 5-fold cross-validation on its own checks ranks them better than zero-shot alone, measured by average precision. For 43 of 51 eligible tags the blend won. The other 8 kept zero-shot. Then every tag still had to pass the same gate.

The result roughly doubled coverage: about twice as many photos tagged and twice as many tags applied, with the gate unchanged. A linear probe on cached embeddings trains in seconds and runs on the same GPU. It turns review work you were doing anyway into a better model.

Writing back to the system of record

The last step pushed tags into the client's photo platform through its API. I built it to be boring on purpose. It runs as a dry run unless you pass a live flag. It only ever adds tags and never removes one. Every photo and tag it sends gets recorded, so a rerun resumes where it stopped and never sends anything twice. I pushed one photo live first and checked it in the app before running the rest.

What I'd tell someone doing this next

Decide which mistake is the expensive one before you build anything, and design the gate around it. For marketing search, a missing tag costs nothing and a wrong tag costs trust. That one decision gave me everything else: per-tag thresholds instead of a global cutoff, negatives written from real misses, random samples instead of top matches, and a model upgrade that has to beat the baseline on held-out checks before it's allowed to run.

If you're tagging images with a zero-shot model, start by caching the embeddings and checking random photos above the cutoff, not the best matches. Keep every verdict. Within a day those verdicts are a training set, and a few lines of logistic regression will get you further than another round of prompt tweaks.