Duplicate-Gate Blind Spot: Vector Coverage and 0.50
A cosine-similarity duplicate gate is only as good as its vector coverage. Lessons from a 32-article blind spot and a self-measured 0.50 cutoff.
TL;DR
I nearly trusted a duplicate check running on a half-populated index, since 164 of 196 articles had vectors and the missing 16% never showed up. The fix was a backfill plus a self-check confirming each article's own title returns its slug. The 0.50 cutoff isn't universal either; measure your corpus's own score distribution first.
I ran articles.py check-title on a candidate topic one morning. The tool printed JSON with a best_slug and a similarity score that looked convincing. I nearly took it at face value. An old habit saved me: open work/embeddings.json before trusting any number. The index held 164 article vectors while the blog had 196 published articles (commit fea59b1). Roughly 16% of the corpus never showed up on the duplicate gate's radar.
My first assumption was wrong. Once the check-title subcommand was exposed (commit e3c443a), I felt the duplicate-check work was done. A gate answering from a half-populated index only produces confident nonsense. A new-topic check can return a score computed against a corpus missing a chunk of itself, and I would never know which results were misleading.
The vector coverage blind spot
The math worked exactly on whatever sat in the index. Cosine similarity is computed as 1 − (u·v)/(‖u‖‖v‖), and scipy.spatial.distance.cosine does this for 1-D arrays [1]. The formula cannot tell which articles were never vectorized. As long as 32 articles had no vector, they were invisible to duplicate detection: a topic check that should have collided with one of them came back with a low score and passed.
The fix was mechanical. The backfill grew the index from 164 to 196 rows (commit fea59b1), and every new article gets embedded at publish time now. I also added a self-check: after embedding, check-title on the article's own title must return its own slug as best_slug. When it does not, the embed failed and I know before the gate gets trusted for a decision.
The sneaky part: the gate never errors on missing data. It keeps answering, just from the articles it happens to hold. Nothing turns red. The score looks normal. This failure mode resembles an uncalibrated scale more than a crash; the digits render fine while the base has drifted. That is why I treat the self-check as mandatory rather than decorative.
0.50 is not a universal law
The second mistake was subtler. With the gate running, I nearly treated 0.50 as a constant of nature. The number makes sense only because I measured it from the score distribution of real pairs in my own corpus: unrelated pairs cluster at or below 0.46, same-domain pairs at or above 0.68, and 0.50 sits in the gap between them.
A score of 0.50 does not read like a percentage, either. The distribution depends on the corpus. Every article on this blog touches deploys, APIs, or docker, so even two different topics share baseline vocabulary. The similarity floor sits higher than it would for a random-topic corpus. Another corpus could have the gap between unrelated and same-domain pairs land somewhere else entirely. Copying someone else's cutoff is guessing with extra steps.
Cosine similarity is the standard recommendation for measuring embedding relatedness [2], and length-1 normalized vectors make the dot product produce identical rankings. That recommendation covers the distance formula. It says nothing about where your cutoff goes. OpenAI's own docs describe embeddings as float vectors measuring text relatedness [2]; no documentation page knows your corpus's score distribution.
Benchmarks score the model, not your cutoff
MTEB is the reference evaluation of embeddings across languages and modalities [3]. A good MTEB score justifies picking that model, end of story. It cannot hand you a duplicate cutoff for a mixed-language article corpus, because the cutoff comes from your data's spread, and the benchmark never saw your data.
The sequence I follow now: fill the index, measure the score distribution from real pairs, then draw the line in the clear gap. The order matters as much as the numbers. A gate calibrated on its own corpus has earned the right to call a topic new.
Sources