A detection vendor tells you their model is 99.4% accurate. It sounds like a strong number, it is probably not a lie, and it tells you almost nothing about what will happen when you point it at your traffic.
This is not a subtlety. For rare-event detection the headline accuracy figure is close to unusable, and the arithmetic that shows why is simple enough to do on a napkin during the sales call. What follows is that arithmetic, the two numbers you should be asking for instead, and how to get an answer that is about your platform rather than someone's test set.
Accuracy is a ratio you can win by doing nothing
Accuracy is the share of all decisions that were correct. It counts the negatives, and for CSAM detection the negatives are essentially everything.
Suppose one upload in a hundred thousand is abuse material. Build a classifier that returns "clean" unconditionally — twelve characters of code, no model — and point it at a million uploads. It is wrong ten times and right 999,990 times.
Its accuracy is 99.999%.
That classifier is worthless and it beats the number most vendors quote. Any metric a null model can win is not measuring the thing you care about. The problem is not that vendors are dishonest; it is that accuracy averages over a population where the interesting cases are a rounding error.
The two numbers that mean something
Recall — of the abuse material that was present, how much did the system catch? This is the number your legal exposure feels. Missed material stays live until someone reports it, and for this category the users most likely to encounter it are the least likely to report it.
Precision — of everything the system flagged, how much was actually abuse material? This is the number your review queue feels, and your users feel through it. Low precision means moderators spending their day on innocent photographs while the real cases wait behind them.
They trade against each other, and the exchange rate is set by where you put the threshold. Nothing improves both at once except a better model.
The base rate is the entire argument
Here is the calculation the accuracy figure hides. Take the same million uploads and the same one-in-a-hundred-thousand base rate, so ten genuine positives. Now use a genuinely good classifier: 95% recall, and a false-positive rate of 0.1%.
- It catches 9 or 10 of the 10 real cases. Good.
- It also flags 0.1% of the 999,990 clean uploads — about 1,000 images.
Your review queue holds roughly 1,010 items, of which about 9 are real. Precision is under 1%. Ninety-nine out of every hundred items your moderators open are somebody's holiday photograph.
Meanwhile that system's accuracy is 99.9%, and the vendor is not lying when they say so.
Now notice what it takes to fix it. To get precision to a coin flip — half the queue being real — the false-positive rate has to fall to about 0.001%. That is a hundredfold reduction, not a tuning pass. This is why rare-event detection is hard, and why anyone quoting you precision without telling you the base rate they measured it at is quoting a number that does not transfer.
The lesson is not that detection does not work. It is that precision is a property of your traffic, not of the model, and any figure measured on a curated test set where positives are 10% of the data will collapse when it meets a real upload stream.
What the vendor's number is actually about
A model's reported metrics describe one model, at one threshold, on one dataset. Three of those are theirs and one of them should be yours.
Ask what the evaluation set was. If positives were a meaningful fraction of it — and on any hand-built benchmark they are — then the precision figure is a laboratory measurement. It is still useful: it tells you something real about the model's discrimination. It is just not a forecast of your queue depth.
Ask what threshold the numbers were taken at. A single precision/recall pair is one point on a curve. Two vendors quoting different points are not comparable, and a vendor who cannot produce the curve has not looked at it.
Ask about the negatives. A classifier's false-positive behaviour is mostly a story about what it confuses. For this task the categories that matter are predictable and uncomfortable: family photographs at the beach, medical images, teenagers who are lawfully of age, art. A vendor who has never had to answer for those has not deployed against real traffic.
The threshold is a policy dial, not a quality setting
Because precision and recall trade off, where you set the threshold is a decision about which failure you can better afford — and that differs sharply by platform.
A dating service that quarantines aggressively and reviews afterwards can accept a queue full of false positives, because a wrongly held photograph is a minor annoyance and a missed one is a catastrophe. A hosting provider where a flag takes a customer's whole site down cannot; there, a false positive is an outage with a contract attached.
That is why Visork returns a probabilistic score rather than a verdict, with the threshold profile set per platform during scoping. Detection reports what it sees; your policy decides what happens. Any vendor that ships one global threshold has made your risk decision for you, and made it without knowing your business.
The useful consequence: you can often run two thresholds. A high one that acts automatically, a lower one that only queues for review. Most platforms get more out of that than out of arguing about a single cut-off.
What to ask, in order
- What was the base rate in your evaluation set? If they cannot say, the precision figure is not interpretable.
- Show me the precision–recall curve, not a point. Then ask which point they would recommend for a platform like yours, and why.
- At the recall you recommend, how many items per million uploads land in my queue? This converts the whole conversation into the only unit that matters: moderator hours.
- What does it get wrong? A vendor who names their failure categories unprompted has looked. One who says "very few false positives" has not.
- Can I measure this myself, on my traffic, before I commit? This is the one that separates claims from evidence.
Running the benchmark yourself
You do not need a data science team. You need a sample and some discipline.
Use real traffic, sampled the way it actually arrives. A deliberately assembled set of hard cases measures something, but not your queue depth. If you want both, run both, and keep the numbers apart.
Measure recall on known material if you can get it. Most platforms cannot lawfully hold a positive set, which is a real limitation and worth saying plainly: in practice you measure recall against previously confirmed cases from your own moderation history, and you accept that the estimate is rough. Anyone who tells you this part is easy is skipping it.
Measure precision properly, because you can. Take everything the system flags over a fixed window and have a reviewer adjudicate all of it. That gives you precision at that threshold with no statistics required — it is a count.
Record the threshold with every number. A precision figure without its threshold is not a measurement, and six weeks later nobody will remember which run it came from.
Log score, threshold, verdict, and timestamp from the first day. You need this for the benchmark, and then you need it permanently: it is what makes a DSA Article 17 statement of reasons specific, and what your transparency report is counted from. Retrofitting it is miserable.
The Visork pilot exists for exactly this: 100 analyses at no cost, run against your own traffic, before commercial terms are discussed. Not because it is generous — because a number measured anywhere else does not predict what you are buying.
What a good answer sounds like
Not "99% accurate."
Something closer to: at this threshold, on a stream with your base rate, expect to catch roughly this share of real material and to see roughly this many items per million uploads in your queue; here is the curve if you want to move that trade; here is what we most often get wrong; and here is how to check all of it on your own data before you sign anything.
That answer is longer, less quotable, and the only kind worth acting on.
The figures in this post are illustrative arithmetic chosen to show how base rates behave, not measurements of any particular system. Your own numbers are the only ones that describe your platform.