What public Sherlock contest data tells us about the LLM-era submission flood — and the benchmark we are building to measure both bug-finding and bug validation.
Sherlock public findings · 24 contests ending Sep 2024 – Jul 2025 · Solidity, Go, and Rust
LLMs have made it cheap to produce a convincing security report. That does not mean they have made it cheap to produce a correct one. Our earlier “Fog of Slop” post described the incentive problem: automated submissions can overwhelm review systems even when most reports are wrong, low-value, or copies of the same underlying issue.
This analysis puts numbers behind that argument. We assembled a benchmark from Sherlock’s public contest data: 13,251 submissions from 24 recent audits. After labeling and deduplication, those submissions resolve to just 303 unique, paid medium- or high-severity bugs.
That gap is the point of the benchmark. Existing evaluations often begin with a small, curated list of known bugs. EVMBench, for example, contains 22 items. This dataset starts with the mess that judges actually see: thousands of plausible claims, an uneven class balance, and extensive duplication.
Four out of five labeled reports are not bountiable
Of the 12,090 labeled submissions, 9,580 — 79% — are invalid, informational, or low severity. Medium and high findings make up the remaining 21%.
The imbalance is not a quirk of one audit. The chart below shows the share of labeled reports that judges paid as medium or high severity each month. Higher points mean more real bugs surfaced that month. Even the most bug-rich month was still majority noise.
Monthly bountiable share — percent of labeled submissions that were paid as valid findings
Why May spikes: LEND, one unusually large and bug-rich contest, produced 42% bountiable reports that month. The share then falls back to roughly 18% in June and July.
2,510 paid submissions collapse to 303 bugs
The valid reports contain another kind of noise: duplication. Sherlock may receive many descriptions of the same bug, but those reports do not represent independent discoveries for benchmark purposes.
Scoring on raw reports would reward a tool 2,207 extra times for recognizing bugs it had already seen. We therefore score by bug cluster. The effective benchmark size is 303, not 2,510.
The bugs only one person found look different
Eighty-four bugs were reported by exactly one auditor. These “singletons” may be the most interesting part of the dataset: they are the findings least likely to emerge from many people asking a model to search broadly.
A singleton is 11 percentage points more likely to be medium severity, not high severity.
Switch views to compare severity mix with judge disagreement. Percentages are calculated within each group.Singletons are 71% medium severity, compared with 60% for bugs found by multiple auditors. They are also harder for judges to resolve: 45% were contested during escalation, versus 28% of crowd-found bugs.
One possible interpretation is that sophisticated, targeted LLM use helps skilled auditors pursue subtle issues, while undirected “find all bugs” use pushes many people toward the same recognizable patterns. The data is consistent with that explanation, but it does not prove it. We do not observe which tools each auditor used.
Solidity dominates the validated set
Of the 303 unique bugs, 259 come from Solidity contests, 41 from Go, and three from Rust.
The non-EVM sample is too small for strong conclusions about model capability by language. Go and Rust appear to underperform here, but the more likely explanation is selection: these particular contests produced fewer paid findings. For now, this is primarily an EVM benchmark with limited non-EVM coverage.
What the benchmark measures next
The next phase will run our validation tool against all 303 confirmed bugs and their audited codebases, then pair those positives with representative invalid reports. That creates three separate, measurable tasks:
- Find: Starting from the code alone, how many of the 303 known bugs does the tool independently surface?
- Validate: When given a confirmed report, can the tool trace the claim through the code and reproduce the judge’s reasoning?
- Confirm or refute: Across known-valid and invalid reports, does the tool reach the same disposition as the contest judges?
Separating these tasks matters. A system can be good at finding familiar vulnerability patterns but poor at validating a specific claim. It can also be a strong validator without independently discovering every bug. Reporting one aggregate score would hide those differences.
The short version
This dataset turns the Fog of Slop problem into a concrete benchmark: 13,251 reports, 79% not bountiable, and 2,510 paid submissions that reduce to 303 distinct bugs. The difficult part is not producing another plausible report. It is preserving the rare, subtle finding while rejecting invalid claims and recognizing duplicates.
Our next results will show where our validation tool succeeds and fails across discovery, validation, and final judgment. That should make the benchmark more than a description of today’s submission flood: it becomes a repeatable way to compare tools against the work security reviewers actually have to do.
Open questions
- How much judge workload can a filter remove before the risk of missing a singleton becomes unacceptable?
- Should discovery and validation be evaluated as separate products, rather than combined into one score?
- Should a benchmark score only the final verdict, or also the quality of the code trace and evidence used to reach it?
- Does source-grounded validation improve most on subtle singleton bugs, or on the repeated findings that dominate the crowd?