Our eval said F1 = 1.00. It was lying, and we published the retraction.

For a while, erabot's public eval page reported detection F1 = 1.0000. Precision 1.0, recall 1.0, zero false positives, zero false negatives across 595 call sites.

A perfect score should have been the red flag. The ground-truth manifests were generated by the scanner itself, so the scanner was being graded against its own answers. Circular. We retracted the number, rebuilt the ground truth with an independent labeling pipeline that never sees the scanner's output, and started over. Today the eval page opens with the retraction note, because we would rather show you 0.76 that is real than 1.00 that is not.

What honest measurement unlocked

The independent labels (520 repos, 3,049 call sites) said our structural detector's recall was 0.42. That number stung, so we did the only useful thing: categorized every miss.

The false negatives clustered: framework idioms like agent.run() and chain.invoke() where the billable call happens inside library code; wrapped clients resolved in another file; dynamic dispatch. We also tested the flattering hypothesis that our line-matching was undercounting — it accounted for 1% of misses. The recall problem was real.

Build, measure, and sometimes delete

We built two deterministic fixes. A framework-idiom pass with receiver deny-lists (so Flask's app.run() does not become an LLM call) and string guards. And a wrapper-resolution pass that flagged call sites of functions wrapping SDK calls.

Then we measured both on the same 520 repos. The idiom pass: +6.2 points of recall at flat precision. Shipped. The wrapper pass: precision 0.141. Our premise about how wrapper calls should be counted was simply wrong. We deleted the pass and left a tombstone in the module explaining why, so nobody rebuilds it without reading the evidence.

Where detection stands now

Detection is a four-layer system, and each layer's contribution is measured on the same corpus:

| Configuration | Precision | Recall | F1 | |---------------|-----------|--------|-----| | Structural (tree-sitter) | 0.833 | 0.423 | 0.561 | | + deterministic idiom extensions | 0.831 | 0.485 | 0.612 | | + LLM recall backstop (full stack) | 0.799 | 0.731 | 0.764 |

A fourth layer — the audit agent actively searching the codebase during an audit — operates beyond what this benchmark can score.

The moral

Evals are only useful if they can hurt you. Ours went from a number that flattered us to a system that steers engineering: every detection change now ships with a before/after on the same 3,049 labeled call sites, or it does not ship. The full tables, including everything above: /eval.