Eval
How well erabot works
Two things matter: does erabot correctly find the LLM calls in your code, and are its fixes worth applying? Both are measured below — on independently-labeled repos and with a quality gate on every fix.
Detection accuracy
Scored against ground-truth labels generated independently of the scanner — 520 real open-source repos, 3,049 call sites. Detection is layered; each layer adds recall.
| Layer | Precision | Recall | F1 |
|---|---|---|---|
| Structural (tree-sitter) | 0.83 | 0.42 | 0.56 |
| + framework-idiom detection | 0.83 | 0.49 | 0.61 |
| + LLM recall backstop (full stack) | 0.80 | 0.73 | 0.76 |
The full stack lifts recall +73% over the structural layer alone while precision holds at ~0.80. And this is the raw scanner — in a real audit the agentic engine re-verifies every finding before you see it, so the false positives you actually encounter are far fewer.
Run at scale
The full pipeline, end-to-end on 15 substantial open-source AI products.
Fixes are quality-gated
Before a model-downgrade is recommended, the shadow harness replays real traffic through the old and new setup and a strict judge checks the output still holds up. A downgrade that fails the bar is blocked — not shipped.
| Strategy | Saving | Quality | Result |
|---|---|---|---|
| Blanket model downgrade | 34.1% | 90% | Blocked |
| Prompt-hardened downgrade | −0.4% | 95% | No saving |
| Escalation routing | 24.7% | 100% | Shipped |
Model-swaps are ~1% of findings; the rest (caching, redundant-call removal, context trimming, batching) carry no quality trade-off. Structural fixes also pass a build/test gate before they ship.
A note on an earlier number
An earlier version of this page reported F1 = 1.00. That eval was circular — the ground-truth labels were generated by the scanner itself. We retired it, rebuilt the ground truth independently, and published the real figures above. We’d rather show a true 0.76 than a false 1.00.