We audited 15 open-source AI products for LLM cost waste. All 58 patches applied cleanly.
To find out whether an AI agent can audit a codebase for LLM cost waste the way a senior engineer would, we ran the erabot agentic engine across 15 substantial open-source AI products: LibreChat, Dify, Flowise, Langflow, CrewAI, MetaGPT, OpenHands, Onyx, Quivr, Continue, Lobe Chat, Khoj, Aider, GPT Researcher and PrivateGPT. Over 31,000 source files in total.
The headline numbers:
| Metric | Result |
|--------|--------|
| Products audited | 15/15, zero failures |
| Root-cause findings | 29 (45% span multiple files) |
| Patches generated | 58, all 58 apply cleanly (git apply --check) |
| Model swaps blocked by our own quality gate | 12 of 31 |
| Average cost per audit | $2.18, about 8.5 minutes |
The findings we did not expect
A live timestamp defeating prompt caching across an entire framework. MetaGPT embeds the current time into its RoleZero/TeamLeader system prompt. Provider prompt caching requires an exact prefix match, so this one line invalidates the cache on every request, forfeiting the 50-90% cached-input discount for the whole agent runtime.
A default no-op reranker stuffing 40 unranked chunks into every RAG call. Quivr's retrieval path skips top_n when no reranker is configured, which is the default. Answer generation receives up to 40 chunks instead of a handful. At typical volumes we model it around $3,250/month per busy deployment.
The most expensive model as the default for every new user. OpenHands defaults new users to the priciest Claude model in the frontend, in the local deployment script, and doubled its context condenser threshold platform-wide.
A component library in every message. LibreChat injects its full shadcn/ui component reference (~4,400 tokens) into the system prompt of every message when Artifacts mode is on, not just messages that create artifacts.
The number we are proudest of is the rejections
The agent proposes fixes; a shadow harness replays representative traffic through the old and new configuration, measures the real cost delta, and has a strict judge compare outputs. It rejected 12 of the agent's 31 model-swap proposals — including one where the "cheaper" model rewrote an entire page structure. A verification layer that never says no is marketing. One that blocks its own agent's bad ideas 12 times in a single run is engineering.
Honest caveats
Savings figures are modeled at stated call volumes (disclosed per finding) and scale with real traffic; they are estimates until measured against a deployment's own usage. "Applies cleanly" means the patch applies to the audited revision; maintainers may reasonably prefer different fixes. We are filing the findings upstream with patches attached.
Every audit in this study ran with the same engine that ships in the CLI: erabot audit . runs the whole thing inside your own environment, and your code never reaches our servers. Full methodology and layer-by-layer detection accuracy: /eval.