Case study: two structural LLM-cost bugs in a production RAG platform

Most LLM cost tools tell you what you spent — a dashboard, a per-model breakdown, a bill that already arrived. erabot is built to do the opposite: read the code and find the reason the bill is high, before the next invoice. To show what that looks like on real software, we ran erabot's agentic auditor against a real target.

We picked a widely-used, open-source enterprise RAG platform — we're keeping it unnamed here, because the point of this write-up is the class of bug, not the project (both patterns are common, and the codebase is genuinely well-engineered). It's a good stress test for two reasons: it runs LLMs at real volume on real user traffic, so anything erabot finds has to be a subtle structural issue, not low-hanging fruit. We scoped the audit to the chat path (18 files), because that's the code that fires on every user message.

Here's what came back.

The setup: 4 detected call sites, 2 real

The structural detector flagged 4 candidate LLM call sites. The agentic layer then read each one and threw out two false positives — database .query() calls that a naive matcher mistakes for model calls. That left 2 real call sites, and both led to genuine, cross-file findings.

We're deliberately open about those false positives, because it's the whole point: a regex or a naive AST match over call names will over-report. The agentic pass exists to verify by reading the code before anything reaches you. A finding erabot shows you is a finding it defended.

Finding 1 — a frontier model doing a job a cheap model should do

Category: model mismatch · Estimated savings: ~$2,700/mo (see methodology)

When a conversation grows past a set fraction of the model's context window, the platform compresses the chat history: it keeps the most recent portion verbatim and asks an LLM to summarize the rest. That's a sound design. The cost bug is which model does the summarizing.

The compression routine never receives a model of its own. It's handed whatever LLM the user or persona selected for the actual conversation — the same frontier, tool-calling-grade instance used for the live chat completion. So the summary call ships the bulk of a long history (tens of thousands of tokens) to a frontier model (GPT-4o-class or Opus-class), for a task that is pure bulk-text condensation. Summarization doesn't need frontier reasoning or tool use; a utility-tier model (gpt-4o-mini, claude-haiku) does it at a fraction of the input price.

This isn't a one-off — it's structural. There's no separate "utility model" tier for internal LLM housekeeping, so every persona and model choice inherits the same overspend. The fix, in summary: introduce a configurable utility-model seam and route internal summarization through it, leaving the user's model choice for the conversation itself. A small, well-scoped change — erabot produced the patch alongside the finding.

Finding 2 — a prompt cache quietly busted by message ordering

Category: caching · Estimated savings: ~$130/mo (see methodology)

This is the kind of bug a human reviewer almost never catches, because every individual line looks correct.

Each request is assembled as an ordered list of messages, with a flag marking which ones are cacheable. The system prompt and the kept history messages are correctly flagged cacheable. The request builder then finds the longest cacheable prefix — but with a single monotonic rule: the moment it reaches the first message that isn't flagged, the cache boundary freezes there for the rest of the request.

Here's the trap. Two things get inserted inside that prefix rather than after it: the persona's custom instruction block, and a context message that embeds the entire content of every attached file. Neither is flagged cacheable, so both default to "off." Sitting inside the scanned prefix, they freeze the cache boundary early — and the full text of every attached file, plus the persona block, gets re-tokenized and billed at full input price on every single turn, even though it's byte-identical turn to turn. The surrounding history caches; the attachments don't.

The fix, in summary: flag those two message types cacheable so they fall inside the cacheable prefix like the history around them. Effectively a one-property change — but one no dashboard could ever point you to.

The numbers, and exactly what they rest on

On erabot's default assumptions, the two findings total ~$2,830/mo. Read that number honestly — here's precisely what's behind it:

• erabot ran as a static + agentic audit with no runtime telemetry connected. Where it can't measure traffic, it assumes a deliberately simple 1,000 calls/month per call site and a 50/50 input/output token split, priced at current public rates for the models found in the code.
• The dollar figure therefore scales linearly with real traffic. For a busy deployment, 1,000 compression events/month is conservative — the real number is higher. For a small self-host, it's lower. The estimate is a floor keyed to an assumption, not a measurement.
• Connect a runtime source (the erabot proxy, or a Helicone/Langfuse importer) and the engine substitutes measured call volumes for the assumption. That's when the number stops being an estimate.

What is not an estimate is the findings themselves. Both were verified against the platform's actual source, both are cross-file and structural, and both came with an apply-ready patch. The dollar amount is a hypothesis about traffic; the bug is a fact about the code.

Why a dashboard would have missed both

Neither of these is visible at the line level. Finding 1 only exists because a model chosen in one module flows, several hops later, into a summary call in another — the waste is in the data flow, not any single line. Finding 2 only exists because of the interaction between where messages are appended in one place and how a cache prefix is scanned in another. A spend dashboard sees the total go up; it can't tell you the reason is a cache flag that was never set two modules away.

That's the gap erabot is built for: cost bugs that live in the architecture, found by reading the code the way an engineer would — then handed back as patches you can apply.

If you run LLMs at volume

The audit runs entirely in your own environment — your code never leaves your VPC. You get the same output: verified, cross-file findings with apply-ready patches, and a dollar estimate that gets sharper the moment you connect real traffic. Read how the CLI works, or book a demo.

Findings verified against a real production codebase at the time of writing (July 2026). Dollar estimates use erabot's default no-telemetry assumptions described above and will differ from your actual spend.