Inside the shadow harness: how we prove a cost fix won't degrade quality

Every AI cost tool eventually gives you the same advice: use a cheaper model. GPT-4o-mini instead of GPT-4o. Haiku instead of Sonnet. On a spreadsheet it is irresistible — a 10–20x cost multiplier, gone with a one-line change.

It is also the one recommendation that can quietly wreck your product. A cheaper model is cheaper because it is worse at something. If that something is your task, you did not save money — you shipped a regression and lowered your bill as a side effect. A dashboard that only watches spend will happily report the "win."

So erabot does not recommend a model swap on faith. Before a swap reaches your report, it goes through the shadow harness.

What the harness does

The idea is simple: don't argue about whether the cheaper model is good enough — measure it, on your actual inputs.

1. Replay. Take real sample inputs for the call site in question and run them through both the current model and the proposed cheaper one.
2. Measure the delta. Compute the true cost difference on those inputs — not a list-price estimate, the actual token math.
3. Judge the quality. An LLM judge compares the two outputs and scores whether the cheaper model's answer is acceptable for the task.
4. Gate. The call site gets one of three verdicts:
   • SAFE — quality holds, real savings. Recommend it.
   • UNSAFE — the cheaper model degraded the output. Reject the swap, no matter how good the savings looked.
   • NO_SAVING — the delta did not survive contact with real inputs. Don't bother.

Only SAFE swaps make it into the headline savings number. UNSAFE ones are shown separately, flagged, and excluded from what we claim you can realize.

It rejects our own suggestions

The harness is not decoration. When we ran the erabot agentic engine across 15 open-source AI products, it proposed 31 model swaps and the quality gate blocked 12 of them. Our own agent wanted to make those changes; the harness said no.

One concrete case: on Onyx, a RAG chat platform, the engine flagged a context-compression swap that modeled real savings. The shadow harness replayed it, judged the compressed outputs against the originals, and returned UNSAFE — the compression lost information the answers depended on. That finding never became a recommendation. Without the harness, it would have been a confident, wrong, money-saving suggestion.

This isn't a one-off. A 2026 pre-registered randomized trial of prompt compression in production found that moderate compression cut cost ~28%, but aggressive compression backfired — degrading quality enough to erase the win. And the "Token Complexity" research (2025) shows every task has an intrinsic minimum token budget below which accuracy collapses. In other words: there's a real floor, it's task-specific, and you can't find it by guessing. You find it by measuring — which is exactly what the harness does, per call site, on your own inputs.

Why this is the honest way to count savings

There is a temptation in this category to sum every theoretical saving into one big number. We don't, for two reasons:

• Quality-affecting fixes must earn their place. A model swap only counts if it passed shadow verification. Quality-neutral fixes (caching, dedup, batching, context-trim) don't change the model's output at all, so they are the safe core of the savings; risky swaps are additive only when proven.
• You can't sum overlapping fixes. Two findings on the same call site don't stack, so the realizable total is lower than the sum. We say so.

The result is a smaller headline number than a spreadsheet would give you — and one you can actually bank.

Why observability can't do this

A runtime observability tool sees the requests you already sent. It has no counterfactual: it cannot run your input through a different model, compare the outputs, and tell you whether the cheaper one would have been good enough. That requires replaying inputs and judging results — which requires being in the loop at analysis time, not just watching traffic.

That is the moat. Anyone can tell you a smaller model is cheaper. Proving it is safe on your inputs, before you ship it, is the hard part — and the whole point.

erabot audit runs the harness inside your own environment, on your own model keys. The savings it reports are the ones that survived being tested.