Why we’d rather refuse than fabricate a savings number

Every LLM gateway will happily tell you it saved you money. Point it at a cheaper model, watch the per-request cost drop, and there’s your number. The problem is that the number is answering the wrong question. Cost went down — but did quality hold? If you can’t answer that with evidence, the savings figure is a guess wearing a dollar sign.

TokenTriage draws a hard line here. When you ask it whether routing beat the zero-router baseline, it can give one of three answers:

• It beat the baseline — but only when there is a real counterfactual to compare against (judged shadow duplication). Then you get a number, and the number survives an audit.
• It sits on the frontier — when all it has are served-model labels, it can say routing is no worse, but it cannot claim a win, because it never saw the road not taken.
• It can’t be measured yet — when the labels are insufficient, it refuses. Not a zero, not a shrug: an explicit refusal that tells you what evidence is missing.

That third answer is the one that matters. Anyone can build the first. The discipline is in shipping software that will look you in the eye and say “insufficient labels to judge routing” instead of inventing a percentage that would fall apart the moment a diligence team pulled the thread.

Why this protects you, not us

A fabricated savings number is a liability that compounds. You make a roadmap decision on it. Finance forecasts on it. An investor diligences it. When it turns out the cheaper route quietly degraded quality on 8% of requests, every downstream decision built on that number was wrong too.

An honest gateway is slower to impress and impossible to embarrass. The savings you do see are ones you can defend line by line — because each one carries its method, its evidence, and its provenance. That is the whole trade: a smaller, truer number beats a bigger, breakable one every time you have to stand behind it.