Token-efficiency evaluation (cost per outcome)

How it works

Every AI system has a spend per outcome, and in multi-agent systems most of it goes on models talking to each other in prose that was shaped for human readers. Whether natural language is the right wire format between two agents is an empirical question, not a given: terser encodings — structured fields instead of sentences, a compressed prompt, an identifier passed by reference instead of the whole document restated, a schema the receiving agent already knows — cut tokens per turn, but they also change what the receiving model can do with what it was handed, and the saving is easily eaten by extra turns, retries, or a quality drop that only shows up two steps downstream. The measurement is differential: fix a task set and a success oracle, then run the same tasks through each candidate variant — verbose natural language, terse natural language, structured exchange, compressed prompt, reference-passing, a smaller model for the sub-step — and report success rate together with tokens in and out, turn count, latency and cost. The unit that decides is cost per successful outcome: a variant that halves tokens while dropping success from 90% to 70% made the task more expensive, not less. The same design prices the quieter sources of waste — history re-sent every turn, a system prompt that grew by accretion, a cache-unfriendly prefix — which are invisible without a standing number to compare against.

When to use it

Multi-agent and agent-to-agent systems where one model's output is another's input and nobody has yet priced the conversation; before adopting prompt compression, a format change, or a cheaper model, so the saving is demonstrated at equal quality rather than assumed; when a feature's unit economics have to be defended against a per-request budget; and as a standing regression gate, since prompt bloat arrives incrementally and no single commit looks expensive.

Limitations

Only as trustworthy as the success oracle behind it — without one strong enough to make a quality drop visible, efficiency work optimizes toward cheap failure, which is the characteristic way this method goes wrong. Results are specific to one model, its tokenizer and its pricing, so a format that is terse for one vendor need not be for another and savings don't transfer. Constraining or compressing what a model reads and writes can itself cost accuracy, so the cost and quality numbers have to come from the same run rather than be compared across sessions. Token count and latency are also separate axes that can move in opposite directions — a compression step is itself a model call — and like any differential method this compares the variants somebody thought to write, which is not the same as finding the efficient one.

Cite this