Agent trajectory evaluation

How it works

Tool-contract tests prove an agent can call a tool correctly; end-state assertions prove it finished. Neither says anything about the multi-step behaviour in between, which is where agentic systems actually fail. Trajectory evaluation runs the agent against scripted tasks in a controlled environment and scores the full sequence: task completion against a verifiable end state, step and token cost against a reasonable baseline, tool-choice appropriateness at each decision point, and pathologies — repeated identical calls, oscillation between two states, silent give-ups, actions outside the intended scope. Because agents are stochastic, the same task is run many times and reported as a success rate rather than a pass/fail, which also exposes the reliability gap between 'worked once in a demo' and 'works every time'.

When to use it

Any multi-step or tool-using agent; before widening an agent's autonomy or permissions; regression-testing agent behaviour across model or prompt upgrades.

Limitations

Needs a scripted environment with verifiable end states, which is significant setup and never covers the open-ended tasks real users bring. A trajectory can also be judged 'good' by a scoring rubric that simply encodes the author's assumptions about how the task should be done.

Cite this