AI agents in production: evaluation before autonomy

Building an agent that completes a task once is now easy. Building one that completes it reliably, with known failure modes, is still engineering work. We treat agents like any other production service: they get tests, metrics and a rollback plan.

Start with a golden dataset

Before tuning prompts, collect 50 to 200 real examples of the task with known-good outcomes. This dataset becomes the agent's test suite. Every prompt or model change runs against it, and regressions block the release just like a failing unit test.

Constrain the tools, not just the prompt

• Give each tool the narrowest permission it needs
• Validate tool inputs with schemas before execution
• Require human approval for irreversible actions
• Log every tool call with its inputs and outputs

Measure what the business cares about

Accuracy on the golden set matters, but so do cost per task, latency and the escalation rate to humans. An agent that is 3% more accurate but twice as expensive may not be an improvement.

const result = await runEval(agent, goldenSet);

assert(result.accuracy >= 0.94);
assert(result.p95LatencyMs <= 8000);
assert(result.costPerTask <= 0.04);

Autonomy is something an agent earns. We start with suggest-only mode, graduate to act-with-approval, and only then to full automation for the cases the data shows it handles well.