Practical Evals for Agentic Systems

Thesis

Agentic AI systems need a different evaluation discipline than ordinary chatbots. A chatbot can be evaluated on whether the answer is helpful, safe, and grounded. An agent must be evaluated on the full path from goal to action: planning, tool choice, state tracking, policy compliance, recovery, cost, latency, and final outcome.

The central question is not simply, did the model say the right thing? It is, did the system do the right work under bounded authority? This is the strongest follow-on to governance because it answers the proof question: how do we know the AI system works and stays safe after we change a prompt, model, tool, policy, retriever, or workflow?

Evening Star thesis An agent is not production-ready until its behavior can be tested, scored, reproduced, and improved across traces, tools, outcomes, and human review.

What To Evaluate

Agent evals should measure both capability and control. Capability evals ask whether the system can complete valuable work. Control evals ask whether it completes that work safely, with the right authority, evidence, and escalation. A good program needs both. Capability without control creates reckless automation. Control without capability creates polished theater.

Eval surface	Question	Evidence
Task success	Did the agent achieve the intended outcome?	Environment state, tests passed, ticket closed, record updated.
Path quality	Did the agent take a reasonable route?	Trace steps, tool sequence, retries, unnecessary calls.
Policy compliance	Did it stay inside allowed authority?	Policy decision, approval state, denied actions.
Tool safety	Were tool calls valid and bounded?	Schemas, arguments, destinations, side effects.
Recovery	Did it notice failure, uncertainty, or drift?	Fallback path, escalation, rollback, incident marker.
Human usefulness	Could an operator understand and challenge the result?	Explanation, evidence bundle, confidence decomposition.

The Eval Stack

OpenAI frames agent evals around traces, graders, datasets, and eval runs. Anthropic emphasizes that agent evaluations combine code-based, model-based, and human graders. The UK AI Security Institute Inspect framework provides an open evaluation system with datasets, solvers, scorers, logs, agents, tools, and sandboxing. These are not competing ideas. They are the pieces of a practical stack.

A trace captures what happened. A dataset defines repeatable scenarios. A grader scores behavior. An eval run compares the current system to the expected bar. A dashboard tracks trend, regression, and failure class. A review workflow turns ambiguous results into human calibration. The important move is to make evals part of the release pipeline and runtime improvement loop, not a one-time research exercise.

Layer	Implementation pattern
Trace capture	Record prompts, retrieved context, tool calls, policies, approvals, outputs, and environment outcomes.
Scenario dataset	Use realistic tasks with initial state, hidden hazards, expected safe behavior, and final-state checks.
Grader mix	Use deterministic tests when possible, model judges for nuance, and human review for calibration.
Thresholds	Separate launch threshold, regression threshold, red-team threshold, and incident-trigger threshold.
Improvement loop	Convert production failures and overrides into new eval cases.

Capability, Regression, And Adversarial Suites

Evening Star should recommend three standing suites. The capability suite measures where the agent can create value. These tasks should be hard enough that the pass rate is not perfect; they show the hill to climb. The regression suite protects known-good behavior. It should pass nearly all the time and run on every meaningful change. The adversarial suite tests misuse, prompt injection, unsafe tool selection, context poisoning, data leakage, and escalation failures.

Long-horizon benchmarks add an important lens. METR proposes measuring the human time length of tasks that AI agents can complete reliably. HCAST similarly grounds autonomy tasks in human baselines. This matters for applied AI leaders because task duration is more intuitive than an abstract benchmark score. A system that reliably completes five-minute tasks is not the same risk profile as a system that can complete four-hour workflows with tools and persistent state.

A Practical Eval Runbook

Start by selecting one production workflow. Define the intended outcome, unacceptable outcomes, available tools, allowed data, and human escalation points. Capture traces from real or staged runs. Turn the best and worst runs into scenarios.

Write deterministic graders for anything that can be checked by code: tests passing, database state, JSON schema, number of tool calls, forbidden destinations, or policy result. Add model-based graders for judgment quality, instruction following, and evidence sufficiency. Use human review to calibrate the judges and resolve edge cases.

The release rule should be simple: no consequential agent ships without a passing regression suite, documented red-team results, trace completeness, and a rollback plan. The runtime rule is equally simple: every serious production failure becomes a permanent eval. That is how the system learns.

Closing

Practical evals are the difference between impressive demos and deployable systems. They let builders move fast without flying blind. They let security teams evaluate behavior instead of arguing about vibes. They let executives ask for proof instead of promises. Evening Star AI should make this a core doctrine: applied AI engineering is not real until the agent can be evaluated as a system.

References

Continue through the AI Security and Runtime Governance research thread.