Evaluating and Monitoring Agents

27.1. Evaluating and Monitoring Agents#

An agent’s output looks confident whether or not it is correct, and a long run can quietly use up a budget. Ask three questions of any agentic workflow: is it right, what did it do, and what did it cost. The last section asks a fourth: whether the task suits an agent at all.

27.1.1. Evaluate against a baseline#

Before you trust an agent with real work, measure it on tasks whose answers you know. Pick five to ten tasks you have already solved, give each to the agent with the same prompt, and score its output against your answer. Rerun the set whenever you change the model, the prompt, or the setup, so you can see whether a change helps or hurts.

Use the same set to resist needless complexity. A multi-agent pipeline adds coordination overhead and can pass errors along, so compare it with a single agent on your own tasks, and keep whichever is more accurate and cheaper; see Multi-Agent Orchestration. Multi-Agent Workflow Example makes this comparison for one task.

27.1.2. Check the output#

A baseline shows how an agent does on average; each result still needs its own check:

Run the tests yourself. Keep or write tests before the agent changes code, and read the test output, not the agent’s summary of it.
Check that the tests were not weakened. Agents under pressure to finish sometimes skip a failing test, loosen a tolerance, edit a reference file, or hard-code an expected value. Read the diff of your test files as carefully as the code.
Compare against known-good outputs. Compare new output with a reference result, within a tolerance you chose on purpose; see Refactoring Research Code into Packages.
Reproduce the key numbers. Recompute a headline number another way, rerun with a different seed, or plot the data and look. Two independent paths agreeing is far stronger evidence than one confident run.
Read the full diff. An agent’s summary of its changes can leave things out; the diff cannot.
Check every source. Open each citation, link, and quoted figure the agent relies on; see Literature Review and Data Exploration.

27.1.3. Watch what it did#

To see where an agent went wrong, not just that it did, keep a record of each run. Three records cover most needs:

The session transcript. claude --resume reopens any conversation, with every tool call and result.
The print-mode result. --output-format json records the final answer, session ID, and estimated cost of a batch run; --output-format stream-json --verbose records every step.
A command log. A hook can append every shell command the agent tries to a file; see Hooks.

For tracing many runs in a standard format, see the OpenTelemetry GenAI semantic conventions.

27.1.4. Watch cost and context#

Track usage and cost. Your account has usage limits, which Claude Code shows with /usage. FASRC makes you responsible for monitoring your agents’ usage and any costs they incur for you, your lab, or the university; see its AI Agents guidance. Cluster jobs draw on your fairshare allocation; see Fairshare Policy.
Let caching work. Claude Code caches the unchanging start of each request, such as its system prompt, tools, and project instructions, automatically. Run /clear between unrelated tasks, so the old conversation is not sent again with every request.
Keep the context small. Give the agent what the task needs, not the whole repository; a smaller context is cheaper and often more accurate.
Bound long runs. Set --max-turns, --max-budget-usd, and a SLURM --time before an unattended run; see Caps on unattended runs.

27.1.5. When not to use an agent#

Some tasks are better done yourself, or with the agent preparing a command that you review and run:

You cannot check the result. With no tests, no reference, and no expertise to review it, a confident answer is a liability.
The data is above what the tool is approved for. On the cluster, a cloud agent may work only with public data (Level 1) unless your school has an agreement with the provider; see Before you start. For a model served on the cluster, see HPC Agentic Recipes, including the data rules that apply to it.
The action is hard to undo. Deleting or moving shared data, overwriting results, submitting or canceling many jobs, changing permissions, and publishing are yours to run.
The judgment is the science. Choosing a hypothesis, deciding what a result means, and standing behind a claim stay with you; see Agentic AI in Research.
The task is small or one-off. If reviewing the agent’s work takes longer than doing it, do it yourself.
It would cost more than it saves. Open-ended runs over a large context can spend more than the task is worth.