Multi-Agent Workflow Example

29. Multi-Agent Workflow Example#

This chapter runs the task of End-to-End Agentic Workflow Example with four agent roles: a planner, a critic, a writer, and an auditor. The loop script still submits the jobs and writes the records. The code is in the agentic-workflow-example repository.

        flowchart TB
    subgraph SUP [" "]
        direction LR
        S(["Setup, baseline,<br/>and round 1"]) --> P["Plan-only job:<br/>plan wave 1"] --> R(["You read<br/>the plan"])
    end
    subgraph LOOP ["Loop job"]
        direction LR
        W["Planner and<br/>critic"] --> J["Up to 3<br/>round jobs"]
        J -->|next wave| W
        J --> F["Final<br/>steps"]
    end
    SUP -->|you start the loop job| LOOP
    LOOP --> V(["You check<br/>the result"])
    classDef agent fill:#14154C,color:#ffffff,stroke:#3D3E82;
    classDef you fill:#A51C30,color:#ffffff,stroke:#A51C30;
    class P,W,J,F agent;
    class S,R,V you;
    style SUP fill:none,stroke:#c8c8d0
    style LOOP fill:none,stroke:#c8c8d0

Part	Single-agent example	This chapter
Agents	One agent step for each round, and a report step	Planner, critic, writer, and auditor
Rounds	One round job at a time	Waves of up to 3 round jobs
Check before a round	The schema	The schema, a repeat check, and the critic
GPUs	Full GPUs	A100 MIG slices
Confirmation	One confirmation round	Two confirmation rounds, and the means of 3 seeds
Check of the report	You	The auditor and two code checks, then you

Several agents cost more tokens. Use them only when a test shows a gain; see Start with one agent. This chapter runs that test on this task. It showed no gain from the extra agents; see Results.

29.1. Before you start#

Single-agent example. Do the setup, the baseline, and round 1 of End-to-End Agentic Workflow Example. This loop copies round 1 to its own state directory, state_multi/.
Code. In your clone, run git pull. Then git status must show no changes.
Records. If you ran the single-agent loop in this clone, the agents can read its records in state/. For a separate result, use a new clone.
GPUs. The round jobs run on kempner_requeue with --constraint=a100-mig. This selects A100 MIG slices (3g.20gb): parts of an A100 GPU with 20 GB of memory each. One round takes 1 to 3 minutes.
Cost. The budget is 1,250,000 tokens, the plan-only job included, and 150 GPU minutes for the rounds. Both are estimates: a step can use more than planned. Plan for 10% more tokens, because the auditor can use more than its reserve. The confirmation rounds can add up to 20 GPU minutes, and each requeue up to 10.
Your changes. The jobs refuse to start while a tracked file has changes. So commit each change to experiment_multi.yaml. For a test, use a copy in local/, as the README shows.

29.2. Step 1: Check the settings#

Read experiment_multi.yaml, slurm/multi_loop.sbatch, slurm/round_array.sbatch, and the prompts in prompts/multi/. These files set what the agents and the jobs can do.

Role	Model	Effort	Task
Planner	`claude-opus-5-5`	`xhigh`	Plans up to 3 configurations for each wave, each with a hypothesis, a predicted validation accuracy, and a pass probability. Revises rejected configurations one time.
Critic	`claude-sonnet-5-5`	`high`	Accepts or rejects each configuration before any GPU job runs
Writer	`claude-sonnet-5-5`	`medium`	Writes the draft report
Auditor	`claude-opus-5-5`	`max`	Checks each claim of the report against the records

Change a role. Edit its model and effort in roles. Use full model names, because an alias such as opus moves to newer models. Keep a different model for a checker and the role that it checks.
One agent. Set roles.critic.enabled: false. Then one agent plans the waves.
Model check. Claude Code can use another model with only a warning, for example when your organization does not allow a model. A planner or critic step with another model stops the loop with model_check. For the writer and the auditor, audit.json records it in failures.

Limit	Value
Threshold	Validation accuracy of at least 0.905
Plateau	Stop after 6 complete rounds in a row, in round order, without a smaller passing model
Tokens	1,250,000: 800,000 for the search, 450,000 for the writer and the auditor. The search stops when the next wave could pass 800,000.
GPU	150 GPU minutes. Each round of a wave reserves its 10-minute time limit.
Rounds	45 round attempts, round 1 included
Failed waves	Stop after 2 waves in a row with no complete round
Round jobs at the same time	At most 3, as one array job; see Array Jobs
Queue	45 minutes for one array job. Then the loop script cancels the waiting tasks and stops.
Loop job	`shared`, 2 CPUs, 8 GB, 6 hours. No new wave starts in the last 3 hours, which are for the role steps and the final steps.

The agents have the guardrails of the single-agent example. They have no shell, no write tools, and no MCP tools. They cannot read outside the project directory, or read the README; see Guardrails and Prohibited activities. Only the scale changes: up to 3 GPU jobs at a time, within a fixed GPU budget.

Before an unattended run, answered for this example

Question	Answer
Task boundary	Plan or check configurations, or write or check the report
Read scope	Only the project directory, without the README
Write scope	None. The loop script writes all files.
Credentials	None. The agents have no shell.
Network	Only the model provider
Data policy	CIFAR-10 is public data (Level 1).
Untrusted input	Project files, and the outputs of earlier role steps in `state_multi/`
Resource scope	At most 3 GPU jobs at a time, at most 10 minutes for each run of a job; 150 GPU minutes; 45 round attempts; 1,250,000 tokens; 6 hours
Evidence	The `run_meta.json` file of each round, `history.tsv`, `ledger.tsv`, `waves/`, `hypotheses.tsv`, `summary.json`, `audit.json`, the JSON result of each role step, and the loop log
Stop condition	The limits above. SIGTERM stops the loop cleanly.

See Before an unattended run.

29.3. Step 2: Plan wave 1#

Check the first plan before any GPU job runs.

On a login node, in the project directory, submit the plan-only job:

source cluster.env
sbatch --time=01:30:00 slurm/multi_loop.sbatch --plan-only

When squeue -j <job_id> shows nothing, the log logs/prune-multi-loop_<job_id>.out must end with wave 1 planned: N configuration(s) accepted.
Read state_multi/waves/wave_01/plan.md. It shows each configuration and the verdict of the critic. A rejected configuration gets one revision. If the critic rejects the revision too, the configuration is dropped.
To accept the plan, start the loop; see Step 3: Start the loop.
To plan again, remove state_multi/waves/wave_01/, and submit the job again. A plan of wave 1 uses about 90,000 tokens. Do not edit the files of the plan by hand.

Example: a plan of wave 1 with the default roles

Item	Sparsity	Steps	Epochs	Learning rate	Excluded	Predicted validation accuracy	Pass probability	Status
1	0.92	2	10	0.02	none	0.928	0.92	accepted
2	0.96	4	16	0.03	none	0.917	0.75	accepted
3	0.985	5	20	0.05	`conv1`, `fc`	0.906	0.45	accepted

29.4. Step 3: Start the loop#

On a login node, in the project directory, submit the loop job:

source cluster.env
sbatch slurm/multi_loop.sbatch

Wave 1 runs the plan that you read. For each later wave, the planner plans, the code and the critic check the plan, and the loop script submits one array job. This is the pattern of Run an agent unattended in a batch job.

Watch. Run tail -f logs/prune-multi-loop_<job_id>.out. Do not turn on remote control; see Remote control.
End. When the loop job is done, its log ends with final steps done: see state_multi/summary.json, final_report.md, and audit.json.
Stop. Run scancel <loop_job_id>. The loop script cancels its array job. Check with squeue --me that no prune-round task is left.
Continue. Submit the loop job again. It counts the GPU minutes of the canceled tasks, and it continues the search.
Finish later. Run sbatch slurm/multi_loop.sbatch --finish. It runs only the final steps, and it keeps the selected round. After the final steps, the loop job refuses to start in the same state directory. To search again, use a new one.
Timeouts. If sacct -j <array_job_id> shows TIMEOUT, and the logs logs/prune-round_<job_id>.out have no epoch line, reads from DATA_DIR hang. Each such round uses 10 GPU minutes. Stop the loop, and tell FASRC.
Problems. If the agents behave unexpectedly, follow Reporting a concern.

29.5. Step 4: Check the result#

After the last wave, the loop script selects the passing round with the fewest weights left. If no round passes, it selects the round with the highest validation accuracy. The search rounds use seed 0. The loop script runs two confirmation rounds of the selected round, with seeds 1 and 2. It evaluates the three models on the test set, copies the selected model to results/state_multi/, and runs the writer and the auditor. Then check these files in state_multi/; see Check the output:

File	Check
`summary.json`	`stop_reason` is a normal stop. `integrity`, `agent_format`, and `model_check` mean a fault.
`summary.json`	`means.holds` is `true`: the 3-seed means are at least 0.905 for validation and 0.900 for test. If it is `false`, the result does not hold, so report the 3-seed means. If it is `null`, a confirmation round is missing. Run `--finish`.
`audit.json`	`auditor_step_failed` is `false`. Each claim is `supported`. Check each claim that is not. If the auditor step failed, read `failures`.
`audit.json`	Check each number in `unmatched_numbers`. The code check `check_report` did not find it in the records. A derived number, such as a margin, is often correct.
`audit.json`	`check_round.ok` is `true`.
`final_report.md`	Read it after the checks above. Its tokens and cost are from before the writer and auditor steps.
`ledger.tsv`	The tokens of each role, and the GPU minutes of each job

The auditor and the code checks do not replace your check; see Best practices for trustworthy agentic research.

To audit the report of the single-agent example, run source cluster.env, then STATE_DIR=state sbatch --time=01:00:00 slurm/multi_loop.sbatch --audit-only. It writes state/audit.json. The audit of a single-agent report uses about 130,000 to 360,000 tokens.

29.6. Results#

Three designs ran the task, one time each:

Design	Setup
Sequential	One agent, one round at a time: the loop of the single-agent example, with the limits below
Waves	One agent, waves of 3: this loop with the critic off
Waves with critic	Planner and critic, waves of 3: this loop with the default settings

Same conditions. All designs start from the same baseline and round 1, each in its own clone. In each design, claude-opus-5-5 at xhigh decides, claude-sonnet-5-5 writes the report, and the auditor checks it. Each design has 800,000 tokens for the search, 45 round attempts, a plateau of 6 rounds, two confirmation rounds, and 150 GPU minutes on MIG slices.
Rule. Fix the rule before the runs. A result holds if its 3-seed means are at least 0.905 for validation and 0.900 for test. A design is better than another only if both results hold, and its weights left are at least 0.15 percentage point lower, for example 0.60% against 0.75%.

Result	Sequential	Waves	Waves with critic
Stop	Plateau	Plateau	Token budget
Rounds after round 1	14	21	21
Selected round	Round 9	Round 16	Round 21
Weights left	0.700%	0.485%	0.661%
Validation accuracy of seeds 0, 1, and 2	0.9058, 0.9032, 0.9040	0.9052, 0.9046, 0.9016	0.9088, 0.9030, 0.9028
Means of the 3 seeds: validation, test	0.9043, 0.9001	0.9038, 0.8978	0.9049, 0.9017
Result holds	No (validation)	No (both)	No (validation)
Tokens of the search	626,272	616,201	692,090
Tokens of the report and the audit	157,494	262,900	656,481
Estimated cost	$2.79	$4.22	$5.99
GPU minutes, confirmation included	41.8	59.9	61.8
Time of the search, plan-only job included	62 minutes	45 minutes	52 minutes

No result holds. So the runs show no difference between the designs. Waves with critic found its smallest passing model in its last wave, and then the token budget stopped it.
Selection. The loop selects the smallest model that passed with seed 0. Near the threshold, a lucky seed helps a model pass. In each design, both confirmation rounds of the selected round were below 0.905.
Tokens. Waves with critic used 1,348,571 tokens, 8% more than its budget. Its auditor used 619,300 tokens, more than the 450,000 kept for the writer and the auditor.
Cluster. All rounds after round 1, and the confirmation rounds, ran on A100 MIG slices (3g.20gb), with no requeue. In the search, each design waited 4 to 6 minutes in the queue in total. Your waits can be longer.
One run. One run of each design cannot prove a difference. The tokens of each role change from run to run.

Tests in a CPU job checked the critic and the auditor on the records of a single-agent run; see Evaluating and Monitoring Agents.

Test	Result
Critic: 5 configurations that break a rule, 3 runs each	Rejected in 15 of 15 runs, each for the right reason
Critic: the 14 configurations of the single-agent run, 3 runs each	Accepted in 41 of 42 runs
Auditor: the report of the single-agent run, with 3 known errors	Found the 3 errors, and 2 other errors. `check_report` found 1.
Auditor: the report with the 3 known errors fixed	Found the 2 other errors again, and gave 1 false alarm
Auditor: the fixed report with 5 added errors	Found the 5 errors, and 1 of the 2 other errors. `check_report` found 2.

In the three runs, the critic accepted all 21 configurations. The auditor marked 3, 2, and 3 claims as wrong in the reports of Sequential, Waves, and Waves with critic. check_report found 2 of them: the tokens and the cost in the report of Sequential, which are from before the report step; see Step 4: Check the result. Each report gave the selected round and the 3-seed means correctly.

29.7. When to use several agents#

Check of Before you trust the result	Evidence
A single agent was tried on the same task, and the multi-agent version did measurably better.	No. No design has a result that holds, so the runs show no gain.
Each agent had only the tools and budget its step needed.	Partly. Read-only tools, role caps, and the model check. The auditor of Waves with critic used more tokens than its reserve.
You read each step’s output, not only the last.	Yes. The plan of wave 1 before the loop. The later plans, `hypotheses.tsv`, and `audit.json` after it.
The reviewer worked from the diff in a fresh context, not from the implementer’s summary.	Partly. The critic and the auditor are separate `claude -p` calls. The critic reads the reason of the planner for each configuration, as its rules need.
The plan, the review, and each step’s JSON result are kept, and the total cost is recorded.	Yes. `state_multi/` and `summary.json`

Waves. Use them when time matters. Waves ran 21 rounds in 45 minutes, and Sequential ran 14 rounds in 62 minutes, with about 2.6 GPU minutes for each round in both. Waves also used fewer search tokens for each round.
Critic. Test it on your task first. In the runs, it rejected no configuration, and it used 231,560 of the 692,090 search tokens of Waves with critic. It can help when a bad round uses many GPU hours.
Auditor. Use it for each report that you share. It found wrong claims that check_report did not find, in the tests and in the reports of Waves and Waves with critic.
Confirmation rounds. Keep them. With seed 0 alone, each selected round passed the validation threshold, but no result held.

29.8. Adapt the example#

Roles. Change the models, the effort levels, and the caps in experiment_multi.yaml. Commit the change.
Waves. Change wave.size and wave.max_parallel. More GPU jobs at a time make the unattended run larger, so read Prohibited activities first.
GPUs. Set round.constraint in experiment_multi.yaml, for example a100|h100|h200|rtx6000pro. A full GPU is faster, so lower gpu_minutes too, but keep 10 minutes for each round of a wave.
Subagents. To use a role in your own interactive sessions, make its prompt a subagent; see Subagents in Configuring Agents for Your Project. Do not add it to the clone of this example: a new file in .claude/ stops the jobs with integrity.