End-to-End Agentic Workflow Example

28. End-to-End Agentic Workflow Example#

This chapter shows one complete agentic workflow on the Kempner AI cluster, from setup to a checked result. A Claude Code agent prunes a ResNet-18 model for CIFAR-10. The goal is the smallest model that keeps a test accuracy of at least 90%. The agentic-workflow-example repository holds the code, the job scripts, and the agent settings.

Round. One configuration in one GPU job. The job prunes the weights, fine-tunes the model, and measures it on 5,000 validation images.
Threshold. A round passes when its validation accuracy is at least 0.905: 0.900 plus a margin for noise. Only the final steps use the test set.
Weights left. The fraction of nonzero weights in the 20 convolution layers and the linear layer. Fewer is better.

        flowchart TB
    subgraph SUP [" "]
        direction LR
        S["You set up and<br/>train the baseline"] --> R1(["Round 1:<br/>you approve"]) --> C(["You check<br/>the limits"])
    end
    subgraph LOOP ["Loop job"]
        direction LR
        A["Agent step"] --> J["Round job"]
        J -->|next round| A
        J --> F["Final steps"]
    end
    SUP -->|you start the loop job| LOOP
    LOOP --> V(["You check<br/>the result"])
    classDef agent fill:#14154C,color:#ffffff,stroke:#3D3E82;
    classDef you fill:#A51C30,color:#ffffff,stroke:#A51C30;
    class A,J,F agent;
    class S,R1,C,V you;
    style SUP fill:none,stroke:#c8c8d0
    style LOOP fill:none,stroke:#c8c8d0

You supervise round 1. Then a loop job runs more rounds without a person, within limits that you check first. This is the bounded loop of Workflow patterns, with a stopping criterion and a budget.

28.1. Before you start#

Data. CIFAR-10 and the code of the example are public (Level 1), so a cloud agent may read them; see Before you start.
Agent. Install Claude Code with the native installer, because the loop job looks for claude in ~/.local/bin. Sign in with the Claude access that HUIT provides through Harvard SSO, and leave ANTHROPIC_API_KEY unset; see Running a terminal agent.
uv. Install uv; see uv Environment.
Partitions. You need a SLURM account that can use kempner_requeue for the GPU jobs and shared for the loop job; see SLURM Accounts. kempner_requeue can preempt a job with no warning. The GPU jobs save a checkpoint after each epoch, and they continue after a requeue; see Checkpoint and resume.
Cost. In a test run, the loop job used about 320,000 tokens and 17 GPU minutes in 66 minutes. The baseline and round 1 added about 2 GPU minutes and one interactive session.

28.2. Set up the project#

The project keeps its files in three places; see Storage Options.

What	Where	Reason
Code, environment, and records	Lab storage	Lab storage is persistent. The home directory has a 100 GB limit.
uv cache and Python interpreter	Lab storage	The cache is large. The agent sandbox hides your home directory, so an interpreter there cannot run in sandboxed commands; see Agent sandboxing.
Round checkpoints	Scratch (`RUN_ROOT`)	Scratch is fast. FASRC deletes scratch files after 90 days.
CIFAR-10	The read-only shared copy	No download; see Shared Data/Model Repository.

Clone the repository into lab storage, and copy the settings file:

mkdir -p /n/holylabs/LABS/<lab_name>/Users/<username>
cd /n/holylabs/LABS/<lab_name>/Users/<username>
git clone https://github.com/KempnerInstitute/agentic-workflow-example.git
cd agentic-workflow-example
cp cluster.env.example cluster.env

Edit cluster.env: set your SLURM account in SBATCH_ACCOUNT, and your lab name and user name in the three paths. sbatch reads the account from SBATCH_ACCOUNT, so the job scripts need no edit. Then load the file, and build the environment on a compute node:

source cluster.env && echo "$SBATCH_ACCOUNT $RUN_ROOT"   # check the values
mkdir -p logs && chmod 700 logs          # the hook writes logs/agent-commands.log here
salloc --partition=test --time=0-01:00 --mem=16G --cpus-per-task=4
uv sync --locked --extra gpu             # about 5 GB of packages
readlink -f .venv/bin/python             # must not be in your home directory
exit

Before you start Claude Code in the project directory, read its agent configuration, as for any repository that you did not write; see Untrusted repositories.

28.3. Train the baseline#

source cluster.env
sbatch slurm/baseline.sbatch

The job trains the dense model for 40 epochs on one GPU, in about 2 minutes. SLURM writes the log to logs/, which you created in the setup; see Write a batch script. The job is complete when squeue -j <job_id> shows nothing and the log ends with val_accuracy=. In a test run, state/baseline/metrics.json showed a validation accuracy of 0.937.

28.4. Run round 1 with the agent#

Round 1 tests the pipeline, not the method: the approvals, the job, and the records. Approve a mild configuration, for example a sparsity of 0.5 to 0.8 with a few epochs. Approve only the exact command sbatch --wait slurm/round.sbatch agent/next_config.json. Refuse added options such as --partition, --gres, or --time, because they change the fixed resources of the job.

On a login node, start tmux. In tmux, request a CPU job: salloc --partition=test --time=0-02:00 --mem=16G --cpus-per-task=4; see Keeping a session alive.
In the project directory, run source cluster.env, then claude --permission-mode default. The trust prompt lists the allow rules of the project, and its default choice is “No, exit”. Trust the directory only after you read the agent configuration.
Run /status, and check that the organization is Harvard’s.
Give this prompt:

Read AGENTS.md and state/baseline/metrics.json. Propose a cautious first round that tests the pipeline, explain it, and write it to agent/next_config.json. Then run sbatch --wait slurm/round.sbatch agent/next_config.json as a background command.
Read the configuration, and approve the write. Read the sbatch command, and approve it; see Submit and monitor.
When the job ends, type /check-round 1. Each line of the output must start with OK. If a line starts with DIFF, do not start the loop.
Type /exit. Then type exit two times: one time to end the job, and one time to close tmux.

This is the workflow of Your First Agentic Workflow on the Cluster, with a SLURM job as the task. In a test run, round 1 removed half the weights with a 2-epoch fine-tune and reached a validation accuracy of 0.932.

28.5. Check the limits, then start the loop#

In the unattended phase, no one approves each round. You approve the whole budget one time, so read the limits first:

experiment.yaml. The threshold, the plateau rule, the token budget, the GPU budget, the round limit, and the queue and time allowances.
slurm/loop.sbatch. The 6-hour time limit of the loop job, which is the final stop.

The limits that matter are outside the model; see Prefer enforceable boundaries over prompt instructions. The agent has different powers in the two phases:

Guardrail	Round 1 (supervised)	Unattended phase
Permission mode	`default` (manual); see Permission modes	`dontAsk`; see Caps on unattended runs
Tools	All built-in tools	Read, Grep, and Glob only
Configuration	You read it, then approve the write	The loop script validates it
Writes	The agent asks first. Deny rules block edits of tracked files, `cluster.env`, and the output directories; see Permission rules.	None. The loop script writes all files.
Reads	File tools: only the project directory. Sandboxed commands: not the home directory.	Only the project directory
Shell	Sandbox on, SLURM commands outside it, and a hook that blocks `scancel` and `rm -r`	No shell
Jobs	`sbatch` asks	The loop script submits one job at a time, with fixed resources
Integrity	The round job stops if a tracked file changed	The same, and the loop script checks too

Agentic AI in Research advises fuller autonomy only for well-scoped, low-stakes tasks. This task fits: public data, a fixed evaluator, one GPU job at a time, and fixed budgets. Prohibited activities forbids “unsupervised or insufficiently controlled agentic workflows at scale” that could interact with cluster infrastructure, networks, user data, credentials, or external systems in ways that create security risk. Here, the unattended agent has no shell, no write tools, and no MCP tools, and it reads only the project directory.

Before an unattended run, answered for this example

Question	Answer
Task boundary	Choose the next pruning configuration, and return it as JSON.
Read scope	Only the project directory. The prompt points to `state/history.tsv` and `state/notes.md`.
Write scope	None. The loop script writes all files.
Credentials	None, because the agent step has no shell. Keep secrets out of `cluster.env` and out of commands.
Network	Only the model provider. The agent step has no web tools.
Data policy	CIFAR-10 is public data (Level 1).
Untrusted input	Only project files, and the reasons that earlier agent steps wrote in `state/notes.md`.
Resource scope	One GPU job at a time, at most 10 minutes for each run of the job; 60 GPU minutes; 15 round attempts; 1,000,000 tokens; 6 hours.
Evidence	The `run_meta.json` file of each round, `state/history.tsv`, `state/ledger.tsv`, the JSON result of each agent step, and the loop log.
Stop condition	The stop rules below. SIGTERM stops the loop cleanly.

See Before an unattended run.

On a login node, in the project directory, start the loop job:

source cluster.env
sbatch slurm/loop.sbatch

For each round, the loop script runs an agent step, validates the configuration, submits one round job, and records the result. The agent step is the print-mode call below. It uses the limits of Run an agent unattended in a batch job and the fixed JSON shape of Chain steps with print mode. The loop script runs it for you, with $PROMPT from prompts/agent_step.md:

claude -p "$PROMPT" \
  --permission-mode dontAsk --tools "Read,Grep,Glob" --disallowedTools "mcp__*" \
  --json-schema "$(cat configs/round.schema.json)" \
  --max-turns 8 --max-budget-usd 1.00 --output-format json

Watch. Use tail -f logs/prune-loop_<job_id>.out, state/history.tsv, and state/ledger.tsv. Do not turn on remote control; see Remote control.
Stop. Run scancel <loop_job_id>. The loop script cancels its own round job and writes state/summary.json. Check with squeue --me that the round job is gone.
Finish later. After an early stop, sbatch slurm/loop.sbatch --finish runs only the final steps.
Problems. If the agent behaves unexpectedly, follow Reporting a concern.

28.6. Check the result#

After the last round, the loop script selects the passing round with the fewest weights left. If no round passes, it selects the round with the highest validation accuracy. Then it runs the final steps:

A confirmation round: the same configuration with a different seed for the order and augmentation of the fine-tuning data.
One test evaluation of each of the two models. logs/final_eval.log records each call, and state/test_eval/ holds the two results.
A copy of the selected model in results/state/round_NN_model.pt on lab storage, because scratch files are temporary.
state/summary.json, with the stop reason, the totals, the agent model, and both seeds.
If the token budget allows, a draft report from a read-only agent step, in state/final_report.md.

Then check the result yourself; see Check the output and Watch what it did:

Report. Compare each number in the report with state/summary.json. The report gives the tokens and the cost before its own step, so these two numbers are lower than in the summary. In a test run, the report wrote one wrong interval, then corrected it in the next line.
Records. In an interactive job, run .venv/bin/python -m pruning_example.check_round N, where N is selected.round in the summary. An exit status of 0 means that each check passed.
Seeds. The confirmation round agrees when it also keeps the threshold, and its test accuracy is in the 95% interval of the selected round.
Stop reason. round_limit, plateau, token_budget, gpu_budget, failed_attempts, queue_limit, and time_left are normal stops. integrity and agent_format mean a fault. "terminated": true means an early stop.
Code. Check that the commit in each run_meta.json is the commit that you read.
Cost. Read the tokens, the estimated cost, and the GPU minutes in the summary and in state/ledger.tsv; see Watch cost and context.
Test set. Change nothing after the test evaluation.

A rerun can take a different path. Its numbers are close, not identical, because the jobs can run on different GPU types, and the code does not force deterministic algorithms. Keep the records with the model; see Best practices for trustworthy agentic research and Reproducible Research.

28.7. Results of a test run#

Result	Value
Rounds	Round 1 supervised, rounds 2 to 15 in the loop job. The loop stopped at the round limit. 14 of 15 rounds kept the threshold.
Selected model	Round 15: sparsity 0.9927 with `conv1` kept dense, 83,215 nonzero weights (0.745%)
Size	1.56 MB as a bitmask with FP16 values, against 44.7 MB for the dense model in FP32
Accuracy	Validation 0.9076 and test 0.9045 (95% interval 0.899 to 0.910). Second seed: validation 0.9050 and test 0.9032.
Use of the loop job	323,072 tokens (about $1.11), 17 GPU minutes, 66 minutes

Until round 7, the agent raised the sparsity in each round. Round 8 changed three settings at once and fell below the threshold. From round 9, the agent went back to the recipe of round 7. It then raised the learning rate by 0.01 in each round, from 0.02 to 0.08. All later rounds kept the threshold.

All 15 rounds

Round	Sparsity	Steps	Epochs in total	Learning rate	Validation accuracy	Weights left
1	0.5	1	2	0.01	0.9320	50%
2	0.85	2	4	0.01	0.9356	15%
3	0.95	3	6	0.01	0.9180	5.01%
4	0.97	4	8	0.01	0.9196	3.02%
5	0.98	5	10	0.01	0.9074	2.02%
6	0.985	6	18	0.01	0.9090	1.52%
7	0.988	5	20	0.01	0.9052	1.22%
8	0.989	4	20	0.01	0.8976 (below)	1.12%
9	0.9885	5	20	0.02	0.9126	1.17%
10	0.9895	5	20	0.03	0.9154	1.07%
11	0.991	5	20	0.04	0.9078	0.92%
12	0.9915	5	20	0.05	0.9096	0.87%
13	0.992	5	20	0.06	0.9074	0.82%
14	0.9923	5	20	0.07	0.9100	0.79%
15	0.9927	5	20	0.08	0.9076	0.75%

From round 3, conv1 stayed dense. Round 8 also used a weight decay of 0.0002 in place of 0.0005.

The confirmation round passed exactly at the threshold, and both test values are below 0.905. The draft report flagged both points. Treat the result as a lead to check, not as a finding.

28.8. Adapt the example#

Change. Replace the task: the data, the model, the metric, the configuration schema, the threshold, and the limits.
Keep. Keep the parts that make the run safe: the fixed evaluator, the read-only agent step, the loop script, and the limits outside the model. Also keep the parts that make the result checkable: the run records, the confirmation round, and the one-time test evaluation.
Data. Check that the data level of your data and your results allows the agent to read them. Most unpublished research data is Level 2; see Before you start.
Rounds. Decide what the agent may change in each round, as in Experiment Management with Agents.

To extend the code with the agent, use the two skills from Coding Agent Skills in an interactive session. The project settings deny edits of src/ and tests/. So work on a development branch, and remove those two deny rules from .claude/settings.json on that branch. When the code is done, put the deny rules back, and commit everything. The round job refuses to start while a tracked file has uncommitted changes.

Tests. For a new metric or model, ask: “Use the testing-and-continuous-integration skill: add tests for the new code and run them.” The tests use generated images, so they need no GPU.
Run records. For a new dataset or a new output, ask: “Use the reproducible-research skill: check that each run still records its seed, commit, and data version.”
Checks. The check-round skill is specific to this project. To write your own skill, see Building Custom Tools and MCP Servers.