All chapters
  1. Start hereOne change, from diff to production
  2. Before stage 1The team setup: what changes when you are not alone
  3. Stage 1Code change: catch it while the agent is still editing
  4. Stage 2Commit: what must be true before a commit exists
  5. Stage 3Pull request: arrive already reviewed, with evidence
  6. Stage 4Review: who reviews what, at hundreds a day
  7. Stage 5CI/CD: spend scarce capacity like a budget
  8. Deep diveCI/CD agents: where they live and how they run
  9. Deep diveBuild the agents: pinpoint the change, then fix it
  10. Stage 6When the change is an agent
  11. OperateVolume and cost: what breaks at 300 a day
  12. AdoptAdopt it in steps
  13. ReferenceHook events: which one, with a script for each
  14. ReferenceGlossary
  15. AboutAbout this site
Stage 6

When the change is an agent

A skill file, a prompt or a tool definition is also a change. How to review text like code, and how to build an eval that is fast, parallel and still honest.

Acme's CI triage agent has been right about red builds for weeks. When a run on the main branch fails, it reads the logs, looks at the changes since the last good build, and names the one that most likely broke it. People have started to trust it.

Today a developer opens a pull request that adds one sentence to one of the agent's instruction files. The sentence is sensible. The diff is three lines long. A reviewer reads it and has nothing to object to.

And yet nobody on the team can say whether the agent is now better or worse. The effect of that sentence is not in the diff. It is in what the agent will do on a failure nobody has seen yet. This chapter is about how a team finds out, before the change merges, and does it fast enough to use on every change. The numbers in it are illustrative: they show the method, not a measurement.

An edit to an agent is a change to behavior, and behavior cannot be read from a diff. It has to be measured. The hard part is not measuring; it is measuring fast, cheaply and honestly enough to do it on every change.

Act 1 · Understand#

1. What the triage agent does on an ordinary day#

Before asking what a change can break, it helps to see what the agent does when nothing has changed.

At 11:46 the run called e2e-full fails as job 4411. The failing test waits for a confirmation email that never arrives. The triage agent starts. It asks for the log of the failing job and reads a timeout. A timeout says something went wrong somewhere else, so it asks for the log of the email service in the same run, and finds the real error: a null value where a string was expected. It asks which changes were merged since the last good build, sees seven, and notices that one of them, pull request #4812, added a field to the order event. It asks for that one diff, sees the new field can be null, and checks the bug history for this test. Nothing. It answers: culprit #4812, confidence 0.82, with the log lines it relied on. The full design of this agent, and its input and output, are on Build the agents.

Several different things worked together in that paragraph, and a change to any one of them is a different kind of change. It is worth naming them precisely, because the rest of the chapter depends on the distinction.

PartWhat it isIn the triage agentA change to it looks like
ModelThe language model that reads text and writes text, including requests to call a toolThe model named in the run scriptOne line in a script: a new model version
AgentThe program that runs the model in a loop: send the context, receive a request, run the tool, return the result, repeat until an answerThe headless run started by the CI workflowA new cap, a new permission, a new start command
Instructions and skillsText the agent loads to guide the model. A skill is a file of instructions for one kind of task.The system prompt, and the skills read-build-log and search-bug-historyAn edited sentence. The case on this page.
ToolOne function the agent may call, with a name, a description and an input shapeget_build_log, list_changes, search_bugsA different default, a renamed field, a longer answer
Tool serverA separate program that offers tools to the agent, here through the Model Context Protocol (MCP), the common standard for thisThe triage tool server, which wraps the CI system, git and the bug trackerA new tool, a changed endpoint

None of these changes is visible in the agent's answers until the agent runs. That is the root of the problem.

2. Why a reviewer cannot judge the change from the diff#

Code has a property that agent instructions do not. A reviewer of code can usually trace what a line does: this branch runs when the input is empty, this call returns that type. An instruction to a model does not work that way. The model reads it together with everything else in its context, and how much weight it gets depends on wording, position and the case in front of it. The same is true, in a different way, of a tool that now returns 100 lines instead of 200, or a model that is one version newer. Each one changes behavior on some cases and not on others, and nothing in the diff says which.

The natural answer is the one used for code: keep a set of cases with known right answers, and run the changed agent on them. For the triage agent, a case is a past red build whose culprit was confirmed afterward, by re-runs and by the revert that fixed main. Run the agent on the case and compare its answer with the confirmed culprit.

That collection of cases is called a golden set, and using it to check that a change did not make the agent worse is called a regression eval (evaluation). The rest of this chapter assumes Acme has one: 60 past red builds with a confirmed culprit.

A useful golden set is not 60 copies of the easy case. It is chosen so that each kind of mistake the agent could make has a case that catches it.

Twelve of the 60 casesWhat failed, the right answer, and the mistake each case is there to catch

One case file is shown in full on Build the agents.

CaseWhat failedThe right answerWhy it is in the setTags
case-4411A confirmation email never arrives#4812The cause is in a second service's logread-build-log, second-service-log
case-3644A checkout test times out#3630The real error is 80 lines after the first errorread-build-log, long-log
case-3990A type error in the cart service#3987The plain case. A sentinel: it has caught two regressions before.read-build-log
case-4127A report job returns wrong totals#4101, the older of two changes to the same fileCatches "blame the newest change"list-changes, two-candidates
case-4302A sign-in test fails now and thenUnknown: an open flaky-test record existsCatches blaming a change for a flaky testsearch-bug-history, unknown
case-4051Every job fails at set-upUnknown: the package mirror was downCatches blaming a change for an outageread-build-log, unknown
case-3902An error inside a library#3899, a lock file changeThe diff holds no product codelist-changes, dependency
case-3655A new test fails#3651, which added the testThe defect may be older than the changelist-changes, new-test
case-4688Two changes fail only together#4680 with #4671The answer has two changestwo-candidates, together
case-4215A migration fails on seeded data#4209The error is in a database logsecond-service-log
case-4530A configuration value was renamed#4522No code near the failure changedlist-changes, config
case-4399A slow query times outUnknown: no change touches the areaThe list of suspects is wrongunknown

Three of the twelve have the answer "unknown". A set without them rewards an agent that always blames somebody. The tags say which skill or tool each case exercises; they matter in Act 3.

3. The same agent does not give the same answer twice#

One more fact has to be in place before any test design makes sense. Run the unchanged agent on the same case twice, and it may take a different path: read the change list before the log, ask for a different range, call the bug search earlier or not at all. Usually it reaches the same answer. Sometimes it does not.

This happens because the model chooses each next step from probabilities, and a small difference early in a run changes everything after it. Lowering the model's temperature makes its choices less random. It does not make the agent deterministic. An agent's context is built from tool answers as well as the prompt, and those can arrive in a different order or differ in detail between runs. And unless a provider explicitly guarantees identical output for identical input, a sampling setting should not be relied on for repeatability at all. So treat the agent as variable, and measure how variable.

Acme does that first. It runs the unchanged agent on all 60 cases three times.

Result over three runs of the same versionCasesWhat it means
Passed three times49Stable pass, in these three runs
Failed three times6Stable fail, in these three runs
Mixed5The agent did not change, and the result did

Three different statements could be made from this table, and only two of them are supported.

What the table does provide is context. Five cases changed result on an agent that did not change, so a change in the total number of passing cases, by itself, is a weak signal: some movement in the total is expected even when nothing changed. This baseline of observed instability is often called the noise floor. It is not a threshold. It does not mean that a change affecting fewer than five cases is unimportant. Two cases can matter a great deal, for example when both are cases the agent must never get wrong, or when both fail in every run.

So the rest of this chapter keeps three levels apart, and uses each for a different decision.

LevelWhat it isWhat it decides
Observed instabilityCases whose result varied across identical runs of the same version, like the five aboveWhich cases need more runs before anyone reads much into them
Investigation triggerA rule the team sets in advance. At Acme: any case that passed in every old run and failed in every new run; any change at all on a sentinel or high-risk case; any change in the "unknown" cases.Stop the change and look. It does not say the change is bad, only that it must be explained.
Statistically supported conclusionA claim that the new version is worse or better overall, with a stated method: enough paired runs per case, and a test chosen before the runs, such as a paired comparison of per-case resultsAccept or reject a change on the evidence, rather than on a reading of the numbers

Most day-to-day decisions use the second level. The third is worth the cost for changes that can move every case, such as a new model.

Act 2 · Explore#

The team now knows what it needs: run the changed agent on the golden set, compare with the confirmed answers, and do it before the change merges. Each obvious way to do that runs into a wall. The walls are worth understanding one at a time, because the design in Act 3 is built from their causes.

4. The first idea: run every case, one after another#

The simplest test is faithful to production: start the real agent against the real tools, give it the first case, wait for the answer, then the next. Nothing is simulated, so a pass means something.

The trouble is time. One investigation takes about 12 minutes, because the agent waits on slow internal services and on the model. Ten cases take two hours. Sixty take twelve. A developer who changed one sentence will not wait two hours to learn whether it was safe, so in practice the test stops being run, or is run only on a handful of cases.

Ten cases, one after another (illustrative)
One case
12 min
Ten cases, in a row
120 minutes before the first answer about the change

The time comes from waiting, not from work. Most of each run is the agent idle while a log service or the model answers. Waiting can overlap. That leads to the second idea.

5. The second idea: run them all at once#

If each run is mostly waiting, start all ten at the same time. Ideally the whole set finishes in about the time of one run.

In practice it often does not, and the reason is the tools. Each run makes about seven to twelve tool calls: logs, change lists, diffs, bug searches. Ten runs at once send about a hundred requests to the same internal services within minutes, where they normally see a few. Those services were sized for people, not for a burst. They slow down, then start refusing requests. If the agent or its tool client retries a refused call, each retry adds another request to a service that is already overloaded. Runs take longer than they did alone, and some time out. Now the eval is slow and flaky, and a failure no longer says whether the agent is wrong or the log service was busy.

Ten cases at once, against shared tools (illustrative)
What we hoped
about 12 min
What happens
longer: throttling, retries, some runs time out

This does not happen to every team. A tool service with spare capacity and rate limits that fail fast can absorb the burst. But the failure has a clear cause: many agents sharing endpoints that have no limit and no cache. Act 3 removes that cause instead of hoping for headroom.

6. The third idea: record the tool answers and replay them#

If the live tools are the problem, take them out. Run each case once against the real tools and record every request and its answer. From then on, a test run gets its answers from the recording. It is fast, it touches no real service, and it can run in parallel.

It breaks the first time the edited agent asks a question the recording does not hold. A recording is a list of the questions one run happened to ask, in the order it asked them. Section 3 showed that the agent does not ask the same questions every time, and an edit to its instructions is designed to change what it asks. Below, the recorded run went straight from the test log to the change list. The edited agent, told to look harder at timeouts, asks for the email service's log and for the bug history. The recording has no answer for either.

The recording knows the questions one run asked

Recorded run

get_build_log(4411)recorded
list_changes(a7be113, c41d7e0)recorded
diff of #4812recorded

Edited agent, replayed

get_build_log(4411)hit
get_build_log(4411, service=notifications, search=Error)no recording
search_bugs(confirmation_email)no recording

The question is reasonable and the data exists. The recording simply never saw it.

A common next step is to make the agent repeat itself: fix the random seed and set the temperature to zero, so that the replayed run asks the same questions. Replay works more often. But section 3 already said why this cannot be relied on: temperature zero lowers randomness and does not guarantee identical runs. And it tests a configuration that is not the one in production, while the edit under test is meant to change the questions anyway.

7. The fourth idea: run a few cases first#

The last obvious idea accepts that the full set is expensive and puts a cheap filter in front of it. Run three cases. Read the agent's trace, the record of every request it made and every answer it received, by eye or with a second model as a judge. If both look fine, run the full set. This is the best of the four. It catches gross mistakes in minutes and saves the cost of the full set on changes that are plainly broken.

It still has a known failure: sometimes the smoke test passes and the full set fails. The arithmetic shows why that is not bad luck. Suppose an edit harms 6 of the 60 cases. Three cases picked without looking at the edit all miss those six with probability (54 x 53 x 52) / (60 x 59 x 58), which is about 72%. And even a harmed case can pass once, because section 3 showed results vary between runs. A passing smoke test of three fixed cases, each run once, says very little about an edit.

IdeaWhy it seems rightWhat breaks itThe cause underneath
One after anotherFaithful to productionTwo hours for ten casesMost of the time is waiting on slow services
All at onceWaiting can overlapShared tools throttle; retries add load; runs get slower and flakyMany agents share endpoints with no limit and no cache
Record and replayFast, no load, parallelThe edited agent asks questions the recording never sawA recording stores answers to one run's questions, not the data behind them
Make it repeat itselfReplay works more oftenNot guaranteed; tests a non-production settingLower randomness is not determinism
A few cases firstCheap early filterSmoke passes, full set failsFixed cases miss what the edit touched; one run per case is one sample

Read down the last column. Each failure has a cause that can be designed away, and that is what Act 3 does.

Act 3 · Engineer#

The causes from Act 2 point to the design. Remove the dependence on one run's questions. Put limits where many runs meet. Choose test cases from the change. Run each case more than once. Grade with code wherever code can decide. Then wire it into the pipeline so that it runs on every change to the agent.

8. Freeze the world, not the path#

Replay failed because it stored the answers to the questions one run asked. Store instead the data those questions were asked of. For each golden case, export a snapshot of the world as it was when the failure happened: the build logs of that run, the list of changes, the diffs, the bug records. Then run a small stand-in tool server that offers the same tools, with the same names and inputs, and computes each answer from the snapshot.

Because the answer is computed, the question does not have to be one that was asked before. The agent can read the change list first or the log first, search for any word, ask for any service's log. As long as the data was exported, there is an answer. If it was not exported, the stand-in says so clearly, and the run is marked incomplete rather than wrong.

The edited agent's questions, answered from the frozen snapshot

Edited agent asks

get_build_log(4411)
get_build_log(4411, service=notifications, search=Error)
search_bugs(confirmation_email)
get_build_log(4411, service=payments)

Stand-in answers

the timeout line, from the test logcomputed
the null-value error, from the email logcomputed
no matching bug recordscomputed
not in snapshotmarked incomplete
The stand-in, in codeThree tools answered from one frozen case, including questions no run asked before

What this code is meant to show. It is a simplified demonstration of one idea: a stand-in that computes its answers from frozen data can answer questions that no recorded run ever asked. If it works, the output below shows answers to three new questions, and an explicit refusal for data that was never exported. It does not show that a production stand-in is easy, or that it matches the real tools.

Each tool reads the exported files and computes its answer, so a search word or a service name that no earlier run used still works. Data that was never exported returns an explicit error instead of an empty answer, because an empty answer would look like evidence.

#!/usr/bin/env python3
"""agents/triage/evals/standin.py — the stand-in tool server's three tools, answered from one frozen case.
Simplified demonstration: it ignores some arguments (see below). A real server must honor every argument
and wraps these functions in the tool protocol the agent already uses."""
import json, pathlib

class FrozenCase:
    def __init__(self, case_dir):
        self.dir = pathlib.Path(case_dir)
        self.changes = json.loads((self.dir / "changes.json").read_text())   # the changes in the build, as exported
        self.bugs = json.loads((self.dir / "bugs.json").read_text())         # bug records as they were that day

    def get_build_log(self, job_id, service="test-runner", search=None, window=200):
        path = self.dir / "logs" / f"{service}.log"
        if not path.exists():                       # data that was never exported: say so, never invent it
            return {"error": "not in snapshot", "service": service}
        lines = path.read_text().splitlines()
        if search:                                  # any search word works: the answer is computed, not recorded
            return {"lines": [l for l in lines if search in l]}
        first = next((i for i, l in enumerate(lines) if "Error" in l), 0)
        return {"lines": lines[max(0, first - window // 2): first + window // 2]}

    def list_changes(self, good_sha, bad_sha):
        return {"changes": [{k: c[k] for k in ("pr", "title", "files")} for c in self.changes]}

    def search_bugs(self, query):
        return {"bugs": [b for b in self.bugs if query.lower() in (b["title"] + " " + b["text"]).lower()]}

Run against a small sample snapshot of case 4411 (three of its seven changes), it answers the two questions the recorded run asked, and the three it never asked:

recorded   11:48:02 TimeoutError: no confirmation email after 120 s
recorded   [4805, 4812, 4816]
new        ["11:46:03 AttributeError: 'NoneType' object has no attribute 'strip' (email/confirmation.py, line 61)"]
new        {'bugs': []}
new        {'error': 'not in snapshot', 'service': 'payments'}

The last line is the important one. The agent asked for a service whose log was never exported. The stand-in does not guess. The grader marks that run incomplete and the exporter learns that payments logs are sometimes needed.

What this demonstration leaves out. To stay short, it ignores some of its own arguments: list_changes returns every exported change whatever commit range it is given, and get_build_log does not use job_id to choose a job. A production stand-in must honor every argument the real tool accepts, and reproduce the behavior the agent can observe: filtering by range, ordering, page sizes and pagination, truncation, error responses and their codes, and permission boundaries, so that a question the real tool would refuse is also refused. Contract tests against the real tools, described below, are how a team keeps it honest.

Diagram: Freeze the world, not the path · how the eval runs without production

Read left to right. A snapshot holds the data of one past failure. A stand-in server computes tool answers from it. The agent under test may take a different path each run; the data stays fixed.

  1. Once per golden case
    1. A real past failure: A red build whose culprit is confirmed.
    2. Snapshot exporter: Copies the build log, the changes in that build and the bug records of the time.
    3. Frozen snapshot: Read-only data for one case.
  2. Every eval run
    1. Stand-in tool server: The same read-only tool names and arguments as production. Answers by querying the snapshot. A query outside the snapshot marks the run incomplete.
    2. Agent under test: The candidate triage agent, unmodified, connected to the stand-in instead of production.
    3. Answer + trace: The culprit, the confidence score and every tool call.
  3. Grading
    1. Outcome: Does the culprit match the label? Code decides.
    2. Path rules: Do the log lines it cites exist and contain the error? Did it stay under the call limit? Code decides.
    3. Reasoning: Was the conclusion supported by what it read? A judge model, measured against people first.

The properties that made Act 2 fail are now gone, and it is worth saying exactly which.

PropertyWhy it follows
Different paths workAnswers are computed from data, so a question no earlier run asked still has an answer, if the data was exported
Parallel is safeNo production service is touched. Ten runs or a hundred.
FasterLocal reads replace slow internal services. This removes tool waiting only. Model time, rate limits and repeat runs remain, so measure the new duration instead of assuming it.
ComparableThe facts cannot drift between runs. What remains is the agent's own variation, which section 3 measured.

This is not free. A stand-in is a second implementation of the tools, and it has to be kept faithful.

CostHow to contain it
Building the exporter is real workStart with the three most-used tools. Unsupported tools return a clear "not in snapshot" error, and the run is marked incomplete, not failed.
Snapshots age when a tool's schema changesVersion each snapshot against the tool schema. For small changes, an adapter in the stand-in maps old data to the new shape. Expire old cases on a schedule, for example after a quarter, so the set stays small enough to re-export when a change is large.
The stand-in can differ from the real tool in ordering, truncation, paging and error behaviorContract tests that send the same requests to both and compare the response shapes. A weekly run of the same cases against the live tools measures the remaining gap.
The agent gains a new toolOld snapshots hold no data for it, so old cases become incomplete. Backfill the export where history exists. Where it does not, run those cases without the new tool and report them as a separate group. Never return an invented empty answer.
Tools that write, or that depend on the current timeKeep them out of the snapshot world. Evaluate them separately with a fake that records the intended action.

A pass against the stand-in is evidence about the agent working on frozen data. The weekly live run is what keeps that evidence connected to production.

9. When a snapshot exporter is too much for now#

Some teams cannot build the exporter yet. A cheaper first step keeps most of the benefit of replay without its main flaw. Cache each read-only tool answer under a key built from the case, the tool and its schema version, and the normalized arguments, not from the position in the call order. On a hit, return the cached answer. On a miss, call the live tool, record the answer, and count the miss.

This is hybrid replay. A run with a miss has mixed old answers with today's data, so it is marked not comparable for the merge gate, and the miss is used to extend the cache. Report the miss rate per tool. A rising rate after a change shows the agent is asking different questions. A low rate does not show the answers are right.

10. When runs must share live tools: limits, not hope#

Some evals have to hit live tools: the weekly faithfulness run, or a team that has no stand-in yet. Section 5's failure came from many agents sending uncached reads to endpoints with no limit. Three controls remove that cause.

CostHow to contain it
A shared cache can leak across permissionsThe key includes who is asking and the data version. Cache only answers that are the same for every caller in the eval.
Limits move the wait into a queueMeasure queue time and error rate. The eval is only faster if both drop.

11. Choose the smoke cases from the change, and run each more than once#

Section 7's smoke test failed for two reasons, and each has a fix.

The first reason was that the three cases were fixed. They tested the same three behaviors whatever the edit touched. Instead, tag each golden case with the skills and tools it exercises, as in the table in section 2. A change to the skill read-build-log then runs the cases tagged with it, a few sentinel cases that have caught regressions before, and a seeded random sample from the rest. The smoke set looks where the edit can do harm. It is still an early filter, not the gate: a small sample can miss a rare side effect, and the full set runs before merge.

The second reason was that each case ran once. Run each smoke case several times on the old version and on the new one, and compare case by case. A case that passed every time before and fails every time after is a much stronger signal than one changed total. Three runs is a starting point, not a proof: with three runs a case can only show 0, 1, 2 or 3 passes. Choose the number of runs, and how many flips count as a failure, from the variation measured in section 3.

The comparison is done by a short script. What it is meant to show: that comparing old and new case by case reveals what a single total hides. It reads one line per run, classifies each case as passing every run, failing every run, or mixed, for each version, and prints only the cases whose class changed. When it runs cleanly, its output lists exactly the cases that need a person's attention; its exit code fails the check when any case moved from passing every run to failing every run. It has been run on sample data.

#!/usr/bin/env python3
"""agents/triage/evals/flips.py  RESULTS.jsonl  ->  which cases changed, old version against new.
Each line: {"case": "case-4411", "version": "old" or "new", "run": 1, "passed": true}"""
import collections, json, sys

passes = collections.defaultdict(lambda: {"old": [], "new": []})
for line in open(sys.argv[1]):
    if line.strip():
        r = json.loads(line)
        passes[r["case"]][r["version"]].append(r["passed"])

def stable(results):                      # "pass", "fail" or "unstable"
    share = sum(results) / len(results)
    return "pass" if share == 1 else "fail" if share == 0 else "unstable"

groups = collections.defaultdict(list)
for case, p in sorted(passes.items()):
    groups[(stable(p["old"]), stable(p["new"]))].append(case)

regressions = groups[("pass", "fail")]
for (old, new), cases in sorted(groups.items()):
    if old != new:
        print(f"{old:>8} -> {new:<8} {len(cases):2d}  {' '.join(cases)}")
print(f"unchanged: {sum(len(c) for (o, n), c in groups.items() if o == n)}")
print(f"regressions (pass -> fail): {len(regressions)}")
sys.exit(1 if regressions else 0)

Its exit code makes it usable as a check: a case that moved from stable pass to stable fail fails the step. The words "stable" and "unstable" here mean stable within the runs made, nothing more.

12. Grade the answer, the path and the reasoning#

A run can reach the right culprit for the wrong reason, or reason well and land on the wrong change. So grading has three layers, and code does as much of it as code can.

LayerWhat it checksDone by
OutcomeThe culprit matches the confirmed answerCode
Path rulesTool calls stayed under the limit. No call to a tool outside the agent's list. The log lines the agent cites exist in what the tools returned, and contain the error it names.Code, over the trace
Reasoning qualityWas the conclusion supported by what it read?A judge model, for this layer only

A path rule such as "it read the log" is weak alone, because an agent can read a log and ignore it. Checking that each cited line really came back from a tool is much stronger, and it is still code. The grader that does this is shown on Build the agents.

The third layer needs judgment, and a judge model, a second model that reads a run and grades it, is the usual tool. It is a measuring instrument, so it has to be measured before anyone relies on it.

Way to use a judgeWhat it buysWhat goes wrong
One judge, a score from 1 to 5Easy to set upScores drift, and most runs get a 4. A score has no fixed meaning.
One judge, yes or no against a short rubricAnswers can be compared with people's answersThe rubric must be narrow: "does the reason follow from the quoted lines?"
Compare two runs, blindJudges are better at "which of these two is better supported" than at absolute scores. Fits old against new.Two runs for each judgment. The order must be shuffled.
A judge from a different model family than the agentIt does not share the agent's blind spots, and does not favor its own styleA second provider to pay and to keep available
Several judges from different families, majority decidesOne judge's bias is outvoted. Disagreement between judges is itself a signal: send those runs to a person.Three times the cost. Use it on the changed cases only, not on all 60.
Rule for every judgeWhy
Measure it before it may block anythingGive it 30 runs that people graded, including ones people disagreed on. Until its agreement with people is known, it advises. Measure again when the judge changes.
It reads the changed cases, not the whole setThe flip table already says which cases matter. That keeps the cost small.
It never overrules codeWhen the culprit is wrong, the run failed, however good the reasoning reads

13. Is the confidence still honest?#

The agent outputs a confidence score, and later steps may act on it, for example by opening a revert pull request. So a change to the agent must not only keep its accuracy; it must keep the score meaningful. The property is called calibration: of the answers where the agent said about 0.9, were about 90% right? A change can keep accuracy and break calibration, typically by making the agent more confident when it is wrong. How the score is produced and how the calibration table is built are on Build the agents.

14. Shadow mode keeps the golden set growing#

Shadow mode only makes sense once four moments in an agent change's life are kept apart, because they are easy to blur.

MomentWhat happensWhat it changesWhat gates it
1. MergeThe pull request with the new instructions, tool or model setting enters the main branchThe code and files in the repository. Nothing acts differently yet.The evals in section 15: smoke on the pull request, the full golden set in the merge queue
2. Deploy as a candidateThe new version is installed beside the current one, without authorityA second version exists and can be runThe merge
3. ShadowThe candidate runs on live red builds in parallel with the current version. Its claims are stored, never acted on.Evidence on today's failuresDeployment as a candidate
4. ActivateA setting makes the candidate the version whose claims start the verifier and the revert flowProduction authorityPromotion criteria set in advance, met in shadow

This sequence needs one thing from the architecture: the version that has authority is chosen by a setting, separately from which version is merged. At Acme the triage workflow reads the active version from .github/agent-versions.yml, a gate file owned by the platform team. A team whose agent takes effect the moment it merges cannot run shadow mode after merging. It must run its shadow evaluation before the merge instead, and accept that the merge is also the activation.

A golden set ages. The failures of six months ago are not today's failures. Shadow mode runs the candidate agent beside the current one on live red builds, with no authority to act. For a triage agent, evidence about the real culprit arrives later: the controlled re-runs, the revert, the incident review. A case enters the labelled set only when that evidence is firm. Cases with two causes, or a flaky test, go into a separate ambiguous set: useful to read, not used as pass or fail.

CostHow to contain it
Model calls and tool load while it runsShadow a sample of failures, not all
Only investigated failures get a labelThe set drifts toward the failures people cared about. Track the share of shadow cases that never get a label.
Labels arrive lateIt is a nightly signal, not a gate on a pull request
Not every agent has a later ground truthThen a person labels a weekly sample

15. Putting it in the pipeline#

Each piece above answers one failure from Act 2. In the pipeline they become tiers: cheap checks run on every push, the expensive ones only when they can see something new.

Before any eval runs, a change to a skill file goes through checks that need no model at all, because a skill is text with a structure and rules.

LayerWhat it checksWhoFails the pull request?
1. LintHeader present. The description says when to use the skill. Length within budget. Links resolve. No secrets or personal data. Every tool it names exists and is allowed.A script in CIYes
1b. ReachWhat the skill pulls in: helper scripts, referenced files, remote resources. Are remote resources pinned? Did the effective tool permissions change?A script, plus a person for anything newYes
2. Overlap checkThe description is not a near-copy of another skill's descriptionA script, or a small model, comparing all descriptionsWarn; a person decides
3. Trigger testDoes the agent pick this skill when it should, and leave it alone when it should not?An eval runYes
4. Outcome evalWith the skill in use, does the task still succeed?The tiers belowYes
5. Human reviewIs the instruction right? Does the evidence support it?A person, red laneApproval

The trigger test needs a word of explanation, because it tests something specific to how skills load. In Claude Code, only a skill's name and description are in the model's context at the start of a run; the body loads when the model decides to use the skill, and it decides from the description. So a skill can be well written and never run, because its description loses to another one. Other agent frameworks load skills differently; check yours. The test is a table of prompts with the skills allowed and forbidden for each, run with the whole skill set loaded, as in production.

Prompt in the test tableExpected skillWhy it is in the table
"Why did job 4411 fail?"read-build-logThe plain case
"Is job 4411's failure a known bug?"search-bug-history; read-build-log is allowed after itThe nearest neighbor must come first
"Summarize yesterday's end-to-end environment usage"none of the triage skillsOut of scope; must not fire

Real tasks often use more than one skill, in an order that can vary, so each row states an allowed set and a forbidden set, and an order only where the order matters. The result is precision and recall for each skill, and a matrix of which skill was picked when another was expected. When two skills overlap, the matrix shows it, and the fix is usually in the two descriptions. Some skills are picked because of what happened earlier in a session, such as a recovery skill after a failed tool call; those rows start from a short scripted history. And a skill configured to load only on explicit request has no description in the context at all, so it will never be picked by itself; run the test with the production loading configuration.

The person who reviews the pull request gets a behavior diff beside the text diff: which eval cases changed result, and which trigger-test rows changed. A text change with no behavior change is either harmless or untested, and the reviewer asks which.

Diagram: A change to an agent · five tiers, cheapest first

Read left to right. The first two columns block in seconds and catch most mistakes. The eval tiers run on frozen snapshots, so they are fast and parallel. The last column does not block; it keeps the others honest.

  1. Text checks
    1. Skill lint + reach: Frontmatter, when-to-use description, length, links, no secrets, tools exist. Plus what the skill pulls in: scripts, files, remote resources. The skill-lint check.
    2. Overlap check: Compares the description with every other skill's description.
  2. Selection
    1. Trigger test: With the whole skill set loaded: does the agent pick an allowed skill, and none of the forbidden ones? The trigger-test check.
  3. Smoke
    1. Cases chosen from the diff: Cases tagged with the changed skill, fixed sentinel cases, and a seeded random few. The agent-smoke check, on the pull request.
    2. Several runs, base and candidate: A pass rate per case on both versions, not one lucky run.
  4. Full set
    1. Golden set, in parallel: Every case, on frozen snapshots, at once. The agent-eval check, in the merge queue.
    2. Three graders: Outcome by code, path rules and cited evidence by code, reasoning by a judge model that advises until it is measured.
    3. Behaviour diff: Which cases changed result. The human reviewer reads this next to the text.
  5. After merge
    1. Shadow mode: Deployed as a candidate beside the active version, it runs on live red builds with no authority. Activation is a switch in .github/agent-versions.yml.
    2. Weekly live run: The same cases against real tools, in sequence. Measures the stand-in gap.
TierRuns whenCostBlocks the merge?
Lint and overlapEvery pushSecondsYes
Trigger testA skill description or the set of skills changedA minuteYes
Smoke: cases chosen from the change, several runs each on old and new, on frozen snapshotsEvery pull request that touches the agentMinutesYes
Full golden set on frozen snapshots, in parallelBefore merge, in the merge queueMeasure it; tool waiting is gone, model time is notYes
Live run, one case after anotherWeeklyHoursNo. It measures how far the stand-in has drifted.
Shadow modeAfter merge and deployment as a candidate, before activation; and continuously on a sampleOngoingIt does not block the merge. It gates activation, and it grows the golden set.

Notice that the first idea from Act 2, the slow sequential run against live tools, is still here. It moved from the merge gate to a weekly job, where its faithfulness is worth its cost.

16. Four changes, worked start to finish#

The design is easier to trust once it has been seen working. The four changes below are the four kinds from section 1: what a tool returns, an instruction file, the model, and the second agent that fixes builds. Each kind has a cheapest test that can catch it, and the habit worth building is to run that one first. The results shown are illustrative, not measured.

The change is inCheapest test that can catch a regressionNeeds the model?Then
What a tool returnsDoes the tool still return the evidence each case needs?NoRun the cases that lost evidence first; the full set still runs before merge
An instruction or skill fileTests of the one sub-task the text governs, then a smoke set picked from the changeYes, smallThe full set before merge
The modelNothing cheap sees it. Run the full set, old model against new, several times.Yes, largeMerge, deploy as a candidate, shadow, then activate
The red-build fix agentSix checks by code on past fixesYes, for the run; no, for the gradingA weekly sample read by people

16.1 Change 1: what a tool returns#

The change. To save tokens, the get_build_log tool will return 100 lines around the first error by default, not 200. The agent's instructions, skills and model stay the same.

  agents/triage/tools/get_build_log.py
-     DEFAULT_WINDOW = 200
+     DEFAULT_WINDOW = 100

The test that fits. One question needs no agent run at all: does the tool's default answer still contain the line each case needs? Each case file lists those lines, so a script can check all 60 against the frozen logs in seconds. What the script is meant to show: which cases can no longer see their evidence with the new default. A clean run proves only that the required lines are still returned. It does not prove the agent behaves the same, because the agent now sees less context around them. It has been run on sample data.

#!/usr/bin/env python3
"""agents/triage/evals/reach.py  WINDOW  CASES_DIR  ->  which cases lose their evidence.
No model. For each golden case: does the tool's default answer still contain the lines
that a right answer must cite?"""
import json, pathlib, sys

window, cases_dir = int(sys.argv[1]), pathlib.Path(sys.argv[2])

def default_answer(log_lines, n):
    """What get_build_log returns with no range: n lines around the first error."""
    first = next((i for i, line in enumerate(log_lines) if "Error" in line), 0)
    return log_lines[max(0, first - n // 2): first + n // 2]

lost = []
for case_file in sorted(cases_dir.glob("case-*/case.json")):
    case = json.loads(case_file.read_text())
    for service, needed in case["must_cite_in_log"].items():
        log = (case_file.parent / "logs" / f"{service}.log").read_text().splitlines()
        answer = "\n".join(default_answer(log, window))
        if not all(text in answer for text in needed):
            lost.append((case["case"], service))

print(f"window={window}  cases={len(list(cases_dir.glob('case-*')))}  lose evidence={len(lost)}")
for case, service in lost:
    print(f"  {case}: the required line is outside the default answer for '{service}'")
sys.exit(1 if lost else 0)

The result.

window=100  cases=60  lose evidence=4
  case-3644: the required line is outside the default answer for 'checkout'
  case-4215: the required line is outside the default answer for 'database'
  case-4127: the required line is outside the default answer for 'reports'
  case-3718: the required line is outside the default answer for 'orders'
StepWhat ranResult
1The script above, 60 cases, no model, 4 seconds56 cases still receive their required lines. That makes them less likely to be affected; it does not show their behavior is unchanged, because the agent now sees less context. 4 cases lose a required line.
2The agent, on those 4 cases only, three runs each, old tool and newIn 3 cases the agent asked for a second range and still found the line. case-3644 failed in all three runs: it answered from the first 100 lines.
3DecisionThe change is not merged as it is. The tool now adds one line to a shortened answer, saying that more lines exist and which range to ask for. Steps 1 and 2 are run again: all pass. The full set then runs before merge.
What this approach buysWhat goes wrong
56 of 60 cases are screened with no model callEvery case must list its required lines. Someone writes them.
The 4 agent runs go where the risk isIt only sees evidence that became unreachable. A tool change that adds confusing text is not caught here; the full set before merge covers that.
The reason for the failure is known before the agent runsIt needs the frozen logs. Without them there is nothing to check against.

16.2 Change 2: an instruction in a skill file#

The change. Pull request #4827 gives the skill read-build-log one new rule, meant to help the agent choose when two changes touch the same file.

  agents/triage/skills/read-build-log/SKILL.md
   2. Find the first line that is an error of the program, not of the test runner.
+     When two changes touch the file named in the error, suspect the most recent one first.

The rule sounds reasonable. It is also exactly the mistake case-4127 exists to catch.

StepWhat runsCasesTimeResult
1. Tests of the one sub-taskOne model call each: here is a log excerpt and a list of changes; which change do you suspect? No tools, no loop.40Under a minute38 pass. 2 fail: both have an older culprit.
2. Smoke set chosen from the changeThe full agent on frozen data, three runs each, old skill and new8: the cases tagged read-build-log and two-candidates, the sentinel case-3990, and two picked at random6 minutes, in parallelSee below
3. The full setOnly if step 2 passes60Not run

The flip script from section 11 prints only what changed:

    pass -> fail      2  case-4127 case-4688
unstable -> pass      1  case-3644
unchanged: 5
regressions (pass -> fail): 2

The edit fixed one unstable case and broke two stable ones. The count of passing cases went from 7 to 6, a movement in the total that alone would say little. The flip table shows what the total hides: two cases that passed in all three old runs failed in all three new runs. That meets Acme's investigation trigger from section 3, and one of the two, case-4127, is the case built to catch exactly this mistake. It is not a statistical conclusion about the agent as a whole. It is a concrete, explainable failure, which is enough to stop the change. Judge an edit by the cases that changed, not by the total. Note also that case-4127 was in the smoke set only because the cases were chosen by tag; three fixed cases would very likely have missed it, as section 7 computed.

The decision. The rule is rewritten: "When two changes touch the file, read both diffs before choosing." Steps 1 and 2 pass. The full set then runs before merge.

What this approach buysWhat goes wrong
Step 1 gave a warning in under a minuteA sub-task can pass alone and fail inside the loop. It does not replace steps 2 and 3.
The smoke set looks where the edit can hurtTags must be kept current. A wrong tag hides a case.
The flip table separates a consistent change from chance variationThree runs for each version: six times the runs of a single pass

16.3 Change 3: the model#

The change. The provider releases a new model version. One line changes in the run script. No instruction, skill or tool changes, so there is nothing to choose a smoke set from. Every case may move.

The test that fits. The full set, old model against new, three runs each, on frozen data, in parallel. Then four comparisons, because one number would hide the important one.

ComparisonOld modelNew modelReading
Cases that changed5 fail to pass, 2 pass to fail, 4 between stable and unstableNet better. The two regressions must still be read.
Right "unknown" (10 cases)86Worse. Both regressions are here: the new model names a change for case-4302 and case-4399.
Wrong and sure (wrong above 0.8)25Worse. It is more confident, right or wrong.
Tool calls for each case, median75Better: cheaper and faster

A single accuracy number would say "upgrade": 52 right against 49. The second and third rows say the new model blames changes it should not, and is surer when wrong. For this agent, that is the costly kind of error. With 10 "unknown" cases, a drop from 8 to 6 is a small sample; it justifies holding the switch and looking, not a final verdict.

StepAction
1Do not merge the model change as it is. The new model is better at finding and worse at declining.
2Strengthen the rule for "unknown" in the system prompt. That is an instruction change: run the test of section 16.2 on the new model, then the full set again.
3Merge when the full set no longer shows those regressions, with the old model still active.
4Deploy the new model as a candidate and run it in shadow mode on live failures for two weeks. Rebuild the confidence table from its shadow claims; the old table described the old model.
5Activate the new model when right-unknown and wrong-and-sure in shadow are at least as good as the active model, as set out before the shadow run started.
What this approach buysWhat goes wrong
It sees every case, which a model change requiresIt is the most expensive run: 60 cases, three times, two models
The per-measure comparison finds what a total hidesNeeds enough "unknown" cases to say anything about them. Ten is thin.
Shadow mode checks cases the golden set does not haveTwo weeks is slow. A model that is retired on a date may force the pace.

16.4 Change 4: the agent that fixes#

The triage agent names a change. A second agent, the red-build fix agent, proposes a repair after a claim is verified. It is changed and tested the same way, with a different set of checks. Its design is on Build the agents.

The change. The red-build fix agent's task file gains one permission.

  agents/fix/prompts/task.md
   You may not edit the failing test.
+  You may add or change tests in the tests/contract/ folder.

The test that fits. The eval set is 25 past red builds that a person fixed. The agent gets the verified claim and the code at the culprit commit, so the failing test fails before its repair. Six checks by code grade each attempt.

MeasureBeforeAfterReading
Fix rate (all six checks pass)11 of 15 fixable cases13 of 15Better
Correct hand-over (10 cases only a person may fix)10 of 109 of 10One regression
A new lower-tier test that fails without the fix6 of 1112 of 13The purpose of the change, achieved
Median changed lines1431Larger changes: more for a reviewer to read

The regression: in one case the agent "fixed" a contract mismatch by changing the existing contract test to accept the new behavior. The failing test was untouched, so the old checks passed. The permission was too wide.

The decision. The line becomes "You may add tests in tests/contract/. You may not change an existing one." A seventh check by code is added: no existing test file is modified. All 25 cases run again.

What this approach buysWhat goes wrong
Graded by code: a test passes or it does not"The checks pass" is not "the fix is right". A weekly sample must be read by people.
The hand-over cases catch an agent that fixes what it should leavePast fixes are few. 25 cases is a small set.
Each regression becomes a new checkThe checks only know the mistakes already seen

16.5 What the four have in common#

QuestionAnswer from the four examples
What do I run first?The cheapest test that can see this kind of change. For a tool change, a script with no model.
How do I know a difference is real?Keep three levels apart. Observed instability says which cases need more runs. An investigation trigger, set in advance, says when to stop and look: here, a case that moved from passing every run to failing every run, or any change on a sentinel case. A claim that the new version is worse overall needs enough paired runs and a test chosen in advance.
Why did the smoke test pass and the full set fail?The smoke cases were not chosen from the change, each ran once, or the change was to the model.
What is the one number?There is none. Right culprit, right "unknown" and wrong-and-sure can move in opposite directions.
When is it safe to merge, and when to activate?Merge after the full golden set passes. Activate after shadow mode meets criteria set in advance. For a change that can move every case, such as a model, both steps are needed; for a small instruction change, a team may activate at merge if its evals are strong enough.

17. Side questions#

QuestionShort answer
How big should the golden set be?Big enough that one case is not 10% of the score. Ten cases cannot tell an 80% agent from a 90% one. Shadow mode is how it grows without hand labelling.
Is a judge model reliable?For narrow questions with a rubric, often. It is a measuring tool, so calibrate it like one.
Should prompts and skills live in the code repository?Yes. Then they get lanes, owners, history and the same pipeline as code. Where they live and how a pipeline loads them is on CI/CD agents.
Does the coding agent have tooling for this?Claude Code documents comparing a task with and without a skill, and a report that shows skills that are never invoked. The stand-in tool server, the trigger table and the shadow loop are patterns you build.
What ties the evidence together?A provenance record for each pull request and each eval run: base and head, rules and owners in force, model and tool versions, snapshot versions, each verdict and which system produced it, signed by the server. It is the same idea as the attested review receipt, applied to everything.

The team at the start had a three-line diff that nobody could judge. What it needed was not a faster agent or a bigger test set. It needed a test that could ask the changed agent any question and still get the answers of the day the failure happened, that ran the cases the change could touch, more than once, and that knew how much the agent varies when nothing changes at all. With those three things, the same three-line diff arrives with a short report: which cases moved, in which direction, and whether that is more than the agent's usual variation. That report is something a reviewer can judge.

Next: every model run behind #4812 added up, and what breaks at 300 changes a day: Volume and cost.