Acme's CI triage agent has been right about red builds for weeks. When a run on the main branch fails, it reads the logs, looks at the changes since the last good build, and names the one that most likely broke it. People have started to trust it.
Today a developer opens a pull request that adds one sentence to one of the agent's instruction files. The sentence is sensible. The diff is three lines long. A reviewer reads it and has nothing to object to.
And yet nobody on the team can say whether the agent is now better or worse. The effect of that sentence is not in the diff. It is in what the agent will do on a failure nobody has seen yet. This chapter is about how a team finds out, before the change merges, and does it fast enough to use on every change. The numbers in it are illustrative: they show the method, not a measurement.
An edit to an agent is a change to behavior, and behavior cannot be read from a diff. It has to be measured. The hard part is not measuring; it is measuring fast, cheaply and honestly enough to do it on every change.
Act 1 · Understand#
1. What the triage agent does on an ordinary day#
Before asking what a change can break, it helps to see what the agent does when nothing has changed.
At 11:46 the run called e2e-full fails as job 4411. The failing test waits for a confirmation email that never arrives. The triage agent starts. It asks for the log of the failing job and reads a timeout. A timeout says something went wrong somewhere else, so it asks for the log of the email service in the same run, and finds the real error: a null value where a string was expected. It asks which changes were merged since the last good build, sees seven, and notices that one of them, pull request #4812, added a field to the order event. It asks for that one diff, sees the new field can be null, and checks the bug history for this test. Nothing. It answers: culprit #4812, confidence 0.82, with the log lines it relied on. The full design of this agent, and its input and output, are on Build the agents.
Several different things worked together in that paragraph, and a change to any one of them is a different kind of change. It is worth naming them precisely, because the rest of the chapter depends on the distinction.
| Part | What it is | In the triage agent | A change to it looks like |
|---|---|---|---|
| Model | The language model that reads text and writes text, including requests to call a tool | The model named in the run script | One line in a script: a new model version |
| Agent | The program that runs the model in a loop: send the context, receive a request, run the tool, return the result, repeat until an answer | The headless run started by the CI workflow | A new cap, a new permission, a new start command |
| Instructions and skills | Text the agent loads to guide the model. A skill is a file of instructions for one kind of task. | The system prompt, and the skills read-build-log and search-bug-history | An edited sentence. The case on this page. |
| Tool | One function the agent may call, with a name, a description and an input shape | get_build_log, list_changes, search_bugs | A different default, a renamed field, a longer answer |
| Tool server | A separate program that offers tools to the agent, here through the Model Context Protocol (MCP), the common standard for this | The triage tool server, which wraps the CI system, git and the bug tracker | A new tool, a changed endpoint |
None of these changes is visible in the agent's answers until the agent runs. That is the root of the problem.
2. Why a reviewer cannot judge the change from the diff#
Code has a property that agent instructions do not. A reviewer of code can usually trace what a line does: this branch runs when the input is empty, this call returns that type. An instruction to a model does not work that way. The model reads it together with everything else in its context, and how much weight it gets depends on wording, position and the case in front of it. The same is true, in a different way, of a tool that now returns 100 lines instead of 200, or a model that is one version newer. Each one changes behavior on some cases and not on others, and nothing in the diff says which.
The natural answer is the one used for code: keep a set of cases with known right answers, and run the changed agent on them. For the triage agent, a case is a past red build whose culprit was confirmed afterward, by re-runs and by the revert that fixed main. Run the agent on the case and compare its answer with the confirmed culprit.
That collection of cases is called a golden set, and using it to check that a change did not make the agent worse is called a regression eval (evaluation). The rest of this chapter assumes Acme has one: 60 past red builds with a confirmed culprit.
A useful golden set is not 60 copies of the easy case. It is chosen so that each kind of mistake the agent could make has a case that catches it.
Twelve of the 60 casesWhat failed, the right answer, and the mistake each case is there to catch
One case file is shown in full on Build the agents.
| Case | What failed | The right answer | Why it is in the set | Tags |
|---|---|---|---|---|
case-4411 | A confirmation email never arrives | #4812 | The cause is in a second service's log | read-build-log, second-service-log |
case-3644 | A checkout test times out | #3630 | The real error is 80 lines after the first error | read-build-log, long-log |
case-3990 | A type error in the cart service | #3987 | The plain case. A sentinel: it has caught two regressions before. | read-build-log |
case-4127 | A report job returns wrong totals | #4101, the older of two changes to the same file | Catches "blame the newest change" | list-changes, two-candidates |
case-4302 | A sign-in test fails now and then | Unknown: an open flaky-test record exists | Catches blaming a change for a flaky test | search-bug-history, unknown |
case-4051 | Every job fails at set-up | Unknown: the package mirror was down | Catches blaming a change for an outage | read-build-log, unknown |
case-3902 | An error inside a library | #3899, a lock file change | The diff holds no product code | list-changes, dependency |
case-3655 | A new test fails | #3651, which added the test | The defect may be older than the change | list-changes, new-test |
case-4688 | Two changes fail only together | #4680 with #4671 | The answer has two changes | two-candidates, together |
case-4215 | A migration fails on seeded data | #4209 | The error is in a database log | second-service-log |
case-4530 | A configuration value was renamed | #4522 | No code near the failure changed | list-changes, config |
case-4399 | A slow query times out | Unknown: no change touches the area | The list of suspects is wrong | unknown |
Three of the twelve have the answer "unknown". A set without them rewards an agent that always blames somebody. The tags say which skill or tool each case exercises; they matter in Act 3.
3. The same agent does not give the same answer twice#
One more fact has to be in place before any test design makes sense. Run the unchanged agent on the same case twice, and it may take a different path: read the change list before the log, ask for a different range, call the bug search earlier or not at all. Usually it reaches the same answer. Sometimes it does not.
This happens because the model chooses each next step from probabilities, and a small difference early in a run changes everything after it. Lowering the model's temperature makes its choices less random. It does not make the agent deterministic. An agent's context is built from tool answers as well as the prompt, and those can arrive in a different order or differ in detail between runs. And unless a provider explicitly guarantees identical output for identical input, a sampling setting should not be relied on for repeatability at all. So treat the agent as variable, and measure how variable.
Acme does that first. It runs the unchanged agent on all 60 cases three times.
| Result over three runs of the same version | Cases | What it means |
|---|---|---|
| Passed three times | 49 | Stable pass, in these three runs |
| Failed three times | 6 | Stable fail, in these three runs |
| Mixed | 5 | The agent did not change, and the result did |
Three different statements could be made from this table, and only two of them are supported.
- An observation: five of the 60 cases gave a different result across three identical runs. That is what happened.
- Instability: the agent varies on at least those five cases. Three runs show that variation exists; they do not measure how large it is. A case that passed three times may still fail on a fourth run.
- Significance: a claim that a later difference between two versions is real and not chance. That needs more runs, a measured baseline like this one, and a stated method. This table does not provide it.
What the table does provide is context. Five cases changed result on an agent that did not change, so a change in the total number of passing cases, by itself, is a weak signal: some movement in the total is expected even when nothing changed. This baseline of observed instability is often called the noise floor. It is not a threshold. It does not mean that a change affecting fewer than five cases is unimportant. Two cases can matter a great deal, for example when both are cases the agent must never get wrong, or when both fail in every run.
So the rest of this chapter keeps three levels apart, and uses each for a different decision.
| Level | What it is | What it decides |
|---|---|---|
| Observed instability | Cases whose result varied across identical runs of the same version, like the five above | Which cases need more runs before anyone reads much into them |
| Investigation trigger | A rule the team sets in advance. At Acme: any case that passed in every old run and failed in every new run; any change at all on a sentinel or high-risk case; any change in the "unknown" cases. | Stop the change and look. It does not say the change is bad, only that it must be explained. |
| Statistically supported conclusion | A claim that the new version is worse or better overall, with a stated method: enough paired runs per case, and a test chosen before the runs, such as a paired comparison of per-case results | Accept or reject a change on the evidence, rather than on a reading of the numbers |
Most day-to-day decisions use the second level. The third is worth the cost for changes that can move every case, such as a new model.
Act 2 · Explore#
The team now knows what it needs: run the changed agent on the golden set, compare with the confirmed answers, and do it before the change merges. Each obvious way to do that runs into a wall. The walls are worth understanding one at a time, because the design in Act 3 is built from their causes.
4. The first idea: run every case, one after another#
The simplest test is faithful to production: start the real agent against the real tools, give it the first case, wait for the answer, then the next. Nothing is simulated, so a pass means something.
The trouble is time. One investigation takes about 12 minutes, because the agent waits on slow internal services and on the model. Ten cases take two hours. Sixty take twelve. A developer who changed one sentence will not wait two hours to learn whether it was safe, so in practice the test stops being run, or is run only on a handful of cases.
The time comes from waiting, not from work. Most of each run is the agent idle while a log service or the model answers. Waiting can overlap. That leads to the second idea.
5. The second idea: run them all at once#
If each run is mostly waiting, start all ten at the same time. Ideally the whole set finishes in about the time of one run.
In practice it often does not, and the reason is the tools. Each run makes about seven to twelve tool calls: logs, change lists, diffs, bug searches. Ten runs at once send about a hundred requests to the same internal services within minutes, where they normally see a few. Those services were sized for people, not for a burst. They slow down, then start refusing requests. If the agent or its tool client retries a refused call, each retry adds another request to a service that is already overloaded. Runs take longer than they did alone, and some time out. Now the eval is slow and flaky, and a failure no longer says whether the agent is wrong or the log service was busy.
This does not happen to every team. A tool service with spare capacity and rate limits that fail fast can absorb the burst. But the failure has a clear cause: many agents sharing endpoints that have no limit and no cache. Act 3 removes that cause instead of hoping for headroom.
6. The third idea: record the tool answers and replay them#
If the live tools are the problem, take them out. Run each case once against the real tools and record every request and its answer. From then on, a test run gets its answers from the recording. It is fast, it touches no real service, and it can run in parallel.
It breaks the first time the edited agent asks a question the recording does not hold. A recording is a list of the questions one run happened to ask, in the order it asked them. Section 3 showed that the agent does not ask the same questions every time, and an edit to its instructions is designed to change what it asks. Below, the recorded run went straight from the test log to the change list. The edited agent, told to look harder at timeouts, asks for the email service's log and for the bug history. The recording has no answer for either.
Recorded run
Edited agent, replayed
The question is reasonable and the data exists. The recording simply never saw it.
A common next step is to make the agent repeat itself: fix the random seed and set the temperature to zero, so that the replayed run asks the same questions. Replay works more often. But section 3 already said why this cannot be relied on: temperature zero lowers randomness and does not guarantee identical runs. And it tests a configuration that is not the one in production, while the edit under test is meant to change the questions anyway.
7. The fourth idea: run a few cases first#
The last obvious idea accepts that the full set is expensive and puts a cheap filter in front of it. Run three cases. Read the agent's trace, the record of every request it made and every answer it received, by eye or with a second model as a judge. If both look fine, run the full set. This is the best of the four. It catches gross mistakes in minutes and saves the cost of the full set on changes that are plainly broken.
It still has a known failure: sometimes the smoke test passes and the full set fails. The arithmetic shows why that is not bad luck. Suppose an edit harms 6 of the 60 cases. Three cases picked without looking at the edit all miss those six with probability (54 x 53 x 52) / (60 x 59 x 58), which is about 72%. And even a harmed case can pass once, because section 3 showed results vary between runs. A passing smoke test of three fixed cases, each run once, says very little about an edit.
| Idea | Why it seems right | What breaks it | The cause underneath |
|---|---|---|---|
| One after another | Faithful to production | Two hours for ten cases | Most of the time is waiting on slow services |
| All at once | Waiting can overlap | Shared tools throttle; retries add load; runs get slower and flaky | Many agents share endpoints with no limit and no cache |
| Record and replay | Fast, no load, parallel | The edited agent asks questions the recording never saw | A recording stores answers to one run's questions, not the data behind them |
| Make it repeat itself | Replay works more often | Not guaranteed; tests a non-production setting | Lower randomness is not determinism |
| A few cases first | Cheap early filter | Smoke passes, full set fails | Fixed cases miss what the edit touched; one run per case is one sample |
Read down the last column. Each failure has a cause that can be designed away, and that is what Act 3 does.
Act 3 · Engineer#
The causes from Act 2 point to the design. Remove the dependence on one run's questions. Put limits where many runs meet. Choose test cases from the change. Run each case more than once. Grade with code wherever code can decide. Then wire it into the pipeline so that it runs on every change to the agent.
8. Freeze the world, not the path#
Replay failed because it stored the answers to the questions one run asked. Store instead the data those questions were asked of. For each golden case, export a snapshot of the world as it was when the failure happened: the build logs of that run, the list of changes, the diffs, the bug records. Then run a small stand-in tool server that offers the same tools, with the same names and inputs, and computes each answer from the snapshot.
Because the answer is computed, the question does not have to be one that was asked before. The agent can read the change list first or the log first, search for any word, ask for any service's log. As long as the data was exported, there is an answer. If it was not exported, the stand-in says so clearly, and the run is marked incomplete rather than wrong.
Edited agent asks
Stand-in answers
The stand-in, in codeThree tools answered from one frozen case, including questions no run asked before
What this code is meant to show. It is a simplified demonstration of one idea: a stand-in that computes its answers from frozen data can answer questions that no recorded run ever asked. If it works, the output below shows answers to three new questions, and an explicit refusal for data that was never exported. It does not show that a production stand-in is easy, or that it matches the real tools.
Each tool reads the exported files and computes its answer, so a search word or a service name that no earlier run used still works. Data that was never exported returns an explicit error instead of an empty answer, because an empty answer would look like evidence.
#!/usr/bin/env python3
"""agents/triage/evals/standin.py — the stand-in tool server's three tools, answered from one frozen case.
Simplified demonstration: it ignores some arguments (see below). A real server must honor every argument
and wraps these functions in the tool protocol the agent already uses."""
import json, pathlib
class FrozenCase:
def __init__(self, case_dir):
self.dir = pathlib.Path(case_dir)
self.changes = json.loads((self.dir / "changes.json").read_text()) # the changes in the build, as exported
self.bugs = json.loads((self.dir / "bugs.json").read_text()) # bug records as they were that day
def get_build_log(self, job_id, service="test-runner", search=None, window=200):
path = self.dir / "logs" / f"{service}.log"
if not path.exists(): # data that was never exported: say so, never invent it
return {"error": "not in snapshot", "service": service}
lines = path.read_text().splitlines()
if search: # any search word works: the answer is computed, not recorded
return {"lines": [l for l in lines if search in l]}
first = next((i for i, l in enumerate(lines) if "Error" in l), 0)
return {"lines": lines[max(0, first - window // 2): first + window // 2]}
def list_changes(self, good_sha, bad_sha):
return {"changes": [{k: c[k] for k in ("pr", "title", "files")} for c in self.changes]}
def search_bugs(self, query):
return {"bugs": [b for b in self.bugs if query.lower() in (b["title"] + " " + b["text"]).lower()]}
Run against a small sample snapshot of case 4411 (three of its seven changes), it answers the two questions the recorded run asked, and the three it never asked:
recorded 11:48:02 TimeoutError: no confirmation email after 120 s
recorded [4805, 4812, 4816]
new ["11:46:03 AttributeError: 'NoneType' object has no attribute 'strip' (email/confirmation.py, line 61)"]
new {'bugs': []}
new {'error': 'not in snapshot', 'service': 'payments'}
The last line is the important one. The agent asked for a service whose log was never exported. The stand-in does not guess. The grader marks that run incomplete and the exporter learns that payments logs are sometimes needed.
What this demonstration leaves out. To stay short, it ignores some of its own arguments: list_changes returns every exported change whatever commit range it is given, and get_build_log does not use job_id to choose a job. A production stand-in must honor every argument the real tool accepts, and reproduce the behavior the agent can observe: filtering by range, ordering, page sizes and pagination, truncation, error responses and their codes, and permission boundaries, so that a question the real tool would refuse is also refused. Contract tests against the real tools, described below, are how a team keeps it honest.
Diagram: Freeze the world, not the path · how the eval runs without production
Read left to right. A snapshot holds the data of one past failure. A stand-in server computes tool answers from it. The agent under test may take a different path each run; the data stays fixed.
- Once per golden case
- A real past failure: A red build whose culprit is confirmed.
- Snapshot exporter: Copies the build log, the changes in that build and the bug records of the time.
- Frozen snapshot: Read-only data for one case.
- Every eval run
- Stand-in tool server: The same read-only tool names and arguments as production. Answers by querying the snapshot. A query outside the snapshot marks the run incomplete.
- Agent under test: The candidate triage agent, unmodified, connected to the stand-in instead of production.
- Answer + trace: The culprit, the confidence score and every tool call.
- Grading
- Outcome: Does the culprit match the label? Code decides.
- Path rules: Do the log lines it cites exist and contain the error? Did it stay under the call limit? Code decides.
- Reasoning: Was the conclusion supported by what it read? A judge model, measured against people first.
The properties that made Act 2 fail are now gone, and it is worth saying exactly which.
| Property | Why it follows |
|---|---|
| Different paths work | Answers are computed from data, so a question no earlier run asked still has an answer, if the data was exported |
| Parallel is safe | No production service is touched. Ten runs or a hundred. |
| Faster | Local reads replace slow internal services. This removes tool waiting only. Model time, rate limits and repeat runs remain, so measure the new duration instead of assuming it. |
| Comparable | The facts cannot drift between runs. What remains is the agent's own variation, which section 3 measured. |
This is not free. A stand-in is a second implementation of the tools, and it has to be kept faithful.
| Cost | How to contain it |
|---|---|
| Building the exporter is real work | Start with the three most-used tools. Unsupported tools return a clear "not in snapshot" error, and the run is marked incomplete, not failed. |
| Snapshots age when a tool's schema changes | Version each snapshot against the tool schema. For small changes, an adapter in the stand-in maps old data to the new shape. Expire old cases on a schedule, for example after a quarter, so the set stays small enough to re-export when a change is large. |
| The stand-in can differ from the real tool in ordering, truncation, paging and error behavior | Contract tests that send the same requests to both and compare the response shapes. A weekly run of the same cases against the live tools measures the remaining gap. |
| The agent gains a new tool | Old snapshots hold no data for it, so old cases become incomplete. Backfill the export where history exists. Where it does not, run those cases without the new tool and report them as a separate group. Never return an invented empty answer. |
| Tools that write, or that depend on the current time | Keep them out of the snapshot world. Evaluate them separately with a fake that records the intended action. |
A pass against the stand-in is evidence about the agent working on frozen data. The weekly live run is what keeps that evidence connected to production.
9. When a snapshot exporter is too much for now#
Some teams cannot build the exporter yet. A cheaper first step keeps most of the benefit of replay without its main flaw. Cache each read-only tool answer under a key built from the case, the tool and its schema version, and the normalized arguments, not from the position in the call order. On a hit, return the cached answer. On a miss, call the live tool, record the answer, and count the miss.
This is hybrid replay. A run with a miss has mixed old answers with today's data, so it is marked not comparable for the merge gate, and the miss is used to extend the cache. Report the miss rate per tool. A rising rate after a change shows the agent is asking different questions. A low rate does not show the answers are right.
10. When runs must share live tools: limits, not hope#
Some evals have to hit live tools: the weekly faithfulness run, or a team that has no stand-in yet. Section 5's failure came from many agents sending uncached reads to endpoints with no limit. Three controls remove that cause.
- A shared read-through cache for reads whose answer cannot change, such as the log of a finished job.
- A concurrency limit per endpoint and overall, so the burst becomes a queue the service can handle.
- One in-flight request per key, so ten identical misses become one call and nine waiters.
| Cost | How to contain it |
|---|---|
| A shared cache can leak across permissions | The key includes who is asking and the data version. Cache only answers that are the same for every caller in the eval. |
| Limits move the wait into a queue | Measure queue time and error rate. The eval is only faster if both drop. |
11. Choose the smoke cases from the change, and run each more than once#
Section 7's smoke test failed for two reasons, and each has a fix.
The first reason was that the three cases were fixed. They tested the same three behaviors whatever the edit touched. Instead, tag each golden case with the skills and tools it exercises, as in the table in section 2. A change to the skill read-build-log then runs the cases tagged with it, a few sentinel cases that have caught regressions before, and a seeded random sample from the rest. The smoke set looks where the edit can do harm. It is still an early filter, not the gate: a small sample can miss a rare side effect, and the full set runs before merge.
The second reason was that each case ran once. Run each smoke case several times on the old version and on the new one, and compare case by case. A case that passed every time before and fails every time after is a much stronger signal than one changed total. Three runs is a starting point, not a proof: with three runs a case can only show 0, 1, 2 or 3 passes. Choose the number of runs, and how many flips count as a failure, from the variation measured in section 3.
The comparison is done by a short script. What it is meant to show: that comparing old and new case by case reveals what a single total hides. It reads one line per run, classifies each case as passing every run, failing every run, or mixed, for each version, and prints only the cases whose class changed. When it runs cleanly, its output lists exactly the cases that need a person's attention; its exit code fails the check when any case moved from passing every run to failing every run. It has been run on sample data.
#!/usr/bin/env python3
"""agents/triage/evals/flips.py RESULTS.jsonl -> which cases changed, old version against new.
Each line: {"case": "case-4411", "version": "old" or "new", "run": 1, "passed": true}"""
import collections, json, sys
passes = collections.defaultdict(lambda: {"old": [], "new": []})
for line in open(sys.argv[1]):
if line.strip():
r = json.loads(line)
passes[r["case"]][r["version"]].append(r["passed"])
def stable(results): # "pass", "fail" or "unstable"
share = sum(results) / len(results)
return "pass" if share == 1 else "fail" if share == 0 else "unstable"
groups = collections.defaultdict(list)
for case, p in sorted(passes.items()):
groups[(stable(p["old"]), stable(p["new"]))].append(case)
regressions = groups[("pass", "fail")]
for (old, new), cases in sorted(groups.items()):
if old != new:
print(f"{old:>8} -> {new:<8} {len(cases):2d} {' '.join(cases)}")
print(f"unchanged: {sum(len(c) for (o, n), c in groups.items() if o == n)}")
print(f"regressions (pass -> fail): {len(regressions)}")
sys.exit(1 if regressions else 0)
Its exit code makes it usable as a check: a case that moved from stable pass to stable fail fails the step. The words "stable" and "unstable" here mean stable within the runs made, nothing more.
12. Grade the answer, the path and the reasoning#
A run can reach the right culprit for the wrong reason, or reason well and land on the wrong change. So grading has three layers, and code does as much of it as code can.
| Layer | What it checks | Done by |
|---|---|---|
| Outcome | The culprit matches the confirmed answer | Code |
| Path rules | Tool calls stayed under the limit. No call to a tool outside the agent's list. The log lines the agent cites exist in what the tools returned, and contain the error it names. | Code, over the trace |
| Reasoning quality | Was the conclusion supported by what it read? | A judge model, for this layer only |
A path rule such as "it read the log" is weak alone, because an agent can read a log and ignore it. Checking that each cited line really came back from a tool is much stronger, and it is still code. The grader that does this is shown on Build the agents.
The third layer needs judgment, and a judge model, a second model that reads a run and grades it, is the usual tool. It is a measuring instrument, so it has to be measured before anyone relies on it.
| Way to use a judge | What it buys | What goes wrong |
|---|---|---|
| One judge, a score from 1 to 5 | Easy to set up | Scores drift, and most runs get a 4. A score has no fixed meaning. |
| One judge, yes or no against a short rubric | Answers can be compared with people's answers | The rubric must be narrow: "does the reason follow from the quoted lines?" |
| Compare two runs, blind | Judges are better at "which of these two is better supported" than at absolute scores. Fits old against new. | Two runs for each judgment. The order must be shuffled. |
| A judge from a different model family than the agent | It does not share the agent's blind spots, and does not favor its own style | A second provider to pay and to keep available |
| Several judges from different families, majority decides | One judge's bias is outvoted. Disagreement between judges is itself a signal: send those runs to a person. | Three times the cost. Use it on the changed cases only, not on all 60. |
| Rule for every judge | Why |
|---|---|
| Measure it before it may block anything | Give it 30 runs that people graded, including ones people disagreed on. Until its agreement with people is known, it advises. Measure again when the judge changes. |
| It reads the changed cases, not the whole set | The flip table already says which cases matter. That keeps the cost small. |
| It never overrules code | When the culprit is wrong, the run failed, however good the reasoning reads |
13. Is the confidence still honest?#
The agent outputs a confidence score, and later steps may act on it, for example by opening a revert pull request. So a change to the agent must not only keep its accuracy; it must keep the score meaningful. The property is called calibration: of the answers where the agent said about 0.9, were about 90% right? A change can keep accuracy and break calibration, typically by making the agent more confident when it is wrong. How the score is produced and how the calibration table is built are on Build the agents.
14. Shadow mode keeps the golden set growing#
Shadow mode only makes sense once four moments in an agent change's life are kept apart, because they are easy to blur.
| Moment | What happens | What it changes | What gates it |
|---|---|---|---|
| 1. Merge | The pull request with the new instructions, tool or model setting enters the main branch | The code and files in the repository. Nothing acts differently yet. | The evals in section 15: smoke on the pull request, the full golden set in the merge queue |
| 2. Deploy as a candidate | The new version is installed beside the current one, without authority | A second version exists and can be run | The merge |
| 3. Shadow | The candidate runs on live red builds in parallel with the current version. Its claims are stored, never acted on. | Evidence on today's failures | Deployment as a candidate |
| 4. Activate | A setting makes the candidate the version whose claims start the verifier and the revert flow | Production authority | Promotion criteria set in advance, met in shadow |
This sequence needs one thing from the architecture: the version that has authority is chosen by a setting, separately from which version is merged. At Acme the triage workflow reads the active version from .github/agent-versions.yml, a gate file owned by the platform team. A team whose agent takes effect the moment it merges cannot run shadow mode after merging. It must run its shadow evaluation before the merge instead, and accept that the merge is also the activation.
A golden set ages. The failures of six months ago are not today's failures. Shadow mode runs the candidate agent beside the current one on live red builds, with no authority to act. For a triage agent, evidence about the real culprit arrives later: the controlled re-runs, the revert, the incident review. A case enters the labelled set only when that evidence is firm. Cases with two causes, or a flaky test, go into a separate ambiguous set: useful to read, not used as pass or fail.
| Cost | How to contain it |
|---|---|
| Model calls and tool load while it runs | Shadow a sample of failures, not all |
| Only investigated failures get a label | The set drifts toward the failures people cared about. Track the share of shadow cases that never get a label. |
| Labels arrive late | It is a nightly signal, not a gate on a pull request |
| Not every agent has a later ground truth | Then a person labels a weekly sample |
15. Putting it in the pipeline#
Each piece above answers one failure from Act 2. In the pipeline they become tiers: cheap checks run on every push, the expensive ones only when they can see something new.
Before any eval runs, a change to a skill file goes through checks that need no model at all, because a skill is text with a structure and rules.
| Layer | What it checks | Who | Fails the pull request? |
|---|---|---|---|
| 1. Lint | Header present. The description says when to use the skill. Length within budget. Links resolve. No secrets or personal data. Every tool it names exists and is allowed. | A script in CI | Yes |
| 1b. Reach | What the skill pulls in: helper scripts, referenced files, remote resources. Are remote resources pinned? Did the effective tool permissions change? | A script, plus a person for anything new | Yes |
| 2. Overlap check | The description is not a near-copy of another skill's description | A script, or a small model, comparing all descriptions | Warn; a person decides |
| 3. Trigger test | Does the agent pick this skill when it should, and leave it alone when it should not? | An eval run | Yes |
| 4. Outcome eval | With the skill in use, does the task still succeed? | The tiers below | Yes |
| 5. Human review | Is the instruction right? Does the evidence support it? | A person, red lane | Approval |
The trigger test needs a word of explanation, because it tests something specific to how skills load. In Claude Code, only a skill's name and description are in the model's context at the start of a run; the body loads when the model decides to use the skill, and it decides from the description. So a skill can be well written and never run, because its description loses to another one. Other agent frameworks load skills differently; check yours. The test is a table of prompts with the skills allowed and forbidden for each, run with the whole skill set loaded, as in production.
| Prompt in the test table | Expected skill | Why it is in the table |
|---|---|---|
| "Why did job 4411 fail?" | read-build-log | The plain case |
| "Is job 4411's failure a known bug?" | search-bug-history; read-build-log is allowed after it | The nearest neighbor must come first |
| "Summarize yesterday's end-to-end environment usage" | none of the triage skills | Out of scope; must not fire |
Real tasks often use more than one skill, in an order that can vary, so each row states an allowed set and a forbidden set, and an order only where the order matters. The result is precision and recall for each skill, and a matrix of which skill was picked when another was expected. When two skills overlap, the matrix shows it, and the fix is usually in the two descriptions. Some skills are picked because of what happened earlier in a session, such as a recovery skill after a failed tool call; those rows start from a short scripted history. And a skill configured to load only on explicit request has no description in the context at all, so it will never be picked by itself; run the test with the production loading configuration.
The person who reviews the pull request gets a behavior diff beside the text diff: which eval cases changed result, and which trigger-test rows changed. A text change with no behavior change is either harmless or untested, and the reviewer asks which.
Diagram: A change to an agent · five tiers, cheapest first
Read left to right. The first two columns block in seconds and catch most mistakes. The eval tiers run on frozen snapshots, so they are fast and parallel. The last column does not block; it keeps the others honest.
- Text checks
- Skill lint + reach: Frontmatter, when-to-use description, length, links, no secrets, tools exist. Plus what the skill pulls in: scripts, files, remote resources. The skill-lint check.
- Overlap check: Compares the description with every other skill's description.
- Selection
- Trigger test: With the whole skill set loaded: does the agent pick an allowed skill, and none of the forbidden ones? The trigger-test check.
- Smoke
- Cases chosen from the diff: Cases tagged with the changed skill, fixed sentinel cases, and a seeded random few. The agent-smoke check, on the pull request.
- Several runs, base and candidate: A pass rate per case on both versions, not one lucky run.
- Full set
- Golden set, in parallel: Every case, on frozen snapshots, at once. The agent-eval check, in the merge queue.
- Three graders: Outcome by code, path rules and cited evidence by code, reasoning by a judge model that advises until it is measured.
- Behaviour diff: Which cases changed result. The human reviewer reads this next to the text.
- After merge
- Shadow mode: Deployed as a candidate beside the active version, it runs on live red builds with no authority. Activation is a switch in .github/agent-versions.yml.
- Weekly live run: The same cases against real tools, in sequence. Measures the stand-in gap.
| Tier | Runs when | Cost | Blocks the merge? |
|---|---|---|---|
| Lint and overlap | Every push | Seconds | Yes |
| Trigger test | A skill description or the set of skills changed | A minute | Yes |
| Smoke: cases chosen from the change, several runs each on old and new, on frozen snapshots | Every pull request that touches the agent | Minutes | Yes |
| Full golden set on frozen snapshots, in parallel | Before merge, in the merge queue | Measure it; tool waiting is gone, model time is not | Yes |
| Live run, one case after another | Weekly | Hours | No. It measures how far the stand-in has drifted. |
| Shadow mode | After merge and deployment as a candidate, before activation; and continuously on a sample | Ongoing | It does not block the merge. It gates activation, and it grows the golden set. |
Notice that the first idea from Act 2, the slow sequential run against live tools, is still here. It moved from the merge gate to a weekly job, where its faithfulness is worth its cost.
16. Four changes, worked start to finish#
The design is easier to trust once it has been seen working. The four changes below are the four kinds from section 1: what a tool returns, an instruction file, the model, and the second agent that fixes builds. Each kind has a cheapest test that can catch it, and the habit worth building is to run that one first. The results shown are illustrative, not measured.
| The change is in | Cheapest test that can catch a regression | Needs the model? | Then |
|---|---|---|---|
| What a tool returns | Does the tool still return the evidence each case needs? | No | Run the cases that lost evidence first; the full set still runs before merge |
| An instruction or skill file | Tests of the one sub-task the text governs, then a smoke set picked from the change | Yes, small | The full set before merge |
| The model | Nothing cheap sees it. Run the full set, old model against new, several times. | Yes, large | Merge, deploy as a candidate, shadow, then activate |
| The red-build fix agent | Six checks by code on past fixes | Yes, for the run; no, for the grading | A weekly sample read by people |
16.1 Change 1: what a tool returns#
The change. To save tokens, the get_build_log tool will return 100 lines around the first error by default, not 200. The agent's instructions, skills and model stay the same.
agents/triage/tools/get_build_log.py
- DEFAULT_WINDOW = 200
+ DEFAULT_WINDOW = 100
The test that fits. One question needs no agent run at all: does the tool's default answer still contain the line each case needs? Each case file lists those lines, so a script can check all 60 against the frozen logs in seconds. What the script is meant to show: which cases can no longer see their evidence with the new default. A clean run proves only that the required lines are still returned. It does not prove the agent behaves the same, because the agent now sees less context around them. It has been run on sample data.
#!/usr/bin/env python3
"""agents/triage/evals/reach.py WINDOW CASES_DIR -> which cases lose their evidence.
No model. For each golden case: does the tool's default answer still contain the lines
that a right answer must cite?"""
import json, pathlib, sys
window, cases_dir = int(sys.argv[1]), pathlib.Path(sys.argv[2])
def default_answer(log_lines, n):
"""What get_build_log returns with no range: n lines around the first error."""
first = next((i for i, line in enumerate(log_lines) if "Error" in line), 0)
return log_lines[max(0, first - n // 2): first + n // 2]
lost = []
for case_file in sorted(cases_dir.glob("case-*/case.json")):
case = json.loads(case_file.read_text())
for service, needed in case["must_cite_in_log"].items():
log = (case_file.parent / "logs" / f"{service}.log").read_text().splitlines()
answer = "\n".join(default_answer(log, window))
if not all(text in answer for text in needed):
lost.append((case["case"], service))
print(f"window={window} cases={len(list(cases_dir.glob('case-*')))} lose evidence={len(lost)}")
for case, service in lost:
print(f" {case}: the required line is outside the default answer for '{service}'")
sys.exit(1 if lost else 0)
The result.
window=100 cases=60 lose evidence=4
case-3644: the required line is outside the default answer for 'checkout'
case-4215: the required line is outside the default answer for 'database'
case-4127: the required line is outside the default answer for 'reports'
case-3718: the required line is outside the default answer for 'orders'
| Step | What ran | Result |
|---|---|---|
| 1 | The script above, 60 cases, no model, 4 seconds | 56 cases still receive their required lines. That makes them less likely to be affected; it does not show their behavior is unchanged, because the agent now sees less context. 4 cases lose a required line. |
| 2 | The agent, on those 4 cases only, three runs each, old tool and new | In 3 cases the agent asked for a second range and still found the line. case-3644 failed in all three runs: it answered from the first 100 lines. |
| 3 | Decision | The change is not merged as it is. The tool now adds one line to a shortened answer, saying that more lines exist and which range to ask for. Steps 1 and 2 are run again: all pass. The full set then runs before merge. |
| What this approach buys | What goes wrong |
|---|---|
| 56 of 60 cases are screened with no model call | Every case must list its required lines. Someone writes them. |
| The 4 agent runs go where the risk is | It only sees evidence that became unreachable. A tool change that adds confusing text is not caught here; the full set before merge covers that. |
| The reason for the failure is known before the agent runs | It needs the frozen logs. Without them there is nothing to check against. |
16.2 Change 2: an instruction in a skill file#
The change. Pull request #4827 gives the skill read-build-log one new rule, meant to help the agent choose when two changes touch the same file.
agents/triage/skills/read-build-log/SKILL.md
2. Find the first line that is an error of the program, not of the test runner.
+ When two changes touch the file named in the error, suspect the most recent one first.
The rule sounds reasonable. It is also exactly the mistake case-4127 exists to catch.
| Step | What runs | Cases | Time | Result |
|---|---|---|---|---|
| 1. Tests of the one sub-task | One model call each: here is a log excerpt and a list of changes; which change do you suspect? No tools, no loop. | 40 | Under a minute | 38 pass. 2 fail: both have an older culprit. |
| 2. Smoke set chosen from the change | The full agent on frozen data, three runs each, old skill and new | 8: the cases tagged read-build-log and two-candidates, the sentinel case-3990, and two picked at random | 6 minutes, in parallel | See below |
| 3. The full set | Only if step 2 passes | 60 | Not run |
The flip script from section 11 prints only what changed:
pass -> fail 2 case-4127 case-4688
unstable -> pass 1 case-3644
unchanged: 5
regressions (pass -> fail): 2
The edit fixed one unstable case and broke two stable ones. The count of passing cases went from 7 to 6, a movement in the total that alone would say little. The flip table shows what the total hides: two cases that passed in all three old runs failed in all three new runs. That meets Acme's investigation trigger from section 3, and one of the two, case-4127, is the case built to catch exactly this mistake. It is not a statistical conclusion about the agent as a whole. It is a concrete, explainable failure, which is enough to stop the change. Judge an edit by the cases that changed, not by the total. Note also that case-4127 was in the smoke set only because the cases were chosen by tag; three fixed cases would very likely have missed it, as section 7 computed.
The decision. The rule is rewritten: "When two changes touch the file, read both diffs before choosing." Steps 1 and 2 pass. The full set then runs before merge.
| What this approach buys | What goes wrong |
|---|---|
| Step 1 gave a warning in under a minute | A sub-task can pass alone and fail inside the loop. It does not replace steps 2 and 3. |
| The smoke set looks where the edit can hurt | Tags must be kept current. A wrong tag hides a case. |
| The flip table separates a consistent change from chance variation | Three runs for each version: six times the runs of a single pass |
16.3 Change 3: the model#
The change. The provider releases a new model version. One line changes in the run script. No instruction, skill or tool changes, so there is nothing to choose a smoke set from. Every case may move.
The test that fits. The full set, old model against new, three runs each, on frozen data, in parallel. Then four comparisons, because one number would hide the important one.
| Comparison | Old model | New model | Reading |
|---|---|---|---|
| Cases that changed | 5 fail to pass, 2 pass to fail, 4 between stable and unstable | Net better. The two regressions must still be read. | |
| Right "unknown" (10 cases) | 8 | 6 | Worse. Both regressions are here: the new model names a change for case-4302 and case-4399. |
| Wrong and sure (wrong above 0.8) | 2 | 5 | Worse. It is more confident, right or wrong. |
| Tool calls for each case, median | 7 | 5 | Better: cheaper and faster |
A single accuracy number would say "upgrade": 52 right against 49. The second and third rows say the new model blames changes it should not, and is surer when wrong. For this agent, that is the costly kind of error. With 10 "unknown" cases, a drop from 8 to 6 is a small sample; it justifies holding the switch and looking, not a final verdict.
| Step | Action |
|---|---|
| 1 | Do not merge the model change as it is. The new model is better at finding and worse at declining. |
| 2 | Strengthen the rule for "unknown" in the system prompt. That is an instruction change: run the test of section 16.2 on the new model, then the full set again. |
| 3 | Merge when the full set no longer shows those regressions, with the old model still active. |
| 4 | Deploy the new model as a candidate and run it in shadow mode on live failures for two weeks. Rebuild the confidence table from its shadow claims; the old table described the old model. |
| 5 | Activate the new model when right-unknown and wrong-and-sure in shadow are at least as good as the active model, as set out before the shadow run started. |
| What this approach buys | What goes wrong |
|---|---|
| It sees every case, which a model change requires | It is the most expensive run: 60 cases, three times, two models |
| The per-measure comparison finds what a total hides | Needs enough "unknown" cases to say anything about them. Ten is thin. |
| Shadow mode checks cases the golden set does not have | Two weeks is slow. A model that is retired on a date may force the pace. |
16.4 Change 4: the agent that fixes#
The triage agent names a change. A second agent, the red-build fix agent, proposes a repair after a claim is verified. It is changed and tested the same way, with a different set of checks. Its design is on Build the agents.
The change. The red-build fix agent's task file gains one permission.
agents/fix/prompts/task.md
You may not edit the failing test.
+ You may add or change tests in the tests/contract/ folder.
The test that fits. The eval set is 25 past red builds that a person fixed. The agent gets the verified claim and the code at the culprit commit, so the failing test fails before its repair. Six checks by code grade each attempt.
| Measure | Before | After | Reading |
|---|---|---|---|
| Fix rate (all six checks pass) | 11 of 15 fixable cases | 13 of 15 | Better |
| Correct hand-over (10 cases only a person may fix) | 10 of 10 | 9 of 10 | One regression |
| A new lower-tier test that fails without the fix | 6 of 11 | 12 of 13 | The purpose of the change, achieved |
| Median changed lines | 14 | 31 | Larger changes: more for a reviewer to read |
The regression: in one case the agent "fixed" a contract mismatch by changing the existing contract test to accept the new behavior. The failing test was untouched, so the old checks passed. The permission was too wide.
The decision. The line becomes "You may add tests in tests/contract/. You may not change an existing one." A seventh check by code is added: no existing test file is modified. All 25 cases run again.
| What this approach buys | What goes wrong |
|---|---|
| Graded by code: a test passes or it does not | "The checks pass" is not "the fix is right". A weekly sample must be read by people. |
| The hand-over cases catch an agent that fixes what it should leave | Past fixes are few. 25 cases is a small set. |
| Each regression becomes a new check | The checks only know the mistakes already seen |
16.5 What the four have in common#
| Question | Answer from the four examples |
|---|---|
| What do I run first? | The cheapest test that can see this kind of change. For a tool change, a script with no model. |
| How do I know a difference is real? | Keep three levels apart. Observed instability says which cases need more runs. An investigation trigger, set in advance, says when to stop and look: here, a case that moved from passing every run to failing every run, or any change on a sentinel case. A claim that the new version is worse overall needs enough paired runs and a test chosen in advance. |
| Why did the smoke test pass and the full set fail? | The smoke cases were not chosen from the change, each ran once, or the change was to the model. |
| What is the one number? | There is none. Right culprit, right "unknown" and wrong-and-sure can move in opposite directions. |
| When is it safe to merge, and when to activate? | Merge after the full golden set passes. Activate after shadow mode meets criteria set in advance. For a change that can move every case, such as a model, both steps are needed; for a small instruction change, a team may activate at merge if its evals are strong enough. |
17. Side questions#
| Question | Short answer |
|---|---|
| How big should the golden set be? | Big enough that one case is not 10% of the score. Ten cases cannot tell an 80% agent from a 90% one. Shadow mode is how it grows without hand labelling. |
| Is a judge model reliable? | For narrow questions with a rubric, often. It is a measuring tool, so calibrate it like one. |
| Should prompts and skills live in the code repository? | Yes. Then they get lanes, owners, history and the same pipeline as code. Where they live and how a pipeline loads them is on CI/CD agents. |
| Does the coding agent have tooling for this? | Claude Code documents comparing a task with and without a skill, and a report that shows skills that are never invoked. The stand-in tool server, the trigger table and the shadow loop are patterns you build. |
| What ties the evidence together? | A provenance record for each pull request and each eval run: base and head, rules and owners in force, model and tool versions, snapshot versions, each verdict and which system produced it, signed by the server. It is the same idea as the attested review receipt, applied to everything. |
The team at the start had a three-line diff that nobody could judge. What it needed was not a faster agent or a bigger test set. It needed a test that could ask the changed agent any question and still get the answers of the day the failure happened, that ran the cases the change could touch, more than once, and that knew how much the agent varies when nothing changes at all. With those three things, the same three-line diff arrives with a short report: which cases moved, in which direction, and whether that is more than the agent's usual variation. That report is something a reviewer can judge.
Next: every model run behind #4812 added up, and what breaks at 300 changes a day: Volume and cost.