All chapters
  1. Start hereOne change, from diff to production
  2. Before stage 1The team setup: what changes when you are not alone
  3. Stage 1Code change: catch it while the agent is still editing
  4. Stage 2Commit: what must be true before a commit exists
  5. Stage 3Pull request: arrive already reviewed, with evidence
  6. Stage 4Review: who reviews what, at hundreds a day
  7. Stage 5CI/CD: spend scarce capacity like a budget
  8. Deep diveCI/CD agents: where they live and how they run
  9. Deep diveBuild the agents: pinpoint the change, then fix it
  10. Stage 6When the change is an agent
  11. OperateVolume and cost: what breaks at 300 a day
  12. AdoptAdopt it in steps
  13. ReferenceHook events: which one, with a script for each
  14. ReferenceGlossary
  15. AboutAbout this site
Stage 5

CI/CD: spend scarce capacity like a budget

When tests need a complete running copy of the system, CI capacity is fixed. Merge queues, test selection, priority for scarce end-to-end environments, and a verified loop for red builds.

At 10:48 pull request #4812 has its two owner approvals. It adds one optional field, delivery_note, to Acme's order API, and every check so far is green. One step is left before it reaches the main branch: a run on one of Acme's 12 end-to-end test environments, the only place where every service runs together.

It is not the only change that wants one. About 300 pull requests open at Acme on a day like this, and a coding agent that sees a failed run can ask for another in a second. The short end-to-end run in the merge queue takes 8 minutes of an environment; the full run takes 40. Nothing on this page makes an environment faster.

And passing is not the end. At 11:46, 44 minutes after #4812 merges, the wider end-to-end run on main fails, and seven changes are suspects. So this stage has two problems that need different tools: before merge, how to spend a fixed capacity; after merge, how to find and undo the cause without letting a guess act on its own. The numbers on this page are illustrative: they show the method, not a measurement.

Cheap checks run on everything. Scarce checks run on what survived. An agent's guess about a red build is a claim until controlled re-runs, with no model in them, support it.

Act 1 · Before merge: a fixed budget#

Before merge the problem is capacity: a fixed number of environments and more changes than they can test.

1. Why there are only twelve environments#

An end-to-end test environment is a complete running copy of the system: every service, a seeded database, queues, and sandbox connections to the payment, email and sign-in providers. Acme has 12 of them. The obvious question is why a company that runs in the cloud does not simply start more. The cloud removes the wait for machines. It does not remove four other limits.

LimitWhy it holds in the cloud
CostEach environment runs every service, a database and queues, all day. Twelve is what the budget pays for.
Set-up timeA fresh environment needs minutes to deploy every service, run the migrations and seed the data. A new one per run adds that time to every run.
Shared provider sandboxesThe payment, email and sign-in providers give the company a few sandbox accounts, each with a rate limit. More environments share the same accounts and reach the limit sooner.
Shared test dataTest users, cards and mailboxes in a sandbox account are shared. Two runs that use the same test customer at the same time break each other.

So the number is fixed, and the arithmetic is short. A full run takes 40 minutes, so one environment can do 36 a day, and twelve can do 12 x 36 = 432. That is a ceiling, and it assumes every environment is busy every minute. Set-up and reset, data seeding, provider rate limits and flaky outside services take a share, and a queue that runs near its ceiling has long waits. Against the real capacity stand 300 pull requests a day, plus every retry an agent asks for.

2. Spend the scarce tier last#

A fixed resource is treated like a budget: spend it only on what is worth it. In CI (continuous integration, the jobs that run on every change) that means ordering the checks by cost. The cheap ones run on everything and stop most bad changes. The scarce one runs only on changes that survived the rest.

TierCheckWhat runsWhere
Unit testsunit-testsLint, build and unit testsOrdinary runners, minutes
Integration and contract testsintegration-testsOne service with its real database and queue in containers, plus contract tests between servicesOrdinary runners
Tests chosen from the diffaffected-testsThe tests mapped to the changed filesOrdinary runners
Merge queuee2e-smokeA short end-to-end runOne of the 12 environments
After mergee2e-fullThe wider end-to-end suite. Job 4411 fails here, which is how main can still go red.The 12 environments
Nightlynightly-fullEverything, with no selectionThe 12 environments, off-peak

Diagram: CI as a budget · cheap checks on everything, scarce checks on what survived

Read left to right. Every column filters, so fewer changes reach the next one. The end-to-end tier is last and is entered by priority. The numbers are illustrative.

  1. Every push
    1. Lint, build, unit tests: Minutes, on ordinary runners. Most bad changes stop here.
    2. Integration and contract tests: One service with its real database and queue in containers, and each interface between services against its contract. Catches much of what an end-to-end run would, without an environment.
    3. Lane + receipt check: Compute the lane from the base branch rules. Verify a signed review receipt, or run the review.
    4. Tests chosen from the diff: Integration tests mapped to the changed files.
  2. Approved changes
    1. Merge queue: Each change is tested on top of the ones ahead of it, so main stays green.
    2. Batch of five: Several changes share one run. A red batch is split to find the bad change.
  3. Scarce tier
    1. Scheduling classes: Release branch, red lane, people's changes, agent retries. Each class has a share and a maximum wait.
    2. End-to-end run: The whole system runs in one environment, with seeded data and provider sandboxes. In the queue, e2e-smoke (about 8 minutes); after merge, e2e-full (40 minutes).
  4. After merge
    1. Progressive deploy: Only after e2e-full passes on that main commit. Small steps, with the metrics named in the risk note watched.
    2. Nightly full run: Everything on main, once a night, with no selection. It blocks nothing; it is the safety net for a wrong dependency map.

Each column of the diagram filters, so fewer changes reach the next one. Notice where the risk sits: whatever runs only after merge is a place where main can turn red. Act 2 is about that last column.

3. What 300 changes a day cost, and five levers#

The scarce tier is not the only cost. The ordinary tiers run far more often, and at volume they need their own budget. Take a large repository where one full build with its tests takes 45 minutes, and 300 pull requests merge in a day.

QuestionArithmeticAnswer
Can main build one merge at a time?300 merges x 45 minutes = 13,500 minutes = 225 hoursNo. A day has 24 hours. One at a time, 32 changes merge in a day.
How many builds really run?300 pull requests x about 2.5 pushes each = 750, plus 300 queue buildsAbout 1,050 builds a day
How much machine time is that?1,050 x 45 minutesAbout 790 runner-hours a day, if nothing is done

The 790 runner-hours come from two multipliers. Every push of every pull request pays the full 45 minutes, so the cost grows with pushes, not with merges; and the merge queue adds one more full build for each of the 300 merges. Cut the minutes per build or the number of builds, and the total falls in proportion.

Five levers bring this down. They are listed in the order to use them, because each one makes the next cheaper.

LeverWhat it doesEffect on the numbersWhat goes wrong
1. Do not rebuild what did not changeThe build tool knows which parts depend on the changed files. A shared cache holds the output of every part that anyone already built.Most changes touch a few parts. A typical build falls from 45 minutes to about 8. Only a change to a core library pays the full 45.A dependency the build tool does not know about gives a stale result. The build must declare every input.
2. Fail in the first two minutesLint and the fastest tests run first. A new push cancels the run of the old commit.Runs that were going to fail stop early. Superseded runs stop at once.Nothing worth naming. Do this first.
3. Build pull requests in parallelOrdinary runners are elastic. Many builds run at the same time.Waiting time per change stays 8 to 45 minutes whatever the volume. The cost is machine time, not waiting.Money, and a peak hour that needs several times the average.
4. Test the merge queue in parallelThe queue tests many pull requests at the same time, each on top of the ones ahead of it. Section 4 shows how.With 45-minute checks, one slot merges 1.3 an hour, so 30 an hour needs 23 slots. With lever 1, checks take about 12 minutes and 6 slots are enough.A failure in the middle makes every build behind it start again. The longer the checks, the more that costs.
5. Do not build main twiceThe queue already built and tested the exact content that lands on main. A job you add to the main pipeline looks that build up by its content hash and reuses it.The 300 builds on main cost almost nothing. Only the wider end-to-end run is new work.Whatever runs only after merge is where a red main comes from. That is the e2e-full run in Act 2.

The levers fix the ordinary tiers. None of them makes an environment faster, so the end-to-end tier stays a separate, fixed budget. The rest of Act 1 is about spending it well, and the merge queue is where most of it is spent.

4. How the merge queue finds a bad change with no model#

A merge queue is a line of approved changes, each tested on top of the ones ahead of it before main moves. It exists because two changes can each pass alone and fail together. Two questions come up first. Does the next build wait for the one before it? And are ten queued pull requests built as one?

Neither, in the common design, which is how GitHub's merge queue works. Every queued pull request gets its own build, and that build contains every pull request ahead of it. The builds run at the same time. Nothing waits, and nothing is lumped together. Five pull requests are in Acme's queue at 10:50, and #4812 is the fifth.

PositionPull requestIts temporary branch containsRuns
1#4805main + 4805At the same time as the others
2#4807main + 4805 + 4807At the same time
3#4808main + 4805 + 4807 + 4808At the same time
4#4810main + 4805 + 4807 + 4808 + 4810At the same time
5#4812main + all four + 4812At the same time

Each build assumes that the ones ahead of it will pass. That is a bet. When every build passes, the bet paid: main moves to the content of the last build, and all five are merged after one build time, not five.

When one of them is bad#

On the real day all five passed. To see how the queue isolates a bad change, suppose instead that #4808, in position 3, breaks a test.

StepWhat happensWho acts
1Builds 1 and 2 pass. Build 3 fails. Builds 4 and 5 fail too, because they contain #4808.CI jobs
2The queue reads the results in order. Build 2 passed and build 3 failed. The only difference between them is #4808. So #4808 is the bad one.The repository host. No person and no model.
3#4808 is removed from the queue. Its author is told, with the failing log.The repository host
4#4805 and #4807 merge. They were never affected.The repository host
5New temporary branches are created for #4810 and #4812, without #4808. Their builds start again.The repository host; CI
6The author's agent reads the log, fixes the branch and the pull request joins the queue again, at the end.The author's agent, under the retry budget (section 6)

Finding the bad change cost no tokens and no person. The order of the queue is the search: before merge, isolation is arithmetic on build results. Keep that in mind for Act 2, where this arithmetic is no longer available.

The price is step 5: two builds were thrown away and run again. That is why the checks before the queue matter. A pull request that already passed its own checks rarely fails in the queue, and a rare failure is cheap. How rare it must be follows from the chance that some change ahead of you fails. The table is for the 45-minute case from section 3, with 23 slots and so up to 22 changes ahead.

Queue failure rate per pull requestChance that at least one of the 22 ahead of you failsMeaning
0.5%About 10%One build in ten is repeated. Healthy.
2%About 36%One build in three is repeated. Shorten the queue or the checks.
5%About 68%Most builds are wasted. The queue is the wrong tool until the failure rate falls.

Acme's queue is smaller: its build concurrency is 5 (section 7), so at most 4 changes are ahead. At the same failure rates the chance is about 2%, about 8% and about 19%. The shorter the queue, the more failures it can afford.

Three ways to build a queue, and when batching pays#

Stacking spends one build per pull request. On ordinary runners that is fine. On the end-to-end tier, where each build may need an environment, a team may want one build to prove several changes at once. That is a batch, and it trades fewer builds for a harder search when the batch fails.

DesignBuilds for 10 good pull requestsWhen one is badHow the bad one is foundUse it when
One at a time10, one after another: 10 build times of waiting1 build lostIt is the one that failedLow volume. At 300 a day and 45 minutes it cannot keep up.
Stacked, in parallel (above)10, all at once: 1 build time of waitingThe builds behind it run againThe first failing build in queue orderThe default. Fast when failures are rare.
Batched (one build for the whole group)1The batch is split in halves and each half is built: about 4 more builds for a batch of 10The half that fails is split again until one pull request is leftThe build or the test is expensive and scarce: an end-to-end environment, a long test on special machines

A batch is searched by halves. Ten changes fail together. Build the first five alone: they pass, so the bad one is in the second five. Build three of those: they fail. Build one of the three: it passes. Build the next: it fails. Four extra builds found one bad change in ten, with no model.

Acme uses stacking now and plans batching. The ordinary checks run stacked, because runners are plentiful. The e2e-smoke run can be batched for green and yellow changes, because there are only 12 environments, but only after measuring the failure rate. On GitHub this is a job Acme assembles: the merge queue documentation says "Merge limits do not combine merge_group builds" (checked 2026-11-16). A red-lane change such as #4812 gets a smoke run of its own.

5. Approaches weighed#

Each piece is now in place. Side by side, with the cost of each, they look like this.

ApproachWhat it buysWhat goes wrongUse it when
A. Full pipeline on every pushSimple, completeEnvironments are full of runs for changes that would have failed a lint. The queue is hours long.Low volume, no scarce tier
B. Tiered pipeline (cheap checks, then integration and contract tests, end-to-end environments last)Environment time goes only to changes that passed everything else. The integration tier catches much of what an end-to-end run would.A change can pass alone and fail combined with another. A stand-in for an outside provider can disagree with the real provider, and a contract test proves each side, not the whole flow.Always
B2. Shrink what needs an end-to-end environment (pick the end-to-end suites by affected service and user flow; reuse one build for the whole merge group; move checks down to contract tests)Fewer and shorter end-to-end runs per change, before any queueing trickNeeds a map from code to service and user flow, kept current. A check moved down to a contract test no longer sees the whole flow.Always, and before batching
C. Merge queue (test each change on top of the ones ahead of it)Main stays green on the checks the queue runs; "passed alone, failed together" is caughtOne bad change delays everyone behind itHigh volume
D. Batched merge queue (test several changes as one group)One end-to-end run proves five changesA red batch needs a search to find the culprit. Extra runs. Not every host's queue shares one build across a group (GitHub's merge limits do not, as of 2026-11-16), so this can be a job you assemble: the end-to-end run targets the last entry of the group, which contains the others.Scarce tier, low failure rate
E. Test selection from the diffRuns the tests that the change can affectA wrong dependency map skips the test that would have failedWith nightly-full as the net
F. Scheduling classes for environmentsRelease and red-lane work goes firstUnder steady high-priority load, plain priority starves everything elseWith reserved capacity or weighted fair shares per class, and a published maximum wait

Recommended default: B, B2, C and F from the start. Add D only after measuring: batching pays when failures on the end-to-end tier are rare, and its real cost depends on how a red batch is split and re-tested. Add E with a nightly full run. Approach E is also why affected-tests runs on every pull request but is not a required check: a wrong dependency map can skip a test, and nightly-full is its net.

6. Rules that stop agents from spending the budget#

Agents change one thing about this budget. A person thinks twice before re-running a 40-minute job. An agent that reads a failed log and is told to make the checks pass can re-trigger CI again and again, at no cost to itself. Without rules, the scarce tier fills with retries.

RuleReason
An agent gets a retry budget per changeCount end-to-end attempts by change lineage, on the server, so a new pull request or a rebase does not reset the count. After two agent-requested end-to-end retries, a person must approve the next one. Re-runs for an environment fault (a provider sandbox outage, a failed data seed) are counted separately.
Speculative agent work never preempts a person's runClasses, in order: release branch, red lane, people's pull requests, agent retries. Each class has a share and a maximum wait. An emergency bypass leaves an audit record.
A flaky test is a defect, with an ownerRetrying past flakiness spends the scarce tier to hide a problem. Quarantine the test and open a work item.
Environment time per change is measuredYou cannot budget what you do not count.

All four rules move the decision to retry away from the agent: the server counts, the scheduler orders, and a person approves past the limit.

The queue needs three rules of its own, because a queue that removes innocent changes wastes every build behind them.

RuleWhy
A failed test is run once more inside the same job before the build is reported as failedA team rule, set in the test runner or the CI job, not in the queue. A flaky test would otherwise remove an innocent pull request and throw away every build behind it. A pass on the second run is recorded as a flake, with an owner, and the build continues.
Jumping the queue is for emergenciesMoving one pull request to the front changes what every build contains, so GitHub rebuilds every in-progress entry. Keep an emergency path, and record each use.
Agents do not re-queue foreverA removed pull request may be fixed and re-queued by its author's agent twice. The third time needs a person. This is the retry budget above, applied to the queue.

The queue rules protect the changes behind a failure: an innocent pull request should never be removed, and a guilty one should not come back without limit.

7. #4812 through the queue#

With the budget rules in place, the real run of #4812 is short. It is red lane, so it gets an e2e-smoke run of its own, and it passes.

TimeWhat happensWho actsResult
10:50The pull request enters the merge queue. Four others are ahead of it.The repository hostA temporary branch: main, plus the four ahead, plus #4812
10:50–11:02The required checks run on that temporary branch: build, unit-tests, integration-tests, and e2e-smoke on one end-to-end environment (8 minutes). lane and agent-review report success at once in the queue, because they already judged the pull request. The four builds ahead run at the same time.CI jobsAll five pass. The combination is tested, not only each change alone.
11:02The five changes merge. Main moves forward.The repository host#4812 is on main
11:02The build for the new main is not repeated. The queue already built this exact content, so the stored build is reused (lever 5). This is Acme's own job, not a GitHub feature.A CI job that looks up the build by its content hashBuild ready in under a minute
11:05e2e-full starts on the new main: the wider end-to-end suite, 40 minutes.The environment schedulerJob 4411 takes one of the 12 environments
How the queue is wired on GitHubThe test workflow's triggers and jobs, and the three queue settings

What this shows. Which event starts which tier, so that every required check reports in the queue and e2e-full belongs to the workflow that the red-build loop listens to (section 10). It is an outline: the steps of each job are left out. A successful parse proves the shape, not that the jobs work.

A required check must also run on the event merge_group. Without it, the check is never reported, the entry waits until the status check timeout, and the merge fails. The same holds for the separate lane.yml and agent-review.yml workflows: each needs a merge_group trigger with a job that reports success at once there.

# .github/workflows/tests.yml  (outline: steps omitted)
name: Run Tests
on:
  pull_request:
  merge_group:            # the temporary branch the queue builds
  push:
    branches: [main]      # e2e-full runs on each new main
  schedule:
    - cron: "0 2 * * *"   # nightly-full, off-peak (illustrative time)
jobs:
  unit-tests:
    runs-on: ubuntu-latest
  integration-tests:
    runs-on: ubuntu-latest
  affected-tests:         # runs on every pull request; not a required check
    if: ${{ github.event_name == 'pull_request' }}
    runs-on: ubuntu-latest
  e2e-smoke:              # one end-to-end environment, in the queue only
    if: ${{ github.event_name == 'merge_group' }}
    runs-on: [self-hosted, e2e-gateway]
  e2e-full:               # job 4411 on the day; a failure here starts red-build.yml
    if: ${{ github.event_name == 'push' }}
    runs-on: [self-hosted, e2e-gateway]
  nightly-full:
    if: ${{ github.event_name == 'schedule' }}
    runs-on: [self-hosted, e2e-gateway]
Queue settingAcme's valueWhy
Build concurrency5Up to five temporary branches are tested at the same time, because each may need an end-to-end environment for e2e-smoke. Section 3 shows that ordinary checks alone would need about 6.
Only merge non-failing pull requestsOnA pull request that fails is removed. It never merges on the strength of the ones behind it.
Status check timeout240 minutesLonger than the longest environment wait plus a 40-minute run. A stuck run then fails the entry and does not block the queue.

Everything before merge worked as designed. The queue proved the combination of five changes on the checks it runs. It could not prove what those checks do not cover, and that is where Act 2 begins.

Act 2 · After merge: a red main#

After merge the problem changes: one failing run covers several changes, and the queue's order no longer says which one caused it.

8. 11:46: why every earlier check passed#

At 11:46 job 4411, the e2e-full run on main, fails. The test checkout_sends_confirmation_email waits for a confirmation email that never arrives: TimeoutError: no confirmation email after 120 s. Main is red. While it stays red, every change in the queue is tested on top of a broken base, and nothing new can reach production (section 14).

The cause sits in two services. The order.created event now carries delivery_note. When the customer gives no note, the value is null. The notifications service, owned by another team, builds the confirmation email and trims the note. Trimming null raises an error, so no email is sent. Each earlier check had a reason to miss it.

CheckWhy it did not see the problem
The agent's stop hook and unit-testsThe orders tests always sent a note. No test ordered without one.
integration-testsThe contract test between orders and notifications checked that the field exists, not that it may be null.
agent-reviewThe reviewer read the diff. The code that breaks is in another service and is not in the diff.
e2e-smoke in the merge queueThe smoke run covers sign-in and checkout. It does not wait for the email.
e2e-full after mergeIt caught it. This is the first check that runs both services and orders without a note.

This is the foundational fact of Act 2. A change can be correct in its own service and still break the system, and only a run of the whole system can see it. The queue's trick does not help here: the failing run covers every change since the last passing e2e-full at a7be113. That is seven changes, the five of the merge group and two earlier ones, and the failing build is main at c41d7e0. Something has to say which of the seven to undo.

9. Two obvious answers, and why neither is enough#

The first answer needs no model: search by halves. Re-run the failing test in the middle of the list of seven changes, then in the middle of the failing half, until one change is left. It is the same search the batched queue uses, and its answer is checked by the test itself. Its cost is time, because each step must wait for the result of the one before it, and each step takes an environment.

The second answer is fast: ask an agent. A model can read the log and the list of changes and name the likely culprit in minutes. But a model's answer is a guess with a confidence score. If a script reverts whatever the agent names, a confident wrong answer reverts good work and leaves the real cause on main. Speed without a check moves the risk; it does not remove it.

How the two compare depends on how many suspects there are. The re-runs here target the one failing test, about 12 minutes each.

SuspectsSearch by halvesTriage agent, then verify with three runs
7 (a busy hour)3 runs in a row: about 36 minutes, and main stays red for all of it7 minutes of model time, then 3 runs: about 19 minutes if the runs go side by side, about 43 if they go one after another (on the day the three runs took 40 minutes, 11:55 to 12:35)
60 (a nightly run)About 6 runs in a row: about 72 minutes for one targeted test, about 4 hours if each step must run the 40-minute suiteThe same 7 minutes and 3 runs. When the claim is refuted, the search by halves starts, and nothing was lost but the 7 minutes.

Read the table honestly. With seven suspects the two paths are close, and the agent path wins only when its re-runs can go side by side. Its clear gain is the case with many suspects. Neither answer alone is the design. The design takes the agent's speed for ranking and the re-runs' honesty for deciding.

10. The loop that holds: name, verify, revert#

The loop has three parts. A triage agent names one suspect. A verifier with no model tests that claim with controlled re-runs. Only a supported claim leads to a revert pull request, and a person approves it.

The triage agent, in one paragraphWhat it reads, what it may do, and where its design lives

The triage agent is a coding agent run headless in a CI job, started by .github/workflows/red-build.yml when the test workflow fails on main. It has three read tools, get_build_log, list_changes and search_bugs, offered by a tool server (a separate program that offers tools to the agent, here through MCP, the Model Context Protocol). It has no shell and no tool that changes anything. For job 4411 it read the failing log, then the notifications service's log, then the list of seven changes, then one diff, then the bug history, and named #4812 at 0.82 in six tool calls: those five reads and one call that loaded a skill. A run that reaches its turn or spend cap produces no claim and counts as "unknown". Its design (parts, prompt, three skills, input, claim and where the confidence comes from) and the call-by-call trace are on Build the agents. Where its files live, the workflow and the start command ci/triage.sh with its flags are on CI/CD agents. All its files are gate files: a change to any of them is a red-lane pull request with its own evals (When the change is an agent).

Diagram: A red build · triage, then verify, then fix

Time runs down the page. The agent's answer is a claim. The verifier, which contains no model, tests it with controlled re-runs. Only a supported claim leads to a revert or a fix, and a person decides.

  1. CI pipeline to Triage agent: Job 4411 failed. A red build on main starts the triage agent.
  2. Triage agent to Verifier: Culprit: #4812, confidence 0.82. A claim, with a score. Nothing acts on it yet.
  3. Verifier: Reproduce, then compare suspect and parent. First the failure is reproduced at the failing build. Then the target runs at the suspect commit and at its parent, with pinned inputs. Fails at the suspect and passes at the parent: supported. An interaction with a later change, a merge commit or an unreproducible old build: inconclusive, and a person looks.
  4. Verifier to Revert script, then red-build fix agent: Supported. Evidence, though not proof: a flaky target or two interacting changes can still mislead. The revert script starts only after this; the red-build fix agent only after main is green.
  5. Verifier to Triage agent: Candidate case for the eval set. Supported and refuted claims become candidate cases. They enter the labelled set once the evidence is firm.
  6. Revert script, then red-build fix agent to Owner: Revert pull request, with the evidence. Revert first: restoring main is urgent. A fix pull request can follow through the normal lanes.
  7. Owner to CI pipeline: Approve and merge. A person decides. Main is green again after one run.

The diagram shows the order, and the point of the order: the agent's answer is a claim, and nothing acts on it until the verifier has tested it.

StepWhoOutput
1. TriageThe CI triage agent reads the failing log, the changes in the build and the bug historyA culprit change and a confidence score, or "unknown"
2. VerifyA plain CI job, no modelFirst reproduce the failure at the exact failing build. If it also fails at the suspected commit, compare that commit with its parent: same pinned artifacts and inputs, same environment image and seeded data. Repeat when the target is known to be flaky. Result: supported, refuted or inconclusive.
3. Revert or fixA script, only when step 2 says supportedA revert pull request by default. A repair comes later, from the red-build fix agent or a person. Both go through the lanes like any change. Inconclusive goes to a person.
4. DecideA personMerge the revert or the fix

Step 2 matters twice. It tests a confident guess before anyone acts on it. And its result is a candidate label for the triage agent's golden set, the past red builds with a confirmed culprit that a change to the agent is tested against (When the change is an agent).

The order of work, and where tokens are spent#

A model is used only where it saves environment time, and only once. Every step that a script can answer runs before it.

StepUses a model?CostThe search ends here when
1. Is this failure already known? Compare the failing test and its error text with open flaky-test and environment records.No. A script.SecondsIt matches. The run is linked to the open record. Nothing else starts.
2. Count the suspects. List the changes since the last passing run.No. A script.SecondsThere is one suspect. Go straight to step 4.
3. Rank the suspects. The triage agent reads the log and the list of changes and names one.Yes. One run for each distinct failure, with a turn cap and a budget cap.A few thousand tokens of log and one diffIt answers "unknown". Go to step 5.
4. Verify the named suspect. Three targeted re-runs.NoThree runs of one testThe verdict is supported. A revert pull request is opened.
5. Search by halves. Used when the claim is refuted or unknown.NoFor 7 suspects, 3 runs, one after another. For 60, about 6.One change is left
6. A person.The verdict is inconclusive: the failure does not repeat, or only two changes together cause it

Five rules keep the token cost of step 3 small.

RuleWhy
One triage run for each distinct failure, not for each failing testOne broken service can fail forty tests with the same error. Group by error text first, in a script.
The tools return parts of a log, never the whole logThe 200 lines around the first error are usually enough.
The agent sees the list of changes first, and one diff on requestSeven full diffs would cost far more than the answer needs.
A re-run of the same failing build reuses the claimThe claim is stored with the build. A retry does not start a second triage.
Caps on turns and on spendA run that reaches a cap answers "unknown" and the search by halves takes over.

The loop on the day#

TimeWhat happensWho actsResult
11:46Job 4411 fails.The CI jobMain is red. Seven suspects.
11:48–11:55The triage agent reads the failing log, the seven changes and the bug history.The triage agent, started by red-build.ymlA claim: culprit #4812, confidence 0.82
11:55–12:35Three targeted re-runs of the one failing test, one after another.The verifier: a plain CI job with no model in itVerdict: supported
12:36A revert pull request, #4840, is opened.A script, because the verdict is supportedA red-lane pull request with the evidence attached
12:44An API owner reads the evidence and approves the revert.A personThe revert enters the merge queue
12:58The revert merges as d2e9b61.The repository hostMain is green for merging. 72 minutes red. No person searched for the cause.
About 13:40 (illustrative)The first e2e-full run after the revert passes.CIFrom then the commits on main are eligible to deploy (section 14)

The three boxes below are the day's evidence, in the order it was produced.

The triage claimWhat the CI triage agent wrote at 11:55. A claim, not yet a fact.
{
  "failing_job": "e2e-full / job 4411",
  "failing_test": "checkout_sends_confirmation_email",
  "failing_build": "main at c41d7e0",
  "culprit": { "pull_request": 4812, "commit": "9f3c2ab" },
  "confidence": 0.82,
  "reason": "The notifications log shows an error while trimming delivery_note, 3 seconds after order.created. Pull request 4812 added delivery_note to that event. The test orders without a note, so the value is null.",
  "evidence": [
    "11:48:02 TimeoutError: no confirmation email after 120 s",
    "11:46:03 AttributeError: 'NoneType' object has no attribute 'strip' (email/confirmation.py, line 61)",
    "#4812 feat(orders): add delivery_note to the order API"
  ],
  "other_suspects": [ { "pull_request": 4816, "confidence": 0.09 } ]
}

The agent only writes this file. It does not revert, does not push and does not re-run anything. A confident wrong answer here costs nothing yet. The claim schema is on Build the agents.

The three verifier runsNo model. The same test, three places in history.
RunCode under testExpected if the claim is rightResult
1The failing build of main, c41d7e0Fails again. If it passes, the first failure was not repeatable.Failed
2The commit of #4812 on main, 9f3c2abFailsFailed
3The parent of that commit, 5d0e8f1PassesPassed

Each run uses the same environment image and the same seeded data, and runs only the one failing test: about 12 minutes each. The verifier does not read the confidence score. The verdict is one of three words: supported, refuted or inconclusive. Here it is supported: the evidence supports the claim. It does not prove it, as section 11 explains.

If run 1 had passed, the verdict would be inconclusive and a person would look. A flaky test and a real defect need different work.

The revert pull requestOpened by a script at 12:36, approved by a person at 12:44
#4840  Revert "feat(orders): add delivery_note to the order API" (#4812)

Why:      e2e-full job 4411 has failed on main since 11:46.
Claim:    culprit #4812, confidence 0.82 (CI triage agent)
Verified: supported. Fails at the failing build, fails at #4812, passes at its parent.
Effect:   the delivery_note field is removed from the API and from the order.created event.
Keeps:    the migration file. A migration that has already run is not deleted.
Safe:     the column is nullable and the old code ignores it. No data is removed.
Lane:     red (api/openapi/). Needs an API owner. No data owner: no migration changes.
Not done: the fix. See ORD-2187.
QuestionAnswer
Why revert and not fix?Restoring main is urgent. The fix is new code written under time pressure, and it touches another team's service.
Why does a person still approve?The revert changes a public contract again. And "supported" can be wrong in rare cases, which section 11 lists.
What if clients already used the field?Then the revert is itself a breaking change. The owner sees that in the evidence and may choose the fix instead. Here the field was on main for under two hours and never deployed (section 14), so no client could have used it.
How the red-build loop is wired on GitHubOne workflow: triage, verify, then a revert pull request

What this shows. The order of the three jobs and what separates them: the triage job writes a claim and acts on nothing, the verify job has no model, and the revert job runs only on a supported verdict. It is an outline you assemble, not an official recipe: steps that pass the claim between jobs are left out. A successful parse proves the shape only.

It listens to the workflow named Run Tests, which holds e2e-full (see the queue wiring in section 7). The triage job's full form, with its permissions and the API key passed from the CI secret store, is on CI/CD agents.

# .github/workflows/red-build.yml  (outline)
on:
  workflow_run:
    workflows: [Run Tests]
    types: [completed]
    branches: [main]
jobs:
  triage:                       # writes claim.json, acts on nothing
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - run: ci/triage.sh "${{ github.event.workflow_run.id }}"   # the workflow run id, not a job id
  verify:                       # no model in this job; skipped when the claim is "unknown"
    needs: triage
    runs-on: [self-hosted, e2e-gateway]
    timeout-minutes: 150
    steps:
      - run: ci/rerun.sh "$FAILING_SHA" "$TARGET"     # must fail again
      - run: ci/rerun.sh "$SUSPECT_SHA" "$TARGET"     # should fail
      - run: ci/rerun.sh "$SUSPECT_SHA^" "$TARGET"    # should pass
      - run: ci/verdict.sh                            # supported, refuted or inconclusive
  revert-pr:
    needs: verify
    runs-on: ubuntu-latest
    steps:
      - run: ci/open-revert-pr.sh                     # only when the verdict is supported

On the day, ci/triage.sh received run 88213, the workflow run that contains job 4411.

Every job in this loop runs with nobody at the keyboard: no one answers a permission question, stops a runaway run or reads untrusted text before the agent does. The controls for that (an allow list of tools, a permission mode that refuses what was not allowed, three caps, untrusted input, credentials, commits from an automation identity) are on CI/CD agents, section 8. Claude Code's cloud sessions also offer a built-in auto-fix that reacts to CI failures and review comments on a pull request; the loop above is a pattern you assemble.

Revert first, repair second#

Once the claim is supported, there are three ways to act on it. Each has a cost.

ApproachWhat it buysWhat goes wrong
Auto-revert on supported culpritMain is green again quicklyA revert can break a change that was built on top of it. A wrong "supported" reverts good work.
Agent proposes a fixKeeps the featureThe fix is new code under time pressure. It needs the full lane.
Agent only reportsSafeA person still does the work; the queue stays red longer

Recommended default: revert first, repair second. Restoring main is urgent; the repair is not. At Acme the revert is opened by a script, never merged by one: a person approves it.

After main was green, the red-build fix agent stopped and handed the repair to the notifications team, listing the two valid repairs; the team chose to accept an empty value, and the next day the developer, with a coding agent, wrote that fix as a normal pull request (why the agent stopped).

11. What the verifier cannot do#

A supported verdict is strong evidence, not proof. Each limit below is a case where controlled re-runs can mislead, and each has a design answer.

LimitConsequence
A pass after removal supports the claim; it does not prove itA flaky test, a provider sandbox that recovered, or time-dependent data can make the target pass for another reason. Hence reproduce first, and repeat.
Compare the suspect with its own parent, not with the tip of mainThose two commits differ by exactly one change. Comparing the parent with the tip removes every later change too. Reverting on the tip is a third option, but on a busy branch the revert often conflicts.
Two changes can fail only togetherThen the suspect commit passes alone, and the failure exists only in the later build. Compare two builds on that later base, with and without the suspect. If that is not possible, the result is inconclusive.
The failing test was added in the suspect commitThe test does not exist at the parent. Run the new test against the parent's code: if it fails there too, the commit exposed an older defect and is not the culprit.
History is not always a lineA merge commit has two parents; the job must be told which one. Old builds may no longer be reproducible: the environment image or the seeded data of that day may be gone. Both cases are inconclusive, not refuted.
It costs end-to-end runsThree targeted runs per red build, more for a flaky target. Compare that with the runs wasted while main stays red.

12. Turning a red main into a cheaper check#

The loop restored main in 72 minutes. The better outcome is that the same kind of mistake never reaches main again. A natural first idea is to run the whole pipeline on the developer machine before the pull request. It helps with some failures and cannot help with others, and the reason is the same each time: the developer machine does not have what the failure depends on.

Why a build failsCan the developer machine catch it?What catches it
1. The change is wrong in its own code (lint, types, its own tests)Yes.The agent hooks and the git hooks from stages 1 and 2. This class should never reach CI.
2. The machine differs from CI (tool versions, operating system, environment variables)Mostly.Run the cheap tiers inside the same container image that CI uses. The image tag is pinned in the repository, so both sides move together.
3. The change fails only together with another change that merged in the meantimeNo. The other change may not exist yet when the developer runs the tests.The merge queue. It tests the real combination, in order.
4. The failure needs the whole system (several services, real queues, a provider sandbox)No. A laptop does not run the whole system with seeded data and provider accounts.An end-to-end environment. Pick the suites by what the change touches.
5. A flaky testNo. It passes locally by chance.Quarantine the test and give it an owner.
6. Something outside the code (a provider outage, expired test data)No.The environment scheduler marks it as an environment fault. It is not charged to the change.

So the honest answer has three parts.

PartWhat to do
Move down what can move downClasses 1 and 2 belong on the developer machine, in the CI container image. Every failure caught there saves a CI run and a queue slot.
Do the rest in CI, but before the mergeThe merge queue is the rehearsal: it runs on CI machines, on the real combined content, before main moves. A developer or an agent can also start the CI pipeline on a branch before opening the pull request. That is a remote run in the real environment, with no reviewer waiting.
Turn each red main into a cheaper checkWhen a failure reaches main, ask which lower tier could have seen it, and add that check.

The first two parts lower how often main goes red; only the third lowers it for the next failure of the same kind, so it is the one to apply after every red build.

Job 4411 was class 4: it needed two services, and no local run would have found it. But two cheaper checks could have.

Cheaper checkTierWhy it would have caught it
A contract test that sends nullintegration-tests, minutesThe notifications team publishes what it accepts in the order.created event. A test with delivery_note set to null fails at once.
Suite selection by affected flowe2e-smoke, in the merge queueThe change edited the order.created event. A map from that event to the flows that read it would have added the confirmation-email suite to the smoke run for this pull request.

The next day's fix pull request adds the first one. A null note in order.created now fails in the integration tier, in minutes, before review ends.

13. The same stage, three ways it can end#

#4812 took the longest path. Most changes take the first.

OutcomeWhat it looks likeWhat happens next
1. Everything passesSibling #4819, the rounding fix in the cart, merged at 10:21. e2e-full on that main passed.That commit may deploy, in steps (section 14). Nobody is told anything.
2. A check fails before merge and an agent fixes itThe checks fail on the temporary queue branch. The queue removes that one pull request. The others continue.The author's agent reads the failing log and pushes a fix to the branch. The pull request joins the queue again. After two agent retries of an end-to-end run, a person must approve the next one.
3. Main goes red after merge#4812, above.Triage, then verification, then a revert that a person approves. The repair comes later, as a normal pull request, after a person chooses it.

14. Deploy only what passed the full run#

The last step from diff to production has one rule at Acme: product code on main deploys to production only after e2e-full passes on that exact commit, and then it rolls out in steps. Passing the merge queue is not enough, because the queue runs only e2e-smoke, and Act 2 showed what the smoke run can miss.

The rule decided how the day ended. #4812 reached main at 11:02. Its e2e-full run failed at 11:46, so its commit never became eligible to deploy, and the revert merged at 12:58. The field was on main for under two hours and was never deployed. No customer missed a confirmation email. The rule also has a cost: while main is red, no newer commit is eligible either, so the other six suspects waited for the first green e2e-full after the revert. A red main delays every team's deploy, which is one more reason the loop in section 10 is urgent. The next-day fix deployed the same way, after its own green e2e-full.

A rollout in steps sends the new version to a small share of traffic first, watches it, and widens only while the signals stay healthy. Agents change the volume of deploys, not this practice.

PracticeWith agents
Progressive rolloutUnchanged. More changes per day makes small, reversible deployments more important, not less.
Change recordGenerated from the pull request evidence: intent, lane, tests, approver.
RollbackA person decides. An agent can prepare the rollback pull request and the evidence.
Post-deploy watchAn agent can watch the metrics named in the pull request's risk note and report. It does not decide.

15. Side questions#

QuestionShort answer
Is batch-and-split worth it?If failures are independent, a batch of n passes with probability (1 − p)ⁿ: at 5% and five changes, about 77%. The other 23% pay extra runs to split, more when two changes are bad or fail only together. Measure runs per merged change and the 95th-percentile wait under your own split policy before choosing a batch size.
What does the verifier cost?A few targeted runs per red build. Usually less than the runs wasted while main is red; measure it.
Can the triage agent be wrong with high confidence?Yes. That is what calibration measures, and why the verifier does not look at the confidence score.

The two halves of this stage look alike, since both search for one bad change among many, but they rest on different kinds of evidence. Before merge, the queue's order turns every build result into an answer, so isolation is arithmetic and needs no judgment. After merge, the failing run covers many changes at once, and the fastest answer is a model's guess. The guess is worth having because it is fast. It is safe to act on only because a job with no model in it tests the guess first, and a person still approves what follows.

By 12:58 main is green, the field is off main and was never deployed, and the claim that named #4812 has been tested by three runs with no model in them. The weak point left is the agents themselves. The triage agent and the reviewer both ran in CI jobs with nobody watching, and they behaved as designed only if the runner loaded the files the team meant it to load. Where do those files live, and what does a runner actually load? Next: CI/CD agents.