All chapters
  1. Start hereOne change, from diff to production
  2. Before stage 1The team setup: what changes when you are not alone
  3. Stage 1Code change: catch it while the agent is still editing
  4. Stage 2Commit: what must be true before a commit exists
  5. Stage 3Pull request: arrive already reviewed, with evidence
  6. Stage 4Review: who reviews what, at hundreds a day
  7. Stage 5CI/CD: spend scarce capacity like a budget
  8. Deep diveCI/CD agents: where they live and how they run
  9. Deep diveBuild the agents: pinpoint the change, then fix it
  10. Stage 6When the change is an agent
  11. OperateVolume and cost: what breaks at 300 a day
  12. AdoptAdopt it in steps
  13. ReferenceHook events: which one, with a script for each
  14. ReferenceGlossary
  15. AboutAbout this site

AI coding agents in a real delivery pipeline · approaches weighed

One change, from diff to production

AI (artificial intelligence) coding agents make writing a change cheap. They do not make verifying it cheap. This site follows one change from its first diff to production, whether it was written in an editor or in a terminal agent session, and weighs the approaches at each stage, with the drawback of each.

It is written for the hard case: hundreds of pull requests a day, and tests that need a complete running copy of the system.

Start with the team setup →

At 09:12 on an ordinary Tuesday, a developer at Acme, a software company with one large repository, hands a coding agent a small task: add an optional field, delivery_note, to the order API, so a customer can leave delivery instructions. By 09:31 the agent has edited six files, and every check on the developer machine passes. At 09:40 a reviewer agent finds two problems, one of which only an owner of the public API may decide, and at 10:48 two owners approve pull request #4812. At 11:02 it merges with four other changes, after a short end-to-end test on one of Acme's 12 complete running copies of the system.

At 11:46 the full end-to-end run on the main branch fails. A checkout test waits for a confirmation email that never comes: when a customer leaves no note, the new field is empty (null), and the notifications service, which nobody touched, fails on it. A triage agent names #4812 as the likely cause, re-runs with no model in them support that claim, an owner approves the revert, and main is green again at 12:58. The change was on main for under two hours and was never deployed, because production waits for that full run to pass.

Nobody in that story was careless, and no check failed to do its job. Writing the change took the agent nineteen minutes; verifying it took two owners' attention, one of 12 scarce test environments, a triage run, three re-runs of a 12-minute test, and 72 minutes in which main was red and ordinary work could not merge. Acme opens about 300 pull requests a day, many written with a coding agent, so the question this site asks is where that verification cost lands, and how to make it land earlier and cheaper. The times and counts are illustrative, here and on every page, unless a source is linked.

The claim. When agents write the code, the bottleneck moves from typing to verifying. Every stage of the pipeline has one job: move verification earlier, make it cheaper, and keep the part that must stay with a person small enough for a person to do well.

The path one change takes#

#4812 passed through the same stages as every change at Acme: the code change inside the agent's editing loop, the commit, the pull request, the review, and CI/CD (continuous integration and delivery: the automated pipeline that builds, tests, merges and releases). Each stage has its own checks, and each check sees a different part of the problem.

Diagram: The path of one change · each stage and where its checks live

Read left to right. Each column is a stage. The top box is what happens; the boxes under it are the checks this site puts there. The further right a problem is found, the more it costs. Click any box.

  1. 1 · Code change
    1. Agent edits files: The coding agent reads and edits files to do the task.
    2. Check after each edit: A hook runs ruff on the changed file and sends failures back to the agent.
    3. Tests before it stops: A stop hook runs the type check and the affected tests. Red tests send the agent back to work once.
  2. 2 · Commit
    1. Commit and push: The agent stages, commits and pushes its own branch.
    2. Guard before the push: An agent hook and deny rules stop secrets, protected paths and force pushes.
    3. Server rules: Branch rules, required checks and code owners on the repository host. Code owners on the workflow files keep a pull request from rewriting its own checks. The gate to rely on.
  3. 3 · Pull request
    1. Review before opening: A headless review of the diff before anyone else is notified: on the developer machine for feedback, on a trusted runner for a verdict.
    2. Signed review receipt: A record of exactly what a trusted runner reviewed. CI verifies the signature and the inputs instead of repeating the review.
    3. Pull request with evidence: Intent, lane, tests run, findings fixed or declined, risk note.
  4. 4 · Review
    1. Lane from changed paths: CI computes green, yellow or red from what the change touches.
    2. Reviewer agent on everything: An agent with no edit tools, in a fresh context, reports a few verified findings.
    3. People on yellow and red: One approval for yellow. The named owner, with a reviewed plan, for red.
  5. 5 · CI/CD
    1. Merge queue, cheap tier first: Each change is tested on top of those ahead of it. Unit tests, then integration tests, before any end-to-end environment.
    2. Scarce environments, by class: End-to-end runs only for changes that passed everything else, with a share and a maximum wait per class.
    3. Deploy and watch: Only after e2e-full passes on that main commit; then a progressive rollout. A person decides on rollback.

Read it left to right. Each column is a stage, and the boxes under it are the checks this site puts there; open any box to see it at work on #4812. The further right a mistake is found, the more it has already cost: the missing length check that the reviewer agent caught at 09:40 cost one small fix commit, while the empty note found at 11:46 cost a red main.

Where the verification cost lands, stage by stage#

Before agents, the slow step was writing the change. Review and CI were sized for that pace. With agents, a developer can open several pull requests in the time one took before, and nothing downstream got faster on its own.

The obvious response is to keep the pipeline and add more of what it already has: more reviewers, more test environments. Neither grows with the change rate. People asked to read three times as many diffs start to skim, and an end-to-end environment needs every service, a seeded database and sandbox accounts with outside providers, so adding one is a project, not a click.

So each stage on this site does something else. It moves a check earlier, to where it is cheaper: into the agent's editing loop, or before the pull request opens. It sorts changes by blast radius, how much can go wrong and how far it spreads, into a green, yellow or red lane, so that people read the changes that need a person. And it treats scarce machine time as a budget to spend, not a queue to wait in.

One row in the table below is new. The agents in the pipeline are themselves edited by pull requests, and an edit to an agent's instructions changes behavior that the diff does not show.

StageBefore agentsWith agents, unchanged pipelineWhat this site proposes
Code changeThe developer finds mistakes while typingThe agent writes 400 lines before anything checks themFast checks inside the agent's editing loop: after each edit, and before the agent stops
CommitCommit hooks catch format and secretsThe same hooks, now hit far more oftenThree layers: agent hooks guide, git hooks guard, and only the server's rules are the gate
Pull requestOpened, then reviewedOpened unreviewed; the queue growsIt arrives already reviewed. A review signed by a trusted runner (an attested review receipt) lets CI verify it instead of repeating it.
ReviewEvery change read by a personPeople skim, or the queue stallsLanes by blast radius; people read the lane that needs them, and owners keep the decisions that are theirs
CI/CDCapacity matched the paceScarce end-to-end environments fill with runs that were going to fail anywayEnd-to-end time spent as a budget (merge queue, test selection, priority), and a red main handled by a verified fix loop
Agent changesDid not existSkill and prompt edits merge on a skimskill-lint, trigger-test, agent-smoke and agent-eval: checks that measure behavior, built to be fast and honest

Read down the last column and one pattern repeats. No row removes the person; each makes the checking before the person earlier and cheaper, so what reaches a person is smaller and already argued. On the day of #4812 that is what happened: local checks caught a lint error and a failing test before the commit, the reviewer agent caught the missing length check before any owner looked, and the owners spent their attention on the one decision only they could make, whether the new field was required.

The same day also shows the limit. The empty note passed every check before merge, because the orders tests always sent a note, the contract test only checked that the field exists, and the short run in the merge queue does not wait for the email. That is why a wider run follows the merge, why production waits for it, and why a red main needs a loop that is fast and checked, not a person reading logs under pressure.

The running example#

Every page uses the same fictional team. Acme has a platform team that owns one large repository: a web application, a public API, and backend services for orders, cart, payments, notifications and sign-in (services/auth).

Volume
300 pull requests a day, many written with a coding agent.
Scarce resource
12 end-to-end test environments. A full end-to-end run takes 40 minutes. An integration tier runs first.
The change we follow
Add the optional field delivery_note to the order API. Work item ORD-2187, pull request #4812, six files, red lane.
The agent in production
A CI triage agent: given a failing job, it names the likely culprit change and a confidence score.

The change we follow. A customer wants to add delivery instructions to an order. The change adds an optional text field, delivery_note (at most 500 characters), to POST /v1/orders, stores it, returns it on the order, and carries it in the order.created event, which is how the notifications service came to see it. It touches six files on the branch feat/order-delivery-note, and two of them are high-risk paths: the public API contract and a database migration.

Paths set the minimum lane, so pull request #4812 is in the red lane, where the owners of those paths must approve. Each stage page shows this one pull request passing through the stage, with the pass case and the fail case. The full rules for lanes are on Review.

An end-to-end test environment is a complete running copy of the system: every service, a seeded database, queues, and sandbox connections to the outside providers for payments, email and sign-in. A test there follows a real user flow from start to finish, which is why the missing email surfaced there and in no earlier check. There are only 12, and CI/CD explains why. That number is what makes this example hard.

Five ideas this site adds#

Most of each page weighs familiar practice under new pressure. Five ideas go further. Each tile opens the stage that argues it, with the approaches it was weighed against and what each one costs.

Where to begin#

New to working in a large team, or coming from solo work? Start with the team setup: what exists, where each rule lives, and one change followed from the first edit to a green main. Otherwise, start at the problem you have today.

If you areStart at
Setting up one repository for a coding agentCode change, then Commit
Drowning in pull requestsPull request, then Review
Short on CI capacityCI/CD
Running agents inside CICI/CD agents, then Pinpoint and fix agents
Maintaining agents and skills in productionWhen the change is an agent
Asked what it will costVolume and cost
Planning the rolloutAdopt
A design, not a standard. Each page lists several approaches and what goes wrong with each. Take the shape and tune it. Counts, durations and costs are illustrative unless a source is linked.

The day of #4812 ended with main green and nothing deployed, not because any one check was clever, but because each stage asked the cheapest question it could answer and passed on only what it could not. That is the whole design: let machines answer every question that has a mechanical answer, as early as they can, and spend people on the questions that need an owner.

Product details on this site were checked against the Claude Code documentation on 15 November 2026. Other coding agents have similar features under other names; the stages and trade-offs carry over.

Next: before the first edit, the rules that every change at Acme meets, and where each one lives. The team setup.