By 12:58 on the day of #4812, main was green again and the change had passed through every stage. It is worth going back over that day once more with a different question: how many times did a model run because of this one change? Not the hooks, which ran dozens of times and called no model. The runs where an agent read text, reasoned and answered, each one paid for in tokens.
There were about ten. That sounds small until you remember that #4812 was one of about 300 pull requests Acme opened that day. At that volume the bill, the rate limits and the reviewers' patience all start to break, and they break at the same time, because they are driven by the same few decisions. The numbers on this page are illustrative unless a source is named.
Cost is an engineering property. Four dials control it: when the agent runs, what it reads, which model does the work, and the cap.
1. Every model run behind one change#
The table follows #4812 from 09:12 to 12:58, plus the one run that happened after main was green. The caps come from the pages where each run was set up.
| Time | Run | Who started it | Model? | Caps |
|---|---|---|---|---|
| 09:12 to 09:31 | The editing session: six files, one lint fix, one test fix | The developer, at the keyboard | Yes | A person is watching |
| 09:16, 09:29, 09:31 | The after-edit hook and the Stop hook | The harness (the software that runs the agent and fires its hooks) | No | 20 and 180 seconds |
| 09:34 | The local review in the pre-push git hook, --agent code-reviewer | The developer's push | Yes | 15 turns, 2 USD |
| 09:40 | agent-review, round 1, on 8c1f0a4: two findings | The pull request opening | Yes | 15 turns, 2 USD, 15 minutes |
| 09:41 to 09:44 | The fix-on-request agent fixes finding 1 | The developer's comment | Yes | At most three rounds per finding |
| 09:44 | agent-review again, on 3e9b7d2 | The new commit | Yes | 15 turns, 2 USD |
| 10:20 | The fix-on-request agent applies the API owner's decision | The developer's comment | Yes | At most three rounds per finding |
| 10:31 | agent-review, round 2: no findings | The new commit | Yes | 15 turns, 2 USD |
| 10:50 to 11:02 | lane and agent-review in the merge queue | The merge_group event | No: they report success at once | |
| 11:48 to 11:55 | The triage agent names #4812, confidence 0.82 | The red build on main | Yes | 25 turns, 4 USD |
| 11:55 to 12:35 | The verifier: three re-runs | The claim | No | Three runs |
| 12:36 to 12:58 | The revert #4840 is a pull request too, so its required checks run, agent-review among them | The script that opened it | Yes | 15 turns, 2 USD |
| After 12:58 | The red-build fix agent reads the supported claim, finds two valid repairs, one in another team's service, and hands over | The supported claim, once main is green | Yes | 40 turns, 6 USD |
Two things stand out. First, the runs that made the day's hardest decisions, the hooks, the merge queue and the verifier, called no model at all. Second, the runs fall into two groups that grow very differently with volume.
The first group happens once per change, or once per push: the editing session, the local review, the CI reviews and the fix-on-request runs. That is seven of the runs above. The second group happens once per red build: the triage run, the review of the revert and the red-build fix agent. Acme has a few red builds a day, not 300.
So "multiply by 300" is right for the first group only. If every change looked like #4812, the first group would be 300 editing sessions plus 1,800 unattended runs a day. Most changes are simpler: the typo fix #4822 and the rounding fix #4819 each got one review and no fix.
A better estimate counts reviews per push. With about 2.5 pushes per pull request, the figure the CI/CD page uses, that is about 750 CI reviews and 300 local reviews a day, against a handful of triage runs.
The caps put a ceiling on the first group, not a price. 750 CI reviews with a 2 USD cap each could spend up to 1,500 USD a day before any of them stops. What they actually spend depends on diff sizes, and has to be measured (section 5). In Claude Code, the --max-budget-usd cap is checked against a client-side estimate that can differ from the bill, and the documentation says spend can pass it, so it is a guard rail, not an accounting control.
2. 09:30: the hour when CI takes the developers' capacity#
The cost of a day is one problem. When that cost arrives is a second one, and it shows up before the bill does.
Pull requests do not open evenly through the day. Many developers start in the morning, finish a first task and push within the same hour, as the developer of #4812 did at 09:34. Each push starts a local review, and each new pull request starts a CI review, and all of them call the model provider at once.
If the developers' interactive sessions and the CI jobs authenticate with API keys in the same organization, they share one limit: the Claude Code costs documentation states that these rate limits apply at the organization level, not per individual user, and the API documentation says limits are set for the organization, with optional lower limits per workspace. A burst of CI reviews then slows or fails the sessions of the people who are trying to work.
The documentation's rate-limit recommendations for teams lower the tokens per minute per user as the team grows, because in a large organization fewer people use Claude Code at the same moment. CI does not follow that pattern. It is driven by pushes, and pushes cluster. That is an inference from the documented reasoning, not a documented figure, but it explains why a limit that is comfortable for 300 people can be too small for 300 people plus their pipeline.
There are two controls, and they work together.
| Control | How it works | What goes wrong |
|---|---|---|
| Separate the meters | Give CI its own workspace and API key, and set a workspace rate limit and spend limit on it. The Claude Console supports both per workspace, so CI can use at most its share. | CI reviews now queue among themselves at the peak instead of taking the developers' share. A pull request waits longer for its review. |
| Queue by class | A gateway in front of the model provider orders requests: triage of a red main and release work first, people's interactive work next, routine reviews when there is room, with a longer wait after each rate-limit reply. This is a design, not a product feature. | One more service that every agent run depends on, and one more thing to be down at 09:30 |
How developers sign in changes the picture. On Claude for Teams and Enterprise plans, a member's interactive use draws from a per-seat allowance, while CI should use an API key (section 10), which is billed per token. The two meet at one limit only when both run on API keys in one organization.
To see whether CI is hitting limits, read the retry events: with --output-format stream-json, Claude Code emits a system/api_retry event before each retry, and its error field says rate_limit when that was the cause.
3. Where an agent runs in this pipeline, and which runs need a model#
Section 1 followed one change. The same runs, seen across the whole day, show where a model belongs and where it does not.
| Run | Frequency at 300 pull requests a day | Sensible model class |
|---|---|---|
| Editing session | Per developer, all day | Strong model |
| Hooks after each edit | Thousands | No model at all |
| Local review before push | About 300, plus re-pushes | Mid model |
| CI review | About one per push; fewer with attested receipts | Mid model |
| Fix-on-request runs | Some | Mid to strong |
| Triage of a red build | A few a day | Strong model |
| Red-build fix agent | Only on a supported claim | Strong model |
| Evals for agent changes | Only when an agent changes | The model under test, plus a small judge |
Read the frequency column against the model column. The runs that happen thousands of times need no model, because a linter and a test runner answer them exactly and in seconds. The runs that happen hundreds of times need a mid model, because they read a diff against rules. Only the rare runs justify the strongest model, because a wrong answer there is expensive.
The Claude Code cost guidance makes the same split for its own models: Sonnet for most coding work, Opus for complex reasoning, and model: haiku for simple subagent tasks. The reviewer agent at Acme is defined with model: sonnet.
4. The four dials, each seen on #4812#
Every dial below changed the number of model runs in section 1, or their size. The diagram shows the four in the order they act: a run that never starts costs nothing, so the earlier dials save the most.
Diagram: Four dials · when, what, which model, cap
Read left to right. Every run of an agent in the pipeline passes these four questions. The earlier dials save more: a run that never starts costs nothing.
- When
- On an event, not a timer: A push, a red build, a comment. Never a polling loop.
- Once per change: Fewer fix rounds, and a signed receipt that lets CI skip a second review of unchanged inputs.
- What
- The diff first: The agent starts from the change and fetches what it needs.
- Subagents for wide searches: A side search runs in its own context and returns a summary.
- Which model
- Small: lane, classification: Cheap and fast, for narrow decisions.
- Mid: review: Most reviews.
- Strong: triage, hard fixes: Where a wrong answer is expensive.
- Cap
- Turn limit + run budget: Every unattended run has both.
- Team spend limit: A monthly limit per team, with your own alert before it is reached.
- Measure: Cost per merged change, reuse rate, cap-hit rate.
When. #4812 was reviewed three times in CI because it received two fix commits. Each fix round costs one more review, so the cheapest review is the one a fix round did not need. The workflow's concurrency group cancels the review of a commit that a newer push has replaced, so a burst of pushes pays for one review, not several. An attested review receipt, as for #4819, lets CI check a signature instead of reviewing again.
What. The reviewer starts from the diff and fetches what it needs, instead of loading the repository. The triage agent's tools return 200 lines around the first error, never a whole log, and one diff on request, never all seven. Wide searches go to a subagent, so its output stays in its own context and only a summary returns.
Which model. Section 3 above. Routing is a rule someone writes, and a wrong rule sends a hard case to a weak model.
Cap. Every unattended run in section 1 had a turn cap and a spend cap, and the CI jobs a timeout. A cap that is hit is not free: a run that stops halfway has produced nothing. The triage design turns that into a defined outcome, because a capped run answers "unknown" and the search by halves takes over.
| Dial | Lever | Drawback |
|---|---|---|
| When | Run on an event, not on a timer. Review once per change, not once per push; fewer fix rounds and the attested receipt are this dial. | Event wiring to maintain |
| What | Give the agent the diff and let it fetch what it needs, instead of loading the repository. Use subagents for wide searches so the main context stays small. | Too little context produces confident, wrong findings |
| Which model | Small model for classification and lane checks, mid for review, strong for triage and hard fixes | A wrong routing rule sends a hard case to a weak model |
| Cap | A turn limit and a budget on every unattended run. A spend limit per team or workspace. | A capped run that stops halfway has produced nothing; watch the cap-hit rate |
5. How to estimate your own number#
No page can tell you what a review costs on your repository, this one included, because the cost depends on your diff sizes and your rules. Measure it.
- Run the review on 50 real pull requests with
--output-format json. Each result includestotal_cost_usd, Claude Code's client-side estimate of the run's cost, and a per-model breakdown. - Take the median and the 95th percentile. Large diffs dominate; that is an argument for small pull requests.
- Multiply by the number of reviews that actually run, after receipts and lanes: per push, not per pull request (section 1).
As a reference point, the Claude Code documentation reports an average across enterprise deployments of about 13 US dollars per developer per active day, 150 to 250 US dollars per developer per month, and under 30 US dollars per active day for 90% of users. That is a per-developer figure for people using Claude Code, not a CI budget; review and CI runs depend on your diff sizes. The same documentation recommends starting with a small pilot group to establish a baseline before a wider rollout, which is the method above.
6. Caching: why the same rules cost less the second time#
Every review at Acme sends the model the same system instructions and the same review rules before it sends the diff. Paying full price for that repeated text on every turn would be wasteful, and a prompt cache is what avoids it. The provider keeps the start of a recent request, and when the next request begins with exactly the same text, that part is read from the cache and billed below the standard input rate.
Two facts decide how much this saves in CI. The match is exact and starts at the beginning, so a change anywhere early in the request means everything after it is processed again. And in Claude Code the cache is effectively scoped to one machine and working directory, because the start of each conversation names the directory, the platform and the git branch.
Reviews of different pull requests start from different branches, so expect little sharing between them. The cache does most of its work inside one run, where each turn re-sends all the turns before it.
| Fact (Claude Code documentation) | Consequence |
|---|---|
| Each model has its own cache | A job that switches models in the middle pays to rebuild it |
The default lifetime is five minutes on an API key or a cloud provider; promptCacheTtl set to 1h gives the main conversation an hour, at a higher price for cache writes | Reviews spread thinly through the day share less than reviews that run close together. The one-hour setting pays only when runs come back to the same prefix after a pause. |
Subagents use the five-minute lifetime unless it is set separately with subagentPromptCacheTtl | A review that delegates to a subagent warms a second cache of its own |
| Effectively one machine and directory; parallel sessions in the same directory can read each other's cache | Separate CI runners rarely share. --exclude-dynamic-system-prompt-sections moves per-user context out of the system prompt, which the documentation says improves reuse across machines running the same task. |
The JSON result reports cache writes (usage.cache_creation), and OpenTelemetry reports cache read and creation tokens | You can measure the hit rate instead of guessing |
7. Limits that are not about money#
The morning peak in section 2 is one limit that has nothing to do with the bill. Three others arrive at the same volume, and each one has its own owner on this site.
| Limit | Symptom | Control |
|---|---|---|
| Provider rate limits | Reviews fail or queue at the morning peak, and developers lose their own sessions to CI traffic | A separate workspace with its own rate limit for CI, and a queue by class in front of the provider (section 2). Drawback: one more service that every agent run depends on. |
| Your own tool endpoints | Parallel agents overload an internal service | A shared cache and a per-endpoint limit, as on When the change is an agent |
| Scarce end-to-end environments | The end-to-end queue grows through the day | The budget rules on CI/CD |
| Reviewer attention | People stop reading agent comments | Cap findings per review; verify before posting; track how many findings lead to a change (Review) |
8. Twelve problems that arrive with volume#
Most teams plan for cost and for review load. The twelve below arrive later, and each can stop a pipeline that was working at lower volume. Most of them are explained in full on an earlier page; the last column links there, and this table is the at-volume checklist.
| # | Problem | How it shows | What to do | Where it is explained |
|---|---|---|---|---|
| 1 | A flaky test removes innocent changes from the merge queue | Good pull requests are removed, and every build behind them runs again | Run a failed test once more inside the job. Record a pass on the second run as a flake, with an owner and a deadline. | CI/CD, the queue's rules |
| 2 | Owners become the queue | Red-lane pull requests wait hours for one of three people | A rota with a stated answer time. Review the plan before the code exists, so the code review is short. More owners for narrow paths. | Review, lanes |
| 3 | Agents open more pull requests than people can read | The open count grows every day | A limit on open pull requests for each author and each agent identity. One task, one pull request. | Review |
| 4 | Merge conflicts rise with the number of open branches | Pull requests go stale while they wait | Small changes, short-lived branches, and an agent that rebases a waiting branch when main moves | This page |
| 5 | Approvals go stale | A change is approved, then changed, then merged with the old approval | Dismiss approvals on a new commit. Keep fix commits small so the second approval is quick. | Review, the branch rule |
| 6 | An urgent fix must jump the queue | Jumping makes every running build start again | One emergency path, used rarely, with a record of who used it and why | CI/CD, the queue's rules |
| 7 | The shared build cache holds a wrong result | A build passes in CI and fails for everyone else, or the reverse | Only builds of trusted branches may write to the cache. Pull request builds read from it. | CI/CD, lever 1 |
| 8 | Pull request code reaches a secret | A workflow that runs untrusted code also holds a deploy key | Jobs that run pull request code get no deploy secret. A job that holds one never checks out pull request code. | CI/CD agents, nobody at the keyboard |
| 9 | A revert breaks what was built on top of it | The revert merges and a later change stops compiling | The revert goes through the merge queue like any change, so the combination is tested first, as #4840 did | CI/CD, revert first |
| 10 | Deploys carry too many changes to search | A bad deploy has forty suspects | Deploy often, in small groups, in steps. Cap the number of changes in one deploy. | CI/CD, deploy |
| 11 | An agent in the pipeline gets worse after an edit to its instructions | The triage agent's claims are refuted more often than last month | Treat prompt and skill edits as red-lane changes with the agent checks skill-lint, trigger-test, agent-smoke and agent-eval | When the change is an agent |
| 12 | Text in a diff or a log gives the agent an instruction | A review comment or a claim follows a sentence that someone planted | No edit tools for agents that read untrusted text, and no secret they do not need. Acme's review workflow loads its rules from the pull request's checkout, protected because those files are gate files with an owner; loading them from the base branch is stronger. | CI/CD agents, the self-editing reviewer |
9. What to measure#
Every number in section 1 was reconstructed after the fact. A team running 300 changes a day needs them every day, recorded as they happen.
| Measure | Why |
|---|---|
| Cost per merged pull request, by lane | The number a budget owner understands |
| Reviews per pull request, fix rounds per pull request, and receipt reuse rate | Whether reviewing before the pull request is paying off; section 1 showed that reviews follow pushes |
| Finding acceptance rate | Share of agent findings that led to a code change. Low means noise. |
| End-to-end runs per merged change | Whether the CI budget is working |
| Green-lane audit failures | Whether the lane rules are right |
| Defects that escaped, by lane and by agent involvement | The outcome all of the above exists to protect |
| Cap-hit rate | Runs that stopped at a limit and produced nothing |
| Rate-limit retries by hour | Whether CI is crowding out people at the peak |
Claude Code can export metrics through OpenTelemetry, including per-user token counts, costs and tool activity, so these can live in the monitoring system the team already has. The documentation describes it as the option that works with every way of signing in.
10. Side questions#
| Question | Short answer |
|---|---|
| Seats or metered usage? | People working interactively use seats. For services such as CI, use metered API access, directly or through a cloud provider. The Claude Code documentation allows a subscription token in CI, but notes it is tied to the person who created it, so it recommends an API key for a secret shared across repositories. |
| Does this lock us to one model? | The stages, lanes, receipts and evals are independent of the model. The eval set is what makes a model change safe to try. |
| What is the first cost mistake teams make? | A model call inside the edit loop, or a full review on every push. |
Look again at the day of #4812. The hooks caught two mistakes, the merge queue isolated the change, and the verifier confirmed the culprit, and none of them called a model. The model runs did the work only a model can do: reading a diff against rules, naming a suspect among seven. At 300 changes a day, cost is not mainly the price of a call but the number of calls, and the cheapest call is the one that never runs because an earlier, cheaper check already answered the question.
Next: knowing what each step costs, the question becomes the order in which to turn the steps on: Adopt.