How to Review AI-Generated Code Without Rubber-Stamping It

To review AI-generated code without rubber-stamping it, sort each pull request by what a wrong merge would cost. Require evidence tied to the exact commit before a person reads it. Keep a named approver and a release owner, who also owns the rollback, on every change that touches your public API, SDK behavior, auth, or money. Low-risk changes with complete evidence can move on a lighter review. Everything else goes to someone who can say what the checks would miss.

AI-assisted pull requests can outpace the people who review them. Anthropic reports, for its monorepo alone, around 2,500 code changes merged per day in May 2026 and over 30,000 per day as of September.

In an Ask HN thread on shipping AI-written code you don't understand, commenters split between reading every line, testing more, and matching scrutiny to consequence. The third approach needs written rules: what counts as high consequence, what evidence counts, and who signs.

What changed in the review job

As Brian Houck at DX, a developer productivity research company, puts it, code review is also "how teams build shared understanding of a system." When the author wrote every line, the reviewer could assume the author understood it. An agent that writes the code and its tests breaks that assumption. thoughtbot notes that the person who prompted the change may have read the tests less closely than the implementation.

Anthropic's rules for its Claude Code repos say "Whoever prompted the change is on the hook for it," and an agent's approval never counts as a person's. The OpenSSF guide for AI code assistants agrees: AI-generated code is no shortcut around code review, testing, or static analysis, and you are responsible for any harm it may cause.

Four decisions stay with people:

• Whether the change should exist, and what "done" means.
• Which tier it belongs in when its consequences exceed what its paths suggest.
• Approval on high-risk changes.
• Who owns the release, and how it gets undone.

Keep three layers of review separate

Keep three kinds of signal apart, because each fails in its own way.

Deterministic checks

Tests, type checks, security and dependency scans, and breaking-change diffs give the same answer every time for the same commit. At an API company the contract diff carries the most weight. oasdiff compares two OpenAPI specs, and its breaking command shows only the changes that break existing API clients. buf breaking compares your Protobuf schema against a past version and reports failures as PR comments.

Watch one GitHub detail: a required status check passes with a successful, skipped, or neutral status. A skipped check satisfies the rule, so confirm the check ran on the commit under review.

AI review advice

Cloudflare built an AI reviewer for its merge requests and writes that it "isn't a replacement for human code review, at least not yet with today's models." GitHub says Copilot is not guaranteed to spot all problems in a pull request and should be supplemented with human review. Copilot's reviews do not count toward required approvals by default. Admins can turn that on in repository, organization, and enterprise settings. Treat that approval as advice with a narrow scope.

Human judgment

Cloudflare lists limits of its own AI reviewer. Race conditions that depend on timing or ordering are hard to catch from a static diff. It can flag an API contract change, but it cannot verify that every downstream consumer was updated.

Green checks do not prove semantic correctness. thoughtbot warns that AI will stub or mock the implementation a test is meant to exercise if that gets the test passing. Formation names the general problem: tests and implementation may encode the same misunderstanding, so verification needs an oracle independent of the code under test. By that test, a second model agreeing with the first is not an oracle. One HN commenter worried that code and tests from the same model "might agree with each other and still miss the same thing." For medium and high tiers, require one independent oracle: a contract diff, a test that existed before the change, or a case a human wrote.

Route each change by risk

Anthropic's Claude Code team routes each pull request into a low, medium, or high tier. Low needs no human reviewer once automated review and security checks are green, medium needs one person, and high needs two. A classifier picks the tier from the change. Paths that owners list as sensitive, and every CI workflow file, always need a person. Only trusted authors get the lighter paths. Cloudflare classifies every merge request into one of three tiers from the size and nature of the diff, and anything touching auth/, crypto/, or security-sounding paths always gets a full review. Each describes one company's repos.

At an API or SDK company, we put the public contract at the top: the OpenAPI or Protobuf spec, the SDK's public surface and default behavior, auth, and billing. A wrong merge there reaches customers.

The table is our recommendation, informed by Anthropic's and Cloudflare's tiers. The paths are illustrative; adjust them to your repos.

| Tier | Example changes | Evidence required | Approvers |

| --- | --- | --- | --- |

| Low | Docs and copy, internal-only changes, additive changes behind a feature flag | Green tests, type checks, and security scan on this commit SHA; PR template filled in | The requester signs off; a second person reviews until evidence quality holds |

| Medium | Ordinary product changes, internal endpoints, caching | Low tier evidence, plus failure-oriented tests and one independent oracle | One reviewer other than the last person to push |

| High | Public API contract, SDK public surface or behavior, auth, billing and money movement, CI workflow files, data migrations | Medium tier evidence, plus a breaking-change diff, backward compatibility notes, and a timed rollback plan | Two people, including the code owner, plus a named release owner |

Four escalation rules apply to every tier:

• Set the tier from changed file paths. TradersPost's system sets the floor from changed paths and never reads the description, commit messages, or the agent's explanation.
• Let agent signals raise a tier and never lower it. In TradersPost's words, the authoring agent "must never downgrade a deterministic requirement."
• A PR that spans tiers takes the highest one.
• Stale evidence blocks. Evidence from an earlier commit, or a missing or failed check, sends the PR back. Anthropic treats approvals the same way: an approval covers one specific commit.

A worked PR: retries in a payments SDK

This example is illustrative and did not happen at any real company. We wrote a small fixture for it and ran both snippets with pytest 9.1.1 on Python 3.12.16.

(opens the full-size image in a new tab)

An AI agent is asked to make a payments SDK tolerate network timeouts. It adds automatic retries to the SDK's request function:

def send(transport, method, path, body=None, headers=None, retries=2):
    headers = dict(headers or {})
    for attempt in range(retries + 1):
        try:
            return transport(method, path, body, headers)
        except Timeout:
            if attempt == retries:
                raise

The agent's test has a GET time out once and then succeed. It passes, and CI is green.

The function retries every method, though. Picture a POST that creates a charge. The server commits the charge, then the connection times out before the response arrives. The SDK sends the same POST again with no idempotency key, the server cannot tell it is a retry, and the customer is charged twice.

The agent's test only covers GET retries; nothing exercises a non-idempotent request or a timeout after the server commits. So the reviewer asks for that test, using a fake payments server that commits the charge and then times out before replying on the first call:

def test_post_retry_does_not_charge_twice():
    server = FakePaymentsServer()
    send(server, "POST", "/v1/charges", {"amount": 500, "currency": "usd"})
    assert len(server.charges) == 1

Against the proposed change, the agent's GET retry test passes and this test fails with assert 2 == 1: two charges.

The change is high tier on two counts: it alters public SDK behavior and it moves money. The reviewer requests changes: retry non-idempotent requests only with one Idempotency-Key, set before the retry loop and reused on every attempt. Against the revised change, both tests pass.

The reviewer also asks for a named release owner and a rollback plan that fits how SDKs ship. A git revert does not reach customers who already installed a published SDK version. Rollback means a patch release. Customers pinned to the bad version keep its behavior until they upgrade, and the rollback plan says so.

Copy this PR checklist

Paste this into your pull request template. Anthropic's template asks how the change was tested and how to undo it; an agent can fill that in, and the person who asked for the change signs off.

## Intent and acceptance criteria
- [ ] What this change should do, in one or two sentences:
- [ ] How we know it is done:
 
## Risk tier and why
- [ ] Tier (low, medium, high):
- [ ] Paths or behavior that set the tier:
 
## Evidence tied to this commit SHA
- [ ] Commit SHA:
- [ ] Commands run:
- [ ] Output (paste or link):
 
## Failure-oriented tests
- [ ] Tests for the timeouts, retries, bad input, and permission errors that apply:
- [ ] Independent oracle (pre-existing test, contract diff, or human-written case):
 
## Contract and breaking-change diff
- [ ] OpenAPI or Protobuf diff result:
- [ ] SDK public surface changes:
 
## Security and dependency changes
- [ ] New or updated dependencies, and why:
- [ ] Auth, secrets, or permission changes:
 
## Backward compatibility
- [ ] Who breaks if they do not upgrade:
 
## What the checks would miss
- [ ] Behavior no test or scan covers:
 
## Approver
- [ ] Name:
 
## Release owner
- [ ] Name:
 
## Rollback plan and how fast
- [ ] How to undo (flag, revert, patch release):
- [ ] Time to undo:

When to split or reject a change

Send a PR back when any of these is true:

• It mixes a refactor with a behavior change. Google's review guidance says refactorings usually belong in a separate change and expects logic changes to come with new or updated tests.
• It is too large to explain. Google's guidance says "100 lines is usually a reasonable size for a CL, and 1000 lines is usually too large," leaving room for reviewer judgment, and notes small changes are simpler to roll back.
• Its evidence is not tied to the revision under review.
• Its tests and code come from the same model with no independent oracle, on a medium or high change.
• It adds a dependency nobody can explain.
• It has no rollback path.

Ask for the first two to be split. For the rest, ask for the missing piece.

The tradeoff: gates can move the bottleneck

Every gate here adds work before review. Evidence takes time to gather, so the queue can shift from waiting on a reviewer to waiting on evidence. And when one model writes both the code and the tests, a green suite can reward code that only looks correct.

Anthropic's account shows how an evidence loop can keep going with no stop: one CLI PR went through more than fifty rounds of review and fixes overnight before its author stopped it. Set a stop rule: after a small, fixed number of rounds, a person reads the diff and decides. Google's standard applies: approve once a change "definitely improves the overall code health of the system," even if it is imperfect.

The other objection is to keep reviewing every PR the way you do now. Uniform review gives the retry change the same reading as a typo fix.

To judge the gates, watch four things:

• Time waiting for evidence against time waiting for a reviewer. If evidence wait grows and reviewer wait does not shrink, that tier's gates are too heavy.
• Low-tier changes that later needed a revert. Each points to a path that belongs higher.
• Whether reviewers can say what the checks would miss. Anthropic tells reviewers who cannot to ask for a real review rather than give a quick approval.
• Whether those numbers move. Anthropic writes that it does not yet have circuit breakers that tighten approval when they do, so do it by hand: when low-tier reverts rise, move those paths up a tier.

A four-week adoption plan

1. Week 1: Pick one repo and add the checklist as its pull request template. Require the evidence section on AI-assisted PRs, and record which fields stay empty.
2. Week 2: Protect the high-tier paths. Two approvers there meet SLSA Source Level 4's two-party review, which needs two trusted persons, and the uploader can be one of them.
   • Add CODEOWNERS entries for API specs, the SDK's public modules, auth, billing, CI workflow files, and migrations. Code owners are requested automatically when a PR modifies their code, and "Require review from Code Owners" makes their approval mandatory.
   • In a ruleset for the default branch, require code owner approval, approval from someone other than the last pusher, and passing status checks, including the breaking-change diff.
   • Keep the default option that adds one extra approval when Copilot opens a PR not attributed to a person, and dismiss stale approvals on new commits.
3. Week 3: Apply the medium tier: one reviewer other than the last pusher, plus an independent oracle. Fix the template fields people keep skipping.
4. Week 4 and after: Let low-tier changes skip the second human only while evidence quality holds: complete fields, checks tied to the SHA, no low-tier reverts. If you enable Copilot approvals, limit them to PRs where every changed file matches your low-tier globs. GitHub allows up to 15 globs, and a new commit dismisses Copilot's approval as it does a human's.

Try it on one PR this week

Pick one open AI-generated pull request. Fill in the checklist for it, write down every evidence gap, and name its approver and its release owner. If you cannot name both, assign them before the PR merges. For how one team runs its tiers in full, read Anthropic's account of solving code review.

Sources

• Solving code review at Anthropic, Anthropic
• Redesigning source control at Anthropic, Anthropic
• AI code review, Cloudflare
• What are code reviews even for, DX
• About protected branches, GitHub Docs
• Available rules for rulesets, GitHub Docs
• About code owners, GitHub Docs
• About Copilot code review, GitHub Docs
• Configure code review, GitHub Docs
• Copilot code review can now approve pull requests, GitHub Changelog
• Review Copilot PRs, GitHub Docs
• Small CLs, Google Engineering Practices
• The Standard of Code Review, Google Engineering Practices
• oasdiff, GitHub repository
• Breaking change detection, Buf
• Source requirements, SLSA
• Security-Focused Guide for AI Code Assistant Instructions, OpenSSF
• How to review AI generated PRs, thoughtbot
• How to Review AI-Generated Code Before You Merge It, Formation
• Risk-Proportional Review for AI Agents, TradersPost
• Ask HN: How to deal with shipping AI-written code you don't understand?, Hacker News