Code review with AI, and reviewing AI code

0:00 Code review with AI

As agents write more of your code, review becomes the bottleneck and the last line of defense. In this lecture you will learn what AI reviewers are genuinely good at, how to tune them so developers actually read their comments, and a checklist for humans reviewing agent-written code, which is fluent, confident and occasionally very wrong.

0:24 Analogy: spell-checker and lawyer

Here is an analogy for AI review. Think of proofreading a legal contract. A spell-checker catches every typo, every missing comma, every inconsistent defined term, instantly and without fatigue. But it cannot tell you that the contract gives away your intellectual property. For that you need a lawyer who understands what the deal is supposed to be. The AI reviewer is the world's best spell-checker for code, and more. The human reviewer is the lawyer. You need both, and you should never confuse their jobs.

1:01 Two directions

AI review cuts both ways. First, AI reviewing code: tools like Copilot code review, Cursor's Bugbot, or Claude Code and Codex running in CI give every pull request a first pass. Second, humans reviewing code that agents wrote. Both matter, and they need different techniques. A useful mental model: the AI is a tireless junior reviewer who reads every line. The human is the senior reviewer who judges intent, design and risk.

1:32 Strong vs weak

What is AI review strong at? Local bugs visible in the diff, consistency with nearby code, missing error handling and input validation, common security patterns like injection or leaked secrets, and summarizing big diffs. What is it weak at? Whether the change solves the right problem, architectural fit across services, business rules that are not written down, subtle concurrency issues, and the trade-offs your team has consciously accepted.

2:02 Tune for signal

Noise is what kills AI review. If every pull request collects twenty nitpicks, developers learn to ignore the bot, including the one comment that mattered. So tune for signal. Give the reviewer your standards in its instruction file. Tell it what to prioritize, such as correctness, security, money and public APIs, and what to ignore, like formatting your linter already handles. Ask for severity and confidence on every finding, and cap the number of comments.

2:35 Human checklist

Now the human side. Agent code reads fluently, which makes it dangerously easy to approve. Use a checklist. Intent first: does it do what the issue asked, and only that? Read the tests before the code. Look for plausible fakes, like APIs from a different library version or config keys that do nothing. Check any new dependency is real, maintained and not a typo-squat. Scan for security issues. Look for duplicated helpers. And run it.

3:08 Case: the missing rule

Here is why the checklist matters. A fintech in Lahore had an agent add a refund endpoint. The AI reviewer flagged four minor issues. The human reviewer read the tests first and noticed something missing: no test for a refund larger than the original payment. The code allowed it. Neither the agent nor the AI reviewer could know the rule, because it lived only in the finance team's spreadsheet. The fix was a domain rule, a test, and one new line in the instruction file.

3:45 Self-review with fresh eyes

A powerful habit is self-review with fresh eyes. Before opening a pull request, have a separate agent session, or a subagent, review the diff as if someone else wrote it. Ask only for bugs, security issues, mismatches with the issue, weakened tests and new dependencies, with file, line, severity and fix. A fresh context avoids the author's blind spots and saves your human reviewers time.

4:13 Rules and metrics

Three rules keep the system honest. Never rubber-stamp fluent code. Never let an AI approve a pull request on a protected branch; keep a human approval requirement. And keep comment volume bounded. To track progress, measure how often AI review comments lead to a real code change, how many defects are found after merge, and the median review time for agent pull requests.

4:40 Example: the date parser

A simple example. An agent's pull request adds a function that reads a date string with a parsing library. The AI reviewer comments: this call can throw on invalid input and the error is not handled. Useful, and correct. The human reviewer, reading the tests first, asks a different question: why are we parsing strings at all, when the upstream API already returns a timestamp? Removing the parsing removes the bug class entirely. The AI caught the local problem. The human caught the design problem. That division of labor is what good review looks like.

5:21 Scenario: tuning for signal (illustrative)

Now a realistic scenario with illustrative numbers. A SaaS company in London turns on AI review for all repositories with default settings. In the first month it posts roughly twelve comments per pull request, and developers act on about one in ten. They tune it: priorities in the instruction file, a cap of five findings, and only high-severity comments posted inline. The next month, comments drop to around three per pull request, and developers act on about half of them. Fewer comments, much more value. Common mistake: judging an AI reviewer by how much it says, rather than how often it is right.

6:06 Deeper: two tuning lines that mattered

One level deeper on tuning. The London team's biggest single improvement came from one instruction: ignore formatting and import order, because Prettier and the linter enforce them. That removed roughly half the comments overnight. The second came from requiring a concrete fix with every finding, which filtered out vague observations the reviewer could not justify.

6:30 Watch me do it: reviewing a refund PR

Watch me do it. I'm reviewing an agent's pull request that adds a refund endpoint, using the fresh-eyes self-review first and then the human checklist. I open a new session and paste the self-review prompt: review the diff against ISSUE dot M D, report only bugs, security issues, requirement mismatches, weakened or missing tests and new dependencies, with file, line, severity and a fix. It returns three findings. A missing check that the order belongs to the caller: high severity. An unhandled error when the payment provider times out: medium. And a new dependency for date parsing that duplicates a helper we already have: low. The agent fixes all three before the pull request opens. Now the human pass. Intent: the issue asked for refunds up to the original amount; the pull request does that, and nothing else. Tests first: I read the test file and see happy path, unauthorized user and provider timeout. But there is no test for a refund larger than the payment. I check the code: the amount is validated against the order total, good, so I ask for the missing test anyway, because the rule should be pinned. Plausible fakes: the provider client method it calls exists in our installed version; I check the lock file to be sure. Dependencies: none new after the fix. Then I run it locally. Twelve minutes total, and I actually trust what I am approving.

8:13 Recap

Recap. AI review is a tireless first pass, strongest on local issues. Tune it for signal with priorities, severity and caps. Humans review agent code with a checklist that starts with intent and tests. Your next step: add the review guidelines block from the lesson to your repository, and try the fresh-eyes self-review prompt on your next agent pull request.