Debugging with agents: a scientific protocol

0:00 Debugging with agents

Agents are tireless debuggers. They will read stack traces, add logging and rerun a reproduction two hundred times without complaint. They also share a very human bad habit: changing code until the symptom goes away, without understanding why. In this lecture you will learn a six-step debugging protocol that forces the scientific method, how to use git to narrow the search, and how to debug production issues safely.

0:30 Analogy: the good doctor

Why does debugging need a protocol? Think about a good doctor. When you arrive with a headache, they do not immediately prescribe the strongest painkiller. They ask questions, take your temperature, rule things out, and only then treat the cause. An agent without a protocol behaves like a doctor who prescribes painkillers for everything: the symptom goes away and the illness stays. The protocol turns your agent into the good doctor: reproduce, examine, hypothesize, test, then treat the cause.

1:04 Steps 1–3

Step one: reproduce. Get a failing test or a script that reliably shows the bug. No reproduction, no fix. Step two: gather evidence, like logs, inputs and recent commits touching the affected files. Step three: hypothesize. The agent lists several candidate causes ranked by likelihood, each with a cheap experiment to confirm or refute it. This is where you add the most value, by spotting the hypothesis the agent missed.

1:34 Steps 4–6

Step four: test the hypotheses. Add temporary logging or assertions, run the reproduction, and eliminate causes one by one. Step five: fix the root cause with the smallest possible change, and here the agent should pause for your approval. Step six: prove it. The reproduction passes, the full suite passes, temporary logging is removed, and a regression test stays forever. The lesson has this whole protocol as a copy-paste prompt.

2:04 git bisect

When something worked last week and not today, use git bisect. It performs a binary search through your history to find the commit that introduced the bug. Give it a test command and it runs automatically, which is perfect for an agent. Then ask the agent to read the offending commit and explain why it broke the behavior before it fixes anything. Understanding first, change second.

2:33 Case: the flaky test

A story from a logistics startup in Islamabad. One test failed about once every twenty runs in CI. The first agent attempt added a retry, and the test went green. That would have shipped a real bug. Running the protocol instead, the agent looped the test two hundred times, captured failing cases, and noticed failures only when two jobs had the same timestamp. The hypothesis: an unstable sort on ties. Forcing equal timestamps failed every time. The fix was a secondary sort key, plus a regression test.

3:11 Production debugging

Production incidents need extra care. The agent should work from copies of evidence, not live systems. Export sanitized logs and traces into a scratch folder, and strip customer personal data first. If you connect observability tools through MCP, use scoped, read-only, audited credentials. And never let an agent run commands against a production database during an incident. Use a replica or a snapshot.

3:38 Reading unfamiliar errors

Agents are also superb at explaining unfamiliar errors and stack traces. Ask for evidence along with the explanation. Which frame is the last one in our code? What input would take this path? And for library issues, ask the agent to read the installed version's changelog or source, instead of relying on memory, because its memory might describe a different version than the one you actually run.

4:07 Pitfalls

Four pitfalls to avoid. Retries as fixes, which hide real bugs unless the failure is truly external. Shotgun edits, where several changes at once make it impossible to know what worked. Leftover debug logging, which can leak sensitive data. And trusting remembered APIs. Measure your progress with time to root cause, the share of fixes that include a regression test, and bugs that reopen within thirty days.

4:36 Example: blank search page

A simple example. Users report that the search page is blank for some queries. Step one: the agent writes a failing test using the exact query from a user report, one containing an ampersand. Step two: evidence shows the request returns a server error. Step three: hypotheses are unescaped special characters in the query builder, an encoding issue in the front end, or a timeout. Step four: the test with a plain query passes and the one with an ampersand fails, so the escaping hypothesis survives. Step five: a one-line fix to use parameter binding. Step six: the regression test stays forever.

5:20 Scenario: 'random' checkout errors (illustrative)

Now a realistic scenario with illustrative numbers. An online grocery in Dubai has checkout errors roughly twice a day, which support has been blaming on customers' networks. An engineer pairs with an agent, exports a week of sanitized logs, and asks for hypotheses with evidence. The agent spots that every failure happens within a few seconds of a price update job. A reproduction confirms it: the cart total is computed with the old price and validated against the new one. The fix is a price version check. Before, they had spent weeks adding retries. Common mistake: treating frequency as proof of randomness. Rare is not the same as random.

6:07 Deeper: making 'random' deterministic

One level deeper on the Dubai grocery example. The reproduction was the key move. The agent wrote a test that computes a cart total at price version one, then triggers a price update, then validates checkout. It failed deterministically. Once a rare, timing-based bug becomes a deterministic test, the fix, a price version check, takes minutes, and the test guarantees it never quietly returns.

6:35 Watch me do it: the flaky test

Watch me do it with the flaky test from the Islamabad example, using the debug protocol prompt. Step one, reproduce: I give the agent the loop runner script. It runs the test two hundred times and reports eleven failures, then saves the logs of each failing run. Step two, evidence: it compares failing and passing logs and notices every failure contains two jobs with the same created timestamp. Step three, hypotheses: clock resolution in the test environment, an unstable sort when timestamps tie, or transaction isolation letting a job appear twice. For each it proposes a cheap experiment. Step four: it forces equal timestamps in a new test. Fails every time. It forces distinct timestamps. Passes every time. Isolation is ruled out because the job count is always correct. The unstable-sort hypothesis survives. Step five: it proposes adding job ID as a secondary sort key in the query, and waits. I approve. Step six: it implements the one-line change, removes its temporary logging, runs the loop two hundred more times with zero failures, runs the full suite, and keeps the equal-timestamp test as a regression test. Total time, about twenty minutes. Compare that with the first attempt, a retry that would have hidden a real ordering bug from our customers.

8:06 Recap

Recap. Impose the scientific method: reproduce, gather evidence, hypothesize, experiment, fix the cause, and prove it. Use git bisect to find regressions, debug production from sanitized copies, and verify library behavior against the installed version. Your next step: save the debug protocol prompt from the lesson as a skill or snippet, and use it on the next bug your team sees.