Choosing and setting up tools with a fair trial

0:00 Choosing and setting up your tools

Every team that adopts AI coding tools has the same argument: which one is best? It is the wrong question. The right question is which tool, configured how, helps this team ship reviewed and tested changes on this codebase. In this lecture you will learn the criteria that matter, how to run a fair two-week trial, and the day-one setup that makes any agent safer and more useful.

0:30 Analogy: hiring, not shopping

Here is an analogy for choosing tools. Picking a coding agent is like hiring for a role, not buying a gadget. You would not hire someone because of a flashy portfolio alone. You would give them a realistic work sample, check references, confirm they can follow your company's rules, and look at what they cost. A tool trial is exactly that: a work sample on your real tickets, a check of admin controls and data terms, and an honest look at cost. Keep the hiring mindset and most of the decisions become obvious.

1:10 Criteria

Start with criteria. Workflow fit: does it live in the terminal, the editor or the cloud, and does it work with your Git host? Context handling: does it read shared instruction files and support MCP for your issue tracker? Controls: sandboxing, approval modes, hooks, audit logs. Data and privacy: retention, training use, region and contract terms for your specific plan. Cost model: seats, usage, or both. And automation: is there a headless mode and a CI integration? Weight these for your situation. A client-services agency weights data terms heavily. A solo founder weights speed and cost.

1:52 A fair trial

Now the trial. Pick fifteen to thirty real tickets from your backlog, across bugs, features, tests and refactors. Define done identically for every tool: merged, CI green, approved without major rework. Log the tool, the mode, wall-clock time, reviewer minutes, revision rounds and any defects found later. Rotate developers across tools, and write down in advance what result would make you adopt or reject.

2:20 Feelings ≠ measurements

Why log so carefully? Because self-reported speed is unreliable. A randomized study by the research group METR, run on experienced open-source developers in early twenty twenty-five, found they took longer on average with AI tools, while believing they had been faster. A later, larger cohort showed a smaller and statistically uncertain effect. The takeaway is not that AI is slow. It is that feelings are not measurements. Measure, don't guess.

2:50 Day-one checklist

Whatever you choose, day one looks the same. Add an instruction file with build, test and style commands. Commit a permission baseline. Keep secrets out of git and deny the agent access to env files. Make tests fast and runnable with one command, because agents amplify whatever feedback loop you give them. Protect your main branch with required reviews and checks. And set budget alerts on anything usage based.

3:20 Case: Dubai agency

A quick story. A twelve-person agency in Dubai trialled three tools on twenty tickets. The winner was not the flashiest demo. It was the tool whose admin controls let them block specific client repositories from cloud agents, enforce single sign-on and export audit logs for client security questionnaires. Developers still use a second tool locally for exploration, under a written policy. Governance, not just raw capability, decided it.

3:50 Two criteria that bite later

Two criteria deserve extra attention because they bite later. Data terms first. Before any client code leaves a developer's laptop, check retention, whether your plan allows your code to be used for training, which region processes it, and whether zero data retention is available. Put the answers in writing, because client contracts in the UK, the Gulf and elsewhere increasingly ask. Then cost. Seat pricing is predictable, while usage pricing scales with how long agents run. An agent left looping on a flaky test can burn real money overnight. Set budgets and alerts from the first day, and track spend per team so you can compare it with the value delivered.

4:38 Example: end-to-end time (illustrative)

A simple worked example of a trial log. Ticket one: add pagination to the products endpoint. Tool A took eight minutes to produce a pull request and the reviewer spent twenty-five minutes fixing it. Tool B took twelve minutes and needed only six minutes of review. Which is faster? Tool B, by fifteen minutes end to end, even though it looked slower at first. Now multiply that across twenty tickets and you see why logging reviewer minutes matters more than logging generation time. These numbers are illustrative, but the pattern is one teams report again and again.

5:20 Common mistakes

Common mistakes, so you can avoid them. First, running the trial on a brand-new toy project; every tool looks brilliant on greenfield code. Second, letting one enthusiastic expert do all the trial work, which measures that person, not the tool. Third, skipping the day-one setup for one of the tools, then blaming it for guessing your conventions. And fourth, forgetting to decide in advance what result would make you adopt. Without that, the loudest opinion wins. Ask yourself now: which of these four is most likely in your team?

5:59 Deeper: one complete log row (illustrative)

Let's deepen the trial log example with a full row. Ticket: add pagination to the products endpoint. Tool: B. Mode: delegate. Wall-clock: twelve minutes. Reviewer minutes: six. Revision rounds: one. Defects found later: none after two weeks. Every column matters. If you only record the first two, Tool A looks faster. When you record all six, the picture flips, and you also learn that delegate mode worked well on well-specified API tasks, which tells you where to use the tool first when you roll it out to the wider team.

6:38 Watch me do it: the smoke test

Watch me do it. I'm running the two-agent smoke test from the lesson on a real repository. First, I create a throwaway branch for each tool, so nothing touches main. In the first terminal I start the first agent and paste the identical prompt: read AGENTS dot M D, find one function in the source folder without tests, write focused tests covering normal, edge and error cases, run the suite, and do not modify the source folder. Agent one picks a currency formatting function, writes six tests, runs them, and reports all green. Agent two picks the same kind of function, writes four tests, and also reports green. Now the important part, which most people skip. I deliberately break the function: I change a rounding call so it truncates instead of rounds. I rerun both sets of tests. Agent one's tests catch it, two failures. Agent two's tests all still pass. Its tests only checked that the function returned a string. Next I check the constraint: git diff on the source folder. Agent one changed nothing there. Agent two quietly reformatted the function it was testing. And my review time: about four minutes for agent one, nine for agent two. Three columns in my trial log, constraint followed, tests catch a real bug, review minutes. That tells me far more than any demo.

8:15 Recap

Recap. Choose with a trial, not a debate. Weight criteria for your context. Log review time and rework, not just generation speed. And apply the day-one checklist whatever you pick. Your next step is in the lesson: install two agents, give them the identical test-writing task, and compare how well each followed the constraint and how long your review took.