How to evaluate any AI tool in 30 minutes

0:00 Evaluate any AI tool in 30 minutes

Every AI vendor has a stunning demo. None of those demos are your job. In this lecture you will learn how to evaluate any AI tool in thirty minutes on your own tasks, with a fair test, an honesty check that predicts real world risk, and a scorecard you can defend to your manager or client.

0:24 Why a method matters

Why use a method at all, instead of trying a tool and seeing how it feels? Because first impressions are shaped by polish, speed and confidence, not accuracy. A tool can feel brilliant for a week and quietly invent facts in the one report that matters. Think of it like test driving a car. A spin around the block is fun, but you also want to know about safety, fuel costs and what happens in the rain. The sprint is your structured test drive.

1:01 Why demos mislead

Demos are selected to shine. They use clean inputs and simple questions. The only evaluation that matters is how the tool performs on your recurring tasks, with your kind of material, under your data rules. A demo answers the question, can this tool ever do something impressive? You need the answer to a different question, will it do my work well, week after week?

1:29 Eight criteria

Use eight criteria. Task fit, does it do the job you actually have. Output quality. Grounding, does it cite sources or your files, and admit when it does not know. Integration, does it work where your work lives, through connectors or MCP. Privacy and compliance, including training use, retention and admin controls. Cost per useful output, including checking time. Reliability. And portability, can you export your prompts, files and outputs if you leave.

2:01 The sprint: minutes 0 to 20

Here is the sprint. In the first five minutes, pick three real, non confidential tasks, one writing, one analysis and one research or fact task. Then for fifteen minutes, run identical prompts with identical attachments in each tool. Paste the outputs into a document labelled A, B and C, so you can score blind without knowing which tool wrote what.

2:27 The honesty test

Minutes twenty to twenty five, the honesty test. Ask something where the correct answer is I do not know, or not in the document, such as a figure that is not in the attached report. A tool that invents an answer here will invent answers in your real work. This single test predicts risk better than any benchmark.

2:52 Score and note

In the last five minutes, score each output from one to five on accuracy, instruction following, usefulness and effort to fix, and note anything alarming, like an invented citation or an ignored constraint. Effort to fix matters most, because it is the hidden cost of every AI output.

3:13 Make it fair

Keep it fair. Compare comparable tiers, not one vendor's free plan with another's premium model. Turn on equivalent features, like search on in all tools or none. Repeat important tests, because outputs vary. And watch your own bias toward a familiar interface or a longer answer.

3:33 Data rules while testing

Until a tool is approved for a data class, test only with public or anonymised material. It is only a test is exactly how client contracts end up in unapproved tools. Before moving to real data, check the business terms, admin controls and data processing agreement.

3:53 Simple example

A simple example. Take a five page company policy and ask two tools to summarise it in five bullets. Paste the outputs as A and B and score them blind. A scores four, B scores three. Then ask both, what does the policy say about remote work from abroad, which the policy does not cover. A says it is not stated. B invents a rule. That single honesty test tells you which tool to trust with factual work, more clearly than any leaderboard.

4:29 Worked example: a Dubai research tool choice

A Dubai agency compared three research tools for competitor scans. Three tasks, blind scoring by two account managers, and an honesty test asking for a competitor's revenue, which is not publicly disclosed. One tool presented an estimate as fact, the others flagged it as unavailable. Quality scores were close, so the decision came down to integration with their existing Workspace and the data terms. They re run the sprint every six months. Six months later they reran the same sprint after a major model release. The scores shifted, and the tool that had failed the honesty test now passed it. Because they had kept the scorecard, the comparison took an hour, and they could make the switch on evidence rather than on headlines. That is the quiet advantage of a repeatable test.

5:26 Try this now

Try this now. Set a thirty minute timer. Pick three real, non confidential tasks and one honesty test question whose correct answer is not in the material. Run them in your current assistant and one alternative, paste the outputs as A and B, and score them blind on accuracy, instructions, usefulness and effort to fix. Finish by writing three sentences, which tool you recommend for which job, and why. Save the scorecard, because you will want it again in six months.

6:01 Common mistakes

Four common evaluation mistakes. Rewarding the longest, most confident answer rather than the most accurate. Testing once and generalising, when outputs vary from run to run. Forgetting that checking and fixing time is part of the cost. And using real client data in a trial because it is only a test. Avoid those, and your evaluation will be better than most vendor comparisons you read online.

6:30 Watch me do it, part 1

Let me run the sprint for the Dubai agency's research tool choice. I prepare three tasks. A competitor scan for a coffee chain, a summary of a public industry report, and a current fact about a platform policy. Plus one honesty test, what was the coffee chain's revenue last year, which is not publicly disclosed. I paste identical prompts, with the same attached report, into both tools. Then I copy the outputs into one document and label them A and B in random order, so whoever scores them does not know which tool wrote which.

7:11 Watch me do it, part 2

A colleague scores both blind. Accuracy, instructions, usefulness and minutes to fix. On the three tasks, the totals are close, eighteen versus seventeen. Then the honesty test. A says the revenue is not publicly disclosed and suggests where estimates might be found. B presents a precise figure as fact. That decides the factual work. My three sentence recommendation says, use A for research and client facing facts, B is fine for brainstorming only, and re run the sprint in six months. The scorecard goes in the shared drive with the date.

7:51 Recap and next step

Recap. Evaluate on your own tasks using eight criteria. Run identical prompts, score blind, include an honesty test, keep the comparison fair, and test only with public or anonymised data until a tool is approved. Your next step: run the thirty minute sprint comparing your main assistant with one alternative on three real tasks, and write a three sentence recommendation.