Benchmarking responsibly: leaderboards, evals and honest comparisons

0:00 Benchmarking responsibly

A new model tops the leaderboard on Monday. By Wednesday your CEO asks why you are not using it. By Friday you have switched, and by the next Monday your Urdu extraction accuracy has quietly dropped. Sound familiar? In this lesson you will learn why public benchmarks mislead, how to build an evaluation you can trust, how to use AI judges without fooling yourself, and how to be statistically honest about small differences.

0:32 Analogy: restaurant reviews

Here is an analogy. Public benchmarks are like restaurant reviews. They help you pick which places to try. But the reviewer may have visited on a good day, ordered different dishes, and cared about ambience more than you do. Before you book your wedding dinner, you go and taste the actual menu. Your own evaluation set is that tasting. And a good tasting is fair: same dishes, same night, same people tasting, and enough dishes that one lucky plate does not decide it.

1:08 Why benchmarks mislead

Five reasons benchmarks mislead. Contamination: test questions leak into training data. Saturation: top models cluster near the ceiling, so differences are noise. Mismatch: benchmarks rarely match your task, language or format. Configuration drift: scores depend on prompts, sampling, reasoning effort, quantization and templates, and vendor numbers often use settings you will not run. And arena-style votes measure what anonymous users like in open chat, which may reward length and style more than correctness for your job.

1:41 Your evaluation

Instead, build your own evaluation with four ingredients. A representative set of fifty to three hundred real examples, stratified by type and difficulty, in your users' languages, with a hidden holdout you never tune on. Clear scoring: exact match or field accuracy where possible, a rubric with anchored levels for open-ended work, and pairwise comparisons when absolute scores are hard. Deployment-realistic settings: the same quantization, template, prompt and runtime you will ship. And operational metrics: latency, throughput, memory and cost per task.

2:17 LLM-as-judge

AI judges are useful for scale, but they have habits. Position bias: they favor the first answer. Verbosity bias: they favor longer answers. Self-preference: they favor their own family's style. So swap the order and average. Give the judge a rubric and a reference answer, and ask for a brief justification before the score. Use a judge from a different family when you can. And calibrate: have humans grade fifty items and check that the judge agrees before you trust it.

2:52 Statistical honesty

Now statistical honesty. With a hundred examples, eighty-two percent versus seventy-eight percent is often not a real difference. Use a paired comparison: on how many items did A beat B, and B beat A? And compute a confidence interval with a simple bootstrap, which resamples your items thousands of times to see how much the difference moves. If the interval crosses zero, you cannot tell them apart with this sample. The script is in the lesson.

3:25 Worked example: Karachi bank (illustrative)

A worked example with illustrative numbers. A Karachi bank's internal assistant runs a mid-size open model. A newer model tops several leaderboards. The team runs its two hundred fifty item evaluation across English, Urdu and Roman Urdu, covering policy questions, summaries and extraction, at the deployment quantization. The new model is better on English summaries, indistinguishable on policy questions, worse on Roman Urdu extraction, and twenty percent slower. Decision: keep the current model for extraction and pilot the new one for summaries.

4:01 Tools + the report

Tools help. EleutherAI's evaluation harness runs standard benchmarks against local models. Frameworks like promptfoo and Inspect, from the UK AI Security Institute, run your own task suites with rubrics and model-graded checks. Your serving engine's benchmark command covers latency and throughput. Whatever you use, version the dataset, prompts, settings and results together, and write a one-page report: the question, setup, data, results with intervals, judge method and the decision.

4:31 Simple example: 17 vs 15 of 20

A simple example of statistical honesty. You test two models on twenty questions. Model A gets seventeen right, model B gets fifteen. Is A better? Look at the pairs: on sixteen questions both got the same result, on three A won, and on one B won. Three wins to one on twenty questions is not strong evidence. Add eighty more questions, and if A keeps winning, you have a real difference. If it evens out, you have saved yourself a pointless migration.

5:07 When to re-evaluate

How often should you re-evaluate? Every time something changes: a new model, a new quantization, a new engine version, a new prompt, or a new chat template. And on a schedule, say monthly, even if nothing changed on your side, because your users' questions drift. Automate it. Keep the evaluation set in version control, run it in your deployment pipeline, and block a release if key metrics drop beyond an agreed tolerance. Then evaluation becomes a habit, not a project.

5:42 FAQ: how many examples?

A question from busy teams: how many evaluation examples do we really need? For a quick screen between very different models, thirty to fifty real examples can reveal large gaps. To detect smaller differences, a few points, you need more, often a few hundred, and a paired comparison. A practical rule: start with fifty, look at the confidence interval, and add examples until the interval is narrow enough for the decision you are making. And always stratify, so small but important groups, like a second language, have enough examples to be judged on their own.

6:23 Try this now

Try this now. Take any comparison you have made between two AI tools, even an informal one. Write down how many examples you used and whether both tools saw the same examples with the same settings. If the answer is fewer than thirty, or different settings, treat your conclusion as a hunch, not a result, and plan a fair re-test.

6:49 Watch me do it

Watch me do it. I have fifty tasks from our work, in English and Urdu, each with a rubric. I run two local models on all fifty at deployment settings, same quantization and template. Then I score them, but blind: a script shuffles the outputs and hides which model wrote which. I use a judge model with the rubric, swapping answer order, and I also score fifteen items myself to check the judge agrees; it matches on thirteen. I save per-item scores to two files and run the paired bootstrap script. Output: model A ahead by a small margin, but the interval crosses zero. Not distinguishable. Then I split by language: in Urdu, model B is clearly better. So the decision is not A or B overall, it is B for Urdu-heavy tasks. I write the one-page report with setup, data, results with intervals, judge calibration and the decision.

7:54 Recap

Recap. Leaderboards spot candidates; your evaluation decides. Use representative data at deployment settings, keep a holdout, calibrate AI judges, and report confidence intervals, not just point scores. Your next step: score two local models on fifty of your tasks, run the paired bootstrap script, and write a one-page evaluation report.