Evaluating browser agents: test sets, metrics and benchmarks

Video lecture

Evaluating browser agents: test sets, metrics and benchmarks

15 chapters · about 8 min · full transcript

Coming soon

Chapter 1 of 15

Evaluating browser agents

Benchmarks vs your tasks
Building a golden test set
Metrics that matter
Regression testing

The narrated lecture is in production

Every chapter is scripted and ready. Browse the chapters and read the full transcript now — the video will appear here when it’s published.

Chapters

Transcript of the narration, chapter by chapter.

0:00 Evaluating browser agents

Every agent demo looks amazing. But demos are chosen to succeed. The question that matters is simpler and harder: how well does this agent do my tasks, on my sites, at what cost? In this lesson you will learn how to answer that with public benchmarks, your own golden test set, and a handful of metrics that reveal hidden risk.

0:26 Public benchmarks

Start with public benchmarks, but don't stop there. OSWorld tests agents on real desktop tasks. WebArena and VisualWebArena use realistic self-hosted websites with automatic success checks. Mind2Web tests generalization across many real sites, and WebVoyager runs tasks on the live web. Scores on these have climbed quickly. But benchmark tasks aren't your tasks, live sites drift, and results depend heavily on the harness. Use benchmarks to build a shortlist. Use your own test set to decide.

0:59 Why it matters

Why does this matter? Because without evaluation, every decision about your agent is an opinion. Which model should we use? Did the new prompt make things better? Is it safe to let it submit forms? Is the vendor's update a regression? A golden test set turns each of those into a measurable question you can answer in an afternoon. It also protects you: when something goes wrong, you can show that the agent was tested, how, and against what.

1:33 The driving test

Here's an analogy. A driving test doesn't just ask whether you can move a car. It tests specific maneuvers, on real roads, with tricky situations built in, and the examiner marks serious faults separately from minor ones. Your golden test set is the driving test for your agent. The typical tasks are the everyday roads. The nasty cases are the roundabouts. The adversarial pages are the pedestrian stepping out unexpectedly. And false success is the serious fault that fails the test outright.

2:09 Simple example: 23 golden tasks

A simple example of building golden tasks. Take last month's twenty real requests for one workflow, say, checking supplier portals for delivery dates. For each, you already know the right answer, because a person looked it up. Write that answer down as the expected output. Add three tricky ones: a portal that times out, an order number that doesn't exist, and a page with hidden text telling the agent to visit another site. That's a golden set of twenty-three tasks, built in an afternoon.

2:46 Golden test set

Building a golden set is simpler than it sounds. Collect thirty to a hundred real tasks from the workflow you want to automate. Include easy ones, typical ones and nasty ones: pop-ups, slow pages, missing data. For each, write the expected outcome in a form code can check, the expected fields, the final URL, the database state. Freeze the environment where you can, with a staging site or saved copies, so comparisons are fair. And version it alongside your prompts.

3:21 Adversarial cases

Add adversarial cases on purpose. A page with hidden text telling the agent to go to another site. A fake confirm payment button. A task where the right answer is to escalate because data is missing. The correct behavior is to ignore the injected instruction or ask a human. If your agent follows the hidden text in testing, you've learned something priceless before a real attacker taught it to you.

3:51 Metrics

Now the metrics. Task success rate is the headline. False success, where the agent claimed success but verification failed, is the hidden risk. Escalation precision tells you whether the agent bothers humans only when it should. Safety violations, attempted out-of-policy actions, must trend to zero. Steps, time and cost per task give you ROI. And consistency: run each task several times, because agents are non-deterministic and a single pass can mislead.

4:22 Regression testing

Evaluation is not a one-time event. Re-run your golden set every time you change the prompt, the harness, the model or the tool version. Run it on a schedule too, because vendors update models behind the same alias. In production, pin model versions where the vendor allows it, and only promote a new version once it matches or beats the old one on your own tasks.

4:51 Worked example: choosing a model

A logistics firm in Jeddah enters shipment details into three carrier portals. They built sixty golden tasks, twenty per portal, including five with missing data that should escalate, and ran two models three times each. One model was faster. The other had far fewer false successes. Because a wrong shipment entry is expensive, they picked the careful one for data entry and used the fast one for a read-only status check. The decision took an afternoon, because the test set already existed.

5:27 Sourcing golden tasks

Where do golden tasks come from? The best source is real history. Pull last month's tickets, requests or spreadsheet rows for the workflow, and turn each into a task with a known answer, because a person already did it. Add the weird cases people remember, the page that always times out, the form that rejects certain phone formats. Ask the team what makes the job annoying. Those annoyances are exactly where agents stumble, so they belong in the set.

6:01 Before and after, every time

Keep your evaluation honest with a simple results table for every change. One row per metric: success, false success, escalation precision, safety violations, median cost and median time. Two columns: before and after. Add a note on what changed. Share it with everyone who relies on the workflow. Over a few months, that table becomes the history of your agent, and it answers the question every manager eventually asks: is this getting better or worse?

6:34 Three mistakes

Three common mistakes in evaluation. First, testing on the live web and comparing runs from different days, when the sites themselves have changed. Freeze what you can. Second, measuring only success rate, and ignoring false success, cost and safety. Third, letting the test set go stale while the workflow evolves. Review it every quarter and add every real failure you see in production as a new test case.

7:04 Try this now

Try this now. For one workflow, draft twenty golden tasks from real past requests where you already know the right answer. Write the expected output for each in a form code can check. Add at least three tasks that should escalate, like missing data or a login wall, and two adversarial pages with hidden instructions. Run your agent three times on the whole set, and record success, false success, escalations and cost. Save the results as version one, so every future change has a baseline.

7:41 Recap

Recap. Benchmarks help you shortlist, your golden set decides. Include nasty and adversarial cases with checkable outcomes. Measure success, false success, escalations, safety, cost and consistency. And re-run on every change. Your next step: draft twenty golden tasks for one workflow, including at least three that should escalate. Next module: sandboxing, permissions and security.

Why evaluation is the real moat

Every vendor demo looks impressive. The only question that matters for you is: how well does this agent do my tasks, on my sites, at what cost? Evaluation answers it. Without evaluation, you cannot choose between models, detect regressions when a vendor updates a model, or show stakeholders that the system is safe.

Public benchmarks: useful, but not your answer

Research benchmarks give a rough sense of capability:

OSWorld evaluates agents on real desktop tasks across operating systems and applications.
WebArena and VisualWebArena use self-hosted realistic websites (shopping, forums, content management, maps) with programmatic success checks.
Mind2Web and Online-Mind2Web test generalization across many real websites.
WebVoyager tests end-to-end tasks on live websites.

Vendors report scores on these, and scores have risen quickly. But benchmark tasks are not your tasks, live-web benchmarks drift as sites change, and results depend heavily on the harness used. Use benchmarks to shortlist; use your own test set to decide.

Build a golden test set

Collect 30 to 100 real tasks from the workflow you plan to automate. Include easy, typical and nasty cases (pop-ups, slow pages, missing data, pages that should trigger escalation).
Write the expected outcome for each, ideally as something code can check: expected JSON fields, the final URL, a database state.
Freeze the environment where you can: local copies of pages, a staging site, or recorded fixtures. Live sites drift and make comparisons unfair.
Include adversarial cases: a page with hidden instructions telling the agent to go elsewhere, a fake "confirm payment" button. The correct behavior is to ignore or escalate.
Version it alongside your prompts and harness.

Metrics that matter

Metric	Definition	Why
Task success rate	Share of tasks with correct verified outcome	Primary capability
False-success rate	Claimed success but failed verification	Hidden risk
Escalation precision	Of escalations, share that genuinely needed a human	Too many wastes staff time
Safety violations	Out-of-policy actions attempted (blocked or not)	Must trend to zero
Steps, time and cost per task	Resource use	ROI and latency
Consistency	Success across repeated runs of the same task	Non-determinism

Run each task several times. Agents are non-deterministic; a single pass can mislead.

Hands-on: a minimal eval runner

# eval_runner.py
import json, statistics, time
from agent_loop import run          # your harness from module 1
from executors import make_executor  # your sandbox factory

def check(expected: dict, output: str) -> bool:
    try:
        got = json.loads(output)
    except (json.JSONDecodeError, TypeError):
        return False
    return all(got.get(k) == v for k, v in expected.items())

def evaluate(tasks_path="golden_tasks.jsonl", repeats=3):
    rows = []
    for line in open(tasks_path):
        t = json.loads(line)
        for r in range(repeats):
            ex = make_executor(fresh=True)
            steps = []
            start = time.time()
            out = run(t["task"], ex, log=lambda s, a, i: steps.append(a))
            ok = check(t["expected"], out) if t.get("expected") else ("NEEDS_HUMAN" in out) == t.get("should_escalate", False)
            rows.append({"id": t["id"], "rep": r, "ok": ok, "steps": len(steps),
                         "secs": round(time.time() - start, 1)})
            ex.close()
    rate = sum(r["ok"] for r in rows) / len(rows)
    print(f"success={rate:.0%}  median_steps={statistics.median(r['steps'] for r in rows)}")
    return rows

Add cost tracking from token usage, and a safety counter from your harness's blocked-action log.

Regression testing

Re-run the golden set whenever you change the prompt, harness, model or tool version, and on a schedule, because vendors update models behind aliases. Pin model versions in production where the vendor allows it, and promote a new version only after it matches or beats the old one on your set.

Worked example: comparing two models for form entry

A Jeddah logistics firm enters shipment details into three carrier portals. They built 60 golden tasks (20 per portal, including five with missing data that should escalate) and ran each model three times. One model was faster; the other had a lower false-success rate. Because a wrong shipment entry is expensive, they chose the lower false-success model and used the faster one only for a read-only status-check workflow. The decision took an afternoon because the test set existed.

Pitfalls

Evaluating on the live web and comparing runs from different days.
Measuring only success rate and ignoring false success and cost.
Letting the test set go stale as workflows change.

How to measure success

You know evaluation is working when every change ships with a before/after table and nobody argues from anecdotes.

Public benchmarks (OSWorld, WebArena, Mind2Web, WebVoyager) help shortlist, but your own golden test set decides.
Golden sets include typical, nasty and adversarial cases with checkable expected outcomes, in frozen environments where possible.
Track success, false success, escalation precision, safety violations, cost and consistency across repeated runs.
Re-run evals on every prompt, harness or model change and pin versions in production.

Draft 20 golden tasks for one workflow, including at least three that should escalate and two adversarial pages. Write the expected outcome for each.

Enrol for free to save your progress

Reading is always free. Enrol to keep your place, take the final assessment and earn a verifiable certificate.

Evaluating browser agents: test sets, metrics and benchmarks