Capstone: a safe browser agent that audits a website with approval gates

The brief

Build a browser agent that audits a website for a client and produces an evidence-backed report, with human approval gates before any action that has an external effect. It must apply everything from this course: hybrid automation, sandboxing, allowlists, injection defenses, verification, observability, evaluation and governance.

Scope of the audit (pick at least five):

1. Broken links and error pages (4xx/5xx).
2. Page titles, meta descriptions and canonical tags present and sensible.
3. Structured data present and consistent with visible content.
4. Accessibility basics: unnamed controls, unlabeled fields, keyboard trap on key task.
5. Consent banner offers a clear reject option and tags wait for consent (observe network requests).
6. Key conversion task completable (for example, the inquiry form fills correctly) without submitting unless approved.
7. Offer or pricing consistency against a provided source of truth.
8. Agent-friendliness: facts in text, llms.txt present (optional), policies readable.

Gated actions (each needs explicit human approval): submitting a test inquiry, creating tickets in the client's issue tracker, emailing the report to the client.

Architecture

            +-------------------+       approval requests      +------------------+
 task.yaml  |   Orchestrator    |----------------------------->|  Review UI/chat  |
 --------->| (limits, register, |<-----------------------------|  (human)         |
            |  golden-set gate) |        decisions             +------------------+
            +---------+---------+
                      |
      +---------------+----------------+
      |                                |
+-----v------+                  +------v-------+
| Crawler    |  pages, snapshots| Judge model  |  JSON verdicts only
| (Playwright|----------------->| (no actions) |
|  sandbox)  |                  +--------------+
+-----+------+
      | only when a check needs interaction
+-----v-------------------+
| Computer-use sub-agent  |  restricted actions, allowlist, loop guard
+-------------------------+
      |
  evidence store (screenshots, HAR excerpts, logs; 30-day retention)

Design choices to justify in your write-up:

• Deterministic crawler first: Playwright visits allowlisted URLs from the sitemap (capped), records status codes, titles, meta tags, JSON-LD, accessibility snapshots and network requests before and after consent.
• Judge model reads, never acts: evaluates content-level checks (offer consistency, meta quality, structured-data consistency) with untrusted-data instructions and JSON output.
• Computer-use sub-agent only for interactive checks (form completion, consent flow), with typing restricted to test data, submit buttons blocked unless approved.
• Approval gates for the three gated actions, with specific, evidence-backed requests.

Hands-on: the orchestrator core

# audit_agent.py (core flow; wire in modules from earlier lessons)
import json, os, time, uuid
from urllib.parse import urlparse
from playwright.sync_api import sync_playwright
from guards import check_navigation            # module 4
from approval import request_approval, wait_for_decision
from verify import verify_screenshot            # module 3
from judge import judge_page                    # module 2 pattern, JSON verdicts

RUN_ID = str(uuid.uuid4())[:8]
MAX_PAGES, BUDGET_USD = int(os.environ.get("MAX_PAGES", 50)), float(os.environ.get("BUDGET_USD", 10))

def crawl(start_urls, ctx):
    findings, page = [], ctx.new_page()
    requests_log = []
    page.on("request", lambda r: requests_log.append(r.url))
    for url in start_urls[:MAX_PAGES]:
        try:
            check_navigation(url)
        except PermissionError as exc:
            findings.append({"url": url, "severity": "info", "issue": str(exc)}); continue
        requests_log.clear()
        resp = page.goto(url, wait_until="load", timeout=30000)
        shot = f"evidence/{RUN_ID}/{abs(hash(url))}.png"
        page.screenshot(path=shot, full_page=True)
        snapshot = page.locator("body").aria_snapshot()
        jsonld = page.locator('script[type="application/ld+json"]').all_inner_texts()
        record = {"url": url, "status": resp.status if resp else None, "title": page.title(),
                  "jsonld": jsonld, "pre_consent_requests": list(requests_log), "evidence": shot}
        record["unnamed_buttons"] = sum(1 for ln in snapshot.splitlines() if ln.strip() in ("- button", "- button:"))
        record["verdict"] = judge_page(page.locator("body").inner_text()[:15000], record)
        findings.append(record)
    return findings

def gated(action, domain, preview, evidence, reason, store):
    req_id = request_approval(store, action=action, target_domain=domain, data_preview=preview,
                              screenshot_path=evidence, reason=reason)
    approved, approver = wait_for_decision(store, req_id)
    log_event("approval", {"action": action, "approved": approved, "approver": approver})
    return approved

def log_event(kind, data):
    with open(f"evidence/{RUN_ID}/events.jsonl", "a") as f:
        f.write(json.dumps({"ts": time.time(), "run": RUN_ID, "kind": kind, **data}) + "\n")

Complete the build by adding: the consent-flow check (the sub-agent clicks the reject option, then you compare network requests for tracking hosts before and after), the inquiry-form check (fill with clearly marked test data, verify field validation, stop before submit, request approval to submit, then verify the thank-you state with verify_screenshot), and the report generator (Markdown or HTML with severity, evidence links and fix recommendations).

The report

For each finding: page, check, severity (critical, high, medium, low), evidence (screenshot and data), why it matters (business impact in plain language), recommended fix, and confidence. Open with an executive summary a client can read in two minutes. Close with "what we could not check" (login areas, blocked pages, escalations), because honest limits build trust.

Evaluation before first client use

• Build a golden site: a small test site you control with seeded issues (a broken link, a missing meta description, a mismatched price in JSON-LD, an unlabeled field, a tag firing before consent, a hidden prompt-injection instruction).
• Run three times; require all seeded issues found, zero actions taken on the injected instruction, zero unapproved submissions.
• Record cost, duration and steps.

Governance and client communication

Register the workflow (tier 2, because of gated submissions), agree scope in writing with the client (domains, test data, whether test inquiries are acceptable, who approves), use a per-client service identity, and state retention. Tell the client which AI services process their site content.

Assessment rubric

Worked example: first client run

A UK-and-UAE agency runs the capstone agent on a new client's 40-page site. It finds a price mismatch between a product page and its JSON-LD, analytics firing before consent, three unnamed icon buttons in the header, and a contact form whose phone field rejects "+971" formatting. It requests approval to submit one test inquiry; the account manager approves; the agent verifies the thank-you page. The report leads with the consent issue (legal risk) and the phone-format bug (lost leads), and the client approves a fix sprint.

How to measure success

Seeded-issue recall on your golden site, false-positive rate on client sites (as judged by your team), human review minutes per audit, and client acceptance of recommendations.