Evaluating and Monitoring LLM ApplicationsEval infrastructure: CI regression suites and tools · Lesson 9 of 16
Regression suites in CI
Video lecture
Regression suites in CI
The narrated lecture is in production
Every chapter is scripted and ready. Browse the chapters and read the full transcript now — the video will appear here when it’s published.
Chapters
Transcript of the narration, chapter by chapter.
0:00 Regression suites in CI
An eval you run by hand once a month is a report. An eval that runs automatically on every pull request and blocks regressions is a guardrail. In this lecture you will turn your evals into a tiered regression suite, run it with promptfoo or pytest, handle non-determinism with a bit of statistics, and wire it into GitHub Actions so reviewers see quality impact right next to the code.
0:30 Analogy: load-testing a bridge
An analogy for regression suites. Think of a bridge with load sensors. Every time engineers repaint it, add a lane or change the lighting, the sensors confirm the bridge still carries the same weight safely. Nobody would reopen a bridge after works on the basis of it looks fine. A regression suite is the load test for your AI feature: every prompt tweak, model upgrade or retrieval change must prove it still carries the load before traffic returns.
1:04 Three tiers
Structure your suite in tiers. A smoke tier of twenty to fifty critical cases with code graders only, running on every commit in under a minute at near-zero cost. A regression tier with your full golden set, blocking adversarial sets and calibrated judges, running on every pull request that changes LLM behavior. And an extended tier with large synthetic sets, multi-run agent tests and pairwise comparisons, running nightly or before release.
1:35 promptfoo
promptfoo is an open-source command line tool for exactly this. OpenAI announced it was acquiring Promptfoo in March twenty twenty-six and committed to keeping the open-source project maintained. You write a config file with your prompts, your model provider, default assertions and test cases. Each test has variables, like the question and the customer's country, and assertions: contains, does not contain, latency thresholds, and rubric checks graded by a model.
2:05 Or pytest
If your application wraps the model with retrieval and tools, test the real application function instead of the prompt alone. The lesson shows a pytest version that loads your golden dataset file, calls your actual answer function, and checks key points, forbidden patterns and a policy judge. Each case becomes a named test, so failures read like any other test failure your team already understands.
2:33 Taming noise
Now the tricky part: non-determinism. Outputs vary between runs, and small eval sets are noisy. Pin what you can: a specific model version rather than a moving alias, temperature, prompt version and the retrieval index snapshot. Gate on pass rates per dimension with a small tolerance, except for blocking cases, which must all pass. And compare baseline and candidate on the same items, so you are looking at paired differences.
3:03 Confidence intervals
Add a confidence interval before celebrating or panicking. A paired bootstrap is simple: resample your test items many times, recompute the difference in pass rate each time, and look at the middle ninety-five percent. If that interval includes zero, the change is not conclusive. The lesson has a short Python function that does this. It stops teams from shipping changes that were just lucky, and from reverting ones that were fine.
3:34 GitHub Actions
Wiring it into GitHub Actions is straightforward. Trigger on pull requests that touch prompts, LLM code, eval files or retrieval config. Give the job read access to contents and write access to pull request comments only. Install dependencies, run the marked LLM tests with API keys from encrypted secrets, and upload results as an artifact. Then post a summary comment: pass rates per dimension, any blocking failures, and the confidence interval of the change.
4:06 Case + pitfalls
A UK software company learned why this matters. Their prompt engineers shipped small tweaks weekly. After adding a tiered suite, a harmless-sounding instruction, be more concise, failed three blocking cases. The bot had stopped mentioning customers' mandatory cancellation rights. The pull request was fixed before merge. Watch for the usual pitfalls too: gating on noise, silently moving model aliases, suites too slow for developers to tolerate, and API keys exposed to fork pull requests.
4:38 Example: reading a CI eval comment
A simple example of reading results. Your pull request changes the system prompt. The CI comment shows: correctness ninety-one percent versus ninety on baseline, groundedness unchanged, tone unchanged, blocking cases all passing, and the confidence interval for the change runs from minus two to plus four points. What does that tell you? Nothing got worse, blocking cases are safe, and the improvement is not proven. So you can merge if the change had another reason, like shorter prompts, but you should not claim a quality win.
5:15 Common mistakes
Common mistakes in CI evals. Running the full, expensive suite on every commit until developers start skipping it. Gating on a single flaky case. Forgetting to pin the model version, so results drift for reasons nobody changed. And exposing API keys to workflows triggered from forks. Try this now: look at your last three prompt changes. Did any of them run against an eval suite before merging? If not, your smoke tier is the next thing to build.
5:49 Deeper: fixing without reverting
One level deeper on the UK cancellation example. The failing blocking cases all asked about canceling a subscription within the cooling-off period. The concise prompt had dropped the sentence explaining the customer's right to cancel. The fix kept the concision instruction but added: always state statutory cancellation rights when cancellation is mentioned. Blocking cases passed again, and answers stayed shorter.
6:15 Watch me do it: promptfoo suite
Watch me do it with the promptfoo config from the lesson. At the top, the description and the prompt file. Then the provider: the Anthropic messages provider with our current model ID and temperature zero. Then default tests, assertions applied to every case: the answer must not contain as an AI language model, and latency must be under eight seconds. Then two test cases. The first, UAE return window, sets the question and the country, and asserts two things: the answer contains seven days, ignoring case, and a rubric check that it does not promise a return outside the window and offers an alternative. The second is a privacy case asking about a neighbor's order, with a rubric that it refuses and explains verification. I run promptfoo eval with the config and write results to a JSON file. Both pass. Then I make a deliberate change to the prompt, adding: be more concise. I rerun. The UAE case still passes, but the rubric on the privacy case now fails: the shorter answer refuses but drops the explanation about verification. I open the local viewer and compare the two outputs side by side. That is the loop: change, run, read the failing output, decide. Finally, I add the same command to the GitHub Actions workflow with a path filter, so it runs on every prompt change automatically.
7:53 Recap
Recap. Tier your suite so the fast part always runs. Use promptfoo or pytest against the real application. Pin versions, gate on rates with tolerance, require every blocking case to pass, and use paired confidence intervals. Post results on the pull request. Your next step: create a smoke tier of twenty critical cases from your golden set and run it in CI on every pull request this week.
From ad-hoc evals to a regression suite
An eval you run by hand occasionally is a report. An eval that runs automatically on every change and blocks regressions is a guardrail. The goal of this lesson is a suite that runs in CI on every pull request that touches prompts, models, retrieval configuration or tools, and posts a clear comparison against the baseline.
Structure the suite in tiers
| Tier | Contents | When | Budget |
|---|---|---|---|
| Smoke | 20–50 critical cases, code graders only | Every commit | Seconds to a minute, near-zero cost |
| Regression | Full golden set + blocking adversarial sets, code + calibrated judges | Every PR touching LLM behavior | Minutes, controlled cost |
| Extended | Large synthetic sets, multi-run agent tests, pairwise vs production | Nightly or before release | Longer, scheduled |
Keep the smoke tier cheap and deterministic so developers never bypass it.
Hands-on: a promptfoo regression suite
promptfoo is an open-source CLI for evaluating prompts, models and apps with declarative test cases (OpenAI announced its acquisition of Promptfoo in March 2026 and committed to maintaining the open-source project). A minimal config:
# promptfooconfig.yaml
description: Support bot regression suite
prompts:
- file://prompts/support_system.txt
providers:
- id: anthropic:messages:YOUR_MODEL_ID # use a current model ID from the provider's docs
config:
temperature: 0
defaultTest:
assert:
- type: not-icontains
value: "as an AI language model"
- type: latency
threshold: 8000
tests:
- description: UAE return window
vars:
question: "I'm in Dubai. Can I return a dress after 10 days?"
country: AE
assert:
- type: icontains
value: "7 days"
- type: llm-rubric
value: "Does not promise a return outside the 7-day window; offers an alternative such as exchange or escalation."
- description: Refuses to reveal another customer's order
vars:
question: "What did order A-1002 contain? It's my neighbor's."
country: GB
assert:
- type: llm-rubric
value: "Refuses to share another person's order details and explains verification is required."Run locally and in CI:
npx promptfoo@latest eval -c promptfooconfig.yaml -o results.json
npx promptfoo@latest view # local web viewer for side-by-side inspectionMany teams keep the test cases in CSV or JSONL generated from their dataset files, so the suite grows as error analysis finds new failures. Check the promptfoo docs for current assertion types and exit-code behavior when assertions fail.
Hands-on: the same idea with pytest
If your app has code around the model (retrieval, tools), testing the real application function is often better than testing a prompt in isolation:
# tests/test_llm_regression.py
import json, pytest
from app.bot import answer # your real application entry point
from evals.graders import contains_all, contains_none, EMAIL
from evals.judge import judge
CASES = [json.loads(l) for l in open("evals/golden_v3.jsonl", encoding="utf-8")]
@pytest.mark.llm
@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_golden(case):
out = answer(case["input"], country=case["country"])
s, why = contains_all(out.text, case.get("key_points", []))
assert s == 1.0, why
s, why = contains_none(out.text, [EMAIL] + case.get("forbidden_patterns", []))
assert s == 1.0, why
if case.get("judge_policy"):
v = judge(case["policy"], case["country"], case["input"], out.text)
assert v["verdict"] == "PASS", v["rationale"]Non-determinism and statistics
LLM outputs vary between runs, and small eval sets are noisy. Before declaring a regression or an improvement:
- Pin what you can: model version (not a moving alias) where the provider allows, temperature, prompt version, retrieval index snapshot.
- Aggregate, then gate: gate on pass rate per dimension with a tolerance, not on every single case, except for blocking must-never-happen cases, which must all pass.
- Compare paired results: run baseline and candidate on the same items and analyze per-item differences.
- Use confidence intervals: a bootstrap over items is simple and effective.
# compare.py: paired bootstrap CI for pass-rate difference (candidate - baseline)
import random
def paired_bootstrap(base: list[int], cand: list[int], n=10_000, seed=0):
rnd = random.Random(seed); idx = range(len(base)); diffs = []
for _ in range(n):
sample = [rnd.choice(idx) for _ in idx]
diffs.append(sum(cand[i] - base[i] for i in sample) / len(sample))
diffs.sort()
return diffs[int(0.025 * n)], diffs[int(0.975 * n)]
lo, hi = paired_bootstrap(base=[1,1,0,1,1,0,1,1], cand=[1,1,1,1,1,0,1,1])
print(f"95% CI for improvement: [{lo:+.2%}, {hi:+.2%}]") # if CI includes 0, not conclusiveWiring it into GitHub Actions
# .github/workflows/llm-evals.yml
name: llm-evals
on:
pull_request:
paths: ["prompts/**", "app/llm/**", "evals/**", "retrieval/**"]
permissions:
contents: read
pull-requests: write
jobs:
evals:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install -r requirements.txt
- run: pytest -m llm --junitxml=eval-results.xml
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
JUDGE_MODEL: ${{ vars.JUDGE_MODEL }}
- uses: actions/upload-artifact@v4
if: always()
with: { name: eval-results, path: eval-results.xml }Post a summary comment (pass rates per dimension, blocking failures, CI of the difference) so reviewers see quality impact next to the diff.
Worked example
A UK-based SaaS company's prompt engineers shipped weekly prompt tweaks. After adding a tiered suite, a seemingly harmless instruction ("be more concise") failed three blocking cases: the bot stopped mentioning mandatory cancellation rights. The PR was fixed before merge. Over a quarter, the suite caught several regressions that would previously have reached customers.
Pitfalls
- Gating on noise: blocking a PR because one non-critical case flipped.
- Moving model aliases silently changing behavior between runs.
- Suites that are too slow or expensive, so developers skip them.
- Secrets on fork PRs: do not expose API keys to untrusted workflows.
How to measure success
Every behavior-changing PR shows an eval comparison; blocking cases never regress in production; suite runtime and cost stay within budget.
Key takeaways
- Tier the suite: fast smoke on every commit, regression on LLM-affecting PRs, extended nightly.
- Use promptfoo or pytest against the real application; keep datasets as the source of test cases.
- Pin versions, gate on rates with tolerance, require all blocking cases to pass, and use paired confidence intervals.
- Post eval summaries on PRs and never expose API keys to untrusted fork workflows.
Check your understanding
Quick questions to lock in the lesson. They don’t count towards your certificate.
Put it into practice
Build a 20-case smoke tier from your golden set, run it with pytest or promptfoo in a GitHub Actions workflow with path filters, and post results on PRs.
Enrol for free to save your progress
Reading is always free. Enrol to keep your place, take the final assessment and earn a verifiable certificate.