A/B test fundamentals and experiment design

0:00 A/B test fundamentals

A test says your new product page lifts conversion by twelve percent. Before you celebrate, can you prove the test itself was sound? Most bad decisions in experimentation don't come from wrong maths. They come from broken tests: uneven splits, flickering variants, missing tracking, overlapping experiments. In this lecture you'll learn what an A/B test really does, the types of tests, how to write a test plan, how to choose the unit of randomisation, and how to catch sample ratio mismatch and flicker. Then you'll watch me check a split with a few lines of Python.

0:42 Why design matters

Why is design so important? Because an A/B test is a measuring instrument. If the instrument is broken, the measurement is meaningless, however impressive it looks. Randomisation is what makes the instrument work. It creates two groups that are the same in every way except the change, so seasonality, campaigns, news and competitor moves affect both equally. Break the randomisation, even slightly, and you're back to comparing apples with oranges.

1:12 Types of tests

Types of tests. A/B: control versus one variant, used for most tests. A/B/n: several variants, which needs more traffic because each variant gets a smaller share. Multivariate tests combine several elements and need very high traffic. Split URL tests send people to different pages, useful for big redesigns. And server-side or feature-flag tests change logic in your code, which suits pricing, search, checkout logic and apps. Every extra variant splits traffic further, so keep variants few unless traffic is large.

1:47 The test plan

Write the plan before you build. Test name, hypothesis and audience. Allocation, usually fifty-fifty. The primary metric, secondary metrics and guardrails. The minimum detectable effect, the sample size and duration, covering at least full weeks. Stop rules: stop early only for guardrail harm or a broken experience. And a QA checklist. The plan is your contract with future you. It stops you from changing the question after you've seen the answer.

2:18 Metrics in the plan

Let's talk about metrics inside the plan, because the plan is where most tests are won or lost. Choose one primary metric that decides the outcome, usually revenue per visitor or purchase conversion for a store, qualified leads for a service. Add a few secondary metrics that explain how the change worked, like add-to-cart rate or checkout starts. And add guardrails that must not get worse: return rate, page speed, support contacts, unsubscribes. Then write the decision rule in one sentence, before launch. For example: ship B if revenue per visitor improves at our planned confidence and no guardrail worsens meaningfully. That single sentence prevents most post-test arguments.

3:05 Unit of randomisation

Unit of randomisation. Most web tests randomise by user, using a cookie or a logged-in user ID, so each person sees a consistent experience every visit. Randomising by page view would show someone different versions on each click, which is confusing and statistically messy. For logged-in products, user-ID assignment works across devices. And think about consent: where consent is required before storing identifiers, the tool may only run for consenting users, so your results describe that group.

3:38 Sample ratio mismatch (SRM)

Now sample ratio mismatch. If you allocate fifty-fifty but see fifty thousand six hundred users in A and forty-nine thousand four hundred in B, something may be wrong. A redirect losing users, a bot filter hitting one variant, or a bug stopping the variant loading on some browsers. A simple chi-square test tells you how likely that imbalance is by chance. If it's very unlikely, commonly a p-value below one in a thousand, the test is invalid until you explain and fix it. Don't read the results first. Check the split first.

4:18 Flicker and QA

Two more threats. Flicker: client-side tools change the page after it starts loading, so the original can flash before the variant appears, which biases results and annoys users. Mitigate by loading the script correctly with anti-flicker handling, keeping variant code light, or testing server-side or at the edge for big changes. And QA: check every variant on key devices, browsers and languages, including right-to-left layouts; confirm tracking fires for all metrics; check the initial split; and make sure no other tests conflict on the same pages.

4:55 Simple example: an invalid 'win'

A simple example of an invalid test. An online store tests a new product page, and after two weeks the variant is winning strongly. The SRM check fails: the variant has noticeably fewer users than expected. Investigation shows the variant breaks on an older mobile browser, so those users, who convert poorly anyway, dropped out of the variant's data but not the control's. The win was an artefact. After fixing the bug and restarting, the effect is much smaller.

5:29 Realistic example: Dubai travel checkout (illustrative)

Now a realistic scenario with illustrative details. A travel company in Dubai runs a checkout test using a client-side tool. Results look flat, and the SRM check passes. But QA had missed something: the variant's layout broke in Arabic, where the right-to-left version pushed the payment button below a sticky banner. Arabic-language users in the variant struggled, dragging the overall result down. The team fixes the layout, adds right-to-left checks to their QA template, and reruns. The lesson: SRM catches missing users, but only QA catches broken experiences for users who are still there.

6:10 Watch me do it: SRM check in Python

Watch me check a split. I have user counts per variant from our data warehouse. I write a tiny function using Python's standard library: it computes the chi-square statistic for the observed split against the expected fifty-fifty and converts it to a p-value. I run it on two tests. Fifty thousand two hundred and ten versus forty-nine thousand seven hundred and ninety gives a p-value of about zero point one eight. Fine. Fifty thousand six hundred versus forty-nine thousand four hundred gives about zero point zero zero zero one five. That's a mismatch. A one point two percent imbalance looks tiny, but on a hundred thousand users it's very unlikely to be chance. So before anyone opens the results dashboard, I start investigating.

7:03 Choosing an approach

Choosing a testing approach. Client-side tools with visual editors let marketers launch copy and layout tests quickly, but they add page weight and flicker, and they can break when the site changes. Server-side testing and feature flags need developers, but handle pricing, search and checkout logic without flicker. Edge testing, at the CDN, serves full-page variants before HTML reaches the browser. Many mature programmes use both client-side and server-side. Whatever you choose, make sure your analytics can see which variant each user received, so you can verify results independently of the tool.

7:43 Common mistakes

Common mistakes. No written plan. Overlapping tests on the same element. Ignoring SRM warnings. Skipping cross-device and right-to-left QA. Too many variants for the traffic. Letting consent settings silently change who's in the test without documenting it. And trusting the tool's dashboard without being able to verify assignments in your own data.

8:06 Recap

Recap. An A/B test is a measuring instrument that works only if randomisation works. Write the plan first, randomise by user, check the split for sample ratio mismatch before reading results, prevent flicker, and QA every variant across devices and languages. Choose client-side, server-side or edge approaches to suit the change. Try this now: take your last test, pull the user counts per variant, run the SRM function from the lesson text, and add right-to-left and in-app browser checks to your QA template.