Almost every company that runs A/B tests has the same trajectory. The first ten tests are exciting. Somewhere around test forty the wheels come off: results contradict each other, a “22% uplift” fails to show up in revenue, two teams ship conflicting changes in the same week, and someone senior starts asking whether any of this is real.
That is not a tooling problem. It is what happens when test volume grows faster than the framework around it. This is the framework: the decisions to make once, so that running a hundred experiments a year produces compounding knowledge instead of accumulating noise.
The problem with scale, stated honestly
At a 5% significance level, one in twenty tests of a completely useless change will show a significant result. Run 100 such tests and you will “discover” five wins that are pure chance. Ship them all and you have shipped nothing, while your dashboard says you have improved conversion by 15%.
Everything below exists to stop that. The single biggest predictor of whether an experimentation programme is trustworthy is not the tool, it is whether decisions were fixed before the data arrived.
Layer 1: the metric hierarchy
Decide this once, at programme level, not per test. Three tiers.
- One OEC (Overall Evaluation Criterion). The single metric that decides ship or not ship. One, not three. If you cannot pick one, you have not agreed what you are optimising, and you will resolve that argument after seeing the results, which is exactly when you should not be resolving it.
- Guardrail metrics. Things that must not get worse, regardless of the OEC: page latency, error rate, unsubscribe rate, refund rate, support contacts. Guardrails are how you catch wins that cause hidden harm, like a checkout change that lifts conversion and quietly raises chargebacks.
- Diagnostic metrics. Everything else. These explain why a result happened. They never decide anything, and treating them as decision metrics is the most common route to fooling yourself.
The rule that makes this work: the OEC and guardrails are written down before the test starts, in the same document as the hypothesis. If they can move afterwards, none of the statistics mean anything.
Layer 2: powering the test before you run it
Sample size is decided in advance from four inputs: your baseline conversion rate, the minimum detectable effect you care about, statistical power (80% is the usual floor) and your significance level (commonly 5%).
The MDE is the honest conversation, and most teams skip it. Ask: what is the smallest improvement that would be worth the engineering cost of shipping and maintaining this? If the answer is 2% and your traffic can only detect 10% in a reasonable timeframe, the test is not worth running. Knowing that before you build the variant is the cheapest win in the whole framework.
Then commit to a fixed end date, and always run in whole weeks. Tuesday traffic does not behave like Sunday traffic, and a test that runs nine days over-weights whichever days it double-counted.
Layer 3: stopping rules, and why peeking breaks everything
Checking a running test repeatedly and stopping the moment it crosses significance inflates your false positive rate far above the 5% you think you are running at. The p-value fluctuates; watch it long enough and it will cross the line by chance.
Two legitimate options, and one illegitimate one.
- Fixed horizon. Calculate the sample size, run to it, look once. Simple, statistically clean, requires discipline.
- Sequential testing. Use a method designed for continuous monitoring, which lets you look whenever you like and stop early with valid inference. This is the grown-up answer at scale, and most serious platforms now support it natively.
- Peeking at a fixed-horizon test and stopping when it looks good. This is not a method. It is the single most common reason experimentation programmes produce results that do not replicate.
Pick one per test, write it in the test doc, and hold the line when a stakeholder wants to call it on day three.
Layer 4: trust checks that run automatically
These are cheap to automate and catch the failures that quietly invalidate results.
- Sample Ratio Mismatch (SRM). If you split 50/50 and observe 51.8/48.2 on a large sample, something is broken: a redirect dropping users, a bot filter hitting one variant, an assignment bug. An SRM invalidates the test entirely. Do not analyse it, fix it and rerun. Check SRM on every single test, automatically, before anyone reads a result.
- An A/A test, periodically. Run your pipeline against two identical variants. It should find nothing. If it finds significant differences, your instrumentation is lying and every result you have is suspect.
- Guardrail alerts. Automatic, and empowered to stop a test without a meeting.
- Pre-registration. Hypothesis, OEC, guardrails, MDE, sample size and stopping rule, written before launch. This one habit does more for result quality than any statistical technique.
Layer 5: getting more out of the same traffic
Most companies are traffic-constrained, so the highest-leverage technical investment is variance reduction rather than more tests.
CUPED (Controlled experiment Using Pre-Experiment Data) uses each user’s pre-experiment behaviour as a covariate to strip out variance that has nothing to do with your change. Because a user who spent a lot last month will probably spend a lot this month regardless of your button colour, removing that predictable component makes the treatment effect far easier to see. In practice it can roughly halve the time to significance, which is equivalent to doubling your traffic for free.
It requires stable user identity across the pre-period and the experiment, which is exactly the kind of thing that breaks quietly under consent-gated tracking. Worth checking before you rely on it.
Layer 6: what makes it scale, rather than just work
Layers one to five make a single test trustworthy. These make a hundred tests a year coherent.
- A global holdout. Keep a small slice of users, one to five percent, permanently excluded from all shipped changes. Once a quarter, compare them against everyone else. This is the only honest answer to “did our forty wins actually add up to anything”, and the answer is frequently humbling.
- Interaction discipline. With many concurrent tests, users land in multiple experiments. Usually this is fine and randomisation handles it. It is not fine when two tests touch the same surface, so maintain a simple registry of what is running where, and use mutually exclusive layers for tests that genuinely conflict.
- A searchable decision log. Every test: hypothesis, result, decision, and crucially the losers and the flat results. Most organisations record only the wins, which guarantees someone retries the same failed idea in eighteen months. The flat results are the cheapest knowledge you will ever own.
- Fixed cadence and ownership. A weekly readout where results are reviewed against the pre-registered criteria, by someone whose job is the integrity of the programme rather than the success of any individual test.
What to expect, so nobody is disappointed
A healthy programme has a low win rate. Most tests are flat, a meaningful minority lose, and the winners are smaller than the case studies you read. If your win rate is 60% and every uplift is double digit, that is not a great team, it is a measurement problem, and the global holdout will eventually say so.
Two more expectations worth setting early. Novelty effects mean changes often look better in week one than in week four, so be wary of short tests on returning users. And a result that is significant is not automatically material: a 0.3% lift that is real can still be worth less than the maintenance cost of the code that produced it.
How to start, if you are starting
- Agree the OEC and the guardrails. Nothing else matters until this exists.
- Automate SRM checks and run one A/A test to prove the pipeline.
- Introduce pre-registration. A one-page template is enough.
- Pick a stopping rule and enforce it.
- Start the decision log, including failures, from test one.
- Add CUPED and a global holdout once you are running enough volume to need them.
None of this requires an expensive platform. It requires deciding things before you see the data and then not moving them, which is a cultural problem far more than a technical one. The tooling is the easy part; the discipline is the product.
If you want a second pair of eyes on an experimentation programme, or you suspect your results are not replicating and want to know why, get in touch. Diagnosing that is one of our favourite jobs.
