Calling an A/B test the moment it shows a leading variant, rather than waiting for the sample size the test actually needs, is the single most common reason CRO programs make decisions that don't hold up when re-tested. This is the sample-size-specific companion to my broader conversion rate optimization checklist; if the underlying tracking is unreliable, none of this matters until you fix that first — see my conversion tracking validation checklist.
Why early results are misleading
Early in a test, small sample sizes produce large swings that look dramatic but are mostly noise. A variant can show a 30% lift after 50 conversions per arm and settle to a 4% lift — or no lift at all — by 500. The fix isn't a gut-feel "let it run a bit longer"; it's calculating the required sample size before the test starts, based on your baseline conversion rate and the minimum detectable effect you actually care about.
The three inputs that determine required sample size
- Baseline conversion rate — your current rate before the test; lower baseline rates require larger sample sizes to detect the same relative lift.
- Minimum detectable effect (MDE) — the smallest lift worth acting on. Chasing a 2% lift requires a dramatically larger sample than chasing a 20% lift, so set the MDE at a size that would actually justify a decision.
- Statistical significance and power — 95% significance and 80% power are the standard defaults; loosening either shrinks the required sample but raises the odds of acting on a false result.
| Baseline Rate | Minimum Detectable Effect | Approx. Sample per Arm |
|---|---|---|
| 2% | 20% relative lift | ~9,000 |
| 5% | 20% relative lift | ~3,400 |
| 10% | 20% relative lift | ~1,500 |
| 10% | 10% relative lift | ~6,000 |
These are illustrative, not exact for every calculator — the pattern that matters is that both a lower baseline rate and a smaller MDE sharply increase the sample required, which is why low-traffic pages testing small copy tweaks are rarely a good use of a test slot.
What to do when traffic can't reach the required sample
- Test bigger changes — a full page redesign or offer change produces a larger effect size, which needs a smaller sample to detect than a subtle button-color tweak.
- Combine lower-traffic pages into one test where the change and the audience intent are similar enough to combine honestly.
- Switch to sequential or Bayesian testing methods designed for lower-traffic contexts, rather than forcing a fixed-sample frequentist test onto traffic that will never reach the number it needs.
- Accept that some pages should be optimized on best practice and qualitative research instead of a formal test that traffic can never adequately power.
The pre-registration habit that prevents most mistakes
Write down the required sample size, the MDE, and the planned test duration before launching the test — not after watching the early results. This single habit prevents the most common failure mode: stopping a test early because the numbers look good, then discovering weeks later that the "win" didn't hold.
FAQ
How long should an A/B test run?
Long enough to reach the pre-calculated sample size for your baseline rate and minimum detectable effect, and at least one full business cycle (commonly one to two weeks) to average out day-of-week effects — not a fixed number of days decided in advance without reference to the actual sample size needed.
- Duration should be driven by required sample size, not a calendar guess.
- Running short of a full weekly cycle risks a result skewed by day-of-week traffic patterns.
What happens if I stop a test as soon as it shows significance?
Stopping the moment a test crosses a significance threshold, rather than at the pre-planned sample size, inflates the false-positive rate substantially — a phenomenon sometimes called "peeking" — because early significance often reflects noise that regresses once more data comes in.
- Peeking at results and stopping early is a well-documented source of false positives.
- Pre-registering the required sample size before launch is the simplest fix.