The Pattern
An early-stage company buys an A/B testing tool and starts testing button colours, headlines and layouts on pages with modest traffic. Tests run for weeks, show a leader that flips back and forth, and end without a result anyone believes.
What Teams Usually Try First
Usually more variants per test, more tests at once, and stopping a test as soon as one variant looks ahead.
Where It Stalls
The pages do not get enough visitors to detect the size of change being tested, the tests are not tied to a specific observed problem, and peeking at results early produces false winners. After a few such rounds the team concludes that testing does not work for them.
Recommended Approach
- Before launching, calculate the sample size needed for the page's baseline conversion rate and the smallest lift worth detecting; if it cannot be reached in a few weeks, do not A/B test that page
- For low-traffic pages, make bigger changes (the offer, the page structure, the proof) and learn from qualitative sources first: session recordings, form-field analytics and customer conversations
- Write every test as a hypothesis tied to an observed problem, with one primary metric and a run time fixed in whole weeks, and do not stop it early on a promising peek
- Prioritize pages by traffic and closeness to revenue (pricing, signup, demo request) so the limited traffic is spent where a win matters
- Keep a shared log of every test and its result, including the inconclusive ones, so the team builds knowledge and does not repeat ideas
What It Aims to Achieve
Aims to lead to fewer but better tests, results the team can trust, and a testing programme matched to the traffic the site actually has.