The A/B Cemetery is where all those countless tests go to die, forgotten, undocumented, and often unreconciled with product strategy. It’s what happens when teams run experiments but fail to learn from them, act on them, or even remember they happened.
ORIGIN
The phrase plays on the idea of a graveyard for A/B tests. In data-driven product cultures, it’s easy to run lots of experiments, “let’s test everything!”, but without discipline, results pile up, conflicting insights emerge, and nobody knows which changes were kept, reverted, or why decisions were made. The cemetery is full of old tests, dead ideas, and unlearned lessons.
WHEN
You know you’re working in an A/B Cemetery when:
- Nobody can explain why the current version of a page looks the way it does.
- Teams repeat experiments that were already run (and failed) in the past.
- There’s no central log of experiments or their outcomes.
- Stakeholders cherry-pick winning variants without understanding tradeoffs.
- Test results are ignored because they don’t fit preconceived opinions.
It’s particularly common in high-velocity growth teams or organizations that celebrate “test velocity” over insight quality.
WHY
The A/B Cemetery is filled up because:
- There’s no process for documenting results and decisions.
- Team turnover erases institutional memory.
- Experimentation is treated as a numbers game rather than a learning opportunity.
- Winning variants are deployed without analyzing side effects or long-term impact.
Without a culture of learning, experiments just become busywork.
HOW
Here’s how to keep your experiments out of the cemetery:
- Log every test. Keep a centralized record of hypothesis, variants, metrics, results, and decisions.
- Share learnings. Make sure findings are presented and discussed across the team.
- Validate long-term. Revisit “winners” to ensure they still perform over time.
- Kill weak tests early. Not all tests deserve to run, be clear about hypotheses and success criteria.
- Train teams. Help everyone understand how to design meaningful experiments.
PRO TIP
Name your tests descriptively (“Homepage CTA placement test #2”) so future teams know what was tested and why, instead of “Test123” buried in some analytics tool.
EXAMPLES
- A signup page with odd copy nobody can justify because it “won” a test years ago.
- A marketing team running yet another button color test that was already run twice.
- A dashboard full of variants but no record of which metrics improved.
CONCLUSION
The A/B Cemetery reminds us that experiments are only as good as the knowledge they produce. Test less, learn more, and don’t let your product become a patchwork of unexamined changes.
Also known as: Test graveyard • Experiment entropy • Data without memory
