What are common errors in A/B testing?

Common A/B testing errors undermine result validity:

Insufficient sample size: Running tests without calculating required samples leads to inconclusive or misleading results.

Testing multiple variables: Changing several elements prevents knowing what caused any difference.

Stopping early: Ending tests when results look good inflates false positive rates.

Ignoring statistical significance: Acting on raw percentage differences without significance testing leads to decisions based on noise.

Not accounting for external factors: Tests spanning holidays or unusual events produce confounded results.

Testing without hypotheses: Random testing without clear questions produces random learnings.

Not documenting results: Failing to record learnings means repeating mistakes and losing institutional knowledge.

Bad testing is worse than no testing. It consumes resources while producing misleading conclusions that damage future decisions.