How do I run a statistically valid A/B test?

Running a statistically valid A/B test requires methodical process:

Form a hypothesis stating what you expect to happen and why. "Personalized subject lines will increase opens because they feel more relevant."

Test one variable at a time. Changing multiple elements makes it impossible to know what caused any difference.

Calculate required sample size before starting. Use calculators that factor in your baseline metrics, minimum detectable effect, and desired confidence level.

Randomize assignment ensuring test and control groups are comparable. Most ESPs handle this automatically.

Run to completion without peeking and stopping early. Premature decisions based on incomplete data lead to false conclusions.

Analyze with statistical tools that calculate significance, not just raw percentage differences.

Shortcuts produce unreliable results. A properly run test tells you truth. A poorly run test tells you what you want to hear.