What’s A/B vs multivariate testing?

A/B testing (split testing) compares two versions of a single variable: Subject Line A versus Subject Line B, with everything else identical. This isolation lets you attribute any performance difference directly to that one change. A/B tests are straightforward to execute, require smaller sample sizes for statistical significance, and produce clear, actionable insights. Most email testing starts here.

Multivariate testing (MVT) tests multiple variables simultaneously across all possible combinations. If you're testing two subject lines and two hero images, you'd create four versions (Subject A + Image 1, Subject A + Image 2, Subject B + Image 1, Subject B + Image 2). MVT reveals not just which individual elements perform best, but how elements interact-maybe Subject A wins overall, but Subject B combined with Image 2 outperforms everything else.

The tradeoff is complexity and sample size requirements. Multivariate tests need substantially larger audiences to reach significance across all combinations-each additional variable multiplies the required sample. For most email programs, A/B testing delivers 80% of the learning value with 20% of the complexity. Use A/B testing as your default; reserve multivariate testing for high-volume programs with sophisticated optimization goals and the list size to support it.