What is a p-value?

The p-value represents the probability of observing results as extreme as yours if there were actually no difference between variants (the null hypothesis).

A p-value of 0.05 means there is a 5% chance you would see this difference by random chance alone. A p-value of 0.01 means only 1% chance.

Lower p-values provide stronger evidence that the difference is real. The conventional threshold is p < 0.05, meaning results are considered significant if there is less than 5% probability they occurred by chance.

P-values do not measure effect size, practical importance, or probability that your hypothesis is true. They only measure how surprising your results would be if there were no real difference.

P-values answer a narrow technical question. Interpreting them correctly requires understanding what they do and do not tell you about your test results.