Not segmenting test results?

Analyzing only aggregate results can hide important segment-level patterns:

Masked effects: Variant A might win for mobile users while Variant B wins for desktop. The aggregate result depends on your mobile/desktop mix, not which is actually better.

Segment-specific insights: New subscribers might respond differently than loyal customers. High-value customers might prefer different approaches than casual browsers.

What to segment by:

Device type (mobile vs desktop). Engagement level (active vs inactive). Customer value (high vs low). Acquisition source. Geography or time zone.

Caution: Segment analysis requires adequate sample size in each segment. Do not over-segment into groups too small for significance.

The average can lie. Segment analysis reveals whether results apply uniformly or differ by audience.