This is basically Simpson's paradox dressed up in an A/B testing costume and I did not immediately recognize it as such, which was embarrassing in retrospect.
First, explain how a shift in segment mix (e.g., more returning users in the treatment group) can create a Simpson's paradox where the overall lift is positive despite negative lift for new users. Then, propose a stratified estimator (like a weighted average of segment-level lifts using control segment proportions) to remove aggregation bias. Finally, describe an interaction test (e.g., a regression with treatment×segment interaction) and a decision rule based on effect size, confidence intervals, and business impact to determine if heterogeneity matters.
Pro tip: Emphasize that the overall lift is not wrong, but it answers a different question: 'What is the average effect across the observed mix?' The stratified estimator answers 'What is the effect if the mix were held constant?' This distinction is crucial for product decisions.
Show how a higher proportion of returning users in the treatment group (or a lower proportion of new users) can pull the overall average up, even if each segment's conversion rate is lower in treatment. Use a numerical example with segment sizes and conversion rates to illustrate.
Define a weighted average of segment-level treatment effects, using the control group's segment proportions as weights (or the overall population proportions). This estimates the average treatment effect if the segment mix were identical across groups.
Fit a regression model with treatment, segment, and their interaction (or a chi-square test for homogeneity). The interaction term tests whether the treatment effect differs significantly between segments.
Consider both statistical significance and practical significance: if the interaction p-value is small and the difference in effects exceeds a pre-specified minimum detectable effect (MDE) or has meaningful business impact, then heterogeneity matters. Otherwise, report the overall effect with caution.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.