I fumbled around with 'use a non-parametric test' and mentioned the Mann-Whitney U, which was fine but felt incomplete.
Start by clarifying that non-normality is common in metrics like revenue or session duration, and it doesn't automatically invalidate the test. Then walk through a structured diagnostic and remediation process, emphasizing that the right fix depends on sample size, metric type, and business context.
Pro tip: At Netflix scale, with millions of users, the Central Limit Theorem often makes the t-test robust to non-normality; but for heavy-tailed metrics, consider capping outliers or using a bootstrap to get reliable confidence intervals.
Plot the metric distribution and compute skewness/kurtosis to understand the nature and severity of non-normality. Check if it's due to outliers, zero-inflation, or a naturally skewed metric like revenue.
Determine if the sample size is large enough for the Central Limit Theorem to apply, or if the non-normality is severe enough to bias results. Consider the test's sensitivity and the metric's importance.
Select an appropriate method: transform the metric (e.g., log), use non-parametric tests (e.g., Mann-Whitney), bootstrap the sampling distribution, or apply robust methods like trimmed means. For heavy tails, consider capping outliers.
Run simulations or sensitivity analyses to confirm the chosen method's validity. Clearly communicate the assumptions and limitations to stakeholders, and align on the decision-making framework.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.