This is two questions stapled together and the second half is where people fall apart.
Start by clearly defining Type I and Type II errors and their implications in A/B testing, then discuss how to set thresholds based on business costs and statistical power. For skewed distributions, explain robust methods like bootstrapping or transformations to estimate lift and confidence intervals.
Pro tip: Emphasize that thresholds should be driven by the relative cost of false positives vs. false negatives, and that for skewed metrics, bootstrapping is often more reliable than parametric methods because it doesn't assume normality.
Explain that Type I error (false positive) is concluding a difference exists when it doesn't, while Type II error (false negative) is failing to detect a real difference. Relate these to significance level (α) and power (1-β).
Discuss how to set α and β based on business context: consider the cost of false positives (e.g., rolling out a bad feature) vs. false negatives (e.g., missing a good feature). Typical α is 0.05, but may be stricter for high-risk changes; power is often 0.8.
Acknowledge that skewed metrics violate normality assumptions, so standard t-tests may be invalid. Mention robust alternatives like bootstrapping, permutation tests, or transformations (e.g., log) to estimate lift and confidence intervals.
Describe using bootstrapping: resample with replacement, compute the metric difference (lift) for each resample, and derive the confidence interval from the bootstrap distribution. Alternatively, use trimmed means or quantile-based methods.
Check that the chosen method is appropriate (e.g., bootstrap converges) and interpret the confidence interval in business terms, noting that outliers may still influence results and considering robust metrics like median or trimmed mean.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.