This tripped me up a bit because I started talking about sessions and they pushed back fast.
Start by outlining the end-to-end analysis process: validate data quality, compute both session-level and user-level metrics, and reconcile differences by explaining the unit of analysis and its implications. Emphasize that the choice depends on the business question and the randomization unit, and discuss how to handle variance and dependencies.
Pro tip: Always check whether the randomization unit matches the analysis unit; if not, use clustered standard errors or delta method to account for within-user correlation. Also, consider the novelty effect and long-term impact by looking at trends over the experiment period.
Confirm the randomization unit (user or session), check for sample ratio mismatch (SRM), and ensure data completeness and consistency across groups.
Calculate session-level and user-level metrics (e.g., conversion rate, revenue per session/user) and their variances, noting that session-level metrics treat sessions as independent while user-level metrics aggregate within users.
Explain why results may differ: session-level analysis can inflate significance due to within-user correlation. Use user-level analysis if randomization is at user level, or apply methods like clustered standard errors or bootstrap for session-level data.
Beyond statistical significance, evaluate effect sizes, confidence intervals, and impact on key business metrics. Consider segment analysis and guardrail metrics.
Summarize results clearly, highlighting the chosen unit of analysis and its rationale, and provide actionable next steps (e.g., rollout, iterate, or stop).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Delta method versus Fieller for ratios, I knew delta method but blanked on Fieller by name.
Start by explaining why ratio metrics (e.g., CTR, conversion rate) and skewed outcomes (e.g., revenue, session time) violate standard assumptions of normality and independence. Then outline a structured approach: define the metric, choose appropriate statistical methods (e.g., delta method, bootstrap, or regression adjustment), and validate assumptions. Emphasize the importance of robust standard errors and practical trade-offs.
Pro tip: Mention that Uber often deals with heavy-tailed metrics like trip fares or wait times, so using methods like the delta method or bootstrap with stratification can be crucial. Also, highlight the need to check for practical significance, not just statistical significance, given large sample sizes.
Clarify whether the metric is a ratio (e.g., clicks/impressions) or a skewed continuous variable (e.g., revenue). Discuss its distributional properties and potential issues like zero-inflation or heavy tails.
Explain why naive methods (e.g., t-test on ratios) fail: ratio metrics have correlated numerator and denominator, and skewed data violate normality. This leads to biased or underestimated standard errors.
Describe methods like the delta method for ratios, bootstrap (especially stratified or clustered), or regression adjustment (e.g., CUPED). Mention when to use each based on sample size and computational constraints.
Discuss checking assumptions via simulation or diagnostics (e.g., QQ plots, bootstrap distributions). Consider sensitivity analyses to ensure robustness.
Emphasize that statistical significance must be paired with practical significance. Discuss how to communicate uncertainty and make decisions in the presence of skewed outcomes.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Two-stage least squares using assignment as the instrument.
Start by clarifying the distinction between intent-to-treat (ITT) and local average treatment effect (LATE), then explain that non-compliance requires instrumental variables (IV) or CACE analysis. Describe how random assignment serves as an instrument for actual treatment receipt, and outline the assumptions and estimation steps.
Pro tip: Mention that you would first check the first-stage F-statistic to ensure the instrument is strong, and discuss the exclusion restriction assumption—showing awareness of its untestable nature and how to argue for its plausibility in an experiment context.
Clarify that the goal is to estimate the effect on compliers (those who engage when assigned to treatment), known as the Complier Average Causal Effect (CACE) or Local Average Treatment Effect (LATE).
Use random assignment as an instrument for actual treatment receipt. The IV estimate is the ratio of the ITT effect on the outcome to the ITT effect on treatment uptake.
Discuss the four IV assumptions: relevance (first-stage), exclusion restriction, independence, and monotonicity. Check the first-stage F-statistic and argue for the plausibility of the exclusion restriction.
Compute the CACE using two-stage least squares (2SLS) or the Wald estimator. Interpret the result as the effect for compliers, noting it does not generalize to always-takers or never-takers.
Mention alternative approaches like per-protocol analysis (with caveats) or principal stratification. Discuss sensitivity analyses for violations of assumptions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
You can't just say 'no effect,' that's the classic mistake.
Start by clarifying that a large p-value does not prove the null hypothesis; it only indicates insufficient evidence against it. Then outline a systematic approach to assess whether the experiment was adequately powered, check for practical significance, and run additional tests to rule out alternative explanations like novelty effects or segment-specific impacts.
Pro tip: Emphasize that at Uber, where experiments run at massive scale, even tiny effects can be statistically significant, so a large p-value often signals a need to check for implementation issues or heterogeneous treatment effects rather than concluding 'no effect'.
Verify that the experiment was properly randomized, had sufficient sample size, and the intended treatment was actually delivered. Calculate the minimum detectable effect (MDE) to see if the experiment was powered to detect a meaningful effect.
Look at the effect size and its confidence interval. A large p-value with a wide interval suggests uncertainty; a narrow interval around zero suggests the effect is likely negligible in practice.
Run subgroup analyses (e.g., by city, user type, device) to see if the treatment had an effect in specific segments that is masked in the overall average. Use interaction tests to check for significant heterogeneity.
Examine the treatment effect over time (e.g., daily or weekly) to detect if early spikes or dips are due to novelty or learning effects. If present, consider extending the experiment or analyzing later periods separately.
Apply non-parametric tests (e.g., Mann-Whitney U) if normality is questionable, or use bootstrapping for robust inference. Also check for metric sensitivity and consider multiple testing corrections if many metrics were analyzed.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that pre-experiment data can be used to create covariates that explain a portion of the outcome variance, thereby increasing the sensitivity of your experiment. Then describe specific techniques like CUPED, stratification, or regression adjustment, and emphasize how these methods reduce variance without introducing bias. Finally, discuss practical considerations such as choosing the right pre-period and validating assumptions.
Pro tip: Mention that using pre-experiment data is especially powerful when the pre-period is long enough to average out noise, but be careful not to include data that could be affected by the treatment (e.g., pre-period must end before experiment start). Also, highlight that CUPED is a standard at many tech companies, including Uber, so showing familiarity with it signals practical experience.
Explain that the primary goal is to reduce the variance of the treatment effect estimate, which increases statistical power and allows detection of smaller effects.
Select pre-experiment covariates that are highly correlated with the outcome metric, such as the same metric measured before the experiment or related behavioral metrics.
Describe methods like CUPED (Controlled-experiment Using Pre-Experiment Data), stratification, or regression adjustment to remove the variance explained by pre-experiment data.
Check that the pre-experiment covariates are balanced across treatment and control groups, and ensure the adjustment does not introduce bias (e.g., by using only pre-treatment data).
Quantify the variance reduction achieved (e.g., via R-squared) and consider combining multiple covariates or using more advanced methods like ML-based adjustments.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Short answer: Bonferroni if you're being conservative, Benjamini-Hochberg if you care more about false discovery rate than family-wise error.
Start by acknowledging the multiple testing problem when analyzing heterogeneous treatment effects across many subgroups. Then describe a structured approach: pre-specify hypotheses, use appropriate corrections (e.g., Bonferroni, Benjamini-Hochberg), and consider hierarchical models or false discovery rate control. Emphasize balancing statistical rigor with practical insights, especially in a product analytics context.
Pro tip: At Uber, where experiments run at scale, it's crucial to distinguish between exploratory and confirmatory subgroup analyses. Pre-register your subgroups and use FDR control for exploratory analyses to avoid false positives while still uncovering actionable insights.
Clearly specify the subgroups you're interested in and whether the analysis is exploratory or confirmatory. Pre-registering subgroups reduces the risk of data dredging.
Select an appropriate method based on the number of tests and desired control: Bonferroni for strict family-wise error rate (FWER), or Benjamini-Hochberg for false discovery rate (FDR) when many tests are involved.
Adjust p-values or use a hierarchical model to shrink estimates. Interpret the results in the context of the business, focusing on effect sizes and confidence intervals, not just statistical significance.
If possible, validate significant subgroup effects in a holdout set or a follow-up experiment to ensure they are not spurious.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.