I went two-sided initially out of habit, then second-guessed myself mid-answer.
Start by clearly defining the metric and the population, then state the null hypothesis as no difference in trip completion rate between control and treatment, and the alternative as a difference. Discuss the choice between one-sided and two-sided tests based on the business context and potential for both positive and negative effects, ultimately recommending a two-sided test for robustness.
Pro tip: At Uber, always consider the two-sided test as the default because it guards against unexpected negative impacts, which can be costly in a marketplace with network effects. However, if the treatment is a strict improvement with no risk of harm, a one-sided test can be justified to increase power.
Clarify that the metric is trip completion rate per request, calculated as the number of completed trips divided by the number of requests, and specify the population (e.g., all riders in the experiment).
Null hypothesis (H0): The trip completion rate per request is equal between the control and treatment groups (p_control = p_treatment).
Alternative hypothesis (H1): The trip completion rate per request is not equal between the control and treatment groups (p_control ≠ p_treatment) for a two-sided test, or specify direction for a one-sided test (e.g., p_treatment > p_control).
Consider the business context: if the treatment could have both positive and negative effects, use a two-sided test. If only an improvement is possible and a decrease is not a concern, a one-sided test may be appropriate. At Uber, due to network effects and potential unintended consequences, a two-sided test is generally recommended.
Explain that a two-sided test is more conservative and guards against detecting effects in either direction, which is important for maintaining marketplace balance and rider/driver experience.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly defining the null and alternative hypotheses, then compute the sample proportions for both groups. Use the pooled proportion to calculate the standard error for the z-test, and the unpooled standard error for the confidence interval. Finally, compute the z-statistic, p-value, and 95% confidence interval, and interpret the results in the context of Uber's experiment.
Pro tip: Always state your assumptions (e.g., independence, large sample size) and check them. Also, clarify whether the test is one-sided or two-sided based on the business question, as this affects the p-value interpretation.
State the null hypothesis (no difference in completion rates) and alternative hypothesis (difference or lift). Define p_control and p_treatment as the true completion rates.
Calculate the sample completion rates for control and treatment. Compute the pooled proportion for the z-test and the unpooled standard error for the confidence interval.
Use the pooled standard error to compute the z-statistic. Find the p-value using the standard normal distribution, considering the appropriate alternative hypothesis.
Use the unpooled standard error to construct a 95% confidence interval for the difference in proportions (treatment - control).
Explain the practical significance of the lift, the p-value, and the confidence interval in the context of Uber's experiment. Discuss any limitations or next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Welch's t-test over pooled because the SDs differ across groups (3.2 vs 3.4) and you shouldn't assume equal variance without a reason to.
Start by clarifying that wait time is a guardrail metric, so the goal is to detect harm, not just significance. Recommend a two-sample t-test (Welch's) for comparing means, but also consider a non-parametric test if the distribution is heavily skewed. Then compute the 95% CI for the mean difference using the appropriate standard error and state assumptions like independence, normality (or large sample), and equal variances (if using Student's t-test).
Pro tip: Mention that guardrail metrics often require one-sided tests or non-inferiority margins to avoid false alarms, and that you'd check for outliers and consider bootstrapping if normality is questionable.
Confirm that wait time is a continuous guardrail metric where we want to ensure the treatment doesn't significantly increase wait times. Define the null and alternative hypotheses (e.g., H0: μ_treatment - μ_control = 0 vs. H1: μ_treatment - μ_control > 0 for harm detection).
Select Welch's t-test because it does not assume equal variances and is robust to unequal sample sizes. If the data is highly skewed or has outliers, consider a non-parametric test like Mann-Whitney U or a bootstrap approach.
Assumptions: independent observations, approximately normal sampling distribution of the mean difference (or large sample sizes due to CLT), and for Welch's t-test, no assumption of equal variances. Mention that if sample sizes are small and data non-normal, use bootstrap.
Calculate the difference in sample means, the standard error using Welch's formula, and the degrees of freedom. Then use the t-distribution to find the critical value and construct the CI: (x̄1 - x̄2) ± t* * SE.
Interpret the CI: if it includes zero, no significant difference; if it lies entirely above zero, the treatment may harm wait times. Consider practical significance and guardrail thresholds.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew the procedure but blanked on the ordering step for a second.
First, clarify the raw p-values for each of the three outcomes and confirm they are from independent tests. Then apply Holm-Bonferroni by ordering the p-values from smallest to largest, comparing each to alpha/(m - rank + 1), and stopping at the first non-significant result. Finally, report which effects survive the correction and discuss the implications for decision-making.
Pro tip: In practice, guardrail metrics often have lower power or higher variance, so a non-significant guardrail after correction doesn't necessarily mean it's safe—consider practical significance and confidence intervals alongside p-values. Also, be prepared to explain why Holm-Bonferroni is preferred over Bonferroni for its greater power while still controlling FWER.
Obtain the raw p-values for the primary metric, cancel rate guardrail, and wait time guardrail. Order them from smallest to largest: p(1) ≤ p(2) ≤ p(3).
For each rank i (1 to 3), compute the threshold α/(m - i + 1) where m=3 and α=0.05. So thresholds are 0.05/3 ≈ 0.0167, 0.05/2 = 0.025, and 0.05/1 = 0.05.
Compare p(1) to 0.0167; if significant, compare p(2) to 0.025; if significant, compare p(3) to 0.05. Stop at the first p-value that exceeds its threshold; all subsequent hypotheses are not rejected.
Identify which outcomes have p-values below their respective thresholds. These are the effects that survive Holm-Bonferroni correction at FWER = 0.05.
Explain what the surviving effects mean for the experiment: e.g., if the primary metric survives but a guardrail does not, consider whether the guardrail is practically violated despite lack of statistical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the p-value in plain language: it's the probability of observing data as extreme or more extreme than what you saw, assuming the null hypothesis is true. Then clearly state what it does not mean—it is not the probability that the null hypothesis is true, nor does it measure effect size or practical significance. Finally, connect this to A/B testing at Uber by emphasizing that decisions should combine p-values with effect sizes, confidence intervals, and business context.
Pro tip: Mention that with Uber's massive sample sizes, tiny and practically irrelevant effects can produce very small p-values, so always pair statistical significance with practical significance and guard against peeking or multiple testing.
State that the p-value is the probability of seeing an effect at least as extreme as observed, assuming the null hypothesis (no effect) is true. Avoid jargon like 'conditional probability' unless you explain it.
Emphasize that it is not the probability that the null hypothesis is true, nor the probability that results are due to chance, nor a measure of effect size or importance.
Explain that in A/B tests, a small p-value suggests the observed difference is unlikely under the null, but it doesn't tell you if the difference is large enough to matter for the business.
Recommend using effect sizes, confidence intervals, and business metrics (e.g., revenue, retention) alongside p-values to make informed decisions.
Note that large samples can yield tiny p-values for trivial effects, and that p-hacking, peeking, and multiple comparisons inflate false positives.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This one tripped me up more than I expected.
Acknowledge that the unplanned interim looks inflate the Type I error rate, so the original p-value is no longer valid. Explain how to apply Pocock or O'Brien-Fleming alpha spending to correct the significance threshold, and discuss the trade-offs between the two approaches. Conclude with a recommendation on how to interpret the results and what to do next.
Pro tip: Mention that Pocock is better for detecting early large effects, while O'Brien-Fleming is more conservative early and preserves power for the final analysis—choose based on the study's goals and expected effect timing.
State that unplanned interim looks without alpha spending inflate the family-wise error rate (FWER), making the naive p-value invalid.
Quantify how the significance conclusion changes: the effective alpha is higher than 0.05, so the observed effect may not be statistically significant after correction.
Describe how to use Pocock or O'Brien-Fleming boundaries to adjust the significance threshold at each look, and compute the adjusted p-value or critical value.
Contrast Pocock (constant boundaries, more power for early effects) and O'Brien-Fleming (conservative early, more power at the end) in terms of Type I error control and power.
Suggest which method fits the context (e.g., O'Brien-Fleming for unknown timing) and state the corrected conclusion, possibly recommending a follow-up experiment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
CUPED variance reduction is 1 minus r-squared, so with r=0.40 you get roughly 16% variance reduction.
Start by recalling the CUPED variance reduction formula: the reduction in variance is equal to the square of the correlation between the pre-experiment covariate and the outcome. Then compute the variance reduction as 0.40^2 = 0.16, meaning a 16% reduction in variance. Finally, explain that this reduction translates to a proportional decrease in required sample size (since sample size is proportional to variance) and discuss how it affects the interpretation of results, such as increased power or narrower confidence intervals.
Pro tip: Emphasize that CUPED uses pre-experiment data to reduce variance without introducing bias, and that the variance reduction is directly proportional to the square of the correlation. Also, mention that while the sample size reduction is substantial, it's important to ensure the pre-experiment covariate is unaffected by the treatment and is measured before the experiment.
State that the proportional variance reduction from CUPED is given by ρ^2, where ρ is the correlation between the pre-experiment covariate and the outcome metric.
Plug in ρ = 0.40 to get ρ^2 = 0.16, meaning a 16% reduction in variance (or equivalently, the variance is multiplied by 0.84).
Explain that required sample size is proportional to variance, so a 16% reduction in variance leads to approximately a 16% reduction in required sample size (or equivalently, the same sample size yields higher power).
Mention that with CUPED, confidence intervals become narrower and p-values smaller for the same effect size, potentially leading to earlier stopping or detection of smaller effects.
Highlight that CUPED requires the pre-experiment covariate to be uncorrelated with treatment assignment and measured before the experiment; otherwise, bias may be introduced.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The cancel rate going up is the real tension here.
Start by restating the decision options and the key evidence from the analysis, including guardrail results. Then weigh the trade-offs between statistical significance, practical impact, and business risk to make a clear recommendation. Defend it by acknowledging limitations and outlining next steps or monitoring plans.
Pro tip: Show that you understand the difference between statistical significance and practical significance, and always tie your recommendation back to Uber's core metrics like rider/driver experience, marketplace efficiency, and long-term growth.
Briefly recap the primary metric results, guardrail metrics, and any segment-level findings from the experiment. Highlight whether the results are statistically significant and practically meaningful.
Discuss the balance between potential gains in the primary metric and any negative impacts on guardrails (e.g., rider cancellations, driver earnings, safety). Consider short-term vs. long-term effects and scalability.
Choose one of the three options—ship, iterate, or stop—based on the evidence and trade-offs. Be decisive and align your choice with Uber's strategic priorities.
Justify your recommendation by referencing specific data points, business context, and potential risks. Address counterarguments and explain why your choice is optimal.
If shipping, describe monitoring and rollback plans. If iterating, specify what changes to test next. If stopping, suggest alternative approaches or learnings to apply.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.