This felt like a warmup but I overthought the test choice.
Start by clearly defining the null and alternative hypotheses for the difference in cancellation rates, then outline the two-proportion z-test procedure, including assumptions and test statistic calculation. Next, explain how to compute a 95% confidence interval for the difference using the standard error and z-critical value, and finally interpret the interval in the context of the product change, emphasizing practical significance and decision-making.
Pro tip: Always check the assumptions of the two-proportion z-test (e.g., independence, sample size) and consider using a continuity correction or exact methods if sample sizes are small. Also, discuss the business impact of the confidence interval—does the interval include zero? What does the range imply for Uber's cancellation rates?
State the null hypothesis (H0: p1 - p2 = 0) and alternative hypothesis (H1: p1 - p2 ≠ 0 or one-sided) based on the product change goal. Clarify whether you're testing for any difference or a specific direction.
Verify independence, random sampling, and that the sample sizes are large enough for the normal approximation (np ≥ 10 and n(1-p) ≥ 10 for both groups). If not, consider Fisher's exact test or other methods.
Calculate the pooled proportion, standard error under the null, and the z-statistic. Find the p-value and compare to the significance level (e.g., α = 0.05) to decide whether to reject H0.
Compute the standard error for the difference using unpooled proportions, then the margin of error as 1.96 * SE. The interval is (p1 - p2) ± margin of error.
Explain what the confidence interval means: we are 95% confident that the true difference in cancellation rates lies within the interval. Discuss practical significance, whether the interval includes zero, and implications for the product change.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by writing the logistic regression formula with city and hour-of-day as fixed effects, including an interaction term that captures a plausible effect (e.g., city × hour-of-day). Then discuss key assumptions such as linearity in the logit, independence, and no unmeasured confounding, and outline methods to check calibration (e.g., calibration plots, Hosmer-Lemeshow) and overdispersion (e.g., deviance/df, dispersion parameter).
Pro tip: At Uber, city and hour-of-day are often proxies for unobserved demand and supply shocks; consider using fixed effects or random effects and mention that you'd validate the interaction's business relevance, not just statistical significance.
Write the logistic regression equation with city and hour-of-day as predictors, including an interaction term. For example: logit(p) = β0 + β1*city + β2*hour + β3*(city*hour) + other covariates.
List assumptions: binary outcome, independent observations, linearity of logit for continuous variables, no perfect multicollinearity, and no unmeasured confounding. Also note that city and hour are treated as categorical (fixed effects) unless specified otherwise.
Justify why the interaction matters—e.g., the effect of hour-of-day on the outcome may differ by city due to varying demand patterns. Mention that you'd test its significance and interpret the coefficients.
Describe methods: calibration plots (predicted vs. observed probabilities), Hosmer-Lemeshow test, and calibration belts. Emphasize that good discrimination (AUC) does not guarantee calibration.
For logistic regression, overdispersion is less common but can occur with grouped data. Check by comparing residual deviance to degrees of freedom; if ratio >> 1, consider quasi-binomial or mixed effects models.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Got the basic power calc done, used the standard formula and plugged in the baseline rate around 7%.
Start by calculating the baseline sample size using the standard formula for comparing two proportions, assuming a baseline cancellation rate (e.g., 5%). Then adjust for the intraclass correlation (ICC) by computing the design effect (1 + (m-1)*ICC) and multiplying the sample size accordingly. Clearly state assumptions and interpret the impact on experiment duration and feasibility.
Pro tip: Always clarify the baseline rate and cluster size upfront; in practice, ICC often varies by metric, so sensitivity analysis around ICC assumptions shows rigor and helps stakeholders understand risks.
Confirm the baseline cancellation rate, significance level (95%), power (80%), and the cluster size (average number of trips per driver) for the ICC adjustment.
Use the formula for two proportions: n = (Zα/2 + Zβ)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2, where p1 is baseline and p2 = p1 + 0.005.
Design effect = 1 + (m - 1) * ICC, where m is average cluster size. Multiply the base sample size by the design effect to get the required sample size per arm.
Explain how the design effect increases sample size, potentially requiring longer experiment duration or more drivers, and discuss trade-offs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Easier than I expected after the cluster design question.
Explain the stratified bootstrap procedure step-by-step, emphasizing how it preserves the city-level structure of the data. Then compare it to normal approximation, highlighting scenarios where stratification and bootstrapping are preferred, such as small sample sizes per city or non-normal distributions.
Pro tip: Mention that stratification ensures each city's unique cancellation rate distribution is represented, which is crucial when city sizes vary widely. Also, note that bootstrapping can handle complex metrics like ratios without relying on asymptotic assumptions.
Clarify that the cancellation rate difference is computed between two groups (e.g., treatment vs control) within each city, and cities are the strata. The overall difference is a weighted average of city-level differences.
For each city, separately resample with replacement from the observed data within each group (treatment and control) to preserve the city-specific sample sizes and distributions. Compute the city-level cancellation rate difference for each bootstrap sample.
Aggregate the city-level differences into an overall difference using the same weighting scheme as the original estimate (e.g., by city sample size). Repeat this process many times (e.g., 10,000) to build the bootstrap distribution.
Use the bootstrap distribution to form a confidence interval, such as the percentile method (2.5th and 97.5th percentiles) or BCa if bias correction is needed.
Discuss when stratified bootstrap is preferred: small sample sizes per city, non-normal cancellation rate distributions, unequal variances, or when the metric is a complex ratio. Normal approximation may fail due to violated assumptions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.