I knew the pooled proportion formula but fumbled the traffic adjustment.
Start by clearly stating the sample size formula for a two-proportion z-test, then plug in the baseline rate (5%) and target rate (5.5% for a 10% relative lift). Compute the required sample per variant, then adjust for targeting match and bucketing success to find the effective daily traffic per variant, and finally divide to get the calendar duration.
Pro tip: Always sanity-check your sample size with a quick rule of thumb (e.g., ~16 * p(1-p)/delta^2 for 80% power and alpha 0.05) and mention that real-world experiments often need a guardrail period or ramp-up, so the calendar estimate is a lower bound.
State the baseline conversion rate (p1 = 0.05), the target rate after a 10% relative lift (p2 = 0.055), alpha = 0.05 (two-sided), and power = 0.80 (beta = 0.20).
Use the standard formula for comparing two proportions: n per variant = (z_{1-alpha/2} + z_{1-beta})^2 * [p1(1-p1) + p2(1-p2)] / (p2 - p1)^2, and explicitly mention the variance assumption (binomial variance under each proportion).
Plug in z-values (1.96 for alpha=0.05 two-sided, 0.84 for 80% power) and the proportions to calculate n; show the intermediate arithmetic and round up to the nearest integer.
Calculate effective daily traffic: 50,000 * 0.60 (targeting match) * 0.90 (bucketing success) = 27,000 users per day, then split 1:1 to get 13,500 per variant per day.
Divide the required sample per variant by the effective daily traffic per variant to get the number of days, and state the final duration in days (round up).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Z-test is the right call for proportions at scale and I said so, but then they pushed on small samples and unequal splits and I got a bit wobbly.
Start by clarifying that for a proportion-based metric, the Z-test is typically used because the variance is determined by the null hypothesis proportion, and with large samples the normal approximation is valid. Then discuss when the T-test might be considered (e.g., small samples, unknown variance) and how unequal group sizes and variance estimation affect the choice. Finally, explain that in practice results diverge mainly for small samples or extreme proportions, and mention corrections like Welch's T-test or continuity correction.
Pro tip: Emphasize that at Uber's scale, the Z-test is standard due to large samples, but showing awareness of edge cases (e.g., small samples, rare events) and practical corrections demonstrates depth beyond textbook knowledge.
State that the metric is a proportion (e.g., conversion rate) and discuss the assumptions: independence, large sample size for normal approximation, and variance estimated under the null.
Explain that for proportions, the Z-test is standard because the variance is a function of the proportion, and with large samples the test statistic follows a normal distribution.
Mention that T-test is used when variance is estimated from data and sample size is small, but for proportions with large samples, Z and T converge; T-test is not typically used for proportions.
Explain that results diverge meaningfully only for small samples or extreme proportions; mention small-sample corrections (e.g., Fisher's exact test, continuity correction) and unequal group sizes (e.g., Welch's T-test for continuous, but for proportions use pooled variance or unpooled Z-test).
Summarize that for Uber's scale, Z-test is appropriate, but always check assumptions and consider exact tests for small samples or rare events.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by acknowledging the stakeholder's desire for early stopping and framing it as a trade-off between speed and statistical rigor. Then propose a sequential testing framework (e.g., group sequential designs with alpha-spending functions) that controls Type I error while allowing interim analyses. Finally, quantify the impact on sample size and expected duration, and discuss practical considerations like monitoring frequency and communication.
Pro tip: Emphasize that you would pre-specify the stopping rules and spending function before the test begins to avoid p-hacking, and use simulation to show stakeholders the trade-offs (e.g., inflated sample size if effect is small). This demonstrates both statistical rigor and business pragmatism.
Explain that early stopping based on promising results inflates Type I error if not properly controlled, and that a sequential testing framework balances speed and error control.
Propose a group sequential design with a pre-specified alpha-spending function (e.g., O'Brien-Fleming or Pocock) that allocates Type I error across interim looks, or a always-valid inference approach like mSPRT.
Specify the number and timing of interim analyses, the spending function, and the decision boundaries (e.g., efficacy and futility) for each look, ensuring they are pre-registered.
Explain that sequential designs typically require a larger maximum sample size than fixed-horizon tests to maintain power, but expected sample size and duration can be lower if the effect is large.
Discuss monitoring frequency, operational overhead, and communication with stakeholders about the pre-specified rules and the trade-offs involved.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the rollout scenario and defining the causal estimand, then sketch a DAG to identify confounders and choose an identification strategy (DiD or IV) that fits the data structure. Finally, explicitly list the key assumptions for that strategy and describe concrete validation checks and sensitivity analyses.
Pro tip: Acknowledge that no observational method is assumption-free; demonstrating how you'd falsify assumptions (e.g., pre-trend tests, placebo outcomes, overidentification tests) signals maturity beyond textbook knowledge.
Define the treatment, outcome, and target population, and confirm what variation exists (e.g., staggered rollout, geographic differences) to determine which quasi-experimental designs are feasible.
Sketch a directed acyclic graph showing treatment, outcome, observed and unobserved confounders, and mediators; use it to identify a valid adjustment set and avoid conditioning on colliders.
Select difference-in-differences if you have pre/post data with a credible control group, or instrumental variables if a valid instrument exists; explain why the design addresses the confounders from the DAG.
For DiD, defend parallel trends with pre-trend tests and placebo outcomes; for IV, defend relevance and exclusion restriction with first-stage strength and overidentification tests; discuss sensitivity analyses for unobserved confounding.
Acknowledge remaining threats (e.g., anticipation effects, SUTVA violations) and propose robustness checks such as alternative control groups, synthetic controls, or bounding methods.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.