Uber·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

This was a deep technical screen for a DS role at Uber, basically one massive experiment design question about a new ETA model in a two-sided marketplace. The level of detail expected was pretty brutal and I left unsure if I'd covered enough ground.

Questions Asked (1)

A new rider ETA model may change perceived wait times and cancellations in a two-sided marketplace where drivers move across zones. Design an experiment to estimate the causal impact on rider experience and marketplace health. Cover metrics, randomization unit, interference controls, instrumentation, power calculations, bias mitigation, monitoring, and pre-registration.

A/B Testing & ExperimentationProduct Analytics & MetricsSystem Design

Author's notes

This was basically the whole interview in one prompt.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment around the two-sided marketplace dynamics, then propose a cluster-randomized design (e.g., by city or zone) to mitigate interference from driver movement. Detail key metrics for rider experience and marketplace health, and outline power analysis, bias mitigation, monitoring, and pre-registration.

Pro tip: Emphasize that interference is the biggest threat to validity in marketplace experiments; propose using a switchback or geo-based randomization with buffer zones to isolate treatment effects. Also, pre-register not just the primary metrics but also guardrail metrics to prevent p-hacking and ensure stakeholder alignment.

1. Define Hypotheses and Metrics

Clearly state the causal hypothesis: new ETA model improves rider experience (e.g., reduced perceived wait time, fewer cancellations) without harming marketplace health (e.g., driver utilization, match rates). Select primary, secondary, and guardrail metrics.

2. Choose Randomization Unit and Interference Controls

Because drivers move across zones, individual-level randomization causes spillover. Use cluster randomization (e.g., by city or zone) or switchback designs. Consider buffer zones or spatial separation to minimize interference.

3. Design Experiment and Power Analysis

Determine sample size and duration via power calculations, accounting for intra-cluster correlation. Specify treatment/control split, and plan for heterogeneous effects across zones.

4. Instrumentation and Bias Mitigation

Ensure accurate logging of ETA predictions, actual wait times, cancellations, and driver movements. Mitigate biases like novelty effects, seasonality, and selection bias via randomization checks and covariate adjustment.

5. Monitoring, Pre-registration, and Analysis

Pre-register the design, metrics, and analysis plan. Set up real-time monitoring for guardrail metrics and early stopping rules. Analyze using appropriate methods (e.g., difference-in-differences, CUPED) to estimate causal impact.

Key Points to Mention

Cluster randomization (e.g., by city/zone) or switchback design to handle interference from driver movement.
Metrics: rider perceived wait time (e.g., via surveys or proxy like cancellation rate), match rate, driver utilization, ETA accuracy, and marketplace liquidity.
Power analysis must account for intra-cluster correlation and potential spillover; consider using simulation-based power.
Bias mitigation: pre-experiment covariate balance checks, novelty effect monitoring, and using holdout groups for long-term effects.
Instrumentation: log ETA model outputs, actual wait times, cancellations, and driver locations; ensure data quality checks.
Pre-registration: publicly document hypotheses, metrics, randomization, and analysis plan to prevent p-hacking and ensure transparency.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.