Uber·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026

Summary

Uber DS interview focused on a pretty specific airport ETA modeling scenario. One question but it had a lot of layers, metric design plus experiment design plus causal inference all bundled together. Felt like a case study more than a standard interview question.

Questions Asked (1)

You have a new model that predicts walk-time from a rider's order location to the airport pickup zone, meant to replace the current ETA model. What metrics would you use to evaluate its impact on both riders and drivers? How would you design an experiment to compare the two models? And since riders might change their walking speed based on what the app shows them, how do you separate that behavioral effect from whether the model itself is actually more accurate?

A/B Testing & ExperimentationProduct Analytics & MetricsRoot Cause Analysis

Author's notes

The metrics part felt manageable.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining success metrics for both riders and drivers, then outline a randomized controlled experiment (A/B test) comparing the new model to the current one, and finally propose methods to disentangle model accuracy from behavioral changes, such as using a holdout group or instrumental variables.

Pro tip: Emphasize the importance of guardrail metrics to ensure the new model doesn't negatively impact overall marketplace health, and suggest using a switchback or staggered rollout to account for time-based confounders.

1. Define Evaluation Metrics

Identify key metrics for riders (e.g., wait time, walk time, cancellation rate, satisfaction) and drivers (e.g., idle time, trip completion rate, earnings per hour) that reflect the impact of the new model.

2. Design Experiment

Propose a randomized controlled trial where riders are randomly assigned to either the new model or the current model, ensuring proper randomization and sample size calculation.

3. Isolate Model Accuracy from Behavior

Use a holdout group that receives no ETA information or a fixed ETA to measure pure model accuracy, or employ causal inference methods like instrumental variables to separate behavioral effects.

4. Analyze and Iterate

Compare metrics between groups, check for statistical significance, and consider segment-level analysis to understand heterogeneous effects.

Key Points to Mention

Use of both primary metrics (e.g., walk time accuracy) and guardrail metrics (e.g., overall ETA accuracy, rider cancellations).
Randomization unit: rider-level or session-level to avoid contamination.
Potential novelty effects and how to account for them (e.g., long-run holdout).
Behavioral response: riders may adjust walking speed based on displayed ETA, so need to measure actual vs. predicted walk time.
Causal inference techniques: difference-in-differences, instrumental variables, or structural models.
Consideration of marketplace equilibrium effects: driver behavior may also change, affecting overall system.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.