Uber·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Uber DS interview that went deep into experimental design for marketplace pricing. The question was dense and very specific to their domain, covering switchback experiments, carryover effects, and variance estimation all in one go. Felt more like a take-home problem crammed into a live session.

Questions Asked (1)

Design a switchback experiment for airport pickup pricing in a marketplace with spillovers. You're given that median trip duration is 22 minutes, demand autocorrelation drops below 0.1 at 75 minutes, and there are strong demand peaks roughly every 4 hours. Walk through how you'd choose block length, detect residual carryover, compute variance and sample size under block randomization, randomize across time-of-day and day-of-week, and apply covariate adjustment to reduce variance.

A/B Testing & ExperimentationPricing & MonetizationProduct Analytics & Metrics

Author's notes

This was a lot.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by using the given metrics to justify a block length that balances spillover mitigation and statistical power, then outline a step-by-step plan covering block length selection, carryover detection, variance and sample size calculation, randomization strategy, and covariate adjustment. Emphasize the importance of accounting for time-of-day and day-of-week patterns to ensure valid inference.

Pro tip: In switchback experiments, always pre-specify the block length and carryover detection method to avoid p-hacking; use a washout period if carryover is significant, and leverage covariate adjustment with pre-period data to boost power.

1. Choose Block Length

Select a block length longer than the autocorrelation decay time (e.g., 75 minutes) to minimize spillover, but short enough to capture demand variations. Given peaks every 4 hours, consider blocks of 1-2 hours, ensuring at least one full cycle per block.

2. Detect Residual Carryover

After the experiment, test for carryover by comparing outcomes in the first part of a block to later parts, or by including lagged treatment indicators in a regression. If significant, increase block length or add washout periods.

3. Compute Variance and Sample Size

Under block randomization, the variance of the treatment effect estimator depends on between-block and within-block variance. Use the intraclass correlation (ICC) to adjust sample size; with strong autocorrelation, more blocks are needed. Calculate required number of blocks using standard formulas for cluster randomized trials.

4. Randomize Across Time-of-Day and Day-of-Week

Stratify randomization by time-of-day (e.g., morning, afternoon, evening, night) and day-of-week (weekday vs. weekend) to ensure balance. Within each stratum, randomly assign treatment to blocks, ensuring each block gets both conditions if using a crossover design.

5. Apply Covariate Adjustment

Use pre-experiment covariates (e.g., historical demand, weather, events) in a regression model to reduce variance. Include time fixed effects and block-level random effects to account for clustering.

Key Points to Mention

Block length should exceed autocorrelation decay time (75 min) to mitigate spillovers, but consider demand cycles (4-hour peaks) to avoid confounding.
Carryover can be detected via washout periods, lagged treatment variables, or comparing early vs. late block outcomes.
Variance under block randomization includes between-block and within-block components; sample size depends on ICC and number of blocks.
Stratified randomization by time-of-day and day-of-week ensures balance and accounts for temporal patterns.
Covariate adjustment using pre-period data and time fixed effects can increase power and reduce confounding.
Consider using a switchback design with continuous blocks and randomizing treatment at block level, possibly with a crossover within blocks.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.