DoorDash·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

DoorDash data science interview that went deep into experiment design for a courier equipment rollout. One big multi-part question that basically covered every corner of causal inference I've ever studied, all at once.

Questions Asked (1)

We want to give couriers thermal bags to cut costs from cold-food refunds. How would you design a rigorous experiment around this, covering randomization unit choice, primary metrics and guardrails, stratification, noncompliance, and the full analysis plan?

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This thing had five sub-parts and I felt the walls closing in around part (c).

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business goal and defining the causal question: does providing thermal bags reduce cold-food refunds? Then outline a randomized controlled trial with courier-level randomization, pre-registered metrics, and a detailed analysis plan that accounts for noncompliance and stratification. Emphasize practical constraints like interference and cost, and propose a robust analysis that estimates the intent-to-treat (ITT) effect and possibly the complier average causal effect (CACE).

Pro tip: Mention that you would stratify by historical cold-food refund rate and market to improve power, and that you'd use a CACE analysis to estimate the effect for couriers who actually use the bags, since noncompliance is likely.

1. Define the causal question and randomization unit

Clarify the treatment (providing thermal bags) and outcome (cold-food refunds). Choose courier-level randomization because the intervention is at the courier level, but discuss potential interference if couriers share bags or if customers order from multiple couriers.

2. Select primary metrics and guardrails

Primary metric: cold-food refund rate per delivery (or per courier). Guardrails: total delivery time, customer ratings, courier satisfaction, and cost of bags. Also consider secondary metrics like overall refund rate and repeat order rate.

3. Design stratification and sample size

Stratify by market, courier tenure, and historical cold-food refund rate to reduce variance and ensure balance. Calculate sample size based on expected effect size, power, and intra-courier correlation (if multiple deliveries per courier).

4. Address noncompliance and treatment assignment

Since not all couriers given bags will use them, plan an ITT analysis as the primary analysis, and a CACE analysis using randomization as an instrument to estimate the effect for compliers. Track bag usage via surveys or app prompts.

5. Pre-register analysis plan and execute

Pre-register the analysis plan including primary and secondary metrics, subgroup analyses, and handling of missing data. Use regression adjustment for stratification variables and cluster-robust standard errors if needed. Monitor for novelty effects and ensure blinding where possible.

Key Points to Mention

Randomization unit: courier-level vs. delivery-level, and implications for interference and analysis.
Primary metric: cold-food refund rate; guardrails: delivery time, customer ratings, courier satisfaction.
Stratification by market, courier tenure, and historical refund rate to improve power.
Noncompliance: ITT vs. CACE analysis, and tracking bag usage.
Sample size calculation accounting for clustering and expected effect size.
Pre-registration and avoiding p-hacking; monitoring for novelty effects.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.