TikTok·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

Got a pretty brutal product experimentation question at TikTok for a DS role. The whole thing was a multi-part case study about designing an A/B test for a bike delivery opt-in program, and it went way deeper than I expected for a single question.

Questions Asked (1)

A delivery platform wants to let existing car dashers opt into using their own bikes or e-bikes for deliveries while keeping car access. You need to decide whether to scale this program next quarter. Walk through the full experiment design: business goals and a falsifiable primary hypothesis, success and guardrail metrics with exact definitions, randomization unit choice and justification, power analysis inputs, pre-rollout instrumentation, analysis plan for heterogeneous effects and selection bias, and a decision framework for scale-up vs rollback.

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Strategy

Author's notes

This thing had seven sub-parts and I definitely didn't pace myself well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer as a full experiment lifecycle: start with business goals and a falsifiable hypothesis, define metrics and randomization, then cover power, instrumentation, analysis, and decision rules. Emphasize how you'd handle selection bias and heterogeneous effects, since dashers self-select into bike mode. End with a clear scale/rollback framework tied to guardrails.

Pro tip: Treat the opt-in nature as a feature, not a bug: propose an encouragement design or intent-to-treat analysis to estimate causal effects while acknowledging self-selection. This shows you understand real-world experimentation constraints beyond textbook A/B tests.

1. Define business goals and falsifiable hypothesis

State the business objective (e.g., increase delivery efficiency and dasher satisfaction while maintaining reliability) and a falsifiable primary hypothesis, such as: 'Enabling bike mode for car dashers will increase deliveries per active hour by at least 5% without degrading on-time delivery rate.'

2. Specify success and guardrail metrics with exact definitions

Define primary success metrics (e.g., deliveries per active hour, dasher retention) and guardrails (e.g., on-time delivery rate, customer rating, cancellation rate, cost per delivery) with precise formulas, time windows, and data sources.

3. Choose randomization unit and justify

Randomize at the dasher level (or city-level if interference is a concern) because treatment is applied to individuals and spillover between dashers is limited; justify why this unit balances bias and power.

4. Conduct power analysis and pre-rollout instrumentation

Specify inputs: baseline metric values, minimum detectable effect (MDE), significance level (α=0.05), power (1-β=0.80), and expected variance; calculate required sample size and duration. Pre-rollout, ensure logging of mode selection, delivery events, and guardrail metrics, and run A/A tests.

5. Analysis plan and decision framework

Analyze intent-to-treat and per-protocol effects, test heterogeneous effects by dasher tenure, city density, and vehicle type, and correct for selection bias using propensity score matching or instrumental variables. Decide scale-up if primary metric improves and guardrails are not violated; rollback if guardrails degrade significantly.

Key Points to Mention

Falsifiable hypothesis with a specific MDE and direction
Guardrail metrics like on-time delivery, customer rating, and cost per delivery
Randomization at dasher level with justification and potential interference
Power analysis inputs: baseline, MDE, α, β, variance, sample size
Heterogeneous treatment effects by dasher segment and city
Selection bias mitigation via intent-to-treat, encouragement design, or propensity scores

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.