Start by clarifying the goal: reducing rider cancellations after request by improving ETA clarity. Then design a randomized controlled experiment at the rider level, with careful metric definitions, power analysis, and guardrails. Emphasize practical constraints like interference and novelty effects, and propose a phased rollout with clear success criteria.
Pro tip: In marketplace experiments, always consider interference between treatment and control (e.g., drivers shared across riders) and use cluster randomization if needed. Also, pre-register your analysis plan to avoid p-hacking.
State the hypothesis that a clearer ETA display reduces rider cancellations. Define primary metric (cancellation rate after request) and secondary metrics (ETA accuracy perception, wait time, completed trips). Include guardrails like driver utilization and overall marketplace health.
Randomize at the rider level to avoid contamination, but consider cluster randomization by city or time if interference is a concern. Target all riders who request a trip, but exclude new users or those in ongoing experiments.
Instrument events: ETA card impression, cancellation action, time to cancellation, and trip completion. Calculate sample size using baseline cancellation rate, minimum detectable effect (e.g., 2% relative reduction), power 80%, alpha 0.05, and account for multiple testing.
Start with a small pilot (e.g., 1% of traffic) to catch bugs, then ramp to 50/50. Use holdout groups, control for novelty effects by running for at least 2 weeks, and monitor for SRM (sample ratio mismatch).
Success if primary metric shows statistically significant reduction in cancellations without degrading guardrails. Pre-register analysis: intention-to-treat, segment by rider tenure, and check for heterogeneous effects.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.