Start by clarifying the business goal (e.g., increase order volume, improve delivery efficiency, or enter new markets) and define success metrics that align with that goal. Then design a randomized controlled experiment across cities, ensuring proper randomization, sample size, and analysis plan to detect meaningful effects while accounting for biases.
Pro tip: Consider the network effects and interference between treatment and control cities, as delivery options may spill over across city boundaries. Use a cluster randomized design with cities as randomization units and possibly a switchback or geo-based holdout to mitigate contamination.
Clarify the primary objective (e.g., increase total orders, improve delivery times, or expand market share) and state null and alternative hypotheses. Identify secondary goals like customer satisfaction or courier utilization.
Choose primary and guardrail metrics (e.g., orders per capita, delivery time, cancellation rate). Decide on randomization unit: city-level cluster randomization is typical for geo experiments to avoid contamination.
Select a subset of comparable cities and randomly assign to treatment (bike delivery) or control (no bike delivery). Address potential biases: selection bias (ensure cities are similar), novelty effect, and spillover effects between cities.
Calculate required sample size using power analysis, considering expected effect size, variance, and intra-cluster correlation. Determine test duration to capture stable behavior and account for seasonality.
Use appropriate statistical methods (e.g., difference-in-differences, mixed-effects models) to estimate treatment effect. Check for heterogeneous effects and validate with robustness checks. Provide clear recommendation based on statistical and practical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.