This looks like a clean experiment question until you realize how many pieces they actually want you to hit.
Start by framing the change as a hypothesis about user behavior, then systematically design the A/B test covering all requested components. Emphasize the importance of guardrail metrics and practical considerations like network effects and sample size.
Pro tip: Mention that Google Maps has a global user base, so consider localization and device differences in your design. Also, highlight the need to monitor long-term effects and user satisfaction, not just short-term engagement.
State a clear hypothesis (e.g., moving the search bar to the bottom increases search engagement due to easier thumb access) and select primary metric (e.g., search queries per user) and guardrail metrics (e.g., app crashes, time to first search, user retention).
Decide on the randomization unit (e.g., user-level) and strategy (e.g., random assignment within each country/device type) to avoid contamination and ensure representative samples.
Use power analysis to determine required sample size based on baseline metrics, minimum detectable effect, significance level, and power. Estimate runtime based on daily traffic and desired sample size.
Launch the experiment, monitor for technical issues and novelty effects, and ensure data quality. Consider running for at least one full week to capture weekly patterns.
Compare primary and guardrail metrics between control and treatment groups using statistical tests. Check for statistical significance, practical significance, and segment-level effects. Decide whether to launch, iterate, or abandon.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.