Google·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

A/B testing design question for a DS role at Google. The whole thing was one big open-ended case about search ranking experiments, and they expected you to cover a lot of ground without much hand-holding.

Questions Asked (1)

Design an A/B test to compare two search ranking algorithms. Walk through your hypothesis, primary and guardrail metrics, randomization unit, sample size and statistical power, test duration, segmentation strategy, novelty effects, and how you'd decide whether to ship.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This question has a lot of moving parts and I think I underestimated how much structure they wanted upfront.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the scientific method: start with a clear hypothesis, define success and guardrail metrics, then detail the experimental design (randomization, sample size, duration) and analysis plan. Emphasize how you'd handle practical challenges like novelty effects and segmentation, and conclude with a decision framework for shipping.

Pro tip: Always pre-register your metrics and analysis plan to avoid p-hacking, and consider running a holdback experiment after launch to measure long-term impact.

1. Define Hypothesis and Metrics

State a clear, testable hypothesis about how the new ranking algorithm will improve user experience. Choose a primary metric (e.g., CTR, task success) and guardrail metrics (e.g., latency, revenue) to ensure no harm.

2. Design Experiment

Decide on the randomization unit (e.g., user, session, query) and calculate sample size using power analysis. Determine test duration based on traffic and expected effect size, accounting for weekly seasonality.

3. Address Validity Threats

Plan for novelty effects by running the test long enough and analyzing trends over time. Consider segmentation (e.g., new vs. returning users, query types) to detect heterogeneous treatment effects.

4. Analyze and Decide

Use appropriate statistical tests (e.g., t-test, bootstrap) to compare metrics. Evaluate practical significance and guardrails, then decide whether to ship, iterate, or abandon based on the overall evidence.

Key Points to Mention

Hypothesis should be specific and tied to user value (e.g., 'New algorithm increases long-click rate by 2%').
Primary metric: user engagement (e.g., CTR, dwell time); guardrail metrics: latency, revenue, and user satisfaction.
Randomization unit: user-level to avoid contamination; consider query-level if user-level not feasible.
Sample size calculation: power (80%), significance level (5%), minimum detectable effect (MDE) from historical data.
Test duration: at least one week to capture weekly patterns; extend if novelty effects suspected.
Segmentation: analyze by device, user type, query intent to understand heterogeneous effects.
Decision framework: ship if primary metric improves significantly without guardrail degradation; consider long-term holdback.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.