Google·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Google DS interview with a single deep-dive question on A/B test metric design for a UI color change. The question sounds deceptively simple but they want you to commit to one metric and defend every layer of the definition, no wiggle room.

Questions Asked (1)

An app changes the color of its primary home-screen CTA button. Pick exactly one primary metric for the A/B test and fully defend it: define numerator, denominator, unit of analysis, inclusion/exclusion rules, exposure definition, attribution window, and time horizon. Then explain why that metric is more sensitive and business-aligned than alternatives like CTR or raw revenue, list your guardrail metrics with explicit breach thresholds, describe how you'd handle novelty effects and returning-user contamination in the first 48 hours, and state the minimum sample size or MDE you'd need to make a ship decision within that window.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This one wrecked me a little.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Choose a primary metric that directly measures the button's purpose, such as 'home-screen CTA click-through to conversion' or 'successful task completion rate', and rigorously define its numerator, denominator, unit of analysis, inclusion/exclusion rules, exposure, attribution window, and time horizon. Defend why this metric is more sensitive and business-aligned than CTR or raw revenue, then outline guardrail metrics with breach thresholds, and address novelty effects, returning-user contamination, and sample size/MDE for a 48-hour ship decision.

Pro tip: Emphasize that the primary metric must be a 'success metric' tied to the button's intended action, not just engagement; and propose a sequential testing or always-valid inference approach to handle peeking within the 48-hour window.

1. Select and define the primary metric

Pick a metric like 'home-screen CTA conversion rate' and specify numerator (users who complete the desired action after clicking the CTA), denominator (users exposed to the home screen with the CTA), unit of analysis (user-level), inclusion/exclusion (new users, exclude bots/internal), exposure (first home-screen view), attribution window (e.g., 24 hours), and time horizon (48 hours).

2. Justify metric sensitivity and business alignment

Explain that CTR only measures clicks, not downstream value, and raw revenue is noisy and influenced by outliers; the chosen metric captures the button's direct impact on a key business outcome and is more sensitive to color changes because it focuses on the immediate action the button is designed to drive.

3. Define guardrail metrics and breach thresholds

List guardrails such as overall revenue per user (breach if drop >2%), session duration (breach if drop >5%), and support tickets (breach if increase >10%); set explicit thresholds based on historical variability and business impact.

4. Address novelty effects and returning-user contamination

For the first 48 hours, analyze new users separately to avoid novelty, use a holdback group for returning users, and consider a washout period; if contamination is detected, re-randomize or extend the test.

5. Determine sample size and MDE for 48-hour decision

Calculate required sample size based on baseline conversion rate, desired power (80%), significance (5%), and a minimum detectable effect (e.g., 2% relative lift); if traffic is insufficient, propose a sequential test or extend the window.

Key Points to Mention

Primary metric must be a success metric tied to the button's purpose, not just engagement.
Unit of analysis should be user-level to avoid pseudo-replication.
Attribution window should be short (e.g., 24 hours) to capture immediate effect.
Guardrail metrics with explicit thresholds (e.g., revenue drop >2%) to prevent harm.
Novelty effects: analyze new vs. returning users separately and use holdback groups.
Sample size/MDE: compute based on baseline rate and desired power; consider sequential testing for early decisions.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.