Coinbase·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Apr 2026Remote

Summary

Coinbase data scientist interview with a deep A/B testing case built around a real KYC drop during a volatile BTC period. The question was layered enough that I kept second-guessing whether I was overcomplicating it or not going far enough.

Questions Asked (1)

KYC completion rate on the mobile onboarding funnel dropped 8% week-over-week during a period of high BTC volatility. Design an A/B test to improve completion without increasing fraud risk. Cover your randomization unit, primary and guardrail metrics, how you'd control for market shocks, sample size and duration, sequential monitoring, country-level compliance variation, and a pre-registration plan.

A/B Testing & ExperimentationProduct Analytics & MetricsRoot Cause Analysis

Author's notes

This thing is basically six questions stapled together.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by diagnosing the funnel to identify where the drop occurs and whether it correlates with volatility, then design an experiment that isolates the effect of a targeted intervention while controlling for market shocks and compliance differences. Emphasize guardrail metrics for fraud and a pre-registration plan to ensure validity.

Pro tip: Propose using volatility as a blocking variable or including it as a covariate in your analysis to increase power and account for market shocks. Also, consider running the experiment only in countries with similar compliance regimes to simplify interpretation, or stratify by country if variation is large.

1. Diagnose the Funnel and Hypothesize

Analyze the KYC funnel to pinpoint the step(s) with the largest drop and check if the decline is uniform across countries or user segments. Form a hypothesis about why volatility affects completion (e.g., users rush and make errors, or abandon due to price swings).

2. Design the Experiment

Choose a randomization unit (e.g., user ID) and define the intervention (e.g., simplified UI, progress saver, or real-time assistance). Specify primary metric (KYC completion rate) and guardrail metrics (fraud rate, false positive rate, time to complete).

3. Control for Market Shocks and Compliance

Use volatility as a covariate or stratify randomization by volatility levels. Account for country-level compliance by either restricting to similar countries or including country as a stratification variable and analyzing separately.

4. Determine Sample Size and Monitoring Plan

Calculate required sample size based on baseline completion rate, minimum detectable effect, power, and significance level. Plan duration to capture enough volatility cycles. Implement sequential monitoring with alpha-spending to allow early stopping for efficacy or futility without inflating Type I error.

5. Pre-register and Analyze

Pre-register the hypothesis, metrics, analysis plan, and stopping rules. After the experiment, analyze using appropriate methods (e.g., CUPED with pre-experiment covariates) and check guardrails before declaring success.

Key Points to Mention

Randomization unit: user-level to avoid contamination and account for multiple sessions.
Primary metric: KYC completion rate; guardrail metrics: fraud rate, false positive rate, manual review rate, and time to complete KYC.
Control for market shocks: include BTC volatility as a covariate, stratify randomization by volatility, or use difference-in-differences if a holdout is available.
Country-level compliance: stratify by country or run separate experiments per compliance regime; ensure intervention complies with local regulations.
Sample size and duration: power analysis based on baseline rate and MDE; duration should cover at least one full volatility cycle (e.g., 2-4 weeks).
Sequential monitoring: use group sequential designs or alpha-spending functions to allow interim looks while controlling false positive rate.
Pre-registration: document hypothesis, metrics, analysis plan, and stopping rules on a platform like AsPredicted or internal wiki before launching.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.