Coinbase·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

May 2026Remote

Summary

Coinbase data scientist interview with a pretty meaty stats question on A/B testing. One question but they really drilled into it, asking follow-ups that branched in directions I wasn't fully prepared for.

Questions Asked (1)

You're running an online experiment comparing conversion rates between a treatment and control group. Walk through how you'd compute a 95% confidence interval for the treatment effect by hand, covering the point estimate for absolute lift, the standard error derivation, the CI formula itself, how things change if they ask for relative lift instead, and when the normal approximation breaks down versus when you'd reach for something like a bootstrap or score-based interval.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

This looked like a clean textbook question at first and I thought I had it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly defining the metric and the estimand (absolute lift), then walk through the point estimate, standard error, and CI formula step by step. After covering absolute lift, explain how to transform to relative lift and discuss the assumptions and alternatives when the normal approximation is questionable.

Pro tip: Always state the assumptions behind the normal approximation (e.g., large sample sizes, independent observations) and mention that in practice you'd check with a bootstrap or use a score interval when conversion rates are extreme or sample sizes are small.

1. Define the estimand and point estimate

Clarify that the treatment effect is the difference in conversion rates: p_t - p_c. Compute the point estimate as the difference in observed proportions.

2. Derive the standard error

Under independence, the variance of the difference is the sum of variances: Var(p_t) + Var(p_c). Estimate each variance as p(1-p)/n using the observed proportions, then take the square root to get the standard error.

3. Construct the 95% confidence interval for absolute lift

Use the formula: (p_t - p_c) ± 1.96 * SE. Explain that 1.96 is the critical value from the standard normal for 95% confidence.

4. Convert to relative lift

For relative lift, the point estimate is (p_t - p_c) / p_c. The standard error can be approximated using the delta method: SE_rel ≈ SE_abs / p_c. Then the CI is relative_lift ± 1.96 * SE_rel.

5. Discuss when the normal approximation fails and alternatives

The normal approximation may be poor when sample sizes are small or conversion rates are near 0 or 1 (e.g., np < 5 or n(1-p) < 5). In such cases, use a bootstrap (resample users and recompute the difference) or a score-based interval (e.g., Wilson score interval for each proportion and combine).

Key Points to Mention

Independence assumption between treatment and control groups
Variance of a proportion: p(1-p)/n
Standard error of the difference: sqrt( p_t(1-p_t)/n_t + p_c(1-p_c)/n_c )
Critical value 1.96 for 95% confidence under normality
Delta method for relative lift standard error
When to use bootstrap or score intervals (small samples, extreme proportions)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.