Stripe·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

Stripe DS interview with a meaty experimentation design question that covers basically everything from OEC definition to quasi-experimental fallbacks. One long multi-part problem, not a series of separate questions. Felt more like a take-home prompt delivered verbally.

Questions Asked (1)

A new recommendation module might cause cross-user interference and traffic seasonality. Design a full evaluation plan: define an OEC and three guardrail metrics with formulas, choose and justify a test design (user-level RCT, geo-cluster, or switchback), describe your ramp and pre-registration strategy including stopping rules and variance reduction, propose a quasi-experimental fallback if randomization isn't feasible, and debug a mid-test scenario where the OEC flatlines while add-to-cart rises and conversion drops.

A/B Testing & ExperimentationProduct Analytics & MetricsRoot Cause Analysis

Author's notes

This was one giant prompt broken into five sub-parts and I did not pace myself well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining a clear OEC that balances user experience and business goals, then select guardrail metrics to monitor potential harms. Justify the test design based on interference and seasonality, outline a phased ramp with pre-registration and stopping rules, and prepare a quasi-experimental fallback. Finally, debug the mid-test scenario by decomposing the metric movements and checking for interference or novelty effects.

Pro tip: Always pre-register your analysis plan and stopping rules to avoid p-hacking and ensure valid inference. When debugging, segment by user cohorts and time to isolate interference or seasonality effects.

1. Define OEC and Guardrail Metrics

Choose an OEC that captures the recommendation module's impact on long-term user value, and select guardrail metrics to detect negative side effects. Provide formulas for each.

2. Choose and Justify Test Design

Evaluate user-level RCT, geo-cluster, or switchback based on interference and seasonality. Justify the choice with trade-offs.

3. Plan Ramp and Pre-registration

Outline a phased rollout with pre-registration of hypotheses, metrics, and stopping rules. Include variance reduction techniques.

4. Propose Quasi-Experimental Fallback

If randomization isn't feasible, suggest a quasi-experimental design like synthetic control or difference-in-differences, and discuss assumptions.

5. Debug Mid-Test Scenario

Analyze why OEC flatlines while add-to-cart rises and conversion drops. Check for interference, seasonality, or metric definition issues, and propose next steps.

Key Points to Mention

OEC formula: e.g., long-term revenue per user or engagement-weighted metric; guardrails: e.g., conversion rate, latency, diversity of recommendations.
Test design: user-level RCT if interference is low; geo-cluster if spillover across users; switchback if time-based seasonality is strong.
Pre-registration: document hypotheses, primary/secondary metrics, sample size, stopping rules (e.g., alpha spending, sequential testing).
Variance reduction: CUPED, stratification, or covariate adjustment to increase power.
Quasi-experimental fallback: synthetic control, difference-in-differences, or interrupted time series with robustness checks.
Debugging: segment by user activity, time, and geography; check for cannibalization, novelty effects, or metric contamination.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.