Plaid·Product Manager·Onsite - Product Sense / Strategy·Senior

Senior

Jun 2026

Summary

Plaid PM interview with a product strategy question about helping Netflix roll out ACH payments in the US. Pretty focused case, one question but it had a lot of layers once you started pulling on it.

Questions Asked (1)

You're helping Netflix introduce ACH as a payment option in the US. How would you design a test to decide whether it should be rolled out broadly?

A/B Testing & ExperimentationProduct StrategyProduct Analytics & Metrics

Author's notes

My first instinct was to jump straight into experiment design but I realized pretty fast I hadn't defined what success even looked like for Netflix here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the goal: to determine if ACH increases conversion, reduces costs, or improves retention without harming key metrics. Then design an experiment that isolates the impact of ACH availability, using a randomized controlled trial with a holdout group, and measure both primary and guardrail metrics. Finally, outline how you would analyze results and make a rollout recommendation based on statistical significance and business impact.

Pro tip: Emphasize the importance of segmenting results by user cohorts (e.g., new vs. existing, high vs. low transaction value) to uncover heterogeneous treatment effects, and consider the long-term impact on customer lifetime value, not just short-term conversion.

1. Define Hypothesis and Success Metrics

Articulate a clear hypothesis (e.g., offering ACH will increase payment success rate and reduce cost per transaction) and define primary metrics (e.g., conversion rate, payment success rate) and guardrail metrics (e.g., involuntary churn, customer satisfaction).

2. Design the Experiment

Choose a randomized controlled trial where a subset of users sees ACH as a payment option (treatment) and a control group does not. Ensure proper randomization, sample size calculation, and duration to detect meaningful effects.

3. Identify Segments and Potential Confounders

Plan to analyze results across key segments (e.g., new vs. existing users, payment amount tiers) and account for confounders like device type or time of day. Consider whether ACH availability might cannibalize other payment methods.

4. Analyze Results and Measure Impact

Compare treatment and control groups on primary and guardrail metrics using statistical tests. Calculate the incremental lift and assess whether it meets the pre-defined threshold for rollout (e.g., >2% increase in conversion).

5. Make Rollout Recommendation

Based on the analysis, recommend a full rollout, limited rollout, or no rollout. Consider operational factors like cost savings, customer support implications, and strategic alignment with Netflix's goals.

Key Points to Mention

Randomized controlled trial (A/B test) with a holdout group to measure causal impact.
Primary metric: payment conversion rate or success rate; secondary metrics: cost per transaction, customer lifetime value.
Guardrail metrics: involuntary churn, customer satisfaction, support ticket volume.
Segment analysis to understand heterogeneous effects (e.g., by user tenure, transaction size).
Statistical significance and power analysis to ensure reliable results.
Business considerations: cost savings from lower fees, impact on cash flow, and scalability.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.