← Capital One Interview Insights

Capital One·Data Scientist·Technical Phone Screen·Senior

Senior
May 2026

Summary

Capital One data scientist interview with a deep-dive experimentation question around a credit card acquisition redesign. Single question but it covered basically every angle of A/B testing you can think of, from randomization to fair lending compliance.

Questions Asked (1)

Q1

You're launching a new credit card acquisition flow with updated APR disclosure and a signup bonus. Walk through a complete A/B test design: randomization unit, eligibility exclusions, primary and guardrail metrics, minimum detectable effect, statistical power, sample size, and expected duration. Also address selection bias from pre-approval and underwriting, cross-channel interference, peeking, regulatory constraints like fair lending, and how you'd analyze heterogeneous treatment effects by segment while controlling the false discovery rate.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs
Author's notes

This question is basically a final exam for experimentation.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment around the business goal of increasing acquisition while managing risk and compliance. Then systematically address each design element—randomization, exclusions, metrics, power, duration—and explicitly tackle the biases, interference, peeking, and regulatory constraints. Finally, discuss advanced analysis for heterogeneous treatment effects with FDR control.

Pro tip: In regulated financial services, always involve legal and compliance early to ensure the test design meets fair lending and disclosure requirements; this prevents costly redesigns and demonstrates cross-functional maturity.

1. Define Objective and Randomization Unit

Clarify the primary goal (e.g., increase approved accounts) and choose the randomization unit (e.g., user-level) to avoid contamination. Consider pre-approval and underwriting as eligibility filters.

2. Specify Eligibility, Exclusions, and Metrics

Define who is eligible (e.g., pre-approved applicants) and exclusions (e.g., existing customers, fraud flags). Select primary metric (e.g., application completion rate) and guardrail metrics (e.g., default rate, APR comprehension).

3. Determine MDE, Power, Sample Size, and Duration

Set a minimum detectable effect based on business relevance, choose power (80%) and significance (5%), then calculate required sample size and duration, accounting for traffic and seasonality.

4. Address Biases, Interference, and Peeking

Mitigate selection bias from pre-approval/underwriting via stratified randomization or modeling. Handle cross-channel interference with channel-specific randomization or holdouts. Prevent peeking by pre-registering analysis and using sequential testing.

5. Ensure Regulatory Compliance and Analyze Heterogeneity

Incorporate fair lending constraints (e.g., no disparate impact) and disclosure requirements. Analyze heterogeneous treatment effects by segment (e.g., credit score, channel) using methods like causal forests, controlling FDR via Benjamini-Hochberg.

Key Points to Mention

  • Randomization unit: user-level to avoid contamination, but consider cluster randomization for channels.
  • Eligibility exclusions: pre-approved only, exclude existing customers, fraud, and underwriting declines.
  • Primary metric: application completion or activation; guardrail metrics: default rate, APR comprehension, customer satisfaction.
  • MDE: based on business impact (e.g., 2% relative lift), power 80%, alpha 5%, sample size calculation.
  • Selection bias: pre-approval and underwriting create non-random samples; use stratified randomization or inverse probability weighting.
  • Cross-channel interference: use channel-specific experiments or holdout groups; consider switchback tests.
  • Peeking: pre-register analysis, use sequential testing or alpha spending.
  • Regulatory: fair lending (ECOA), APR disclosure (TILA), ensure no disparate impact; consult legal.
  • Heterogeneous treatment effects: segment by credit score, channel, demographics; use causal forests or meta-learners; control FDR with Benjamini-Hochberg.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.