← Capital One Interview Insights
I went with user-level pretty quickly and felt okay about that, but fumbled a bit explaining the bucketing key.
Start by clarifying the experiment's goal and the nature of the treatment to determine the appropriate randomization unit. Discuss trade-offs between user-level, device-level, and impression-level randomization, emphasizing user-level for most cases to avoid contamination. Then explain how to prevent cross-contamination using consistent hashing, platform-specific buckets, and time-based segmentation, and recommend a stable hashing key like user ID.
Pro tip: Mention that while user-level randomization is ideal, in practice you may need to handle edge cases like shared devices or logged-out users by incorporating device ID as a fallback, and always validate the randomization with a sample ratio mismatch (SRM) check.
Understand what the experiment aims to measure and any technical constraints (e.g., login requirements, cross-device behavior). This determines whether user, device, or impression is the appropriate unit.
Compare user-level (avoids contamination, measures long-term effects), device-level (useful when users aren't logged in), and impression-level (high variance, risk of contamination). Recommend user-level as default for most cases.
Use consistent hashing with a stable key (e.g., user ID) to assign users to buckets. For cross-platform, include platform in the hash or use separate experiments per platform if needed. For time slots, ensure users remain in the same bucket across sessions.
Select a key that is stable and unique, such as user ID. If user ID is unavailable, use device ID but be aware of limitations. Hash the key with a salt to avoid correlation with other experiments.
Run an A/A test or check for sample ratio mismatch (SRM) to ensure randomization is working. Monitor for contamination and adjust if necessary.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the experiment's goal and available data, then propose a stratification scheme based on key covariates like tenure or region. Explain how CUPED uses pre-period data to reduce variance, specifying the pre-period window, covariate definition, and the adjusted estimator formula. Emphasize the importance of pre-registration and validation to avoid bias.
Pro tip: Use a pre-period that is long enough to capture stable behavior but not so long that it includes unrelated trends; typically 2-4 weeks before the experiment start. Always check that the covariate is balanced across treatment and control before applying CUPED.
Choose strata based on factors that strongly predict the outcome and are measured pre-treatment, such as customer tenure, region, or prior spending. Ensure strata are mutually exclusive and collectively exhaustive.
Select a pre-period window (e.g., 2 weeks before experiment start) and define the covariate as the same metric measured during that period (e.g., pre-period spend). Ensure the covariate is unaffected by the treatment.
Use the covariate to adjust the outcome via CUPED: compute the adjusted estimator as Y_adj = Y - θ*(X - E[X]), where θ is the coefficient from regressing Y on X in the control group. This reduces variance without introducing bias.
Check covariate balance across treatment and control, and verify that the variance reduction is achieved. Then perform the hypothesis test using the adjusted estimator, potentially combining with stratification for further gains.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the primary KPI with a clear numerator and denominator, then specify guardrail metrics that ensure no harm. Explain your choice of intent-to-treat (ITT) versus per-protocol and justify it based on the experiment's goal. Finally, detail how you handle zeros and outliers, including winsorization, and discuss the implications for analysis.
Pro tip: Mention that you pre-register the analysis plan, including ITT and outlier handling, to avoid p-hacking and ensure validity. Also, note that winsorization should be applied consistently across groups and that you check sensitivity to the threshold.
Clearly state the primary metric, its numerator (e.g., number of successful conversions) and denominator (e.g., total users exposed). Explain why it aligns with the business objective.
Identify guardrail metrics that monitor potential negative impacts (e.g., latency, customer satisfaction). Provide numerators and denominators for each, ensuring they are actionable.
Decide between intent-to-treat (ITT) and per-protocol. Justify ITT for unbiased causal inference, but mention per-protocol for adherence insights. Discuss trade-offs.
Explain how you treat zeros (e.g., include if meaningful, or use zero-inflated models) and outliers (e.g., winsorization at 1st/99th percentiles). Emphasize pre-specification and sensitivity checks.
Describe how you validate assumptions (e.g., A/A tests, bootstrapping) and iterate on metric definitions if needed, while maintaining statistical rigor.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Two-tailed felt obvious to me since a new scheduling policy could plausibly hurt watch time too.
Start by clarifying the business objective and whether the primary KPI is a metric where only improvement matters (e.g., conversion rate) or any deviation is important (e.g., error rate). Then justify your choice of one-tailed or two-tailed test based on that objective, and specify the appropriate statistical test (e.g., two-proportion z-test) along with its test statistic formula. Finally, briefly discuss the implications for sample size and decision-making.
Pro tip: Always tie your statistical choice back to the business context and risk tolerance—interviewers at Capital One value data-driven decisions that align with business goals, so mention how a one-tailed test can increase power but requires a strong prior justification.
Determine whether the primary KPI is a metric where only an increase is beneficial (e.g., conversion rate) or where any change (positive or negative) is of interest (e.g., customer satisfaction). This dictates whether a one-tailed or two-tailed test is appropriate.
If the goal is to detect an improvement in a specific direction and a decrease is not actionable, a one-tailed test is justified. If the goal is to detect any difference, use a two-tailed test.
For a binary KPI (e.g., conversion), use a two-proportion z-test. For continuous KPIs, use a t-test. Mention that the choice depends on the metric type and sample size.
For a two-proportion z-test, the test statistic is z = (p1 - p2) / sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2)), where p_pool is the pooled proportion. For a one-tailed test, compare the z-value to the critical value in the chosen direction.
Explain how the choice affects sample size, power, and the interpretation of results. Note that one-tailed tests require a strong prior and can be controversial, while two-tailed tests are more conservative.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I set up the formula correctly: n per variant equals roughly 2 times (z_alpha/2 plus z_beta) squared times sigma squared divided by delta squared.
Start by clarifying the metric and assumptions, then walk through the sample size formula for a two-sample t-test on means, plugging in the given values to compute the required sample size per variant. After that, discuss sequential monitoring using alpha spending functions (e.g., O'Brien-Fleming) to control type I error across interim looks, and mention practical considerations like duration and traffic.
Pro tip: Always state that the sample size calculation assumes independent observations and normally distributed means (via CLT), and that in practice you'd adjust for any ratio imbalance or variance reduction techniques like CUPED. Also, emphasize that sequential monitoring requires pre-registration of the number and timing of interim analyses to avoid inflating false positives.
Confirm the baseline mean (42s), standard deviation (55s), relative lift (5%), alpha (0.05 two-sided), and power (80%). Define the metric as mean watch time per impression and note that the test is two-sided.
Calculate the absolute lift: 5% of 42s = 2.1s. This is the minimum detectable effect (MDE) you want to detect.
Use the formula for two-sample t-test: n per group = 2 * (Z_{α/2} + Z_{β})^2 * σ^2 / Δ^2, where Z_{α/2}=1.96, Z_{β}=0.84, σ=55, Δ=2.1. Compute n and round up.
If interim looks are planned, use an alpha spending function (e.g., O'Brien-Fleming) to adjust significance thresholds at each look, controlling overall type I error. Mention tools like group sequential boundaries or always-valid inference.
Mention that the calculated n is per variant, total sample size is 2n. Discuss duration based on traffic, and potential adjustments for variance reduction (CUPED) or non-compliance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer around the three core components: confidence interval construction, stratified pooling, and multiple testing correction, then address the required diagnostics. Emphasize practical implementation with clear statistical reasoning, showing you understand both theory and business application.
Pro tip: When discussing multiple testing correction, mention that you'd prioritize guardrail metrics by business impact and use a hierarchical testing procedure (e.g., gatekeeping) rather than applying a blanket Bonferroni correction, which can be overly conservative and miss real issues.
Specify the metric (e.g., conversion rate difference), the method (e.g., normal approximation or bootstrap), and how you'd compute the interval. Mention assumptions like independence and large sample size, and note that for ratio metrics you'd use the delta method or bootstrap.
Explain that you'd use a stratified analysis, such as Cochran-Mantel-Haenszel for binary metrics or inverse-variance weighting for continuous metrics, to combine stratum-specific estimates while accounting for within-stratum variability. Discuss when to pool (e.g., consistent effects) versus report separately (e.g., heterogeneous effects).
Describe controlling the family-wise error rate (FWER) or false discovery rate (FDR) across guardrail metrics. Suggest methods like Bonferroni, Holm-Bonferroni, or Benjamini-Hochberg, and explain how you'd choose based on the number of tests and business risk.
List at least two checks: (1) compare pre-experiment covariates between groups using standardized mean differences or t-tests, and (2) check sample ratio mismatch (SRM) to ensure randomization produced expected group sizes. Mention that you'd also inspect distributions of key metrics at baseline.
Describe at least two diagnostics: (1) plot metric trends over time for both groups to detect novelty spikes or decay, and (2) compare results across different time periods (e.g., weekdays vs. weekends) or use a holdback group to assess seasonality. Mention that you'd also look for interaction between treatment and time.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Short answer: trim the readout window to exclude the last two days for any conversion-related metrics, or flag those days as incomplete and report a sensitivity analysis with and without them.
First, acknowledge the issue and propose a systematic approach to handle incomplete offline conversion data, such as truncating the analysis window or using statistical methods to account for missing data. Then, discuss the implications for the readout, including potential bias and reduced power, and suggest sensitivity analyses to validate findings.
Pro tip: Always consider the business impact of the missing data: if offline conversions are a key metric, incomplete data could lead to incorrect decisions, so it's crucial to communicate uncertainty and recommend extending the measurement window if possible.
Determine how much data is missing and whether it's missing completely at random or systematically (e.g., due to reporting lag). This informs the appropriate handling method.
Consider truncating the experiment window to exclude the last 48 hours, or use imputation techniques (e.g., last observation carried forward, model-based imputation) if data is missing at random.
Quantify how the missing data affects the experiment's power and whether it introduces bias. For example, if the treatment effect is expected to be immediate, truncation may be acceptable; if delayed, it could bias results.
Perform analyses under different assumptions (e.g., best-case, worst-case) to understand the robustness of the results. This helps in communicating uncertainty to stakeholders.
Clearly state the limitations of the readout, suggest whether the experiment should be extended or if additional data collection is needed, and provide a recommendation based on the sensitivity analyses.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.