← Capital One Interview Insights
I went with one-tailed since the product team only cares about detecting an increase, not a decrease.
Start by clearly defining the metric (watch time per impression) and the two groups (control and treatment). Formally state the null hypothesis as no difference in means and the alternative as a difference, then discuss whether to use a one-tailed or two-tailed test based on the goal of detecting an increase. Conclude with a recommendation and justification.
Pro tip: Mention that while a one-tailed test is appropriate for detecting an increase, it requires strong justification and pre-registration to avoid p-hacking concerns; in practice, many companies default to two-tailed tests for robustness.
Clarify that the metric is watch time per impression, and identify the control (A) and treatment (B) groups. Ensure the metric is continuous and the comparison is between means.
Null hypothesis (H0): μ_A = μ_B (no difference in mean watch time per impression). Alternative hypothesis (H1): μ_A ≠ μ_B for a two-tailed test, or μ_B > μ_A for a one-tailed test if detecting an increase.
If the goal is specifically to detect an increase (i.e., treatment improves watch time), a one-tailed test is appropriate. However, consider the trade-offs: one-tailed tests have more power to detect an increase but cannot detect a decrease, and may be seen as less rigorous.
Explain that the choice depends on the research question and business context. If the business only cares about detecting an increase and a decrease would not lead to action, a one-tailed test is justified. Otherwise, use a two-tailed test.
Recommend a one-tailed test if the goal is strictly to detect an increase and it is pre-registered; otherwise, default to a two-tailed test for flexibility and to avoid criticism.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This took me longer than I'd like to admit.
Start by clearly stating the null and alternative hypotheses and the assumptions (independent groups, approximately normal or large samples). Then compute the group means, variances, and sample sizes, plug them into the Welch t-statistic formula, calculate the Welch–Satterthwaite degrees of freedom, find the p-value using the t-distribution, and construct the 95% confidence interval for the difference in means using the t-critical value. Show all formulas and round final answers to 3 decimal places.
Pro tip: Always interpret the results in the context of the business problem—e.g., whether the observed difference is practically significant—and mention that Welch's test is robust to unequal variances and sample sizes, which is common in A/B testing.
Define H0: μ1 = μ2 vs. H1: μ1 ≠ μ2 (or one-sided if appropriate). Verify assumptions: independent samples, approximately normal or large n, and note that Welch's test does not assume equal variances.
Calculate the sample means (x̄1, x̄2), sample variances (s1², s2²), and sample sizes (n1, n2) for each group.
Use the formula t = (x̄1 - x̄2) / sqrt(s1²/n1 + s2²/n2). Compute the Welch–Satterthwaite degrees of freedom: df = (s1²/n1 + s2²/n2)² / [ (s1²/n1)²/(n1-1) + (s2²/n2)²/(n2-1) ].
Using the t-distribution with the calculated df, find the two-sided p-value and the critical t-value for a 95% confidence level (t_{0.975, df}).
Compute the 95% CI: (x̄1 - x̄2) ± t_{0.975, df} * sqrt(s1²/n1 + s2²/n2). Interpret the interval and p-value in the context of the experiment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly stating the formula for Cohen's d using pooled standard deviation, then walk through a concrete example with numbers to show the calculation. Finally, interpret the resulting effect size in practical terms, relating it to the business context such as the magnitude of improvement in a metric.
Pro tip: Always relate the effect size back to business impact—e.g., a small effect might still be valuable if it translates to significant revenue. Also, mention that Cohen's d assumes normality and equal variances, so check those assumptions.
Explain that Cohen's d measures the standardized difference between two means, useful for comparing effect sizes across different scales. Mention that it's calculated as the difference in means divided by the pooled standard deviation.
State the formula: s_p = sqrt(((n1-1)s1^2 + (n2-1)s2^2) / (n1+n2-2)), where n1, n2 are sample sizes and s1, s2 are standard deviations. Then Cohen's d = (mean1 - mean2) / s_p.
Use a simple example, such as two groups with means 10 and 8, standard deviations 2 and 2.5, and sample sizes 30 each. Compute pooled SD and then d, showing each step.
Refer to Cohen's benchmarks: 0.2 small, 0.5 medium, 0.8 large. Explain what the calculated d means in context—e.g., a d of 0.8 indicates a large effect, meaning the difference is substantial relative to variability.
Discuss how this effect size translates to real-world outcomes, such as a lift in conversion rate or revenue. Emphasize that statistical significance alone isn't enough; effect size shows practical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The p-value halves when you go one-tailed, so if two-tailed was around 0.023 the one-tailed version is about 0.012.
Start by explaining that a one-tailed test concentrates the entire alpha in one direction, so for an observed effect in the pre-registered direction, the p-value is exactly half of the two-tailed p-value. Then discuss the risks: if the true effect is in the opposite direction, the one-tailed test will never detect it, and if you choose the direction after seeing the data, you inflate Type I error. Finally, emphasize the importance of pre-registration and the trade-offs in practical A/B testing.
Pro tip: Mention that in industry, stakeholders often want to know if a change is 'better' or 'worse', so a one-tailed test can be tempting, but it's safer to use a two-tailed test and report directional conclusions only if the effect is significant in the expected direction. This shows you balance statistical rigor with business needs.
Clarify that a one-tailed test allocates all alpha to one direction, while a two-tailed test splits alpha across both tails. This sets the foundation for understanding p-value changes.
State that for an effect in the pre-registered direction, the one-tailed p-value is half the two-tailed p-value. For an effect in the opposite direction, the one-tailed p-value is 1 minus half the two-tailed p-value (i.e., non-significant).
Highlight that if the true effect is in the opposite direction, you lose power to detect it and may incorrectly conclude no effect. Also, if you choose the direction after peeking at data, you inflate Type I error and invalidate the test.
Stress that pre-registration is crucial to avoid p-hacking. In A/B testing, consider whether the business question truly warrants a directional hypothesis or if a two-tailed test is more robust.
Recommend using one-tailed tests only when there is strong theoretical or practical justification for a single direction, and always pre-register. Otherwise, default to two-tailed tests for flexibility and integrity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Power analysis questions always make me nervous because I mix up the z-score values.
Start by stating the two-sample t-test sample size formula for comparing means, then plug in the given values (alpha=0.05 two-sided, power=80%, delta=2, sigma=30.5). Compute the required per-group sample size and round up to the nearest integer, briefly explaining each component.
Pro tip: Mention that this calculation assumes a two-sided test and equal variances; in practice, you might adjust for unequal variances or use a z-test approximation for large samples. Also, note that the formula gives the minimum sample size, so rounding up is necessary.
Use the standard formula for sample size per group in a two-sample t-test: n = 2 * (Z_{1-α/2} + Z_{1-β})^2 * σ^2 / Δ^2, where Z values are from the standard normal distribution.
For α=0.05 two-sided, Z_{1-α/2} = 1.96. For 80% power, Z_{1-β} = 0.84. Sum them: 1.96 + 0.84 = 2.80.
Substitute σ=30.5, Δ=2, and the sum of Z's into the formula: n = 2 * (2.80)^2 * (30.5)^2 / (2)^2. Calculate step by step: (2.80)^2 = 7.84, (30.5)^2 = 930.25, so numerator = 2 * 7.84 * 930.25 = 14586.32; denominator = 4; n = 3646.58.
Since sample size must be an integer, round up to 3647 per group. Explain that this is the minimum required to achieve 80% power at the given significance level.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.