← Capital One Interview Insights
I knew the formulas but fumbled the pooled variance denominator at first, mixed up whether you use nA+nB-2 or something else.
First, clarify the data structure and assumptions: two independent samples of per-user daily watch time, likely with unequal variances and sample sizes. Then compute the pooled-variance t-statistic (assuming equal variances) and the Welch t-statistic (not assuming equal variances), and derive p-values from the respective t-distributions with appropriate degrees of freedom. Finally, compare the results and discuss which is more appropriate given the context.
Pro tip: Always check the equal-variance assumption (e.g., with Levene's test or by comparing variances) before choosing the pooled t-test; in practice, Welch's t-test is more robust and often preferred unless variances are clearly equal.
Identify the two independent samples (control and treatment) of per-user daily watch time. Note sample sizes (n1, n2), means (x̄1, x̄2), and standard deviations (s1, s2). Check assumptions: independence, normality (or large samples), and homogeneity of variances.
Calculate the pooled variance: s_p^2 = [(n1-1)s1^2 + (n2-1)s2^2] / (n1+n2-2). Then compute the t-statistic: t = (x̄1 - x̄2) / sqrt(s_p^2 * (1/n1 + 1/n2)). Degrees of freedom = n1+n2-2.
Calculate the t-statistic: t = (x̄1 - x̄2) / sqrt(s1^2/n1 + s2^2/n2). Degrees of freedom are approximated using the Welch-Satterthwaite equation: df = (s1^2/n1 + s2^2/n2)^2 / [(s1^2/n1)^2/(n1-1) + (s2^2/n2)^2/(n2-1)].
For each t-statistic, compute the two-sided p-value using the corresponding t-distribution with the appropriate degrees of freedom. Report both p-values and compare them.
Compare the results: if variances are similar, the p-values will be close; if not, Welch's p-value is more reliable. Discuss the implications for the A/B test conclusion, considering practical significance and business impact.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where I got a little tripped up trying to be too clever.
Start by clarifying that the choice between one-tailed and two-tailed tests depends on whether the business goal implies a directional hypothesis and whether you care about detecting effects in both directions. For increasing watch time, a one-tailed test is appropriate if you only care about detecting an increase, but you must rigorously justify that a decrease is either impossible or irrelevant. Acknowledge the trade-offs in statistical power, Type I error, and practical decision-making.
Pro tip: Mention that in industry, two-tailed tests are often the default because they guard against unexpected negative effects, but if the business truly only cares about positive lift and the cost of missing a negative is low, a one-tailed test can be justified. Also, note that the choice should be pre-registered to avoid p-hacking concerns.
State the null and alternative hypotheses. For a one-tailed test, the alternative is that the treatment increases watch time (μ_treatment > μ_control). For a two-tailed test, the alternative is that the means are different (μ_treatment ≠ μ_control).
Determine whether the business only cares about detecting an increase (e.g., if a decrease would not change the decision to launch) or if detecting any difference is important. Consider the cost of a false positive (Type I error) versus a false negative (Type II error).
Explain that a one-tailed test has greater power to detect an effect in the specified direction, but it cannot detect an effect in the opposite direction. A two-tailed test is more conservative and allows detection of effects in both directions.
Discuss how the test choice aligns with the decision rule. If the decision is to launch only if watch time increases, a one-tailed test may be sufficient. However, if a decrease would lead to abandoning the feature or further investigation, a two-tailed test is safer.
Provide a clear recommendation (e.g., one-tailed if the goal is strictly to increase watch time and negative effects are not a concern) and justify it with the above considerations, emphasizing rigor and pre-registration.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Straightforward once you have the Welch t-stat and degrees of freedom.
First, clarify that Welch's method is used when the two groups have unequal variances and sample sizes. Then, compute the difference in means, the standard error using the Welch-Satterthwaite formula, and the degrees of freedom. Finally, construct the 95% confidence interval using the t-distribution with the calculated degrees of freedom.
Pro tip: Always check the assumptions: Welch's method is robust to unequal variances but still assumes approximately normal sampling distributions. If sample sizes are small and data are skewed, consider bootstrapping or a non-parametric approach.
Write the formula for the confidence interval: (x̄1 - x̄2) ± t* * sqrt(s1²/n1 + s2²/n2). Mention that Welch's method does not assume equal variances.
Compute the standard error as sqrt(s1²/n1 + s2²/n2), where s1² and s2² are the sample variances, and n1 and n2 are the sample sizes.
Use the Welch-Satterthwaite equation to calculate the degrees of freedom: df = (s1²/n1 + s2²/n2)² / [(s1²/n1)²/(n1-1) + (s2²/n2)²/(n2-1)].
For a 95% confidence interval, find the t-value corresponding to the calculated degrees of freedom and a two-tailed alpha of 0.05 (i.e., t_{0.975, df}).
Plug in the values to get the lower and upper bounds. Interpret the interval in the context of the watch time difference, e.g., 'We are 95% confident that the true mean difference in watch time lies between X and Y minutes.'
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew Cohen's d, forgot Hedges' g corrects for small sample bias with a scaling factor.
Start by clearly defining the two groups and the metric of interest, then compute the pooled standard deviation and the mean difference. Calculate Cohen's d as the standardized mean difference, and apply the small-sample correction factor to obtain Hedges' g. Finally, interpret the effect size in the context of the business problem.
Pro tip: Always report a confidence interval for the effect size, not just the point estimate, to convey uncertainty and statistical rigor. Also, mention that Hedges' g is preferred for small samples (n < 20 per group) to reduce bias.
Identify the two groups (e.g., control and treatment) and the continuous outcome variable. Ensure the data is clean and the groups are independent.
Calculate the mean and standard deviation for each group. These are the building blocks for the effect size calculation.
Compute the pooled standard deviation using the formula: sqrt(((n1-1)*s1^2 + (n2-1)*s2^2) / (n1+n2-2)). This assumes equal variances; if variances are unequal, consider using a different denominator.
Cohen's d = (mean1 - mean2) / pooled_sd. This standardizes the mean difference.
Apply the correction factor J = 1 - 3/(4*(n1+n2)-9) to Cohen's d: g = d * J. This reduces bias in small samples.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Central limit theorem saves you with large n, that part was easy.
Start by explaining the Central Limit Theorem (CLT) and how it ensures the sampling distribution of the mean is approximately normal for large samples, making the t-test robust to non-normality. Then discuss robust alternatives like the Mann-Whitney U test or bootstrap methods, and compare their trade-offs in terms of power, assumptions, and interpretability. Finally, tie it back to practical decision-making in A/B testing at Capital One.
Pro tip: Emphasize that with large samples, the t-test is often robust even with heavy tails, but always check for extreme outliers or skewness that could affect the mean; in such cases, consider trimmed means or bootstrapping to maintain power while addressing robustness.
Describe how the CLT ensures that the sampling distribution of the sample mean approaches normality as sample size increases, regardless of the population distribution, provided the population has finite variance.
Highlight that the t-test is relatively robust to violations of normality for large samples because the test statistic's distribution is approximately normal, and the t-distribution converges to the normal distribution.
Propose alternatives such as the Mann-Whitney U test (non-parametric), bootstrap hypothesis testing, or trimmed mean tests (e.g., Yuen's test) that are less sensitive to outliers and heavy tails.
Discuss trade-offs: non-parametric tests may have lower power if the distribution is not heavy-tailed, bootstrap methods can be computationally intensive but flexible, and trimmed means reduce the impact of outliers but may lose efficiency if tails are not heavy.
Connect the choice of test to practical considerations in A/B testing, such as sample size, effect size, and the cost of false positives/negatives, to recommend the most appropriate method.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.