← Thumbtack Interview Insights
Start by defining statistical power clearly as the probability of detecting a true effect of a given size, then systematically rank the levers that increase power based on their practical impact and feasibility in an A/B testing context. Emphasize that while sample size and effect size are the most impactful, other levers like variance reduction and stratification can be highly effective with less cost.
Pro tip: In practice, increasing sample size or test duration is often the easiest lever, but focusing on variance reduction techniques (e.g., CUPED) and stratification can yield substantial power gains without extending test time, which is crucial for fast-paced environments like Thumbtack.
Explain that power is the probability of correctly rejecting the null hypothesis when a true effect exists, typically set at 80% or 90%. Relate it to Type II error (beta) as power = 1 - beta.
List the main levers: effect size, variance, sample size, allocation ratio, alpha, variance reduction, stratification, and test duration. Rank them from most to least practical impact, considering ease of implementation and typical constraints.
For each lever, briefly describe how it affects power. For example, larger effect size or sample size increases power; reducing variance or using stratification also increases power; alpha and allocation ratio have smaller or more nuanced effects.
Highlight that while increasing sample size or duration is straightforward, it may be costly or slow. Variance reduction and stratification can be more efficient but require technical implementation. Allocation ratio and alpha adjustments are less impactful and may have ethical or business implications.
Summarize that the most practical levers depend on context: for quick tests, focus on variance reduction and stratification; for longer tests, sample size and duration. Always consider the trade-offs between power and resources.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew the formula but blanked on whether to use pooled or unpooled variance under the null.
Start by clarifying the metrics and assumptions, then compute the baseline and target conversion rates. Use the standard formula for sample size per variant for a two-proportion z-test, and finally adjust for attrition and daily traffic to determine the test duration.
Pro tip: Always state your assumptions explicitly (e.g., normality approximation, no novelty effects) and round up sample size and duration to ensure sufficient power. Mention that in practice, you'd validate with a power analysis tool or simulation.
Identify baseline conversion (p1 = 0.05), relative lift (7%), so target conversion p2 = 0.05 * 1.07 = 0.0535. Note alpha = 0.05 two-sided, power = 0.80, so beta = 0.20, z_alpha/2 = 1.96, z_beta = 0.84.
Use the formula for two-proportion z-test: n = ( (z_alpha/2 * sqrt(2*p_bar*(1-p_bar)) + z_beta * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ) / (p2 - p1)^2, where p_bar = (p1+p2)/2. Compute n and round up.
Since 10% of users drop out post-randomization, the required number of randomized users per variant is n_adjusted = n / (1 - 0.10). Round up to ensure enough completers.
Total eligible users per day = 80,000. With equal allocation, each variant gets 40,000 users per day. Duration in days = n_adjusted / 40,000, rounded up to the nearest whole day.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Straightforward once you know that variance reduction scales sample size linearly.
First, recall the standard sample size formula and how variance enters it, then apply the 30% variance reduction to compute the new sample size. Finally, translate the sample size change into test duration by considering the traffic allocation and any constraints.
Pro tip: Emphasize that variance reduction techniques like CUPED can dramatically shorten experiments, but they require pre-experiment data and careful validation to avoid bias. Mention that the effective variance reduction may be less than the theoretical maximum, so it's prudent to be conservative in planning.
State the formula for sample size per variant: n = (Z_{α/2} + Z_β)^2 * (2 * σ^2) / Δ^2, where σ^2 is the variance of the metric and Δ is the minimum detectable effect. Explain that sample size is directly proportional to variance.
With a 30% relative variance reduction, the new variance is 0.7 * σ^2. Substitute this into the formula to get the new sample size: n_new = 0.7 * n_original. Thus, the required sample size decreases by 30%.
Assuming traffic remains constant, test duration is proportional to sample size. So the new duration is 0.7 times the original duration, i.e., a 30% reduction in time. If traffic is split evenly, each variant gets half the traffic, but the relative reduction remains the same.
Mention that the actual variance reduction may be less than 30% due to imperfect correlation with the covariate. Also, note that CUPED requires pre-experiment data and may not be applicable to all metrics. Consider the trade-off between implementation complexity and time savings.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that power depends on the variance of the treatment effect estimate, which is inversely proportional to the harmonic mean of the group sizes. Then derive the relative efficiency of a 90/10 split versus a 50/50 split, showing that the 90/10 split requires about 1.8 times more total traffic to achieve the same power. Finally, discuss practical implications and when such an unequal split might be justified.
Pro tip: Mention that while 90/10 reduces power, it can be useful when treatment is risky or costly, but you should quantify the trade-off and consider using a more balanced split like 80/20 if possible.
Explain that power is the probability of detecting a true effect, and it depends on effect size, sample size, and variance. For a two-sample test, the variance of the difference in means is proportional to 1/n1 + 1/n2.
For a fixed total sample size N, with allocation proportion p to treatment and 1-p to control, the variance is proportional to 1/(pN) + 1/((1-p)N) = 1/(N p(1-p)). Thus, the variance factor is 1/(p(1-p)).
For 50/50, p=0.5, variance factor = 4. For 90/10, p=0.1 (treatment), variance factor = 1/(0.1*0.9) = 11.11. The ratio of variances is 11.11/4 = 2.78, so the 90/10 split requires 2.78 times more total traffic for the same power. Alternatively, power is reduced.
If total traffic is fixed, the effective sample size is reduced by a factor of 2.78. For example, if a 50/50 split with N=1000 per group gives 80% power, a 90/10 split with the same total N=2000 would give only about 50% power (since the variance is 2.78 times larger, the standard error is 1.67 times larger, so the z-score is reduced by 1.67).
Conclude that unequal splits reduce power, so they should be used only when necessary (e.g., risk mitigation, cost constraints). Suggest alternatives like 80/20 which has variance factor 1/(0.8*0.2)=6.25, requiring only 1.56 times more traffic than 50/50.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by validating the experiment's integrity—check for sample ratio mismatch, instrumentation errors, and data quality issues before interpreting the negative lift. Then, investigate potential biases and confounds such as bot traffic, demographic imbalances, novelty effects, outliers, and segment-level heterogeneity. Finally, assess guardrail metrics and peeking risk to determine if the result is trustworthy and actionable.
Pro tip: Always pre-register your analysis plan and avoid peeking at results prematurely; if you must peek, use sequential testing or alpha spending to control false positives. Document every diagnostic step to build a clear audit trail for stakeholders.
Check for sample ratio mismatch (SRM) using a chi-square test, audit instrumentation for logging errors, and verify data pipeline correctness. Ensure the randomization unit and assignment are consistent.
Identify and filter bot traffic, examine demographic imbalances between control and treatment, and test for novelty effects by analyzing time-series trends. Handle outliers via winsorization or robust methods.
Break down results by key segments (e.g., user type, geography, device) to uncover heterogeneous treatment effects. Evaluate guardrail metrics to ensure no unintended harm.
Assess peeking risk by checking if the analysis was triggered early; if so, apply corrections like sequential testing. Consider multiple testing adjustments for segment analyses.
Integrate findings to determine if the negative lift is real or an artifact. If valid, investigate root causes (e.g., implementation bugs, user experience issues) and decide on next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by emphasizing the importance of diagnosing the negative result to distinguish between a true negative effect and issues like low power, implementation bugs, or novelty effects. Then, propose a decision framework that systematically evaluates the validity of the test, the business impact, and the cost of errors to choose between shipping, iterating, or rerunning. Be specific about the conditions and metrics that trigger each path.
Pro tip: Always consider the cost of a false negative versus a false positive in the context of the business; sometimes shipping a change with a negative result is acceptable if the downside is minimal and the potential upside is large. Also, involve stakeholders early to align on decision criteria.
Check for common pitfalls: sample ratio mismatch, instrumentation errors, novelty effects, and insufficient power. Ensure the negative result is not due to a flawed experiment.
Quantify the effect size and confidence interval. Determine if the negative effect is practically significant and if the confidence level meets pre-defined thresholds.
Consider the cost of shipping a harmful change versus the cost of missing a positive effect. Also, assess if there is a promising variant or segment that warrants iteration.
Based on steps 1-3, choose: ship if the negative effect is negligible or acceptable; iterate if there is a hypothesis for improvement; rerun if the experiment was invalid or underpowered.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.