I knew the textbook answer here but fumbled the delivery a bit.
Start by clarifying that 80% power means an 80% chance of detecting a true effect of a pre-specified size, so 20% of the time you'll miss it even when the effect is real. Then explain that a non-significant result with a positive observed effect can occur due to sampling variability, effect size smaller than assumed, or inflated variance. Finally, discuss practical implications and next steps.
Pro tip: Emphasize that power is about the true effect size, not the observed one; a positive but non-significant result often means the observed effect is smaller than the minimum detectable effect, which is common in practice. Also, mention that peeking or multiple testing can inflate false negatives if not adjusted.
Briefly explain that power is the probability of rejecting the null when a true effect of a specific size exists, and significance is about the observed data. Clarify that they are related but distinct.
Discuss how random sampling can produce a positive effect estimate that is not statistically significant due to chance, even with high power. Mention that the observed effect is a point estimate with uncertainty.
Point out that if the true effect is smaller than the assumed effect size used in power calculation, power drops. Also, if variance is higher than expected, the test is less sensitive.
If applicable, note that looking at data multiple times or testing many metrics can increase the chance of false negatives or require adjustments that reduce power.
Suggest that a non-significant result doesn't prove no effect; it may warrant further investigation, larger sample, or consideration of practical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clearly stating that post-hoc power is not informative and should not be calculated after seeing results. Explain that it is a deterministic function of the p-value and adds no new information. Then, describe what to report instead, such as confidence intervals, effect sizes, and sensitivity analyses, and why these are more useful for interpreting the results.
Pro tip: Emphasize that post-hoc power can be misleading because it conflates effect size and sample size, and instead focus on the practical significance of the observed effect. Mention that reporting confidence intervals allows stakeholders to assess the range of plausible effects, which is more actionable for decision-making.
Explain that post-hoc power is calculated using the observed effect size and is therefore a direct function of the p-value. It does not provide additional information about the true power of the test.
Highlight that post-hoc power is often misinterpreted and can lead to erroneous conclusions, such as dismissing a non-significant result as underpowered when the observed effect is small.
Suggest reporting confidence intervals for the effect size, which convey the precision of the estimate and the range of plausible effects. Also, report the observed effect size and its practical significance.
Mention that if power is a concern, a sensitivity analysis can be done to determine the minimum detectable effect given the sample size, but this should be done a priori or as part of planning, not post-hoc.
Emphasize that the goal is to inform decisions, so focus on the effect size and its uncertainty rather than on power. This helps stakeholders understand the practical implications of the results.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Honestly the hardest part of the whole interview.
Start by translating the asymmetric loss into a concrete decision rule: compare the expected cost of shipping versus not shipping, incorporating the effect size, confidence interval, and the 2:1 cost ratio. Then assess whether the current experiment provides enough evidence to make a confident decision, and if not, determine whether extending the experiment or reducing variance is the most efficient path forward.
Pro tip: Frame your answer around expected loss minimization rather than just statistical significance, and explicitly state the threshold (e.g., posterior probability that the treatment is better) that would justify shipping given the cost asymmetry. This shows you think like a business-minded data scientist, not just a statistician.
Convert the 2:1 cost ratio into a decision rule: ship if the expected loss from false positives is less than the expected loss from false negatives. For example, if the treatment effect is positive but uncertain, compute the posterior probability that the true effect exceeds zero and compare it to a threshold derived from the cost ratio.
Examine the observed effect size, confidence interval, and p-value. Determine whether the result is statistically significant and practically significant given the cost asymmetry. If the confidence interval is wide and includes both positive and negative effects, the evidence is inconclusive.
Calculate the expected cost of shipping if the treatment is actually harmful (false positive) versus not shipping if it is beneficial (false negative). Use the 2:1 ratio to weigh these costs. If the expected cost of a wrong decision is high, more evidence is needed.
If the evidence is inconclusive and the potential gain from continuing is high, extend the experiment. If variance is the main issue and you can reduce it (e.g., by using CUPED, stratification, or more precise metrics), do so. If the expected cost of continuing outweighs the benefit, stop and make a decision based on current evidence.
Clearly explain the decision, the rationale, and the trade-offs to stakeholders. Document the assumptions and the cost-benefit analysis for future reference.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Ran through stratification and CUPED as variance reduction options, then talked about recalibrating the MDE based on what we now know the effect might actually be.
Start by diagnosing why the previous experiment fell short—whether it was underpowered, noisy, or too long—then propose targeted changes to MDE, variance reduction, and duration that balance statistical rigor with product velocity. Frame your redesign as a trade-off exercise, explicitly stating assumptions and how you'd validate them with a pilot or power analysis.
Pro tip: At Meta, speed matters: show you can reduce duration without sacrificing power by combining variance reduction (e.g., CUPED) with a slightly relaxed MDE, but always quantify the risk of false negatives and propose a guardrail metric to catch regressions.
Identify why the last iteration failed to detect an effect: was the MDE too large, variance too high, or duration too short? Use observed data to estimate actual variance and required sample size.
Determine the smallest effect size that is practically meaningful for the business. Balance statistical power (e.g., 80%) with the cost of longer experiments, and consider whether a larger MDE is acceptable for a quick directional read.
Propose methods like CUPED, stratification, or regression adjustment using pre-experiment covariates to reduce variance and increase sensitivity without increasing sample size.
Recalculate required sample size and duration using the new MDE and reduced variance. Consider sequential testing or a fixed-horizon design, and account for novelty effects and weekly seasonality.
Run a pilot or simulation to validate assumptions, and set up monitoring for guardrail metrics. Be prepared to adjust MDE or duration if early results suggest the design is still underpowered.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.