I knew the basics (statistical power, significance threshold, baseline conversion rate) but fumbled a bit when trying to explain the tradeoffs between running a test longer vs.
Start by defining the goal of the A/B test and the primary metric you want to move. Then explain how you'd use statistical power analysis to determine the required sample size, considering baseline conversion rate, minimum detectable effect, significance level, and power. Finally, discuss practical constraints and trade-offs, and how you'd communicate the sample size decision to stakeholders.
Pro tip: Always align on the minimum detectable effect (MDE) with stakeholders early, as it's often the most debated input and directly impacts sample size and test duration. Also, mention that you'd run a power analysis before the test and avoid peeking at results, as that inflates false positives.
Clarify what you're testing and the key metric (e.g., conversion rate, click-through rate) that will determine success. This sets the foundation for all subsequent calculations.
Estimate the current baseline conversion rate from historical data. Decide on the smallest relative or absolute lift (MDE) that would be practically significant for the business.
Choose significance level (alpha, typically 0.05) and power (1-beta, typically 0.80). These reflect the acceptable false positive and false negative rates.
Use a sample size formula or tool (e.g., Evan Miller's calculator) that incorporates baseline, MDE, alpha, and power. For more complex metrics, consider variance and distribution.
Check if the required sample size is achievable given traffic and time constraints. If not, revisit MDE or consider alternative test designs (e.g., sequential testing, variance reduction).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.