I started rattling off the obvious cases like network effects and small sample sizes, but then blanked a bit on what to actually propose as alternatives.
Start by defining when A/B tests are inappropriate, such as when randomization is impossible, ethical concerns exist, or the user base is too small. Then propose an alternative method like observational study, quasi-experiment, or qualitative research, and justify why it fits the scenario. Emphasize the trade-offs and how you would validate findings.
Pro tip: Mention that A/B tests require stable, independent user assignment and sufficient sample size; if these are violated, consider switchback tests or holdout groups. Also, highlight that sometimes the best 'test' is to not experiment but to use causal inference methods like difference-in-differences.
Discuss scenarios like network effects, ethical concerns, low traffic, or when the change is irreversible. Explain why randomization or control groups are problematic.
Select a method such as observational study, quasi-experiment (e.g., difference-in-differences), qualitative user research, or switchback testing. Justify based on the scenario.
Highlight how the alternative addresses the limitations of A/B testing, such as handling interference, ethical constraints, or small sample sizes.
Acknowledge the trade-offs of the alternative method, such as potential confounding variables, and how you would mitigate them (e.g., using propensity score matching).
Provide a concrete example from your experience or a hypothetical scenario to illustrate your reasoning and demonstrate applied knowledge.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.