These two get conflated a lot and I've seen people fumble it by defining one in terms of the other.
Start by defining both concepts clearly, emphasizing that they relate to different types of errors in hypothesis testing. Then contrast them using a concrete A/B testing example, and explain how they are set independently and their implications for experiment design.
Pro tip: Mention that while confidence level (1 - α) is typically set to 95%, power (1 - β) is often set to 80%, and that increasing power usually requires a larger sample size or larger effect size, which is a key trade-off in experiment planning.
Explain that confidence level (1 - α) is the probability of not making a Type I error (false positive) — i.e., correctly failing to reject a true null hypothesis. It is typically set at 95%.
Explain that power (1 - β) is the probability of correctly rejecting a false null hypothesis (true positive) — i.e., detecting an effect when it truly exists. It is typically set at 80%.
Highlight that confidence level concerns avoiding false positives, while power concerns avoiding false negatives. They are independent and controlled separately.
Use a scenario: testing a new feature. Confidence level ensures you don't roll out a useless feature; power ensures you detect a beneficial feature if it exists.
Explain that increasing confidence level (e.g., to 99%) reduces power for a fixed sample size, and increasing power requires larger sample size or larger effect size. Mention how these affect experiment duration and business decisions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.