This is a multiple comparisons problem and I knew that going in, but I fumbled the 'why' part more than I expected.
Start by acknowledging the multiple comparisons problem and the need to control false positives. Then discuss adjustments like Bonferroni or Benjamini-Hochberg, but emphasize that the choice depends on the goal (e.g., family-wise error rate vs. false discovery rate) and the correlation among metrics. Finally, mention practical alternatives like pre-registering a primary metric or using hierarchical testing to balance rigor and power.
Pro tip: At Meta, where experiments often have many metrics, it's common to use a primary metric for decision-making and treat others as guardrails or exploratory, with less stringent thresholds. This avoids over-penalizing and maintains sensitivity to meaningful effects.
Explain that testing multiple metrics inflates the chance of false positives, so p-value thresholds must be adjusted to control error rates.
Discuss options like Bonferroni (controls family-wise error rate) or Benjamini-Hochberg (controls false discovery rate), and justify based on the cost of false positives vs. false negatives.
Propose designating a primary metric with a standard threshold (e.g., 0.05) and treating others as secondary or exploratory with stricter thresholds or no formal testing.
Note that metrics are often correlated, so corrections like Bonferroni may be overly conservative; methods like Holm or permutation tests can be more appropriate.
Emphasize that the goal is to make informed decisions, so consider the trade-off between controlling false positives and detecting true effects, especially in a fast-paced environment like Meta.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.