← Pinterest Interview Insights
I fumbled this a little by writing H1 as just 'Treatment > Control' without anchoring it to the MDE.
Start by clearly defining the primary metric and its current baseline value. Then state the null hypothesis (no effect or non-inferiority) and the alternative hypothesis (two-sided or one-sided) based on the business goal. Finally, specify the minimum detectable effect (MDE) in absolute and relative terms, and justify it using practical significance and power analysis.
Pro tip: Always tie the MDE to a business-relevant threshold (e.g., revenue impact or user engagement lift) rather than just statistical convention, and mention that the direction of the test should align with the product team's expectation to avoid misleading conclusions.
Identify the key metric (e.g., click-through rate, saves per user) and its current baseline value from historical data. Ensure it's a single, well-defined metric that directly measures the experiment's goal.
Formulate H0 as no difference or no effect between control and treatment groups. For a two-sided test, H0: μ_treatment = μ_control; for a one-sided test, specify the direction (e.g., H0: μ_treatment ≤ μ_control).
Formulate H1 as the presence of an effect. For a two-sided test, H1: μ_treatment ≠ μ_control; for a one-sided test, H1: μ_treatment > μ_control (or <). The direction should be based on the expected impact.
Determine the smallest effect size that is practically meaningful and that the experiment is powered to detect. Express MDE in absolute terms (e.g., 0.5% increase) and relative terms (e.g., 5% lift), and justify it with business impact or prior data.
Explain why the test is one-sided or two-sided, and how the MDE was chosen (e.g., based on power analysis, cost of false positives/negatives, or stakeholder requirements). Mention the significance level (α) and power (1-β) used.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, state the null hypothesis that the observed assignment counts follow the expected ratio. Then, compute the chi-square statistic and p-value, and compare the p-value to a pre-defined significance level (e.g., 0.05) to decide whether to reject the null. Finally, interpret the result in the context of SRM: if p-value is below threshold, suspect SRM and investigate potential causes.
Pro tip: Always check the expected counts and ensure they are sufficiently large (e.g., >5) for the chi-square test to be valid. Also, mention that SRM detection should be done early and often, and that even a small deviation can be practically significant in large-scale experiments.
Define the null hypothesis (H0): the observed counts match the expected ratio (e.g., 50/50). The alternative hypothesis (H1): the observed counts do not match the expected ratio.
Compute the expected number of users in each group based on the total sample size and the intended allocation ratio.
Use the formula: χ² = Σ (observed - expected)² / expected. Sum over all groups.
Find the p-value from the chi-square distribution with degrees of freedom = number of groups - 1. Compare to significance level (e.g., α=0.05).
If p-value < α, reject H0 and suspect SRM; otherwise, do not suspect SRM. Discuss potential causes and next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Absolute lift is 0.36 minutes, relative lift is 3%, which clears the 2% MDE.
Start by clearly defining the primary metric and the null and alternative hypotheses. Then compute the absolute lift (difference in means), relative lift (percentage change), standard error, and the 95% confidence interval for the difference. Finally, calculate the one-sided p-value and compare it to alpha = 0.05 to determine significance.
Pro tip: Always state your assumptions (e.g., normality, independence) and mention that for a one-sided test, the p-value is half of the two-sided p-value if the effect is in the hypothesized direction. Also, relate the result to practical significance, not just statistical significance.
Identify the primary metric (e.g., mean clicks per user) and state the null (H0: μ_treatment - μ_control ≤ 0) and alternative (H1: μ_treatment - μ_control > 0) hypotheses for a one-sided test.
Calculate absolute lift as the difference in sample means (treatment - control). Compute relative lift as (absolute lift / control mean) * 100%.
Compute the standard error of the difference using sample variances and sizes. Construct a 95% confidence interval for the difference: (absolute lift ± t*SE) or (absolute lift ± z*SE) for large samples.
Calculate the test statistic (t or z) as absolute lift divided by standard error. Find the one-sided p-value as the probability of observing a test statistic at least as extreme in the direction of the alternative.
Compare the one-sided p-value to alpha = 0.05. If p < 0.05, reject the null hypothesis and conclude the effect is statistically significant.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
CTR comparison is aggregate clicks over impressions, not per-user, so you use a proportion test on those pooled numbers.
First, clarify the setup: two metrics (CTR and D7 retention) tested as secondary metrics, each with a two-proportion z-test. Then explain Holm-Bonferroni: order p-values from smallest to largest, compare the smallest to alpha/2, and if significant, compare the next to alpha/1. Finally, state which metrics remain significant based on the corrected thresholds, emphasizing that without actual p-values, the answer is conditional.
Pro tip: Mention that Holm-Bonferroni is uniformly more powerful than Bonferroni and controls the familywise error rate without assuming independence, which is important for correlated metrics like CTR and retention. Also, note that if the smallest p-value exceeds alpha/2, no metrics are significant.
Confirm that CTR and D7 retention are secondary metrics, each analyzed with a two-proportion z-test, and that the familywise alpha is 0.05.
Describe the step-down method: rank p-values ascending, compare the smallest to alpha/2, the next to alpha/1, and stop at the first non-significant result.
With two metrics, the thresholds are alpha/2 = 0.025 for the smaller p-value and alpha/1 = 0.05 for the larger p-value, provided the smaller is significant.
State that the metric with the smaller p-value is significant if p < 0.025; the other is significant only if the first is significant and its p < 0.05.
Since actual p-values are not provided, conclude that the answer depends on the observed p-values and summarize the decision rule.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
My answer was no-ship pending investigation, mostly because the SRM is a real problem and you can't trust the primary metric result until you know why assignment broke.
Start by clearly stating your ship/no-ship recommendation based on the primary metric, then systematically address validity threats like SRM, novelty effects, and other biases. Explain how each concern affects your confidence and what additional checks or analyses you would run to mitigate them before making a final call.
Pro tip: Always quantify the impact of validity concerns—e.g., estimate how much of the observed effect could be due to novelty—and propose concrete follow-up experiments or holdout groups to validate long-term effects.
Begin with a clear ship/no-ship recommendation based on the primary success metric (e.g., CTR, engagement) and its statistical significance. Briefly mention the observed effect size and confidence interval.
Verify that the actual traffic split matches the intended ratio using a chi-squared test. If SRM is present, investigate potential causes (e.g., logging issues, bot traffic) and consider invalidating the experiment.
Analyze metric trends over time to detect novelty (initial spike then decay) or primacy (initial dip then recovery) effects. Use cohort analysis or compare early vs. late periods to see if the effect persists.
Consider other biases such as instrumentation issues, selection bias, or external factors (e.g., seasonality, concurrent experiments). Check for consistency across segments and guardrail metrics.
Suggest additional checks like holdout groups, long-term follow-up, or qualitative user research. Weigh the evidence and provide a final recommendation, possibly conditional on further validation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.