This one took me a minute to set up correctly.
First, compute the design effect using the intra-cluster correlation and average cluster size, then adjust the effective sample size accordingly. Next, calculate the difference in purchase rates and derive a z-statistic using cluster-robust standard errors that account for the design effect. Finally, compute the two-sided p-value from the z-statistic and interpret the results in the context of the experiment.
Pro tip: Always clarify whether the intra-cluster correlation is at the viewer or creator level, as mis-specifying the clustering level can drastically change the design effect and your conclusions. In practice, use the delta method or bootstrap to validate your cluster-robust standard errors, especially when cluster sizes vary.
Use the formula DE = 1 + (m - 1) * ICC, where m is the average cluster size (100 viewer sessions per creator) and ICC = 0.02. Calculate DE = 1 + (100 - 1) * 0.02 = 2.98.
The effective sample size per arm is the total number of viewer sessions divided by the design effect. Total sessions per arm = 10,000 creators * 100 sessions = 1,000,000. So effective N per arm = 1,000,000 / 2.98 ≈ 335,570.
Compute the difference in purchase rates: 0.052 - 0.05 = 0.002. The standard error under cluster-robust estimation is sqrt( (p1*(1-p1) + p2*(1-p2)) / effective_N_per_arm ) * sqrt(DE) or equivalently sqrt( (p1*(1-p1) + p2*(1-p2)) / (total_sessions/DE) ). Then z = difference / SE. Using p1=0.05, p2=0.052, total sessions per arm=1,000,000, DE=2.98: SE = sqrt( (0.05*0.95 + 0.052*0.948) / (1,000,000/2.98) ) = sqrt( (0.0475 + 0.049296) / 335,570 ) = sqrt(0.096796 / 335,570) ≈ sqrt(2.884e-7) ≈ 0.000537. Then z = 0.002 / 0.000537 ≈ 3.72.
For z = 3.72, the two-sided p-value is approximately 2 * (1 - Φ(3.72)) ≈ 0.0002. This is highly significant, indicating strong evidence that the treatment increases purchase rate.
Discuss the practical significance: a 0.2 percentage point absolute lift (4% relative) may be meaningful for TikTok. Also note assumptions: equal cluster sizes, no other sources of correlation, and that the ICC is correctly specified.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew the general shape of the answer: O'Brien-Fleming spends very little alpha early and more toward the end, so early looks use something like alpha/k^2 roughly, while Bonferroni just splits 0.05 evenly by 5 looks giving 0.01 each.
Explain that O'Brien-Fleming alpha-spending allocates alpha conservatively early and more liberally later, using a spending function like α(t)=2(1-Φ(z_{α/2}/√t)) for two-sided tests. For 4 interim looks plus final, compute boundaries at information fractions 0.2, 0.4, 0.6, 0.8, 1.0, and compare to Bonferroni's equal split (α/5=0.01 per look). Discuss how O'Brien-Fleming preserves power better and may allow earlier stopping for overwhelming effects, while Bonferroni is overly conservative and increases required sample size or duration.
Pro tip: Mention that in practice, you'd use software like gsDesign or rpact to compute exact boundaries, and that TikTok likely uses sequential testing methods to balance speed and rigor in A/B tests.
State that you have 5 analyses total: 4 interim looks and 1 final, with equal spacing of information (e.g., 20%, 40%, 60%, 80%, 100% of data). Assume a two-sided test at overall α=0.05.
Describe the O'Brien-Fleming spending function: it spends very little alpha at early looks (e.g., α≈0.0001 at 20% information) and more at later looks, with the final look using close to 0.05. Provide approximate boundaries: at 20%: z=4.88 (p≈0.000001), 40%: z=2.96 (p≈0.003), 60%: z=2.29 (p≈0.022), 80%: z=1.96 (p≈0.05), 100%: z=1.96 (p≈0.05) — but note these are illustrative; actual boundaries depend on correlation.
Bonferroni splits α equally: each look uses α/5=0.01 (two-sided p<0.01). This is much stricter at the final analysis than O'Brien-Fleming, which uses nearly full α at the end. Bonferroni is simple but conservative, especially for early looks where it may be too permissive relative to OBF.
O'Brien-Fleming maintains power better because it doesn't penalize the final analysis as much; it may allow early stopping if effects are very large. Bonferroni requires a larger sample size to achieve the same power, potentially lengthening the experiment. OBF is more efficient for typical effect sizes.
Recommend using O'Brien-Fleming alpha-spending for its balance of early safety and late power, and note that software tools can compute exact boundaries. Mention that the choice affects experiment duration and resource allocation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The Holm-Bonferroni part I had down: rank p-values, compare the smallest to alpha/k, the next to alpha/(k-1), stop when you fail to reject.
Start by explaining the Holm-Bonferroni procedure step-by-step for the four guardrail metrics, emphasizing control of the family-wise error rate. Then discuss the trade-offs and scenarios where Bayesian posterior intervals with a region of practical equivalence (ROPE) are more appropriate, such as when you need to quantify evidence for the null or incorporate prior knowledge.
Pro tip: Mention that Holm-Bonferroni is uniformly more powerful than Bonferroni, and that in practice you might use it only if guardrails are truly independent; otherwise, consider a hierarchical or multivariate approach. Also, note that Bayesian methods with ROPE can be more intuitive for stakeholders when deciding whether a guardrail is practically equivalent.
Explain that with four guardrail metrics plus the primary metric, there are five hypotheses, increasing the chance of false positives. Holm-Bonferroni controls the family-wise error rate (FWER) at a chosen alpha.
Order the p-values from smallest to largest. Compare the smallest p-value to alpha/5, the next to alpha/4, then alpha/3, alpha/2, and finally alpha. Stop at the first non-significant result and declare all remaining hypotheses non-significant.
If a guardrail metric is significant after correction, investigate whether the effect is practically meaningful. Consider the direction and magnitude, not just statistical significance.
Switch when you need to quantify evidence for the null hypothesis, incorporate prior information, or when stakeholders want to know the probability that the effect is practically equivalent. ROPE defines an interval of negligible effect, and you can compute the posterior probability that the effect falls within it.
Highlight that Holm-Bonferroni is frequentist and controls FWER, while Bayesian ROPE provides direct probability statements about practical equivalence. The choice depends on the goal: strict error control vs. decision-making under uncertainty.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.