← SIG (Susquehanna) Interview Insights
The mechanics here are standard: delta is 0.05 * 42 = 2.1 minutes, then plug into n = 2 * ((z_alpha/2 + z_beta) / delta * sigma)^2.
First, translate the relative improvement into an absolute effect size by multiplying the baseline mean by 5% (since negative improvement means a reduction). Then, use the standard formula for sample size per arm in a two-sample t-test: n = 2*(Z_{α/2} + Z_β)^2 * σ^2 / Δ^2, where Z_{α/2}=1.96 and Z_β=0.84 for 80% power. Finally, compute the numerical value and round up to the nearest integer.
Pro tip: Always clarify whether the relative improvement is a reduction or increase, and explicitly state that you're assuming equal variances and a two-sided test. Also, mention that this is a simplified calculation and that in practice you might use simulation or adjust for unequal variances.
Calculate the absolute change: 5% of 42 minutes = 2.1 minutes. Since it's a negative improvement (reduction), the target mean is 42 - 2.1 = 39.9 minutes, so the effect size Δ = 2.1 minutes.
For a two-sample t-test with equal variances, the per-arm sample size formula is n = 2*(Z_{α/2} + Z_β)^2 * σ^2 / Δ^2. Here, α=0.05 two-sided so Z_{α/2}=1.96, power=80% so Z_β=0.84, σ=15, and Δ=2.1.
Compute (1.96+0.84)^2 = 2.8^2 = 7.84. Then n = 2 * 7.84 * (15^2) / (2.1^2) = 2 * 7.84 * 225 / 4.41. Calculate: 2*7.84=15.68; 15.68*225=3528; 3528/4.41 ≈ 800. So n ≈ 800 per arm.
Round up to the next integer: 800 per arm. Mention assumptions: equal variances, normal distribution, and that this is a simplified calculation (e.g., ignoring continuity correction or unequal allocation).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Non-inferiority framing always trips me up a little because the null and alternative are flipped from what you'd normally write.
First, clarify that this is a non-inferiority test comparing a new treatment to a control with a margin of 0.5 percentage points, and that the baseline cancellation rate is 6% for both groups under the null. Then, use the standard formula for non-inferiority sample size for proportions: n per arm = ( (z_alpha + z_beta)^2 * (p1(1-p1) + p2(1-p2)) ) / (margin)^2, where p1 = p2 = 0.06, z_alpha = 1.645 (one-sided 0.05), z_beta = 0.84 (80% power), and margin = 0.005. Finally, compute the value and round up to the nearest integer.
Pro tip: In practice, always verify whether the margin is absolute or relative, and consider continuity correction or exact methods for small rates; also, be prepared to discuss how clustering or unequal allocation would affect the sample size.
Confirm that the non-inferiority margin is 0.5 percentage points (absolute) and that the baseline rate is 6% for both arms under the null. Identify one-sided alpha = 0.05 and power = 80%.
Use the sample size formula for non-inferiority of two proportions: n per arm = ( (z_alpha + z_beta)^2 * (p1(1-p1) + p2(1-p2)) ) / (delta)^2, where delta is the non-inferiority margin.
Set p1 = p2 = 0.06, delta = 0.005, z_alpha = 1.645, z_beta = 0.84. Compute the numerator and denominator to find n per arm.
Round the calculated sample size up to the next whole number, as you cannot have a fraction of a participant. State the final per-arm sample size clearly.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I went with a gatekeeping structure: primary gets the full alpha budget at 0.05, guardrails get tested for non-inferiority at a shared 0.05 one-sided (no adjustment needed since they're non-inferiority claims), and the secondary only gets tested if the primary passes.
Start by clarifying the metric hierarchy and the risks of multiple testing, then propose a gatekeeping strategy that controls family-wise error rate (FWER) across primary and secondary metrics while using a more lenient false discovery rate (FDR) or independent thresholds for guardrails. Justify the choice based on business impact and statistical power, and explain how to compute and report adjusted confidence intervals that reflect the chosen error control.
Pro tip: In practice, guardrail metrics are often monitored with one-sided tests and less stringent error control because the cost of a false alarm is lower than missing a regression; however, you must pre-register the strategy to avoid p-hacking accusations.
Define the primary metric as the key success measure, guardrails as safety checks, and secondary as supportive. Explain that testing multiple metrics inflates Type I error and requires a pre-specified error control plan.
For the primary and secondary metrics, use a gatekeeping procedure (e.g., fixed-sequence or Holm-Bonferroni) to control FWER at α=0.05. For guardrails, use a less stringent approach like Benjamini-Hochberg FDR or independent one-sided tests at α=0.10, justified by asymmetric costs.
For FWER-controlled metrics, adjust confidence levels using Bonferroni or Holm (e.g., 97.5% CI for primary if two metrics). For FDR-controlled guardrails, use Benjamini-Hochberg adjusted p-values and report CIs at the corresponding adjusted level (e.g., 95% CI but note FDR control).
Present unadjusted and adjusted CIs side-by-side, clearly labeling which metrics are confirmatory vs. exploratory. Emphasize that guardrail CIs are for monitoring, not strict hypothesis testing, and discuss any trade-offs in power.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Design effect is 1 + (m-1)*ICC = 1 + 299*0.03 = about 9.97, so basically 10x.
First, compute the design effect using the formula 1 + (m-1)*ICC, where m is the cluster size (300 orders per zone-day). Then, adjust the per-arm sample size by multiplying the standard sample size by the design effect. Finally, divide the adjusted sample size by the cluster size to get the number of zone-days per arm, rounding up to ensure sufficient power.
Pro tip: Always clarify whether the ICC is at the order or zone-day level; here it's zone-day, so the cluster size is the number of orders per zone-day. Also, remember that the design effect increases sample size, so the number of zone-days will be larger than if you ignored clustering.
Extract the cluster size (m = 300 orders per zone-day) and the intra-cluster correlation (ICC = 0.03) from the problem statement.
Use the formula DE = 1 + (m - 1) * ICC to calculate the design effect. Plug in m = 300 and ICC = 0.03 to get DE = 1 + 299 * 0.03 = 9.97.
Multiply the standard per-arm sample size (n) by the design effect to get the adjusted per-arm sample size: n_adj = n * DE. If n is not given, state that you need it, but typically in such problems n is provided or you can express the answer in terms of n.
Divide the adjusted per-arm sample size by the cluster size (m) to get the number of zone-days per arm: zone-days = n_adj / m. Round up to the nearest whole number.
Explain that the design effect inflates the sample size due to clustering, and the number of zone-days ensures adequate power. Mention that if n is not provided, you would need to compute it based on desired power and effect size.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
OBF is conservative early, so the first look has a critical value way out in the tail, something like z = 4 or higher, meaning you'd only stop early for an enormous effect.
Start by explaining that O'Brien-Fleming spending creates conservative early boundaries that are very stringent, while the final boundary is close to the unadjusted alpha. Then discuss how this design inflates the required sample size to maintain power, which increases the minimum detectable effect (MDE) for a fixed sample size or extends trial runtime to detect the same effect.
Pro tip: Emphasize that O'Brien-Fleming is often chosen for its conservative early stopping, which is ideal when early data is less reliable, but it comes at the cost of a larger maximum sample size. In practice, this means you need to plan for a longer trial or accept a larger MDE if you cannot increase the sample size.
Describe how the alpha spending function allocates very little alpha to early looks and more to later looks, resulting in stringent early boundaries that are hard to cross.
Contrast the early stopping boundaries (e.g., very small p-value thresholds) with the final look boundary, which is close to the nominal alpha (e.g., 0.05) but slightly adjusted for multiple looks.
Explain that because early boundaries are stringent, the trial needs a larger sample size to maintain power, which increases the MDE for a fixed sample size or requires a larger sample to detect the same effect.
Highlight that to achieve the same power and MDE, the trial must run longer to accumulate the required sample size, unless the effect is large enough to stop early.
Conclude that O'Brien-Fleming balances early stopping conservatism with minimal alpha penalty at the final look, but requires careful planning of sample size and runtime to avoid underpowered studies.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.