I went with 7-day purchase rate as primary, which felt obvious.
Start by clarifying the business objective (e.g., increase engagement or revenue) and then define a primary success metric that directly measures that objective, such as incremental revenue per user or conversion rate. Next, select two guardrail metrics that ensure the model doesn't harm user experience or long-term health, like unsubscribe rate and email complaint rate. Explain how you would measure these metrics in an A/B test comparing the targeted model against a control (e.g., no email or random targeting).
Pro tip: Emphasize that guardrail metrics should be chosen to detect unintended consequences of targeting, such as alienating users who receive irrelevant emails, and mention that you'd monitor them for statistical significance and practical significance.
Ask or state the primary objective of the email campaign (e.g., drive purchases, increase engagement) to ensure the success metric aligns with business value.
Choose a metric that directly measures the campaign's impact on the goal, such as incremental revenue per user or conversion rate, and explain how it will be measured in an experiment.
Pick two metrics that capture potential negative side effects, such as unsubscribe rate (user annoyance) and email complaint rate (reputation damage), and justify why they are critical.
Outline an A/B test where the treatment is the targeted email model and the control is either no email or a random/untargeted email, to isolate the model's incremental effect.
Explain how you would monitor both primary and guardrail metrics for statistical significance and practical significance, and decide whether to launch, iterate, or stop.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, convert the relative lift into an absolute difference by multiplying the baseline rate by the lift (5% * 8% = 0.4 percentage points). Then, apply the standard sample size formula for comparing two proportions: n = (Z_α/2 + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2, using Z_α/2 = 1.96 and Z_β = 0.84. Finally, compute the result and round up to the nearest whole number.
Pro tip: Always clarify whether the sample size is per arm or total, and mention that this calculation assumes no peeking and a fixed horizon; in practice, you might adjust for sequential testing or use simulation for more complex designs.
Identify baseline conversion rate (p1 = 0.05), relative lift (8%), and compute absolute lift (Δ = 0.004) and treatment rate (p2 = 0.054). Also note alpha (0.05, two-sided) and power (80%).
For a two-sided test at alpha=0.05, Z_α/2 = 1.96. For 80% power, Z_β = 0.84 (since β=0.20).
Use the formula for two proportions: n = (Z_α/2 + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2. Plug in the values: (1.96+0.84)^2 = 7.84; p1(1-p1)=0.0475; p2(1-p2)=0.051084; sum=0.098584; Δ^2=0.000016; n = 7.84 * 0.098584 / 0.000016 ≈ 48,306 per arm.
Round up to the next whole number (48,307) and state that this is the minimum per-arm sample size. Optionally, discuss practical considerations like traffic and duration.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked through an alpha-spending approach using an O'Brien-Fleming-style boundary.
Start by framing the trade-off between early detection and error control, then propose a group sequential design with alpha spending (e.g., O'Brien-Fleming) and pre-specified interim looks. Explain how you would implement it in practice, including stopping rules for efficacy and futility, and how you would monitor sequentially without inflating type-I error.
Pro tip: Mention that you would pre-register the monitoring plan and simulate the design under the null and alternative to verify operating characteristics—this shows rigor and prevents p-hacking concerns.
State the null and alternative hypotheses, the overall type-I error rate (e.g., 5%), and the desired power. Emphasize that the family-wise error rate must be controlled across all interim looks.
Select a spending function like O'Brien-Fleming or Pocock to allocate alpha across interim analyses. Explain that O'Brien-Fleming is conservative early and liberal later, which is often preferred for efficacy monitoring.
Propose specific interim analysis points, e.g., at 25%, 50%, and 75% of planned sample size or information fraction. Discuss how the number and timing depend on the expected effect size and cost of data collection.
Specify efficacy boundaries (e.g., reject null if test statistic exceeds adjusted critical value) and futility boundaries (e.g., stop if conditional power falls below a threshold). Mention that futility can be non-binding to preserve type-I error.
Describe the sequential monitoring process: after each interim look, compute the test statistic, compare to boundaries, and decide whether to continue, stop for efficacy, or stop for futility. Ensure that the spending function is followed exactly.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered the standard stuff: randomization check via a pre-experiment AA test, covariate balance across treatment and control, checking for holdout leakage if users share households or accounts.
Structure your answer by first outlining pre-experiment checks (e.g., sample size, randomization, instrumentation, guardrail metrics), then address interference and seasonality by proposing design and analysis solutions (e.g., cluster randomization, switchback tests, stratification, time-based controls). Emphasize proactive planning and robustness to real-world complexities.
Pro tip: Mention that you would pre-register the analysis plan and run a power analysis that accounts for potential interference and seasonality, and consider using a holdout group to measure long-term effects. This shows rigor and foresight.
Verify sample size and power, ensure proper randomization (e.g., check covariate balance), validate instrumentation and logging, and define guardrail metrics to monitor for unintended consequences.
Assess whether users interact (e.g., social networks, marketplaces) and if treatment can spill over. Consider cluster randomization or switchback designs to mitigate interference.
Use stratification or blocking by day-of-week, include time fixed effects in analysis, or run experiments over full weeks to balance weekly patterns. Consider switchback tests to control for time-varying confounders.
Pre-register analysis plan, monitor for interference and seasonality during the experiment, and use appropriate statistical methods (e.g., CUPED, regression adjustment) to increase sensitivity.
Have a plan if interference or seasonality is detected (e.g., extend duration, re-randomize, or use alternative designs). Communicate risks and trade-offs to stakeholders.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Proposed a regression discontinuity design on the score threshold since the treatment assignment is literally a cutoff.
Start by acknowledging the constraints and proposing a quasi-experimental design like difference-in-differences or synthetic control, tailored to the specific legal/traffic issue. Clearly state the key assumptions (e.g., parallel trends, no spillover) and how you would validate them. Then outline what you'd present in slides: design, assumptions, diagnostics, results with uncertainty, and sensitivity analyses.
Pro tip: Emphasize that you would pre-register the analysis plan and run placebo tests to build trust, since quasi-experiments are more susceptible to bias. Also, mention that you'd collaborate with legal and product teams early to ensure the design is feasible and compliant.
Identify the exact legal or traffic limitation (e.g., cannot randomize users, limited traffic in certain regions) and select a quasi-experimental method such as difference-in-differences, synthetic control, or regression discontinuity that fits the context.
Explicitly list the assumptions (e.g., parallel trends, no interference, stable composition) and describe how you would test them (e.g., pre-trend checks, placebo tests, robustness checks).
Define the treatment and control groups, the time periods, and the primary and guardrail metrics. Specify the statistical model and how you'll estimate the effect with confidence intervals.
Outline additional analyses to assess how sensitive results are to violations of assumptions, such as varying the control group, using different time windows, or applying alternative estimators.
Present the design rationale, assumptions, diagnostics (e.g., pre-trend plots), main results with uncertainty, sensitivity analyses, and a clear recommendation with caveats.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.