Start by defining a primary metric that directly measures the email's impact on user behavior, such as click-through rate or conversion rate, and then select guardrail metrics to monitor potential negative side effects like unsubscribes or spam complaints. Address eligibility rules by specifying criteria for who can enter the experiment and how to handle users who become ineligible, and explain how to manage multiple email exposures under intent-to-treat by analyzing at the user level and using techniques like clustering or mixed models.
Pro tip: Emphasize that guardrail metrics should be leading indicators of long-term harm, and that under intent-to-treat, you must analyze all randomized users regardless of whether they actually received the email, to avoid selection bias.
Choose a metric that directly reflects the email's goal, such as 7-day conversion rate or click-through rate, and ensure it is measurable and aligned with business objectives.
Pick 2-3 metrics that capture potential negative consequences, such as unsubscribe rate, spam complaint rate, or long-term engagement decline, to ensure the experiment doesn't harm user experience.
Define clear criteria for who is eligible (e.g., active users, opted-in to emails) and how to handle users who become ineligible during the experiment, such as excluding them from analysis or using intention-to-treat principles.
Under intent-to-treat, analyze at the user level by aggregating outcomes across all emails received, and consider using methods like per-user averages or mixed-effects models to account for correlation.
Compare primary and guardrail metrics between treatment and control groups, check for statistical significance, and ensure guardrails are not violated before declaring success.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, compute the required sample size per variant using the standard two-proportion z-test formula with the given baseline conversion, relative lift, alpha, and power. Then, translate the total sample size into calendar days by accounting for daily eligible users, deliverability, and the full attribution window, ensuring the test runs long enough to capture all conversions.
Pro tip: Always clarify whether the attribution window is included in the test duration or added on top; at Coinbase, it's typically added, so the test runs for the recruitment period plus the attribution window to avoid truncating conversions.
Use the formula for two-proportion z-test: n = (Z_{α/2} + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2, where p1 = 0.035, p2 = 0.035 * 1.10 = 0.0385. With α=0.05 (two-sided), Z_{α/2}=1.96; power=80%, Z_β=0.84. Compute n per variant.
Since the split is 1:1, total sample size = 2 * n per variant. This is the total number of users that need to be exposed to the test.
With 500,000 eligible users per day and 85% deliverability, the number of users actually exposed per day is 500,000 * 0.85 = 425,000 users per day.
Divide the total sample size by daily exposed users to get the number of days needed to recruit the required sample. Round up to the nearest whole day.
The test must run for the recruitment period plus the full attribution window (7 days) to ensure all conversions are captured. So total calendar days = recruitment days + 7.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, recognize that this is a sample size calculation for a continuous metric (revenue per user) using a two-sample t-test. Use the formula n = 2*(Z_alpha/2 + Z_beta)^2 * sigma^2 / delta^2, plugging in SD=12, delta=0.10, and standard 80% power at 5% significance. Then compute run length by dividing the per-arm sample size by daily traffic per arm, and discuss practical considerations like variance reduction and business impact.
Pro tip: Revenue metrics are often skewed with high variance; mention that using a t-test is still valid due to CLT with large n, but consider capping outliers or using a more robust test if the distribution is extremely heavy-tailed. Also, always sanity-check the implied run length against business cycles (e.g., weekly seasonality) and avoid stopping early based on peeking.
Recognize that the metric is continuous (7-day revenue per randomized user) and the appropriate test is a two-sample t-test for means. State the null and alternative hypotheses.
List the required inputs: baseline mean $0.90, standard deviation $12.00, minimum detectable effect (absolute lift) $0.10, significance level (typically 0.05), and power (typically 0.80). Write down the sample size formula for comparing two means.
Plug values into the formula: n = 2*(Z_{1-alpha/2} + Z_{1-beta})^2 * sigma^2 / delta^2. With Z_{0.975}=1.96, Z_{0.80}=0.84, sigma=12, delta=0.10, calculate n. Show the arithmetic clearly.
Divide the per-arm sample size by the expected daily traffic per arm (or total daily traffic divided by 2) to get the number of days needed. If daily traffic is not given, state the assumption and express run length as a function of traffic.
Mention assumptions: independence, normality (or large n), equal variances, no peeking. Discuss potential issues like skewed revenue distribution, outliers, and the need for variance reduction techniques (e.g., CUPED) to reduce sample size or run time.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Run one is clearly significant with a big effect.
Start by computing the two-proportion z-test p-value and 95% confidence interval for each experiment run separately, then combine the results using a fixed-effects meta-analysis to obtain a pooled estimate and confidence interval. Finally, interpret the pooled results in the context of the business decision, considering both statistical significance and practical significance to recommend whether to launch.
Pro tip: Always check for consistency between the two runs before pooling; if they show conflicting directions or magnitudes, a fixed-effects meta-analysis may not be appropriate, and you should investigate heterogeneity or consider a random-effects model.
For each experiment run, calculate the conversion rates for control and treatment, the difference in proportions, the z-statistic, the two-sided p-value, and the 95% confidence interval for the difference.
Compare the direction, magnitude, and confidence intervals of the two runs to check for consistency. If they are consistent, proceed with pooling; if not, investigate potential causes.
Compute the pooled effect size (difference in proportions) as a weighted average of the two run-specific effects, using inverse-variance weights. Calculate the pooled standard error, z-statistic, p-value, and 95% confidence interval.
Evaluate whether the pooled effect is statistically significant and practically meaningful. Consider the business context, costs, and risks to make a launch recommendation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Got most of the obvious ones: seasonality, novelty effect wearing off, promo concurrency, population drift.
Structure your answer by categorizing reasons into data quality, experimental design, and external factors. For each reason, specify the exact data checks or queries you would run to confirm or rule it out, demonstrating a systematic root cause analysis.
Pro tip: Prioritize reasons by likelihood and impact, and mention how you would use statistical tests (e.g., t-test, chi-square) to validate differences. Also, emphasize the importance of checking data pipeline integrity before diving into experiment metrics.
Ask clarifying questions about the experiment setup, primary metrics, and what 'different results' means (e.g., direction, magnitude, statistical significance).
List plausible reasons across categories: data quality, experiment design, user behavior, and external events. Aim for at least six distinct reasons.
Specify the exact queries, metrics, or statistical tests you would run to investigate each cause. Include both SQL queries and analytical methods.
Rank causes by likelihood and impact, and outline a step-by-step investigation plan, starting with the most probable and easiest to check.
Summarize how you would present findings to stakeholders and recommend actions, such as re-running the experiment or fixing data issues.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer as a sequential re-analysis plan that addresses each component (pre-registration, covariate adjustment, HTE, sequential monitoring, holdout) while emphasizing the trade-offs and decisions at each stage. For the pooled lift between 0-5%, focus on decision-making under uncertainty: consider practical significance, segment-level effects, and risk management for ramp.
Pro tip: Acknowledge that a 0-5% lift may be practically insignificant depending on the metric and business context; propose a decision framework that weighs statistical significance, effect size, and business impact, and suggest a phased ramp with holdout to mitigate risk.
Outline a pre-registration document that specifies primary and secondary metrics, analysis populations, and statistical tests before re-analysis. This prevents p-hacking and ensures transparency.
Use covariate adjustment (e.g., CUPED) to increase power, and pre-specify subgroups for heterogeneity analysis (e.g., user tenure, region). Apply multiple testing corrections for subgroup analyses.
If data was peeking, apply sequential testing corrections (e.g., alpha spending, group sequential boundaries) to control Type I error. Consider always-valid inference methods.
Design a holdout group (e.g., 5-10% of users) to measure long-term effects during ramp. Plan a phased rollout with clear go/no-go criteria at each stage.
Evaluate practical significance: if lift is small but positive, assess confidence intervals, segment-level lifts, and cost of implementation. Consider a limited ramp with holdout to gather more evidence.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.