I went with 7-day retained sessions per user and spent a while defending it.
Start by clarifying the experiment's goal (e.g., driving incremental bookings) and then propose a primary metric that directly measures that goal, such as incremental booking rate. Justify it by discussing the trade-offs with alternatives like content views (too upstream) and overall booking rate (may not capture push-specific impact), and define the metric precisely with numerator, denominator, and attribution window.
Pro tip: Emphasize that the primary metric should be sensitive to the push notification's effect and aligned with the company's north star. Mention that you would also track guardrail metrics (e.g., opt-outs) to ensure the push doesn't harm user experience.
Ask or state the objective of the push notification experiment: is it to increase bookings, engagement, or retention? This determines the primary success metric.
Suggest a metric that directly measures the goal, such as incremental booking rate (bookings per user in the treatment vs. control). Explain why it's better than alternatives like content views (too upstream) or overall booking rate (not specific to push).
Specify numerator (e.g., number of bookings attributed to the push), denominator (e.g., number of users in the experiment group), and attribution window (e.g., 7 days from push send).
Compare with booking rate (may be affected by other factors) and content views (weak link to business value). Highlight that incremental booking rate isolates the push's impact and ties to revenue.
Mention secondary metrics like notification opt-out rate or app uninstalls to ensure the push doesn't negatively impact user experience.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Uninstall rate, notification unsubscribe rate, and spam-report rate are the obvious three.
Start by categorizing guardrail metrics into business, product, and technical health, then propose specific metrics with justified alert thresholds that balance sensitivity and false positives. Explain sequential monitoring using alpha-spending or group sequential methods to control Type I error, and emphasize pre-registration and practical significance.
Pro tip: At Meta, guardrails are often monitored with sequential tests using O'Brien-Fleming boundaries, and alerts are set at a level that accounts for multiple comparisons. Always tie thresholds to the cost of a false alert versus a missed regression.
List metrics across business (revenue, ads CTR), product (DAU, engagement), and technical (latency, error rates) dimensions that could be negatively impacted by the experiment.
For each metric, define a threshold (e.g., 2% relative drop) based on historical variability, business impact, and acceptable risk, considering multiple testing corrections.
Select a method like group sequential testing, alpha-spending, or always-valid inference to control Type I error while allowing early stopping for harm.
Document the monitoring plan before launch and set up automated alerts that trigger when thresholds are crossed, ensuring timely investigation.
When an alert fires, assess practical significance, check for data quality issues, and decide whether to pause, adjust, or continue the experiment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the randomization unit based on the treatment's mechanism and potential interference, then systematically analyze spillover pathways and cluster size tradeoffs. Use a concrete example like cross-border travelers to illustrate how to balance bias and variance. Conclude with a recommendation that aligns with the experiment's goals and practical constraints.
Pro tip: Acknowledge that the optimal unit often depends on the specific product and metric, and propose a data-driven approach to quantify spillover, such as measuring pre-experiment interactions. This shows maturity beyond textbook answers.
Clarify what the treatment is and how it might affect units beyond those directly treated. Consider direct and indirect effects, and whether interference is likely.
Enumerate possible spillover mechanisms, such as social networks, shared resources, or geographic proximity. For each, assess the direction and magnitude of potential bias.
Discuss how larger clusters reduce spillover but increase variance and reduce effective sample size, while smaller clusters increase spillover risk but improve precision. Mention unequal cluster sizes and their impact on analysis.
Analyze how travelers moving between clusters can introduce contamination. Discuss whether to exclude, track, or model their behavior, and the implications for randomization.
Synthesize the analysis to propose a unit (e.g., user, device, geographic region) that minimizes bias while maintaining statistical power, and suggest sensitivity checks.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining cluster randomization and why it's used (e.g., to avoid contamination). Then define intracluster correlation (ICC) and design effect (DE), and show how DE inflates sample size. Finally, walk through a numerical example comparing individual vs. cluster randomization.
Pro tip: Mention that when cluster sizes vary, you should use the average cluster size and adjust for the coefficient of variation; also note that analysis should use methods like mixed-effects models or GEE to account for clustering.
Explain that in cluster-randomized experiments, groups (clusters) rather than individuals are randomized, often to prevent treatment contamination or for logistical reasons.
ICC (ρ) measures the similarity of responses within a cluster, ranging from 0 to 1. Higher ICC means individuals within a cluster are more alike, reducing effective sample size.
DE = 1 + (m - 1)ρ, where m is the average cluster size. It quantifies the inflation in sample size needed compared to individual randomization.
For example, if ρ = 0.05 and m = 50, DE = 1 + (50-1)*0.05 = 3.45. So you need 3.45 times more individuals than individual randomization to achieve the same power.
Explain that you can either increase the number of clusters or the number of individuals per cluster, but increasing cluster size has diminishing returns due to ICC. Also mention analysis techniques to account for clustering.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Stratify on platform (iOS vs Android), notification eligibility status, and baseline activity tier.
Start by clarifying the experiment's unit of randomization (user vs. cluster) and the need for stratification to balance key covariates. Then describe your randomization method (e.g., hash-based) and stratification variables, and finally explain how you would validate covariate balance at both levels using statistical tests and metrics like SMD. Emphasize practical considerations like sample size and interference.
Pro tip: At Meta, where experiments often involve social networks, always address interference and cluster-level effects; mention techniques like cluster-based randomization and the use of ego-network clustering to mitigate spillover.
Clarify whether randomization is at user or cluster level (e.g., social clusters) and identify key covariates (e.g., age, country, activity level) to stratify on. Explain how stratification reduces variance and ensures balance.
Explain the randomization procedure, such as hash-based assignment using user ID or cluster ID, ensuring deterministic and reproducible assignment. Mention how stratification is incorporated (e.g., stratified randomization within each stratum).
Detail methods to check balance for user-level covariates: compute summary statistics (mean, variance) per arm, use standardized mean differences (SMD), and conduct statistical tests (e.g., t-tests) with corrections for multiple comparisons. Consider visualizations like Love plots.
For cluster-level covariates (e.g., cluster size, density), aggregate user-level metrics to cluster level and compare across arms using similar methods. Address intra-cluster correlation by using cluster-robust standard errors or mixed models.
Discuss sample size implications, potential interference/spillover, and how to handle imbalance (e.g., post-stratification, CUPED). Mention monitoring balance over time and the importance of pre-registration.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Mixed-effects with random intercepts for clusters plus CUPED adjustment using pre-period outcomes was my main answer.
Start by clarifying the experimental design and the unit of randomization, then recommend an approach that aligns with that unit while addressing interference. For cluster-randomized designs, use cluster-level difference-in-means or mixed-effects models; for user-level randomization with partial exposure, use user-level models with cluster-robust standard errors and consider exposure-based trimming or instrumental variables.
Pro tip: Always check for interference and partial exposure before choosing a method; a common mistake is to ignore these and get biased estimates. Also, be prepared to discuss the trade-offs between simplicity and correctness—sometimes a simple cluster-level analysis is more robust than a complex model.
Ask about the randomization unit (user, cluster, or time), the definition of clusters (e.g., social network, geographic region), and the exposure mechanism. This determines the appropriate analysis level.
Determine if there is spillover between clusters or partial exposure within clusters. If interference is present, cluster-level analysis or mixed-effects models may be needed; if partial exposure exists, consider exposure-based analysis or instrumental variables.
For cluster-randomized designs, use cluster-level difference-in-means or mixed-effects models to account for within-cluster correlation. For user-level randomization with clustering, use user-level models with cluster-robust standard errors.
Decide whether to analyze based on initial cluster assignment (intention-to-treat) or actual exposure (as-treated). Use methods like instrumental variables or exposure-based trimming to estimate causal effects.
Run robustness checks, such as comparing results across methods, testing for heterogeneous effects, and ensuring assumptions (e.g., no unmeasured confounding) are plausible.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Two-stage randomization is the clean answer: first randomize cities to treatment or control, then within treatment cities randomize users to actually receive notifications.
Start by defining spillover effects and explaining how they violate SUTVA, then describe methods to detect and quantify them, such as network analysis or spatial correlation. Finally, discuss how spillovers change the estimand from an average treatment effect to a direct/indirect effect decomposition and require adjustments like cluster randomization or exposure modeling.
Pro tip: Emphasize that spillovers can bias results in either direction, so it's crucial to pre-specify detection methods and consider the trade-off between bias and variance when choosing an estimand. Also, mention that at Meta, where social networks are common, spillovers are a first-class concern and often addressed via graph cluster randomization.
Explain that spillover occurs when one unit's treatment affects another's outcome, violating SUTVA. Give examples like social influence or shared resources.
Propose methods to detect spillovers: compare outcomes of control units connected to treated vs. unconnected, use network autocorrelation, or run placebo tests on pre-treatment periods.
Quantify by estimating exposure models (e.g., proportion of treated neighbors) or using spatial regression. Measure the magnitude and direction of indirect effects.
Shift from ATE to estimands like direct, indirect, and total effects. Define them clearly: direct effect of treatment on treated, indirect effect on others, and total effect.
Adapt analysis via cluster randomization, graph cluster randomization, or exposure-based analysis. Use methods like Horvitz-Thompson estimators or two-stage randomization.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start with 1-5% traffic, watch the guardrail metrics for 24-48 hours before expanding.
Start by framing the experiment's risk level and how it determines the safeguards. Then outline a staged rollout plan with specific ramp-up percentages and durations, followed by early-stop criteria based on guardrail metrics and statistical thresholds. Finally, detail the first 48-hour monitoring plan, including real-time dashboards and alerting mechanisms.
Pro tip: Define guardrail metrics and their acceptable thresholds before the experiment starts, and pre-register the early-stop rules to avoid p-hacking or post-hoc rationalization. Also, consider setting up a 'circuit breaker' that automatically pauses the experiment if a guardrail is breached, rather than relying on manual intervention.
Evaluate the potential impact of the experiment on user experience and business metrics. Identify guardrail metrics (e.g., revenue, user retention, latency) and set acceptable thresholds for each.
Plan a gradual rollout schedule (e.g., 1%, 5%, 10%, 50%, 100%) with defined durations at each stage. Include criteria for advancing to the next stage, such as no guardrail breaches and sufficient sample size.
Specify statistical rules for stopping the experiment early if guardrails are breached (e.g., sequential testing, alpha spending). Include both automatic and manual stop triggers.
Set up real-time dashboards to track guardrail metrics, sample ratio mismatch (SRM), and key performance indicators. Define alert thresholds and on-call procedures for immediate response.
Document the safeguards, escalation paths, and decision logs. Communicate the plan to stakeholders and ensure alignment on rollback procedures.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.