Start by clarifying the goal and metrics, then propose randomizing at the user level to avoid contamination across accounts. Explain how you'd handle multi-account users, including potential dilution effects, and outline the analysis plan with guardrail metrics.
Pro tip: Emphasize that the unit of randomization should match the unit of analysis and the level at which the treatment is applied—here, the user—and discuss how to detect and mitigate interference if users share devices or switch accounts frequently.
State the hypothesis that pinning accounts with unread notifications increases engagement with those accounts. Define primary metrics (e.g., notification open rate, account switch rate) and guardrail metrics (e.g., overall session time, user satisfaction).
Randomize at the user level, not the account level, because accounts belonging to the same person are not independent. This prevents treatment spillover: if one account is in treatment and another in control, the user's behavior could contaminate both groups.
Assign each user to treatment or control, and apply the feature to all accounts owned by that user in the treatment group. Ensure exposure is logged at the user level and consider using a holdout group for long-term effects.
Identify potential contamination sources such as shared devices or users switching between accounts. Use user-level randomization and consider excluding users who frequently switch accounts or use a cluster-randomized design if needed.
Analyze results at the user level, accounting for within-user correlation across accounts. Calculate required sample size based on user-level metrics, and pre-register the analysis plan to avoid p-hacking.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I went with unread resolution rate within 24 hours as the OEC because it captures whether the feature actually helped people act on notifications, not just whether they clicked.
Start by clarifying the experiment's goal and the product surface (e.g., feed ranking, notifications) to ground your metric choice. Then propose a single OEC that captures long-term user value, and define it precisely with numerator, denominator, and attribution window. Follow with 3–5 secondary/diagnostic metrics that guard against regressions and explain trade-offs, each with exact definitions.
Pro tip: Anchor your OEC to a long-term value proxy (e.g., meaningful sessions or retained DAU) rather than a short-term engagement metric, and explicitly state the attribution window rationale—this shows you understand Meta's emphasis on durable growth and guardrail metrics.
Ask or state assumptions about the product area, user segment, and what change is being tested. Confirm whether the goal is engagement, retention, monetization, or well-being.
Choose one primary success metric that best proxies long-term user value for this experiment. Define numerator, denominator, and attribution window explicitly, and justify why it's the right OEC.
Pick 3–5 metrics that capture complementary value, guard against regressions, or diagnose why the OEC moved. For each, give exact numerator, denominator, and attribution window.
Discuss how you'd interpret conflicting movements (e.g., OEC up but a guardrail metric down) and what would constitute a ship/no-ship decision.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Retention, session length, and notification latency were my three anchors.
Start by framing guardrails around the experiment's primary metric and key business metrics, then define specific thresholds for pausing or stopping based on statistical and practical significance. Emphasize a pre-registered analysis plan with clear decision rules to avoid peeking and ensure valid inference.
Pro tip: Mention that guardrails should be monitored continuously with sequential testing or group sequential boundaries to allow early stopping without inflating false positives. Also, tie thresholds to the cost of a false positive vs. false negative, showing business acumen.
Select metrics that should not degrade, such as revenue, user retention, latency, or error rates, in addition to the primary success metric.
Define thresholds using statistical significance (e.g., p<0.05) and practical significance (e.g., minimum detectable effect) for both primary and guardrail metrics.
Specify conditions for pausing (e.g., guardrail metric degradation beyond a threshold) and stopping (e.g., primary metric shows harm or strong positive effect) using sequential testing methods.
Describe how metrics will be monitored in real-time, who reviews alerts, and the escalation process for pausing or stopping the experiment.
Emphasize pre-registration of the analysis plan and clear communication of thresholds and decisions to stakeholders before launch.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Novelty effects are easy to say and hard to actually operationalize.
Start by clarifying the experiment's unit of analysis and defining account changes as a metric or covariate. Then propose a robust analysis plan that handles account churn and novelty effects through stratification, sensitivity checks, and time-based analysis.
Pro tip: Emphasize that account changes should be pre-specified in the analysis plan to avoid post-hoc bias, and use holdout or long-term metrics to separate novelty from true treatment effects.
Clarify what constitutes gaining or losing an account (e.g., new sign-ups, deletions, deactivations) and how these events relate to the experiment's unit of randomization. Determine whether these changes are part of the treatment effect or a nuisance factor.
Decide whether to analyze based on intent-to-treat (all users randomized) or per-protocol (only users who maintained accounts). Discuss trade-offs: ITT preserves randomization but may dilute effects; per-protocol can introduce bias.
Propose methods such as stratifying by account status at baseline, using account changes as a covariate, or conducting sensitivity analyses excluding users with changes. Also consider time-to-event analysis for account churn.
Analyze treatment effects over time (e.g., weekly) to detect novelty patterns. Use holdout groups or long-term metrics to assess persistence. Consider exposing a subset to the treatment longer before measuring.
Run robustness checks (e.g., different definitions of account changes, alternative novelty adjustments) and clearly communicate assumptions and limitations to stakeholders.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I assumed a baseline unread resolution rate around 30%, set an MDE of 3 percentage points absolute, and talked through using CUPED with pre-experiment resolution rate as the covariate.
Start by framing the experiment's goal and metric, then systematically walk through each component: baseline assumptions, MDE, variance reduction, and sample size/duration calculation. Use a concrete example with plausible numbers to demonstrate practical understanding, and explain the trade-offs and assumptions behind each choice.
Pro tip: Emphasize that power analysis is an iterative process—after the experiment, you can use observed variance to refine future power calculations, and consider sequential testing or CUPED to reduce required sample size.
Clearly state the primary metric (e.g., click-through rate, conversion rate) and the unit of randomization (e.g., user). Mention the baseline rate and any known seasonality or external factors.
Provide the baseline metric value (e.g., 10% CTR), expected variance (or standard deviation), and any assumptions about traffic distribution and independence. Explain how you obtained these (historical data, pilot).
Explain how you chose the Minimum Detectable Effect (e.g., 2% relative lift) based on business impact. Describe variance reduction techniques like CUPED, stratification, or using pre-experiment data to increase sensitivity.
Use the power formula for two-proportion z-test (or t-test) to compute required sample size per variant, then convert to duration based on daily traffic. Mention alpha=0.05, power=0.8, and any adjustments for multiple testing.
Highlight how changing MDE or variance affects duration, and mention practical constraints like novelty effects, minimum runtime for weekly cycles, and potential guardrail metrics.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Defining these precisely is actually harder than it sounds.
Start by clarifying that definitions must be tied to the business goal (e.g., engagement vs. monetization) and specify the logging event schema for each metric. Then explain cross-device attribution using a deterministic identity graph (logged-in user IDs) with probabilistic fallback, and discuss trade-offs like deduplication and latency.
Pro tip: Emphasize that you would validate definitions with A/B tests or holdout groups to ensure metrics are sensitive to changes and not gameable—this shows you think beyond definitions to experimentation impact.
Ask what decisions these metrics will inform (e.g., ranking, notifications) to align definitions with goals. Mention that definitions may vary by surface (feed vs. stories).
For 'visit', specify session start/end (e.g., 30-min inactivity timeout) and log a visit event with timestamp and surface. For 'dwell time', define active vs. passive time (e.g., foreground, scrolling) and log start/end events with duration. For 'unread resolved', define when a notification is considered read (e.g., impression in viewport for >1s) and log a resolution event.
Use deterministic matching via logged-in user IDs as primary; fall back to probabilistic (device graph, IP+UA) only for logged-out users, with clear confidence thresholds. Log a canonical user_id on every event.
Define a session window (e.g., 30 min) to merge visits across devices; deduplicate events using event_id and user_id. For attribution, use last-touch within window and log both raw and deduped metrics.
Propose A/B tests or holdout groups to check metric sensitivity and guard against gaming. Monitor data quality (e.g., missing user_ids) and refine definitions based on observed behavior.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
No, this is not a success under the OEC I defined.
Start by acknowledging the mixed results and framing the evaluation around the product goal: driving meaningful engagement with pinned accounts. Then, break down the metrics to assess whether the lift in switch CTR is a true success or an artifact, and propose next steps like deeper analysis, qualitative research, and iteration.
Pro tip: Emphasize that statistical significance alone doesn't imply product success; always tie metrics to user value and long-term goals. Show you can balance data with product intuition.
Restate the experiment's objective: to increase meaningful engagement with pinned accounts. Define what success would look like beyond CTR, such as increased dwell time, replies, or unread resolution.
Evaluate the lift in switch CTR alongside the lack of improvement in downstream metrics and the increase in bounce-back. Consider whether the CTR lift is offset by negative behaviors.
Dig into why dwell time is low and bounce-back is high. Segment by user type, account type, or notification type to identify patterns. Check for novelty effects or measurement issues.
Conclude that it's not a clear success; the lift in CTR is not translating to meaningful engagement and may indicate a superficial or negative user experience.
Propose actions: run qualitative research, iterate on the feature to improve dwell time, consider shipping only if long-term metrics improve, or kill the feature if evidence suggests harm.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.