Start by clarifying the business goal (e.g., maximize long-term player LTV) and the ad types (rewarded vs. interstitial). Then outline a structured plan covering metrics, experiment design, incremental impact estimation, validity threats, and a ship/don't ship decision framework. Emphasize the importance of measuring incremental impact and considering long-term effects.
Pro tip: Propose a long-term holdout group to measure the persistent effects of ads on retention and monetization, and use a difference-in-differences approach to isolate the incremental impact from other factors.
Identify primary metrics (e.g., incremental revenue per user, retention) and guardrail metrics (e.g., player satisfaction, churn). Consider both short-term and long-term effects.
Choose a randomized controlled trial with a control group (no ads) and treatment groups (rewarded, interstitial, or both). Determine sample size, duration, and randomization unit (e.g., user-level).
Use methods like difference-in-differences or causal inference to measure the incremental effect of ads on key metrics, accounting for seasonality and external factors.
Consider novelty effects, cannibalization of in-app purchases, selection bias, and network effects. Plan sensitivity analyses and holdout groups to mitigate.
Define decision criteria based on statistical significance, practical significance (effect size), and alignment with long-term strategy. Include a cost-benefit analysis and stakeholder alignment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.