The metric definitions felt manageable at first.
Start by clarifying the survey's purpose and how it fits into the product ecosystem, then define a primary goal metric that directly measures the survey's success (e.g., response rate), driver metrics that influence it (e.g., notification relevance), and guardrail metrics that ensure no harm (e.g., notification opt-out rate). When interpreting, explicitly address network effects (spillovers between users) and novelty effects (temporary behavior changes) by using techniques like cluster randomization, holdout groups, and longitudinal analysis.
Pro tip: Emphasize that guardrail metrics should include both user experience and business health, and that network effects can be mitigated by randomizing at the cluster level (e.g., by social graph clusters) rather than by user.
Understand that the survey is triggered after notifications to gather feedback, and its success depends on user engagement without harming the notification experience.
Choose a metric that directly measures the survey's effectiveness, such as survey response rate or completion rate, which aligns with the goal of collecting feedback.
Select driver metrics that influence the primary goal (e.g., notification click-through rate, survey relevance score) and guardrail metrics to monitor unintended consequences (e.g., notification opt-out rate, app uninstalls, overall engagement).
Recognize that users interact, so survey responses or notification behavior may spill over. Use cluster randomization (e.g., by social clusters) and measure indirect effects via network analysis.
Anticipate that initial responses may be inflated due to curiosity. Use a holdout group, run the experiment longer, and analyze trends over time to separate novelty from sustained impact.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.