This question sounds broad until you realize they want the whole framework, not just 'pick a metric.' I started with the north-star metric and worked outward, covering acquisition through retention, but I fumbled a bit when they pushed on causal inference.
Start by clarifying the product's goals and the stage of its lifecycle, then define success metrics across multiple dimensions (adoption, engagement, retention, and business impact). Emphasize the importance of a counterfactual—ideally through A/B testing or quasi-experimental methods—to isolate the feature's causal effect, and tie the evaluation back to the company's north-star metric.
Pro tip: Show that you think in terms of counterfactuals and guardrail metrics: not just 'did the metric go up?' but 'did it go up because of the feature, and did anything else break?' This demonstrates causal rigor and product sense.
Ask what the feature is, what problem it solves, and whether it's in early testing or full launch. Align on the primary goal (e.g., activation, retention, revenue) and the target user segment.
Choose a hierarchy of metrics: a north-star metric, supporting engagement/adoption metrics, and guardrail metrics (e.g., latency, support tickets, churn). Include both leading and lagging indicators.
Prefer a randomized controlled experiment (A/B test) with sufficient power. If randomization isn't possible, use quasi-experimental methods like difference-in-differences, propensity score matching, or synthetic control.
Check for statistical significance, practical significance (effect size), and segment-level heterogeneity. Validate that the experiment was run correctly (no sample ratio mismatch, no novelty effects) and that guardrails weren't violated.
Combine quantitative results with qualitative insights (user feedback, session replays) to make a ship/no-ship/iterate recommendation. Tie the impact back to business value and suggest next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.