I started with engagement stuff like view-through rate and skip rate, which felt obvious, but then I got a bit tangled trying to separate advertiser-side metrics from user experience metrics.
Start by clarifying the goal of testing new ad formats—likely to increase ad revenue without harming user experience. Then structure your answer around a metrics framework that covers business, user, and advertiser perspectives, and mention how you'd use A/B testing to measure impact.
Pro tip: Emphasize the importance of guardrail metrics to ensure that ad format changes don't degrade core user engagement, and consider the trade-off between short-term revenue and long-term user retention.
Confirm the primary goal: is it to increase ad revenue, improve advertiser ROI, or enhance user experience? This shapes which metrics are most important.
Identify key performance indicators (KPIs) across three dimensions: business (revenue, ARPU), user (engagement, satisfaction), and advertiser (CTR, conversion rate).
Choose metrics that ensure the change doesn't harm core user experience, such as time spent, retention, and churn rate.
Outline an A/B test setup: control vs. treatment groups, sample size, duration, and randomization unit (e.g., user-level).
Plan to measure statistical significance, segment results, and decide whether to roll out, iterate, or abandon the new ad format.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.