This question is basically five questions stapled together, which I did not fully appreciate until I was already two minutes into talking about variance estimation and realized I hadn't touched Simpson's paradox yet.
Start by decomposing the metric into numerator (clicks) and denominator (impressions) to see which component drove the change, then check for composition effects (e.g., traffic mix, device, geography) and instrumentation issues (e.g., logging, bot filtering). Finally, apply appropriate statistical methods for ratio metrics (e.g., delta method, bootstrap) and run follow-up analyses (e.g., segment-level, novelty effects, pre-period A/A tests) to confirm robustness.
Pro tip: Always check the denominator first: a drop in impressions can inflate CTR even if clicks are flat. Also, look for Simpson's paradox by examining segment-level data—aggregate gains can hide losses in key segments.
Break down the change into clicks (numerator) and impressions (denominator) to identify which component is driving the +1.2% relative increase. Check if clicks increased, impressions decreased, or both.
Examine whether the treatment and control groups differ in traffic composition (e.g., device, browser, geography, user demographics) that could affect CTR. Use stratification or regression to adjust for imbalances.
Check for logging errors, bot traffic, or changes in data collection that could artificially inflate CTR. Ensure both groups have consistent instrumentation and no missing data.
Use methods like the delta method, bootstrap, or linearized metrics to compute variance correctly for ratio metrics (CTR). Avoid naive t-tests that assume independent observations.
Perform segment-level analyses, check for novelty effects, run A/A tests, and examine pre-period trends to confirm the change is real and not an artifact.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.