The SQL pressure is the part that separates this from a generic case question. Conceptual answers about cannibalization don't land when someone wants to see you actually think through the join. What I'd do first is confirm the exposure logs are clean: count distinct user-experiment assignments versus count distinct users who fired a carousel impression event, and if those don't match within some tight tolerance you have a logging bug before you have a product insight. That query is maybe four lines and should always come first.

For cannibalization specifically, you want to look at click events on other home feed surfaces, segmented by whether the user was in treatment or control, normalized by session. If treatment users are clicking pins below the carousel at a higher rate than control users click equivalent feed positions, that's strong evidence the carousel is redirecting engagement rather than destroying it. The DAU and time spent staying flat actually makes this hypothesis more plausible, not less.

Scroll depth is the one I'd prioritize after that. If you have scroll position logged against time-in-session, you can check whether treatment users are reaching the same feed depth as control. A carousel that's visually heavy can create a friction point where users mentally 'finish' browsing earlier. Novelty effect you check by plotting the CTR metric day-over-day since launch and looking for a decay curve. If day-one CTR was actually fine and it's been sliding since, that's a very different conversation than if it was low from hour one.

The CTR numerator problem is real and I've fumbled the same thing. Raw clicks on a carousel are almost meaningless without knowing what the denominator should be. A user who scrolls past three cards and clicks the fourth has a very different behavioral signal than someone who clicks the first card reflexively. Pinterest specifically has this problem because the visual density of a carousel means exposure is uneven across positions, so impression-level CTR conflates position bias with genuine interest. Saves per carousel impression, or even downstream engagement like close-up views that lead to board saves, is a much cleaner proxy for whether the surface is doing its actual job of surfacing relevant content.

For guardrails, the ones you landed on are solid but the notification volume angle is genuinely non-obvious unless you've thought about how Pinterest's re-engagement loop works. If the carousel is surfacing content that users don't act on but the system interprets as interest, it can inflate notification triggers downstream, which then inflates DAU in a way that looks healthy but is actually churn-accelerating. That's the kind of second-order effect that makes a flat DAU number misleading. Latency and crash rate are table-stakes guardrails for any surface launch, but I'd also add scroll depth on the home feed below the carousel, because if the carousel is eating vertical real estate and users stop scrolling, you're cannibalizing the rest of the feed without seeing it in a single CTR number.

The vague severity answer is something I've given too and it does fall flat. The follow-up is always 'okay but what's the threshold' and if you don't have a number ready it sounds like you haven't actually shipped anything under real constraints.

The way I think about it now: rollback criteria should be pre-committed around guardrail metrics that have user-harm implications, not primary metrics that are ambiguous. If session length drops more than some percentage in a statistically significant way, or if crash rate spikes, those are rollback triggers because they're hurting users right now regardless of whether the feature eventually works. A drop in carousel CTR alone is not a rollback trigger because the interpretation is too uncertain.

Iterate vs rollback for the ambiguous middle is really a question of whether you have a credible fix hypothesis. If scroll-depth analysis shows the carousel is blocking feed engagement, moving it below the fold is a one-sprint test with a clear prediction. That's worth iterating. If the content quality is just bad because of cold-start problems and you don't have a near-term solution to the recommendation model, holding the feature in production while it degrades trust in the surface is probably worse than pulling it. The concrete thing I'd add to your answer is naming a specific degradation number for one guardrail, even if you caveat it as illustrative, because it shows you've thought about what the metric actually means in practice rather than just naming it.

Notification-referred vs organic is embarrassingly obvious for Pinterest once you think about it, agreed. A huge chunk of Pinterest sessions start from a push or email notification pointing at specific content, and those users arrive with intent that a homepage carousel might actively interrupt. Their CTR behavior on the carousel would naturally be lower because they came to do something specific, not browse. Mixing them into your overall metric without that split makes the carousel look worse than it is for the organic discovery use case it was actually built for.