I spent way too long trying to be thorough and ended up just listing every possible thing that could go wrong.
Start by clarifying the metric definition and scope of the drop, then systematically segment the data to isolate the cause. Form hypotheses across the funnel, user segments, and external factors, and validate them with data and engineering investigation.
Pro tip: Always check for data instrumentation issues first—many 'metric drops' are actually logging or tracking bugs, especially if the drop is sudden and across all segments.
Confirm what 'conversion rate' means (e.g., from app open to order placed), the exact time period, and whether the drop is real or a data anomaly. Check for instrumentation changes or logging errors.
Break down conversion by dimensions like platform (iOS/Android), geography, user cohort (new vs. existing), restaurant category, and time of day to identify where the drop is concentrated.
Map the conversion funnel (e.g., app open → search → add to cart → checkout → order) and compare conversion rates at each step week-over-week to pinpoint where the drop occurs.
Generate hypotheses for the drop (e.g., recent app release, backend issues, pricing changes, competitor promotion) and validate them using data, A/B tests, or engineering logs.
Based on findings, propose immediate fixes or further investigations, and set up monitoring to track recovery and prevent future drops.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.