This is basically two questions stitched together and I treated them separately which I think hurt me.
Start by clarifying requirements and scale, then design a data pipeline that ingests per-second metrics from dozens of servers, computes robust baselines, and applies anomaly detection algorithms. Finally, define a paging policy with ack, snooze, and re-page intervals that balances alert fatigue and regression detection, and discuss trade-offs.
Pro tip: Emphasize that alerting should be based on sustained deviations and use techniques like EWMA or change point detection to avoid false positives from transient spikes. Also, mention that the paging policy should include escalation and auto-resolution to further reduce noise.
Ask questions to understand the number of servers, expected latency patterns, acceptable false positive/negative rates, and on-call expectations. This ensures the design meets business needs.
Outline a pipeline that collects per-second average latency from each server, aggregates them (e.g., per service or globally), and stores time-series data for analysis. Consider using a stream processor for real-time detection.
Select algorithms to detect degraded performance, such as statistical process control (e.g., EWMA, CUSUM), change point detection, or ML-based anomaly detection. Discuss how to set thresholds dynamically based on historical data.
Specify the alerting workflow: when an anomaly is detected, page the on-call engineer. Include ack (acknowledge), snooze (temporarily silence), and re-page intervals (e.g., re-alert if not acked within X minutes). Define escalation paths and auto-resolution.
Address trade-offs between sensitivity and alert fatigue, and how to tune parameters. Mention the importance of feedback loops and continuous improvement.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.