← Google Interview Insights

Google·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Apr 2026

Summary

System design round at Google focused on building a CI/CD pipeline with intelligent failure handling. The problem had a lot of moving parts and I don't think I covered everything as cleanly as I wanted to.

Questions Asked (1)

Q1

Design a CI/CD system that detects test failures, applies retry logic for flaky tests, and sends smart notifications to the right teams.

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

I jumped straight into the data flow before nailing down the requirements and kind of paid for it later.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then design a modular pipeline with distinct stages for test execution, failure detection, retry logic, and notification. Emphasize trade-offs like retry limits, flakiness detection heuristics, and notification routing based on ownership and severity.

Pro tip: Propose a feedback loop where flaky tests are quarantined and tracked over time, and notifications include actionable context like failure logs and recent code changes to reduce mean time to resolution.

1. Clarify Requirements and Scale

Ask about scale (tests per day, number of teams), latency requirements, and existing infrastructure. Define what constitutes a flaky test and the desired notification channels.

2. Design the CI/CD Pipeline Architecture

Outline stages: code commit, build, test execution, failure detection, retry mechanism, and notification service. Use a message queue for asynchronous processing and a database to track test results and flakiness history.

3. Implement Retry Logic for Flaky Tests

Design a retry policy with exponential backoff and a maximum retry count. Use heuristics like historical flakiness scores to decide which tests to retry automatically, and mark persistent failures for human review.

4. Build Smart Notification System

Route notifications based on test ownership (from code metadata), failure severity, and team preferences. Aggregate related failures to avoid alert fatigue, and include links to logs, dashboards, and recent commits.

5. Discuss Trade-offs and Scalability

Address trade-offs: retry cost vs. reliability, notification latency vs. batching, and false positives vs. missed failures. Explain how the system scales with increasing test volume and teams.

Key Points to Mention

  • Use of a message queue (e.g., Pub/Sub) for decoupling test execution from result processing and notifications.
  • Flakiness detection via historical pass/fail patterns and statistical thresholds.
  • Retry strategies: immediate retry, delayed retry with backoff, and quarantine of consistently flaky tests.
  • Notification routing based on code ownership (e.g., from OWNERS files) and severity levels.
  • Integration with existing tools like CI systems (Jenkins, GitHub Actions), monitoring (Prometheus), and alerting (PagerDuty, Slack).
  • Metrics and dashboards to track flakiness rates, retry success, and notification effectiveness.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.