Google·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

System design round at Google focused entirely on distributed systems debugging. One long, deep question about replica inconsistencies that sprawled into CAP theorem territory by the end. Felt like a reasonable session but I probably undercooked the repair/mitigation side.

Questions Asked (1)

A distributed database is showing data inconsistencies across its replicas. Walk through the likely causes, how you'd investigate, what you'd look at in logs and telemetry, how you'd verify consistency guarantees, and how you'd repair the system. Also discuss the trade-offs between consistency, availability, and latency while the incident is ongoing.

System DesignRoot Cause AnalysisTechnical Trade-offs

Author's notes

This is a beast of a question and I kind of underestimated how wide it was.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by systematically categorizing potential causes (e.g., network, hardware, software bugs, configuration) and then describe a methodical investigation using logs, metrics, and tracing. Emphasize verification of consistency guarantees and a safe repair process, while balancing trade-offs between consistency, availability, and latency during the incident.

Pro tip: Demonstrate a bias for action by proposing a temporary mitigation (e.g., routing reads to a single replica) to reduce user impact while investigating, and always consider the CAP theorem implications in your trade-off discussion.

1. Identify and Categorize Likely Causes

List potential causes such as network partitions, clock skew, hardware failures, software bugs, misconfigurations, or concurrent writes. Prioritize based on recent changes and system architecture.

2. Investigate Using Logs and Telemetry

Examine logs for errors, warnings, and unusual patterns; check metrics like replication lag, error rates, and latency; use distributed tracing to pinpoint where inconsistencies arise.

3. Verify Consistency Guarantees

Determine the expected consistency model (e.g., strong, eventual) and test whether the system meets it. Use tools like read-repair, anti-entropy, or checksums to detect and quantify inconsistencies.

4. Repair the System

Choose a repair strategy: if safe, perform online repair (e.g., read-repair, hinted handoff); otherwise, isolate affected replicas, restore from backups, or rebuild. Ensure repairs don't exacerbate inconsistencies.

5. Discuss Trade-offs During Incident

Explain how to balance consistency, availability, and latency: e.g., sacrificing availability for consistency by taking replicas offline, or accepting eventual consistency to maintain availability, and communicate these decisions clearly.

Key Points to Mention

CAP theorem and the specific consistency model of the database (e.g., strong vs. eventual).
Common causes: network partitions, clock skew, hardware faults, software bugs, and misconfigurations.
Tools for investigation: logs (e.g., error patterns), metrics (replication lag, error rates), and distributed tracing.
Consistency verification techniques: read-repair, anti-entropy, Merkle trees, and checksums.
Repair strategies: online repair (read-repair, hinted handoff) vs. offline repair (restore from backup, rebuild replica).
Trade-offs: during incident, may need to choose between consistency and availability/latency; communicate impact and mitigation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.