Databricks·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Apr 2026Remote

Summary

Databricks systems design round, and they went straight for the jugular with a distributed fault-tolerance problem. The layered circuit breaker question was more involved than I expected and I kept second-guessing myself on the thread safety parts.

Questions Asked (1)

Design a two-layer circuit breaker (per-instance and global) that protects a downstream service, supporting CLOSED, OPEN, and HALF_OPEN states with configurable failure thresholds, a rolling window, and a cool-down period. The global breaker should influence local breaker behavior, and both must be passed for a request to go through. Walk through allow(), recordSuccess(), recordFailure(), state transitions, thread safety, and how you'd implement the metrics window.

System DesignTechnical Trade-offsAPI & Integrations

Author's notes

This one ate up the whole session.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and defining the state machine for a single breaker, then extend to two layers with a clear composition rule (both must allow). Walk through the API methods and state transitions with thread-safety in mind, and finish by explaining the rolling window implementation and trade-offs.

Pro tip: Emphasize that the global breaker acts as a coarse-grained safety net while the local breaker provides fine-grained isolation; when the global opens, local breakers should not waste resources probing until the global recovers. Also, mention that metrics should be shared or aggregated efficiently to avoid double-counting.

1. Clarify requirements and define the single-breaker state machine

Confirm failure thresholds, window size, cool-down, and what constitutes a failure. Define CLOSED, OPEN, HALF_OPEN transitions and the conditions for each.

2. Design the two-layer composition and allow() logic

Explain that a request is allowed only if both local and global breakers allow it. Describe how the global breaker influences local behavior (e.g., global OPEN forces local to OPEN or prevents HALF_OPEN probes).

3. Detail recordSuccess() and recordFailure() with thread safety

Describe how outcomes update the rolling window and trigger state changes. Discuss synchronization mechanisms (locks, atomics, concurrent data structures) to ensure thread safety.

4. Implement the rolling window for metrics

Choose a data structure (e.g., time-bucketed ring buffer or sliding window with timestamps) and explain how to compute failure rate over the window. Discuss memory and performance trade-offs.

5. Discuss state transitions and edge cases

Walk through transitions: CLOSED to OPEN when failure threshold exceeded, OPEN to HALF_OPEN after cool-down, HALF_OPEN to CLOSED on success or back to OPEN on failure. Cover edge cases like concurrent probes and global-local interaction.

Key Points to Mention

State machine: CLOSED, OPEN, HALF_OPEN with clear transition conditions and cool-down timer.
Two-layer composition: request allowed only if both local and global breakers allow; global OPEN should short-circuit local probes.
Thread safety: use locks or atomic operations for state changes; ensure rolling window updates are thread-safe.
Rolling window implementation: time-bucketed counters or sliding window with timestamps; consider memory vs. accuracy trade-offs.
Metrics aggregation: how to avoid double-counting when both layers record; consider sharing metrics or having global aggregate from locals.
Failure definition: what counts as a failure (exceptions, timeouts, specific error codes) and how to configure thresholds.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.