Databricks·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

Databricks system design round focused entirely on a concurrent event logger, which sounds scoped but actually opens up into a pretty deep rabbit hole of tradeoffs. Came out feeling okay but not great.

Questions Asked (1)

Design a thread-safe event logger that supports many concurrent producers. The log() call should be non-blocking or low-blocking, and a background process should flush events to a sink in the order they were submitted or timestamped. Walk through your synchronization choices, how you handle back-pressure when the queue fills, graceful shutdown without dropping events, and how you'd test it under heavy contention.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

I started with the obvious lock-plus-queue approach and the interviewer was fine with it but immediately pushed on what happens when the queue fills.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements: throughput, latency, ordering guarantees, and durability. Then propose a lock-free ring buffer with atomic sequence numbers for non-blocking log(), and a dedicated flusher thread that drains in batches. Discuss back-pressure via bounded queue and configurable overflow policies, and graceful shutdown using a poison pill and drain loop.

Pro tip: Emphasize that ordering is maintained by assigning a monotonically increasing sequence number at enqueue time, and the flusher processes in that order. Also, mention that you'd use a memory-mapped file or async I/O for the sink to avoid blocking the flusher.

1. Clarify requirements and constraints

Ask about expected throughput, latency bounds, ordering semantics (submission vs timestamp), durability needs, and acceptable back-pressure strategies.

2. Design the concurrent queue

Propose a bounded, lock-free ring buffer (e.g., LMAX Disruptor style) with atomic sequence numbers for producers and a single consumer. Explain how CAS operations ensure non-blocking enqueue.

3. Handle back-pressure and overflow

Describe policies when the queue is full: block, drop oldest, drop newest, or spill to disk. Discuss trade-offs and how to make it configurable.

4. Implement graceful shutdown

Use a shutdown flag and a poison pill or sentinel event. The flusher drains remaining events before exiting. Ensure producers can detect shutdown and stop logging.

5. Test under contention

Outline stress tests with many producer threads, measure latency and throughput, verify ordering, and test shutdown scenarios. Use tools like JMH or custom harnesses.

Key Points to Mention

Lock-free ring buffer with atomic sequence numbers (e.g., Disruptor pattern) for non-blocking log().
Ordering guaranteed by assigning a monotonic sequence number at enqueue time; flusher processes in order.
Back-pressure strategies: bounded queue with configurable overflow (block, drop, spill) and monitoring.
Graceful shutdown via poison pill and drain loop; ensure no events lost.
Batch flushing to sink (e.g., memory-mapped file, async I/O) to reduce syscalls and improve throughput.
Testing with high contention: many threads, measure p99 latency, verify ordering, and simulate shutdown.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.