← Citadel Interview Insights

Citadel·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
May 2026

Summary

Citadel system design round for a software engineering role. The whole interview was a single escalating problem about a per-key call counter, starting trivial and getting progressively more painful. Walked away feeling like I held up okay on parts 1 and 2 but fumbled some of the cross-process nuance in part 3.

Questions Asked (6)

Q1

Design and implement a per-key call counter with a single method that increments the count for a given string key and returns the new count. Start with a single-threaded, in-process implementation and explain precisely why it breaks under concurrent access.

System DesignAlgorithms & Data StructuresTechnical Trade-offs
Author's notes

The basic version is just a map with a default of zero, nothing hard there.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by implementing a simple single-threaded counter using a HashMap, then explain the read-modify-write race condition that causes lost updates under concurrency. Finally, discuss solutions like synchronization, atomic operations, or concurrent data structures, highlighting trade-offs.

Pro tip: Mention that even with a ConcurrentHashMap, the increment operation (get then put) is not atomic, so you need compute or merge methods or explicit synchronization. This shows depth beyond surface-level knowledge.

1. Clarify requirements and assumptions

Confirm that the counter is per-key, in-memory, and that the method returns the new count. Assume single-threaded initially, then address concurrency.

2. Implement single-threaded version

Use a HashMap<String, Integer> and increment the count for the given key, handling missing keys by initializing to 0.

3. Explain why it breaks under concurrency

Describe the read-modify-write sequence: two threads may read the same value, both increment, and one write overwrites the other, causing lost updates. Also mention visibility issues without synchronization.

4. Propose thread-safe solutions

Discuss options: synchronized methods, AtomicInteger with ConcurrentHashMap, or ConcurrentHashMap.compute. Compare performance and contention trade-offs.

5. Summarize and discuss trade-offs

Conclude with the chosen solution, noting scalability, contention, and suitability for high-throughput scenarios like Citadel's trading systems.

Key Points to Mention

  • Race condition: read-modify-write is not atomic, leading to lost updates.
  • Memory visibility: without synchronization, changes may not be visible to other threads.
  • Synchronized keyword or ReentrantLock for mutual exclusion.
  • AtomicInteger with ConcurrentHashMap: atomic increment but still need atomic get-and-increment.
  • ConcurrentHashMap.compute or merge for atomic updates.
  • Trade-offs: contention, scalability, and performance under high concurrency.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Make the counter thread-safe and compare at least two or three distinct approaches, covering correctness, contention, memory, and complexity. Which would you recommend by default and why?

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

I went coarse lock first as a baseline, then per-key atomic integers with a concurrent map, then briefly touched on striped locking as a middle ground.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the counter's usage pattern (read/write ratio, contention level, latency requirements) and then present at least three distinct thread-safe approaches: atomic operations, mutex-based locking, and sharded counters. For each, analyze correctness, contention, memory overhead, and complexity, and conclude with a default recommendation (e.g., atomic for low contention, sharded for high contention) justified by the trade-offs.

Pro tip: Demonstrate awareness of false sharing and cache-line padding in sharded counters, and mention that atomic operations are not always lock-free on all architectures—this shows depth beyond textbook answers.

1. Clarify requirements and constraints

Ask about expected contention, read/write ratio, latency sensitivity, and memory budget to tailor the solution. This shows you don't jump to code without understanding the problem.

2. Present approach 1: Atomic operations

Use std::atomic<int> with fetch_add for increments. Discuss correctness (linearizable), contention (CAS retries under high contention), memory (minimal), and complexity (simple).

3. Present approach 2: Mutex-based locking

Use a mutex to guard the counter. Discuss correctness (mutual exclusion), contention (serializes all operations, high overhead), memory (mutex overhead), and complexity (simple but prone to deadlocks if misused).

4. Present approach 3: Sharded counters

Use per-thread or per-core counters summed on read. Discuss correctness (eventual consistency for reads), contention (minimal, scales with cores), memory (higher due to padding), and complexity (moderate, requires careful aggregation).

5. Compare and recommend

Summarize trade-offs in a table and recommend a default (e.g., atomic for general use, sharded for high contention) with justification. Mention hybrid approaches if applicable.

Key Points to Mention

  • Atomic operations: lock-free but may spin under contention; memory ordering (relaxed vs. sequential consistency) affects performance.
  • Mutex: simple but high contention leads to context switches and priority inversion; consider spinlocks for short critical sections.
  • Sharded counters: reduce contention by partitioning; need padding to avoid false sharing; read requires summing shards, which may be non-atomic.
  • Correctness: linearizability vs. eventual consistency; atomic and mutex provide linearizability, sharded may not.
  • Contention: atomic and mutex degrade with more threads; sharded scales better but has overhead.
  • Memory: atomic and mutex use minimal memory; sharded uses more due to per-thread counters and padding.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

Now extend the counter to work across multiple independent processes on the same host. Why do the Part 2 techniques fail here, and what are your options?

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

I knew the heap isolation answer cold: separate address spaces, locks and atomics in one process are invisible to others.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, explain why Part 2 techniques (e.g., mutexes, atomics, or single-process synchronization) fail across processes due to separate address spaces. Then, outline options for inter-process synchronization, such as file locks, shared memory with semaphores, or message passing, and discuss trade-offs like performance, complexity, and portability.

Pro tip: Mention that file locks are simple but can be slow and have edge cases (e.g., NFS), while shared memory with semaphores is faster but requires careful initialization and cleanup. This shows you consider real-world constraints.

1. Identify the limitation

Explain that Part 2 techniques rely on shared memory within a single process, so they don't work across processes because each process has its own address space.

2. List synchronization options

Describe mechanisms for inter-process synchronization: file locks (flock, fcntl), POSIX semaphores, shared memory with semaphores, message queues, and sockets.

3. Compare trade-offs

Discuss performance (shared memory fastest, file locks slower), complexity (semaphores need careful init/cleanup), portability (POSIX vs. System V), and failure modes (deadlocks, stale locks).

4. Recommend an approach

Choose a solution based on requirements: for high performance, shared memory with semaphores; for simplicity, file locks; for robustness, message passing.

Key Points to Mention

  • Separate address spaces prevent direct sharing of variables or mutexes.
  • File locks (e.g., flock, fcntl) provide advisory locking but can be slow and have NFS issues.
  • POSIX semaphores (named or unnamed in shared memory) enable atomic operations across processes.
  • Shared memory (e.g., mmap, shmget) combined with semaphores offers high performance but requires synchronization.
  • Message passing (pipes, sockets, message queues) avoids shared state but adds serialization overhead.
  • Consider atomic operations on files or using a dedicated lock server for distributed scenarios.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you handle unbounded key growth? What eviction or TTL policy would you use, and how does eviction interact with the exact-count guarantee?

System DesignTechnical Trade-offs
Author's notes

Shorter follow-up but a good one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that unbounded key growth is a fundamental scalability issue, then propose a combination of eviction policies (e.g., LRU, LFU) and TTL-based expiration to bound memory usage. Crucially, address how these mechanisms interact with the exact-count guarantee, likely requiring a trade-off between strict accuracy and resource constraints, and suggest mitigations like approximate counting or periodic reconciliation.

Pro tip: Demonstrate awareness that in systems like Citadel's, exact counts often come with high memory costs, so propose a hybrid approach: use exact counting for critical keys and approximate counting with eviction for less critical ones, or employ a tiered storage strategy.

1. Identify the problem and constraints

Explain why unbounded key growth occurs (e.g., high cardinality, no expiration) and its impact (memory exhaustion, performance degradation). Clarify the exact-count guarantee requirement and its importance.

2. Propose eviction policies

Discuss eviction strategies like LRU, LFU, or random eviction, and their trade-offs. Explain how they bound memory but may cause loss of exact counts for evicted keys.

3. Propose TTL policies

Describe TTL-based expiration (e.g., sliding or absolute TTL) to automatically remove stale keys. Mention how TTL can be combined with eviction for better control.

4. Analyze interaction with exact-count guarantee

Explain that eviction and TTL inherently break exact counts because evicted keys are forgotten. Discuss potential mitigations: approximate counting (e.g., Count-Min Sketch), persisting evicted counts, or using a two-tier system with exact counts for hot keys and approximate for cold.

5. Recommend a balanced solution

Propose a hybrid approach: use exact counting for a bounded set of critical keys, and approximate counting with eviction/TTL for the rest. Emphasize monitoring and tuning based on workload.

Key Points to Mention

  • Eviction policies: LRU, LFU, FIFO, and their trade-offs in terms of hit rate and fairness.
  • TTL strategies: absolute vs. sliding expiration, and how to set TTLs based on access patterns.
  • Impact on exact-count guarantee: eviction causes undercounting; TTL causes counts to reset or expire.
  • Approximate counting algorithms: Count-Min Sketch, HyperLogLog, or probabilistic data structures that bound memory while providing approximate counts.
  • Hybrid approaches: tiered storage (e.g., exact counts in memory for hot keys, approximate counts on disk for cold keys) or periodic reconciliation with a backing store.
  • Monitoring and adaptive policies: dynamically adjust eviction/TTL based on memory pressure and workload changes.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

Your single-owner local service is a single point of failure. How do you make it durable and restart-safe without losing or double-counting increments?

System DesignTechnical Trade-offs
Author's notes

Write-ahead log came to mind first.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging the single point of failure and its risks, then propose a multi-layered solution combining replication, idempotent operations, and transactional guarantees. Emphasize trade-offs between consistency, availability, and complexity, and how you would validate the design under failure scenarios.

Pro tip: Highlight that idempotency alone isn't enough; you need to persist the deduplication state atomically with the increment, and consider using a write-ahead log or consensus protocol to ensure durability and exactly-once semantics.

1. Identify failure modes and requirements

Enumerate how the single owner can fail (crash, network partition, disk corruption) and clarify requirements: durability, restart-safety, no lost or double-counted increments, and acceptable latency/consistency trade-offs.

2. Design for durability and replication

Propose replicating the service state across multiple nodes using a consensus protocol (e.g., Raft) or a replicated log, ensuring that increments are durably persisted before acknowledging success.

3. Ensure idempotent and exactly-once increments

Introduce unique request IDs and maintain a deduplication store (e.g., a set of processed IDs) that is updated atomically with the increment, so retries don't double-count.

4. Handle restart and recovery

Describe how the service recovers state from the replicated log or snapshot on restart, and how it resumes processing without losing or duplicating increments (e.g., replaying log with idempotent operations).

5. Discuss trade-offs and validation

Compare approaches (e.g., synchronous vs asynchronous replication, consensus overhead) and explain how you would test for correctness under failures (e.g., chaos engineering, fault injection).

Key Points to Mention

  • Replication and consensus (e.g., Raft, Paxos) to eliminate single point of failure
  • Idempotency keys and deduplication to prevent double-counting
  • Atomicity of increment and deduplication state (e.g., using transactions or compare-and-swap)
  • Write-ahead logging and snapshots for durability and fast recovery
  • Trade-offs: consistency vs availability, latency vs durability, complexity vs simplicity
  • Monitoring and alerting for replication lag and failure detection

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q6

Under extreme contention on a small number of hot keys, how do you sustain throughput while still returning an exact per-call count to each caller?

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

This one I fumbled a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: exact per-call counts, high contention on a few hot keys, and the need to sustain throughput. Then propose a hybrid approach that combines sharding the hot keys across multiple counters with a read-time aggregation, or using a lock-free atomic counter with batching. Discuss trade-offs between accuracy, latency, and complexity, and mention how you would handle contention at scale.

Pro tip: Emphasize that exact per-call counts under extreme contention often require a trade-off between strict real-time accuracy and throughput; propose a design that provides exact counts with bounded staleness or uses a two-tiered counting system. Show awareness of false sharing and cache-line contention in atomic operations.

1. Clarify Requirements and Constraints

Ask about the expected throughput, latency requirements, and whether exact counts must be returned immediately or can be eventually consistent. Confirm the number of hot keys and the scale of contention.

2. Explore Naive Approaches and Their Limitations

Discuss simple solutions like a single atomic counter or a mutex-protected counter, and explain why they fail under extreme contention due to serialization and cache-line bouncing.

3. Propose a Scalable Counting Architecture

Suggest sharding the counter across multiple threads or nodes (e.g., per-thread counters, striped counters) and aggregating on read. Alternatively, use a concurrent data structure like a concurrent hash map with atomic updates, or a batching approach where increments are buffered and flushed periodically.

4. Address Exactness and Per-Call Counts

Explain how to return an exact count to each caller: either by reading all shards and summing (if shards are local and fast), or by using a centralized counter with a fast path (e.g., combining a local counter with a global counter). Discuss how to handle concurrent reads and writes to ensure exactness.

5. Discuss Trade-offs and Optimizations

Cover trade-offs: sharding reduces contention but increases read complexity; batching improves throughput but adds latency. Mention optimizations like padding to avoid false sharing, using non-blocking algorithms, and considering hardware atomic instructions.

Key Points to Mention

  • Sharding/striping counters to reduce contention on hot keys
  • Atomic operations and memory ordering (e.g., compare-and-swap, fetch-and-add)
  • False sharing and cache-line padding
  • Batching or buffering increments to amortize synchronization costs
  • Read-time aggregation and consistency guarantees
  • Trade-offs between exactness, latency, and throughput

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.