LinkedIn·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

LinkedIn system design round for an ML Engineer role, which ended up being a deep distributed systems question that felt more backend-heavy than I expected for the position. The breadth of what they wanted covered was a lot to fit into one session.

Questions Asked (1)

Design a distributed key-value storage service that supports high availability across zones, horizontal scalability to billions of keys, low-latency reads and writes, per-key read-after-write consistency, TTL, conditional updates, and optional range scans. Walk through the full design including API, sharding, replication, read/write paths, on-disk storage, failure handling, and observability.

System DesignTechnical Trade-offsData Modeling

Author's notes

This is basically a 'design DynamoDB but explain every decision' question and I was not ready for the full scope of it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then present a high-level architecture using consistent hashing for sharding and quorum-based replication across zones. Walk through the read/write paths, storage engine (LSM-tree), and failure handling, emphasizing trade-offs and how each feature (TTL, conditional updates, range scans) is implemented. Conclude with observability and monitoring strategies.

Pro tip: Relate the design to LinkedIn's data infrastructure (e.g., Espresso, Venice) and highlight how ML feature stores could leverage this KV store for low-latency feature retrieval, showing domain awareness.

1. Clarify Requirements and Scale

Ask about expected throughput, latency SLOs, data size, and consistency needs. Confirm the need for multi-zone availability and the specific semantics of per-key read-after-write consistency.

2. High-Level Architecture

Propose a sharded, replicated system using consistent hashing for partitioning and a quorum-based replication protocol (e.g., Paxos/Raft) across zones. Mention a coordinator service for routing and metadata management.

3. Data Model and API

Define a simple key-value API with operations: Get, Put, Delete, ConditionalPut (CAS), and Scan (range). Include TTL as a parameter on writes. Discuss how to support range scans via ordered sharding (e.g., range-based partitioning).

4. Read/Write Paths and Storage Engine

Detail the write path: client -> coordinator -> shard leader -> replication to followers -> ack. Use LSM-tree (e.g., RocksDB) for storage with WAL for durability. For reads, route to leader for read-after-write consistency or to followers with version checks. Explain TTL via expiration timestamps and conditional updates via versioning.

5. Failure Handling and Observability

Describe failure detection (heartbeats), leader election, and data repair (anti-entropy). For observability, cover metrics (latency, throughput, error rates), tracing, and logging, with alerts on SLO violations.

Key Points to Mention

Consistent hashing with virtual nodes for even distribution and minimal rebalancing.
Quorum-based replication (e.g., R+W > N) for strong consistency and high availability across zones.
LSM-tree storage engine with compaction and Bloom filters for efficient reads.
TTL implementation via expiration timestamps and background compaction.
Conditional updates using version numbers or CAS operations.
Range scans via ordered partitioning (e.g., range-based sharding) and efficient iteration.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.