Reddit·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Apr 2026Remote

Summary

Reddit system design round, one big question about building a distributed job scheduler from scratch. The scope was wide enough that I kept second-guessing whether to go deeper on one area or keep moving. Came away feeling like I covered maybe 70% of what they wanted.

Questions Asked (1)

Design a distributed job scheduler that supports user-submitted jobs with schedules, priorities, and dependencies, executing them across a pool of workers. Your design should cover cron-style scheduling, DAG-based dependencies, retries, idempotency, worker assignment, fault tolerance, scaling, and monitoring.

System DesignTechnical Trade-offsAPI & Integrations

Author's notes

This is one of those questions where the surface area is so big you have to make a call early: go broad and skim, or go deep and risk missing chunks entirely.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then propose a high-level architecture with a scheduler, metadata store, and worker pool. Dive into key components like cron scheduling, DAG execution, retries, idempotency, and fault tolerance, and discuss trade-offs and scaling strategies.

Pro tip: Emphasize idempotency and exactly-once semantics as critical for reliability, and discuss how you'd handle partial failures and duplicate executions. Show awareness of operational concerns like monitoring and alerting from the start.

1. Clarify Requirements and Scale

Ask about job volume, latency requirements, failure handling, and multi-tenancy. Establish assumptions to guide design decisions.

2. High-Level Architecture

Outline core components: API for job submission, metadata store (e.g., SQL/NoSQL), scheduler, message queue, worker pool, and monitoring. Explain how they interact.

3. Scheduling and Dependencies

Describe cron-style scheduling using a timing service, and DAG-based dependencies with topological sorting and event-driven triggers. Discuss priority queues for job ordering.

4. Execution and Reliability

Cover worker assignment (pull vs push), retries with backoff, idempotency via unique job IDs and deduplication, and fault tolerance through heartbeats, leases, and replication.

5. Scaling and Monitoring

Explain horizontal scaling of workers and scheduler, partitioning strategies, and monitoring with metrics, logging, and alerting. Discuss trade-offs like consistency vs availability.

Key Points to Mention

Idempotency: ensure jobs can be safely retried without side effects, using unique job IDs and deduplication.
DAG dependencies: use topological sorting and event-driven triggers to execute jobs in correct order.
Retries and backoff: implement exponential backoff with jitter, and dead-letter queues for failed jobs.
Worker assignment: use pull-based model with leases and heartbeats to detect failures and rebalance.
Fault tolerance: replicate metadata store, use leader election for scheduler, and design for at-least-once execution with idempotency.
Monitoring: track job success/failure rates, latency, queue depths, and worker health; set up alerts for anomalies.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.