Google·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

System design round at Google for a software engineer role. The whole thing was one big question about building a task scheduling system, and it went pretty deep pretty fast.

Questions Asked (1)

Design a task scheduling system that supports task dependencies, persistent storage, scheduling policies, and long-running tasks. Cover data models, APIs, execution engine, fault tolerance, backpressure, worker heartbeats, and stuck task handling. Also discuss scaling, multi-tenant isolation, and observability.

System DesignTechnical Trade-offsData Modeling

Author's notes

This is a monster of a question and I did not pace myself well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then propose a high-level architecture with a persistent metadata store (e.g., SQL/NoSQL) and a distributed queue for task execution. Walk through the data model, APIs, execution engine, and fault tolerance mechanisms, emphasizing trade-offs (e.g., at-least-once vs exactly-once, push vs pull). Conclude with scaling, multi-tenancy, and observability considerations.

Pro tip: Demonstrate maturity by explicitly discussing trade-offs (e.g., consistency vs availability, latency vs throughput) and by proposing incremental improvements (e.g., start with a simple design, then scale). Also, mention real-world systems like Google's Borg or Kubernetes CronJobs to show practical awareness.

1. Clarify Requirements and Scale

Ask questions to understand expected task volume, latency requirements, dependency complexity, and multi-tenancy needs. Define scope and assumptions.

2. Design Data Model and APIs

Propose schemas for tasks, dependencies, and workers. Define REST/gRPC APIs for task submission, status, and cancellation, ensuring idempotency and versioning.

3. Architect Execution Engine and Fault Tolerance

Describe a distributed scheduler with a queue (e.g., Kafka, Pub/Sub) and workers. Cover heartbeats, task leasing, retries, dead-letter queues, and stuck task detection via timeouts.

4. Address Backpressure and Scaling

Explain how to handle load spikes with backpressure (e.g., rate limiting, queue depth monitoring) and scale horizontally by sharding tasks and adding workers.

5. Cover Multi-Tenancy and Observability

Discuss isolation via namespaces/quotas and observability with metrics (latency, success rate), logging, tracing, and alerting.

Key Points to Mention

Persistent storage choices (SQL vs NoSQL) and schema design for tasks, dependencies, and worker state.
Scheduling policies: FIFO, priority, fairness, and dependency resolution (e.g., DAG traversal).
Fault tolerance: at-least-once execution, idempotency, retries with exponential backoff, and dead-letter queues.
Backpressure mechanisms: queue depth limits, rate limiting, and load shedding.
Worker heartbeats and stuck task handling: lease renewal, timeout-based reaping, and task re-queueing.
Multi-tenant isolation: resource quotas, namespace separation, and fair scheduling.
Observability: metrics (task latency, queue size), distributed tracing, and logging for debugging.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.