Dropbox·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

Dropbox system design round, no coding, just a whiteboard conversation about scaling a file crawler. The interviewer wanted to go pretty deep on every layer of the design and I felt underprepared for how far into the weeds they wanted to go on fault tolerance specifically.

Questions Asked (1)

You have a working single-machine file crawler that walks a directory tree and processes files. How would you scale it to handle a much larger workload? Walk through your concurrency model, how you'd partition the file space, queuing and dispatch, deduplication, fault tolerance and retries, throttling, and what metrics you'd track.

System DesignTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This is basically a distributed systems design question dressed up as a file crawler question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints (scale, latency, consistency) before diving into the design. Then propose a distributed architecture that decouples crawling, queuing, and processing, and systematically address each aspect: concurrency, partitioning, deduplication, fault tolerance, throttling, and metrics. Emphasize trade-offs and justify choices based on the specific needs of a file crawler at Dropbox scale.

Pro tip: Tie your design to Dropbox's existing infrastructure (e.g., S3 for storage, Kafka for queuing, and metadata services) to show you understand the company's tech stack and can leverage it effectively. Also, highlight how you'd handle incremental crawling and avoid rescanning unchanged files, which is crucial for efficiency at scale.

1. Clarify Requirements and Constraints

Ask questions to understand the scale (number of files, size, growth rate), latency requirements, consistency needs, and whether the crawler is for indexing, backup, or another purpose. This shapes the entire design.

2. Design the Concurrency Model

Choose between a thread pool, async I/O, or a distributed worker model. For a single machine, use a thread pool with bounded queues; for distributed, use multiple worker processes or containers managed by an orchestrator like Kubernetes.

3. Partition the File Space and Dispatch Work

Partition by directory hierarchy, file hash, or metadata to distribute load evenly. Use a central queue (e.g., Kafka, SQS) or a coordinator to dispatch tasks to workers, ensuring no single point of contention.

4. Implement Deduplication and Fault Tolerance

Use a distributed cache or database (e.g., Redis, DynamoDB) to track processed files and avoid duplicates. Implement retries with exponential backoff, dead-letter queues, and idempotent processing to handle failures.

5. Add Throttling and Monitoring

Introduce rate limiting per worker or globally to avoid overwhelming storage or downstream services. Define key metrics (throughput, latency, error rates, queue depth) and set up dashboards and alerts.

Key Points to Mention

Concurrency model: thread pool vs. async I/O vs. distributed workers, and how to handle backpressure.
Partitioning strategy: consistent hashing, directory-based sharding, or file-based sharding to balance load.
Queuing and dispatch: use of message queues (Kafka, RabbitMQ) or a coordinator service for task distribution.
Deduplication: using a distributed set (Bloom filter, Redis) to track processed files and avoid redundant work.
Fault tolerance: retries with exponential backoff, dead-letter queues, checkpointing, and idempotent operations.
Throttling and metrics: rate limiting, monitoring throughput, latency, error rates, queue depth, and resource utilization.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.