HubSpot·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

HubSpot system design round for a software engineering role. One meaty question that took the whole session, and I left feeling like I'd covered maybe 70% of what they were looking for.

Questions Asked (1)

Design an algorithm to find duplicate files across a large directory tree. You're given an iterator of (path, size) pairs and a read_chunks function. Minimize I/O, handle hash collisions, support datasets too big to fit in memory, and explain how you'd parallelize it.

System DesignAlgorithms & Data StructuresTechnical Trade-offs

Author's notes

I started with the obvious size-bucketing step and they seemed fine with that.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by grouping files by size to eliminate unique sizes without reading content, then compute hashes only for files with matching sizes, using a two-level hash (e.g., fast hash then cryptographic hash) to minimize I/O and handle collisions. Design the algorithm to stream data and use external sorting or disk-based hash tables for datasets too large for memory, and parallelize by partitioning files across workers while ensuring thread-safe access to shared state.

Pro tip: Emphasize that you would first check file sizes to avoid unnecessary reads, and mention that you'd use a fast non-cryptographic hash (like xxHash) for initial grouping and a cryptographic hash (like SHA-256) only for final verification to balance speed and collision resistance.

1. Group by size

Iterate through the (path, size) pairs and group files by size, discarding any size group with only one file. This avoids reading file contents for unique sizes.

2. Hash candidates

For each size group with multiple files, read file chunks and compute a fast hash (e.g., xxHash) for each file. Group files by this hash to further narrow down potential duplicates.

3. Verify with cryptographic hash

For files with matching fast hashes, compute a cryptographic hash (e.g., SHA-256) to confirm duplicates and handle collisions. Only files with identical cryptographic hashes are true duplicates.

4. Handle large datasets

If the dataset is too large for memory, use external sorting or disk-based hash tables to store intermediate results. Process files in batches and write partial results to disk, then merge.

5. Parallelize

Partition files across multiple workers (e.g., by hash of file path) to process in parallel. Use a thread pool or distributed system, ensuring that shared data structures are synchronized or partitioned to avoid contention.

Key Points to Mention

Minimize I/O by reading files only when necessary (after size grouping) and reading in chunks to avoid loading entire files into memory.
Use a two-level hashing strategy: fast hash for initial grouping and cryptographic hash for final verification to handle collisions.
For datasets too large for memory, employ external sorting or disk-based data structures, and process files in streaming fashion.
Parallelize by partitioning files across workers, using consistent hashing to distribute load evenly, and synchronize access to shared state.
Consider trade-offs: fast hash may have collisions but is faster; cryptographic hash is slower but collision-resistant. Balance based on requirements.
Discuss scalability: the algorithm should handle millions of files and terabytes of data by leveraging disk and distributed computing.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.