Anthropic·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jul 2026

Summary

System design round at Anthropic for a software engineering role. The whole question was about deduplicating files across a massive directory tree, which sounds manageable until you start thinking about scale and the interviewer keeps asking follow-ups.

Questions Asked (1)

Design a program to find and deduplicate files across a very large directory tree, without loading entire files into memory. Walk through your hashing strategy, how you handle very large files, hash collisions, hard-linking duplicates safely, and how you'd parallelize the whole thing across cores or machines.

System DesignAlgorithms & Data StructuresTechnical Trade-offs

Author's notes

This question has a lot of layers and I kept getting pulled into the wrong ones first.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer as a multi-stage pipeline: first enumerate files and group by size, then hash only same-size candidates using a streaming hash, and finally verify byte-by-byte before hard-linking. Emphasize memory efficiency, collision handling, and safe parallelization across cores and machines.

Pro tip: Mention that you would use a two-level hashing strategy: a fast non-cryptographic hash (e.g., xxHash) for initial grouping, then a cryptographic hash (e.g., BLAKE3) for verification, and finally a byte-by-byte comparison to eliminate any collision risk. Also note that hard-linking should be done atomically and only after verifying content, and that you'd preserve one canonical copy per inode.

1. Enumerate and group by size

Walk the directory tree and collect file paths with their sizes, grouping files by size. Files with unique sizes cannot be duplicates, so they are skipped, drastically reducing the candidate set.

2. Stream-hash candidates

For each size group, compute a hash of each file using a streaming API (e.g., read in chunks) so files are never fully loaded into memory. Use a fast hash first, then a cryptographic hash for stronger guarantees.

3. Verify and handle collisions

When hashes match, perform a byte-by-byte comparison to confirm true duplicates. This eliminates any chance of hash collisions causing data loss.

4. Deduplicate safely with hard links

For each set of verified duplicates, keep one canonical file and replace the others with hard links to it. Ensure atomicity and handle cross-filesystem cases by copying or skipping.

5. Parallelize across cores and machines

Use a work queue to distribute files across threads/processes for hashing, and shard by directory or hash prefix across machines. Aggregate results and perform linking in a coordinated phase.

Key Points to Mention

Streaming I/O with fixed-size buffers to avoid loading entire files into memory.
Two-level hashing: fast non-cryptographic hash for grouping, cryptographic hash for verification.
Byte-by-byte comparison as the final collision check before deduplication.
Hard-linking duplicates atomically, preserving one canonical copy and handling cross-device links.
Parallelization strategies: thread pool for I/O-bound hashing, sharding by directory or hash for distributed processing.
Trade-offs: hash collision probability, I/O vs CPU bottlenecks, and consistency in distributed settings.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.