Harvey AI·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

System design round at Harvey AI for a software engineering role. The whole interview was basically one long question about file deduplication that kept branching into harder and harder territory. Walked out not totally sure how I did.

Questions Asked (1)

How would you determine whether two files have identical byte content, ignoring filesystem metadata? Design a hashing-based approach and extend it to large-scale deduplication across machines.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This question had like seven sub-parts and I did not fully appreciate that until I was already three minutes into talking about SHA-256 vs MD5.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that byte-identical content means comparing raw bytes, not metadata like timestamps or permissions. For small files, a direct byte-by-byte comparison works, but for efficiency and scale, use cryptographic hashes (e.g., SHA-256) to compare fingerprints. Then extend to large-scale deduplication by discussing distributed hash tables, chunking, and trade-offs between accuracy and performance.

Pro tip: Mention that hashing alone can have collisions, so for critical systems, you might verify with a byte-by-byte check when hashes match. Also, highlight that chunk-level deduplication (like rsync or content-defined chunking) is more effective for large files with small changes.

1. Clarify requirements and constraints

Confirm that 'identical byte content' means ignoring metadata, and ask about file sizes, number of files, and performance needs. This sets the stage for choosing the right approach.

2. Design a basic hashing approach

Compute a cryptographic hash (e.g., SHA-256) of each file's content and compare the digests. If hashes differ, files are different; if they match, they are likely identical (with negligible collision probability).

3. Address hash collisions and verification

Acknowledge that hash collisions are possible, though extremely rare. For absolute certainty, perform a byte-by-byte comparison when hashes match, or use a second independent hash.

4. Extend to large-scale deduplication

For many files across machines, use a distributed hash table or a central index mapping hash to file locations. To handle large files efficiently, chunk them (e.g., fixed-size or content-defined) and hash each chunk, enabling deduplication at chunk level.

5. Discuss trade-offs and optimizations

Compare hashing algorithms (speed vs. collision resistance), chunking strategies (fixed vs. variable), and network overhead. Mention techniques like bloom filters for quick negative checks and Merkle trees for hierarchical verification.

Key Points to Mention

Cryptographic hash functions (SHA-256, BLAKE3) and their properties (collision resistance, speed).
Hash collision probability and the need for verification in critical systems.
Chunking strategies: fixed-size vs. content-defined chunking (CDC) for efficient deduplication.
Distributed hash tables (DHTs) or centralized indexes for cross-machine deduplication.
Trade-offs: hashing speed vs. security, memory vs. accuracy, and network bandwidth.
Real-world examples: Git (content-addressable storage), rsync (delta transfer), and backup systems like Borg.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.