← Abnormal Security Interview Insights

Abnormal Security·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Apr 2026

Summary

System design round at Abnormal Security for a software engineering role. The whole thing was one deep-dive question about deduplicating millions of photos on a single machine, and they really did want you to go all the way down to pseudocode.

Questions Asked (1)

Q1

You have a filesystem with millions of photos. Design an algorithm to detect and remove byte-identical duplicate files on a single machine. Cover how you compute and store file signatures, how you map signatures to canonical paths in memory, how you handle hash collisions before deleting anything, how you treat files that share a name but differ in content or metadata, and what the time/space complexity and I/O tradeoffs look like. Write pseudocode for a function that returns the set of paths safe to delete.

System DesignAlgorithms & Data StructuresTechnical Trade-offs
Author's notes

This one sprawled in a way I didn't expect.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints (e.g., single machine, millions of files, byte-identical duplicates). Then propose a two-phase approach: first compute a strong cryptographic hash (e.g., SHA-256) for each file and group by hash; second, for groups with multiple files, perform byte-by-byte comparison to confirm duplicates and handle collisions. Finally, discuss memory management, I/O tradeoffs, and provide pseudocode for the deletion function.

Pro tip: Emphasize that you never delete based solely on hash; always verify with a byte-by-byte comparison for groups with multiple files. Also, mention that you can optimize by first grouping by file size to avoid hashing unique-sized files.

1. Clarify requirements and constraints

Ask about file size distribution, available memory, whether metadata matters, and if the filesystem is static during the operation. This ensures the solution fits the context.

2. Design signature computation and storage

Choose a strong hash (e.g., SHA-256) and compute it for each file. Store signatures in a map from hash to list of file paths, but consider memory usage and potential need for external sorting or disk-based storage.

3. Handle collisions and confirm duplicates

For each hash with multiple files, perform byte-by-byte comparison to confirm true duplicates. This avoids accidental deletion due to hash collisions.

4. Define deletion policy and pseudocode

Decide which file to keep (e.g., oldest, shortest path) and return the set of paths to delete. Write pseudocode that includes the verification step.

5. Analyze complexity and tradeoffs

Discuss time complexity (O(N) hashing + O(D*S) verification), space complexity (O(N) for hash map), and I/O tradeoffs (hashing reads entire files, but grouping by size reduces I/O).

Key Points to Mention

  • Use a strong cryptographic hash like SHA-256 to minimize collision probability, but still verify with byte-by-byte comparison.
  • Group files by size first to avoid hashing files that cannot have duplicates, reducing I/O.
  • Memory management: store only hash and file paths; if memory is insufficient, use external sorting or a database.
  • Handle files with same name but different content: they are not duplicates; only byte-identical files are considered.
  • Metadata (e.g., timestamps, permissions) should not affect duplicate detection; only content matters.
  • Time complexity: O(N) to hash all files, plus O(D*S) to verify duplicates, where D is number of duplicate groups and S is average file size.
  • Space complexity: O(N) for the hash map, but can be reduced with disk-based storage.
  • I/O tradeoff: hashing reads entire files, but grouping by size and using partial hashes can reduce I/O.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.