← Atlassian Interview Insights
I started with full-file SHA-256 hashing and the interviewer immediately pushed back on what happens when two files differ by one line.
Start by clarifying requirements and scale, then propose a multi-tier architecture that uses content-defined chunking and cryptographic hashing for exact deduplication, and locality-sensitive hashing or embeddings for near-duplicate detection. Discuss trade-offs between storage savings, compute cost, and latency, and outline how to handle updates and deletions.
Pro tip: Emphasize that deduplication is a storage-layer concern and should be transparent to the application; also highlight the importance of a garbage collection strategy to reclaim space when documents are deleted or updated.
Ask about expected document types, size distribution, update frequency, and scale (e.g., petabytes, millions of users). Establish consistency, latency, and durability requirements.
Propose a chunk-based storage system using content-defined chunking (e.g., Rabin fingerprinting) and SHA-256 hashes to identify identical chunks. Store chunks in a distributed object store with a metadata service mapping documents to chunk lists.
Use locality-sensitive hashing (e.g., MinHash, SimHash) or embeddings to generate signatures for documents or chunks. Compare signatures to find near-duplicates and store only one canonical copy with references.
Discuss partitioning, indexing strategies (e.g., inverted index for signatures), and caching. Trade-offs: exact dedup is cheap and precise but misses near-duplicates; near-dup detection saves more space but adds compute and false positives.
Explain how to manage document versions, reference counting for chunks, and background garbage collection to reclaim space. Ensure consistency and avoid data loss during concurrent updates.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that hash collisions are inevitable, so you need a strategy to handle them and verify chunk identity. Describe using a strong cryptographic hash for initial comparison, then a byte-by-byte comparison or a secondary hash to confirm equality. Emphasize the trade-offs between performance and accuracy, and how this applies to deduplication or data integrity systems.
Pro tip: Mention that in practice, you'd use a Merkle tree or similar structure to efficiently verify large datasets, and that cryptographic hashes like SHA-256 make collisions so rare that they're negligible for most applications—but you still need a fallback for absolute certainty.
Explain that hash functions map arbitrary data to fixed-size values, so collisions are possible. State that you must handle them to ensure correctness.
Choose a cryptographic hash like SHA-256 to minimize collision probability. Use it as a fast, probabilistic filter to identify potential duplicates.
When hashes match, perform a direct byte comparison of the chunks to confirm they are identical. This guarantees correctness at the cost of speed.
For large-scale systems, consider using a Merkle tree or a second independent hash to reduce the need for full comparisons, balancing performance and accuracy.
Explain when to use each method: e.g., for deduplication, a strong hash alone may suffice; for critical data, always verify. Mention how this applies to systems like Git or backup software.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying requirements and scale, then describe the chunk store layout (how chunks are stored and indexed) and the manifest structure (how files are represented as sequences of chunk references). Explain how deduplication is achieved through content-addressed chunk IDs and reference counting, and discuss trade-offs like chunk size, metadata overhead, and garbage collection.
Pro tip: Emphasize that the manifest is the source of truth for file reconstruction and must be versioned and immutable; also mention that chunk store should be append-only for performance and simplicity, with garbage collection handled separately.
Ask about scale (number of files, average file size, dedup ratio), consistency needs, and whether the system is distributed. State assumptions to scope the design.
Explain that chunks are stored in a content-addressable store, keyed by cryptographic hash (e.g., SHA-256) of chunk content. Discuss physical storage (e.g., flat files, object storage) and indexing (e.g., hash table, B-tree) for fast lookup.
Explain that a manifest represents a file as an ordered list of chunk IDs, along with metadata (file size, permissions, timestamps). Discuss manifest storage (e.g., database, object store) and versioning for snapshots.
Detail how identical chunks are stored once and referenced multiple times. Describe reference counting to track chunk usage and enable safe deletion during garbage collection.
Cover chunk size trade-offs (smaller chunks = better dedup but more metadata), compression, encryption, and how to handle chunk store scalability (sharding, replication). Mention garbage collection strategies.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining encryption at rest and deduplication, then explain the fundamental conflict: encryption randomizes data, preventing identical plaintext from producing identical ciphertext, which breaks deduplication. Introduce convergent encryption as a solution that derives the encryption key from the plaintext hash, enabling identical plaintext to produce identical ciphertext while maintaining confidentiality. Conclude by discussing trade-offs and practical considerations.
Pro tip: Mention that convergent encryption is vulnerable to confirmation-of-a-file attacks, where an attacker can verify if a known file exists in the system by comparing hashes. Also note that Atlassian products like Bitbucket and Jira may use deduplication in storage layers, so understanding this trade-off is crucial for designing efficient and secure systems.
Briefly define encryption at rest (protecting stored data) and deduplication (eliminating redundant copies of data).
Describe how standard encryption with random keys or IVs produces different ciphertext for identical plaintext, defeating deduplication.
Explain that convergent encryption derives the key from the plaintext (e.g., via a hash), so identical plaintext yields identical ciphertext, enabling deduplication.
Mention security implications like confirmation-of-a-file attacks and the need for additional measures (e.g., proof of ownership).
Connect to real-world systems (e.g., backup services, Atlassian products) and discuss when to prioritize deduplication vs. strong encryption.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the system context—what are chunks, manifests, and how references work—then propose a mark-and-sweep garbage collection approach that periodically scans all manifests to identify live chunks and deletes the rest. Discuss trade-offs between batch vs. incremental collection, consistency guarantees, and how to avoid deleting chunks that are still being written or referenced.
Pro tip: Emphasize safety and idempotency: use a grace period or reference counting with tombstones to prevent race conditions where a chunk is marked for deletion but a new manifest references it before deletion completes. Also, mention monitoring and alerting on orphaned chunk growth to catch leaks early.
Ask questions to understand what chunks and manifests are, how references are stored, and what consistency guarantees are needed (e.g., can manifests be updated concurrently?).
Propose a mark-and-sweep approach: periodically scan all manifests to build a set of live chunk IDs, then compare against all stored chunks to find orphans. Consider incremental or generational variants if scale is large.
Explain how to avoid deleting chunks that become referenced during the scan: use a grace period, reference counting, or a two-phase commit with tombstones. Ensure the GC process is idempotent and can resume after failure.
Compare batch vs. continuous GC, impact on system performance, storage cost savings, and complexity. Mention using bloom filters or time-based expiration for efficiency.
Describe how to implement the GC (e.g., a scheduled job), how to test it, and what metrics to track (orphan count, GC duration, false positives).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Logical bytes stored divided by physical bytes on disk was my answer.
Start by defining deduplication ratio clearly (logical data size divided by physical data size) and explain how you would instrument the system to collect the necessary metrics. Then discuss the key metrics that matter, such as space savings, read/write amplification, and performance impact, and how they relate to Atlassian's scale and reliability needs.
Pro tip: Emphasize that deduplication ratio alone is misleading without context—always pair it with latency and throughput metrics to ensure you're not sacrificing performance for space savings.
Clearly state that deduplication ratio = logical data size / physical data size, and clarify whether it's measured at block, file, or object level.
Explain how to collect data: track unique blocks/chunks written, total logical bytes, and physical bytes stored, using counters or metadata from the deduplication engine.
Describe how to compute the ratio over time and across nodes, considering factors like sampling, aggregation windows, and handling of edge cases (e.g., empty data).
List metrics beyond the ratio: space savings percentage, deduplication hit rate, read/write amplification, CPU/memory overhead, and latency impact.
Discuss how deduplication affects performance, scalability, and cost, and how to balance these factors in a system like Atlassian's.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Full-file dedup is simpler but misses partial overlaps, like two versions of the same doc with minor edits.
Start by clarifying the workload characteristics of typical office documents—small size, frequent edits, and high redundancy across versions. Then compare full-file hashing and chunk-level deduplication on dimensions like storage efficiency, CPU/memory overhead, and implementation complexity, and conclude with a recommendation tailored to the workload.
Pro tip: Mention that chunk-level deduplication can be combined with full-file hashing as a fast-path check to avoid unnecessary chunking for unchanged files, showing you understand hybrid approaches and real-world trade-offs.
Describe typical office documents: small size (KB to low MB), frequent small edits, and high similarity across versions. This sets the context for the comparison.
Explain that full-file hashing computes a single hash for the entire file, enabling exact duplicate detection but missing partial changes.
Explain that chunk-level deduplication splits files into chunks (fixed or variable size), hashes each chunk, and stores unique chunks, enabling storage savings across similar files and versions.
Analyze storage efficiency, CPU/memory overhead, latency, and implementation complexity. For office documents, chunk-level dedup offers better storage savings but higher overhead.
Conclude that for typical office documents, a hybrid approach or chunk-level dedup with small chunk sizes may be optimal, balancing savings and performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked about sharding by user ID and caching hot manifests.
Start by clarifying the requirements and scale (e.g., number of files, users, QPS, read/write ratio, latency SLOs). Then propose a horizontally scalable, sharded metadata store with caching and asynchronous replication, and discuss trade-offs between consistency, availability, and performance.
Pro tip: Emphasize that metadata services are typically read-heavy, so focus on caching and read replicas first; also mention that chunk manifests and user file mappings may have different access patterns and could be stored separately.
Ask about expected data volume, request rate, read/write ratio, latency and consistency requirements, and growth projections.
Propose a schema for chunk manifests and user file mappings, and choose a distributed database (e.g., Cassandra, DynamoDB) that supports horizontal scaling and tunable consistency.
Explain sharding strategies (e.g., by user ID or file ID) to distribute load, and replication for fault tolerance and read scalability.
Introduce multi-level caching (client, CDN, in-memory) to reduce latency and database load, especially for hot metadata.
Discuss trade-offs between strong and eventual consistency, and how to handle failures, rebalancing, and monitoring.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.