← Atlassian Interview Insights
This is the kind of question that feels straightforward for about 90 seconds.
Start by clarifying requirements and scale (e.g., number of URLs, images, depth, politeness). Then design a distributed, fault-tolerant architecture with a URL frontier, fetcher, parser, and storage, addressing key challenges like deduplication, rate limiting, and prioritization. Finally, discuss trade-offs and optimizations for unlimited scale and depth.
Pro tip: Emphasize politeness and legal considerations (robots.txt, rate limiting) early, as this shows production awareness and often impresses interviewers at companies like Atlassian.
Ask questions to understand scale (e.g., number of pages, images, depth), politeness constraints, freshness, and storage needs. Define functional and non-functional requirements.
Outline components: URL frontier (priority queue), fetcher (distributed workers), parser (extract links and images), deduplication (Bloom filter or hash), storage (blob store for images, metadata DB), and scheduler.
Detail the URL frontier (prioritization, politeness), deduplication (exact and near-duplicate), distributed fetching (load balancing, fault tolerance), and image storage (CDN, compression).
Explain how to handle unlimited scale (horizontal scaling, sharding, async I/O) and depth (BFS with depth tracking, avoiding infinite loops). Discuss back-pressure and monitoring.
Cover trade-offs: consistency vs. availability, push vs. pull, batch vs. stream processing. Mention optimizations like caching, compression, and using existing frameworks (e.g., Scrapy, Heritrix).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Went with a bloom filter for fast in-memory checks plus a backing store for confirmed visited URLs.
Start by clarifying the scale and requirements (e.g., number of URLs, freshness needs), then propose a layered deduplication strategy using a combination of in-memory sets, Bloom filters, and persistent storage. Discuss trade-offs between accuracy, memory usage, and performance, and mention how you would handle updates and distributed crawling.
Pro tip: Mention that deduplication should happen at multiple stages: before fetching (URL normalization and Bloom filter), during fetching (checking a distributed cache), and after fetching (content hashing to avoid duplicate content). This shows a holistic approach and awareness of real-world complexities.
Ask about scale (number of URLs, crawl rate), freshness (how often to re-crawl), and whether exact or approximate deduplication is acceptable. This sets the context for your solution.
Explain that URLs must be canonicalized (e.g., lowercasing scheme/host, removing fragments, sorting query parameters) to treat equivalent URLs as the same.
Propose an in-memory set for recently seen URLs, a Bloom filter for probabilistic membership, and a persistent database (e.g., Redis, Cassandra) for exact storage. Discuss trade-offs between memory and accuracy.
Describe how to shard the URL space across crawlers (e.g., consistent hashing) and use atomic operations to avoid race conditions when checking/adding URLs.
Mention that even with URL deduplication, different URLs may serve identical content; use content hashing (e.g., SimHash) to detect and skip duplicates.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Per-host rate limiting is one of those things I knew conceptually but hadn't thought through the mechanics of.
Start by clarifying requirements and scale, then present a high-level architecture of the queue and scheduler, emphasizing decoupling and fault tolerance. Dive into per-host rate limiting using a token bucket or leaky bucket algorithm, and discuss trade-offs like distributed coordination and backpressure. Conclude with how you'd monitor and adapt the system.
Pro tip: Show awareness of real-world constraints by mentioning how you'd handle rate limit state in a distributed system (e.g., using Redis with atomic operations) and the importance of graceful degradation when hosts are unresponsive.
Ask about expected throughput, number of hosts, latency requirements, and whether the system is distributed. This ensures your design meets actual needs.
Describe the queue (e.g., Kafka, RabbitMQ, or custom) and scheduler components, explaining how tasks are enqueued, prioritized, and dispatched to workers.
Explain the algorithm (e.g., token bucket) and how you'd track state per host, including distributed coordination if needed.
Discuss trade-offs between accuracy and performance, and how to handle failures like host downtime or rate limit state loss.
Mention metrics to track (e.g., queue depth, rate limit hits) and how the system could adapt dynamically to changing conditions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked through two main tables: one for URL frontier/crawl state (URL, status, last crawled, retry count, source URL) and one for image records (image URL, content hash, storage path, discovered from, timestamp).
Start by clarifying the requirements: what kind of crawler, scale, and access patterns. Then propose a normalized schema with separate tables for crawl state (e.g., URLs, status, timestamps) and image metadata (e.g., URL, dimensions, format, storage location), and discuss indexing and partitioning for performance.
Pro tip: Mention that crawl state often requires frequent updates and may benefit from a NoSQL store or a write-optimized relational design, while image metadata is more read-heavy and can be denormalized for query speed. Also, consider using a job queue or state machine for crawl state to handle retries and failures.
Ask about scale (number of URLs, images), access patterns (e.g., query by domain, status, image type), and consistency needs. This shows you don't jump to solutions.
Propose a table with columns like url, status (queued, fetching, done, failed), last_crawled_at, next_crawl_at, retry_count, and error_message. Discuss indexing on status and next_crawl_at for efficient scheduling.
Propose a table with columns like image_url, page_url, format, width, height, size_bytes, storage_path, and hash. Consider indexing on page_url and hash for deduplication.
Explain how the tables relate (e.g., foreign key from image to crawl state) and discuss partitioning (e.g., by domain or date) and potential use of NoSQL for crawl state if write-heavy.
Mention trade-offs between SQL and NoSQL, normalization vs. denormalization, and how you might evolve the schema as scale grows.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Said I'd use exponential backoff with a max retry cap, and move URLs to a dead-letter queue after hitting the limit.
Start by clarifying the system's requirements and failure modes, then propose a layered retry strategy with backoff and jitter, and finally outline monitoring metrics and alerts. Emphasize idempotency and graceful degradation to show you understand distributed systems trade-offs.
Pro tip: Tie your answer to Atlassian's scale and reliability needs by mentioning how you'd use tools like exponential backoff with jitter to avoid thundering herds, and how you'd monitor retry rates and error budgets to balance resilience with user experience.
Ask about the system's criticality, expected load, and types of failures (transient vs. permanent). Identify which operations are idempotent and which are not.
Propose a retry policy with exponential backoff and jitter, limited retries, and circuit breakers. Discuss when to retry (e.g., only on transient errors) and how to handle non-idempotent operations.
List key metrics: retry count, success/failure rates, latency, error types, and circuit breaker state. Explain how to set thresholds and alerts based on SLOs.
Describe how you'd use logs, traces, and dashboards to diagnose recurring failures. Mention the importance of post-mortems and iterating on the retry policy.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.