← Atlassian Interview Insights
I started with the obvious stuff, a queue of URLs, workers pulling from it, storing images somewhere.
Start by clarifying requirements such as scale, depth limits, politeness, and storage needs. Then propose a high-level architecture with a crawler, parser, downloader, and storage components, and dive into key design decisions like URL frontier management, deduplication, and trade-offs between depth and resource usage.
Pro tip: Emphasize politeness and legal considerations (robots.txt, rate limiting) early, as this shows production awareness beyond just technical design. Also, discuss how you would handle dynamic content and JavaScript-rendered images, which is often overlooked.
Ask about scale (number of URLs, depth, image volume), politeness constraints, storage requirements, and whether dynamic content needs handling. This ensures the design meets actual needs.
Outline main components: URL frontier (queue), fetcher (HTTP client), parser (HTML parser), image downloader, and storage (file system or object store). Mention distributed crawling if scale demands.
Discuss URL deduplication (Bloom filter or hash set), depth control (BFS with depth tracking), politeness (rate limiting per domain, robots.txt), and handling failures/retries.
Design how to store images (e.g., S3 with metadata in DB) and crawled URLs (e.g., NoSQL for visited URLs). Consider deduplication of images via hashing.
Discuss trade-offs: depth vs. resource usage, breadth-first vs. depth-first, synchronous vs. asynchronous downloading, and scaling horizontally with distributed queues.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the requirements and scale, then propose a normalized schema with separate tables for URLs, fetch state, and image metadata, using foreign keys to represent parent-child relationships. Explain how you would handle crawl depth and ensure efficient querying for state transitions and hierarchy traversal.
Pro tip: Mention that you would use an enum for fetch state and consider adding a 'next_fetch_at' timestamp for scheduling retries, showing you think about operational concerns beyond just data storage.
Ask about expected volume, read/write patterns, and whether the system needs to support concurrent crawlers. This ensures your design meets actual needs.
Create a 'urls' table with columns for id, url (unique), fetch_state (enum), crawl_depth, parent_id (self-referencing FK), and timestamps. This centralizes URL metadata.
Create an 'images' table with a foreign key to the URL it was found on, storing metadata like src, alt, dimensions, and format. This keeps image data separate and scalable.
Add indexes on fetch_state, parent_id, and crawl_depth to support efficient state transitions and hierarchy traversal. Consider partitioning if volume is high.
Mention alternatives like using a graph database for complex relationships or adding a separate table for fetch history. Explain why a relational model is suitable here.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Exponential backoff came out of my mouth pretty fast.
Start by clarifying the crawl pipeline's requirements and failure modes, then propose a layered resilience strategy: retries with backoff, a dead-letter queue for poison messages, and checkpointing for resumability. Emphasize idempotency and observability to ensure safe recovery and debugging.
Pro tip: Atlassian values pragmatic trade-offs: explicitly discuss when to retry versus fail fast, and how you'd monitor DLQ depth and alert on it. Also mention that you'd design for idempotency from the start to avoid duplicate work during retries or resumption.
Ask about the pipeline's scale, latency tolerance, and types of failures (transient vs. permanent). Identify what 'partial failure' means in this context and the expected recovery time.
Propose retries with exponential backoff and jitter for transient errors, with a maximum retry limit. Ensure operations are idempotent to avoid duplicate side effects.
After max retries, route failed messages to a DLQ for later analysis and manual reprocessing. Include metadata like error reason and timestamp for debugging.
Use durable checkpoints (e.g., after each batch or page) to track progress. On restart, resume from the last checkpoint, skipping already-processed items.
Track retry counts, DLQ size, and checkpoint lag. Alert on anomalies and provide dashboards for visibility into pipeline health.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Robots.txt I handled by caching the parsed rules per domain with a TTL.
Start by clarifying requirements and constraints, then propose a distributed architecture with a central coordination service (e.g., Redis) for per-domain rate limits and a shared robots.txt cache. Discuss trade-offs between consistency, latency, and fault tolerance, and explain how workers enforce limits and respect robots.txt before fetching.
Pro tip: Emphasize that rate limiting should be per-domain, not per-worker, and that robots.txt caching must handle updates and failures gracefully. Mention using a token bucket algorithm with Redis and a TTL-based cache for robots.txt to balance freshness and performance.
Ask about scale (number of domains, workers, request rate), latency tolerance, consistency needs, and whether robots.txt changes must be detected quickly. This shows you think before designing.
Propose a central store (e.g., Redis) with atomic operations to implement per-domain token buckets or sliding windows. Workers check and decrement tokens before fetching, ensuring global limits across all workers.
Use a shared cache (e.g., Redis or a dedicated service) to store parsed robots.txt rules per domain with a TTL. Workers fetch and parse robots.txt on cache miss, and respect rules before crawling.
Discuss fallbacks if the central store is unavailable (e.g., local rate limiting with conservative limits) and how to handle stale robots.txt (e.g., serve stale on error, revalidate in background).
Compare centralized vs. decentralized approaches, mention sharding by domain for scalability, and consider using a dedicated service for rate limiting and robots.txt to reduce worker complexity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.