Goldman Sachs·Software Engineer·Onsite - System Design / Architecture·Senior

Senior

Jul 2026

Summary

Goldman Sachs SWE system design round, one question about building a web scraping engine. Pretty open-ended, which I wasn't fully prepared for.

Questions Asked (1)

Design a web scraping engine.

System DesignTechnical Trade-offsAPI & Integrations

Author's notes

I jumped straight into the crawler logic and spent way too long on the fetching layer before anyone asked about rate limiting or robots.txt compliance.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements: scale, target sites, data freshness, and legal constraints. Then propose a modular architecture with components for scheduling, fetching, parsing, storage, and monitoring, and discuss trade-offs like politeness vs. throughput and static vs. dynamic rendering.

Pro tip: Emphasize robustness and ethics: implement rate limiting, respect robots.txt, and design for failure with retries and dead-letter queues. This shows you consider real-world constraints and legal compliance, which is crucial in finance.

1. Clarify Requirements

Ask about scale (pages per day), target sites (static/dynamic), data freshness, and legal/ethical constraints. This scopes the problem and shows you avoid assumptions.

2. High-Level Architecture

Outline main components: scheduler, fetcher (with proxy rotation), parser, storage, and monitoring. Explain how they interact and scale independently.

3. Deep Dive into Key Components

Discuss fetching (HTTP clients, headless browsers), parsing (CSS/XPath, ML), and storage (databases, data lakes). Mention handling dynamic content and anti-scraping measures.

4. Scalability and Reliability

Explain horizontal scaling, distributed queues, rate limiting, retries, and circuit breakers. Address how to handle failures and ensure data consistency.

5. Trade-offs and Alternatives

Compare push vs. pull, batch vs. stream, and custom vs. third-party solutions. Highlight trade-offs like cost, complexity, and freshness.

Key Points to Mention

Respect robots.txt and terms of service; implement rate limiting and backoff to avoid bans.
Use distributed message queues (e.g., Kafka, RabbitMQ) for decoupling and scalability.
Handle dynamic content with headless browsers (Puppeteer, Selenium) or APIs when available.
Design for idempotency and deduplication to avoid reprocessing.
Monitor with metrics (success rate, latency) and alerting; log errors for debugging.
Consider data storage options: relational for structured, NoSQL for flexible, and data lakes for raw storage.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.