Glean·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

May 2026Remote

Summary

Coding round for an MLE role at Glean. The problem was building a rate-limited Wikipedia crawler with some prioritization logic baked in. AI coding tools were allowed, which was a nice change, though you still had to own and explain everything you wrote.

Questions Asked (1)

Design and implement a rate-limited Wikipedia crawler that starts from a given page, extracts outgoing links, and prioritizes crawling pages whose starting letter hasn't been visited yet. Once all 26 letters are covered, fall back to random selection from the frontier. Handle deduplication, configurable rate limits, and common failure cases.

System DesignAlgorithms & Data StructuresTechnical Trade-offs

Author's notes

The alphabetical prioritization was the part I kept second-guessing myself on.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then outline a high-level architecture with a frontier, deduplication, rate limiter, and prioritization logic. Dive into the letter-based prioritization algorithm and discuss trade-offs, failure handling, and scalability considerations.

Pro tip: Emphasize the importance of politeness and legal compliance when crawling Wikipedia, and suggest using a token bucket rate limiter with configurable rates per domain. Also, mention that the letter prioritization can be implemented with a priority queue where each URL's priority is based on whether its starting letter is unvisited.

1. Clarify Requirements and Constraints

Ask about scale (number of pages, rate limits), politeness policies, and whether the crawler should be distributed. Confirm that the goal is to cover all 26 letters as quickly as possible.

2. Design High-Level Architecture

Outline components: URL frontier (priority queue), deduplication (Bloom filter or set), rate limiter (token bucket), HTML parser, and link extractor. Discuss single-machine vs distributed.

3. Detail Prioritization and Deduplication

Explain how to track visited letters and assign priority: URLs with unvisited starting letters get higher priority. Use a set or Bloom filter for deduplication, and handle collisions.

4. Implement Rate Limiting and Failure Handling

Describe a token bucket rate limiter with configurable rate. Handle HTTP errors (retries with backoff), timeouts, and malformed pages. Ensure the crawler is polite and respects robots.txt.

5. Discuss Trade-offs and Scalability

Compare in-memory vs disk-based frontier, exact vs approximate deduplication, and single vs distributed crawling. Mention monitoring and metrics.

Key Points to Mention

Use a priority queue for the frontier, with priority based on whether the starting letter is unvisited.
Deduplication using a set or Bloom filter to avoid revisiting pages.
Token bucket algorithm for configurable rate limiting, with per-domain limits.
Handling common failures: HTTP errors, timeouts, and retries with exponential backoff.
Respecting robots.txt and Wikipedia's crawling policies.
Trade-offs between memory usage and accuracy in deduplication, and between single-machine and distributed crawling.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.