Zillow·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

Zillow ML engineer interview with a meaty system design question about RAG pipelines. Just the one question but it went pretty deep into chunking strategy, which I hadn't fully prepped for.

Questions Asked (1)

You're building a RAG system with an LLM and a vector database. Walk through how you'd design the chunking strategy for documents before creating embeddings, including chunk size, overlap, use of document structure, advanced methods like semantic chunking, and how you'd evaluate and iterate on it.

System DesignTechnical Trade-offsAdaptability & Ambiguity

Author's notes

This one sprawled in a way I wasn't ready for.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the use case and document types, then propose a baseline chunking strategy (e.g., fixed-size with overlap) and explain how you would iterate using evaluation metrics. Discuss advanced methods like semantic chunking and document structure as potential improvements, but emphasize the importance of empirical validation.

Pro tip: Always tie chunking decisions to retrieval and generation metrics—chunk size and overlap are not just preprocessing choices, they directly impact answer quality. Mention that you'd start simple and only add complexity if metrics justify it.

1. Clarify requirements and constraints

Ask about document types (e.g., property listings, legal contracts), query patterns, and latency/accuracy trade-offs to tailor the chunking strategy.

2. Establish a baseline chunking strategy

Propose fixed-size chunks (e.g., 256-512 tokens) with 10-20% overlap, and explain how this balances context and retrieval precision.

3. Leverage document structure

Use headings, sections, and paragraphs to create semantically coherent chunks, avoiding splitting mid-sentence or mid-table.

4. Consider advanced methods

Discuss semantic chunking (e.g., using embeddings to detect topic shifts) and hierarchical chunking for multi-level retrieval.

5. Evaluate and iterate

Define metrics (e.g., retrieval recall, answer faithfulness) and run A/B tests to compare chunking strategies, iterating based on results.

Key Points to Mention

Trade-offs between chunk size and retrieval precision/recall
Importance of overlap to preserve context across boundaries
Using document structure (headings, lists, tables) to guide chunking
Semantic chunking via embedding similarity or topic modeling
Evaluation metrics: retrieval hit rate, MRR, answer correctness, latency
Iterative approach: start simple, measure, then refine

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.