Bloomberg·Software Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

Bloomberg system design follow-up, pretty deep into memory modeling territory for a next-token predictor. One question but it had a lot of moving parts and I don't think I covered all of them well.

Questions Asked (1)

Given a fixed memory budget, how many distinct words can a next-token predictor realistically store? Walk through the per-entry storage cost, how vocabulary size grows over time, and what strategies you'd use to keep memory bounded.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This one tripped me up more than I expected.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the memory budget and the definition of a 'word' (token). Then estimate per-entry storage cost by considering the data structure (e.g., hash table, trie) and the size of each entry (token string, frequency, pointers). Finally, discuss how vocabulary grows over time and propose strategies like pruning, quantization, or external storage to keep memory bounded.

Pro tip: Quantify with concrete numbers: e.g., assume 1 million words, average 10 bytes per word, 4 bytes for frequency, and 8 bytes for pointers, leading to ~22 MB; then scale to the given budget. This shows practical estimation skills.

1. Clarify assumptions and constraints

Ask about the memory budget, whether it's for a single model or distributed, and what 'distinct words' means (tokens vs. words). Confirm if we need to store only the vocabulary or also associated statistics.

2. Estimate per-entry storage cost

Break down the cost: token string (average length), frequency count, and any pointers or metadata. Consider the data structure overhead (e.g., hash table load factor, trie node overhead).

3. Calculate maximum vocabulary size

Divide the memory budget by the per-entry cost to get a rough upper bound. Adjust for overhead and fragmentation. Provide a range (e.g., 1M-10M words) based on different assumptions.

4. Discuss vocabulary growth over time

Explain that new words appear as the model encounters new data. Without bounds, vocabulary grows indefinitely, increasing memory usage and potentially degrading performance due to rare words.

5. Propose strategies to keep memory bounded

Suggest techniques like frequency-based pruning (keep top-K words), subword tokenization (BPE, WordPiece), quantization (e.g., 8-bit counts), external storage (disk-based hash maps), and dynamic eviction (LRU).

Key Points to Mention

Data structure choice: hash table vs. trie vs. sorted array, and their memory overheads.
Token representation: storing strings vs. hashes vs. IDs, and compression techniques.
Frequency pruning: keeping only the most common words and mapping rare words to an <UNK> token.
Subword tokenization: using BPE or WordPiece to handle rare words and reduce vocabulary size.
Quantization: reducing the precision of frequency counts or embeddings to save memory.
External memory: using disk-based storage or sharding across multiple machines.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.