This one sprawled in a way I didn't expect.
Start by clarifying requirements (file size, memory limits, definition of 'word', top-K size) and then propose a streaming approach: read the file in chunks, tokenize and normalize each chunk, and update a frequency map. For files larger than RAM, use external sorting or a distributed/partitioned approach (e.g., hash partitioning) to count frequencies, then use a min-heap or selection algorithm to find the top-K. Discuss time and space complexity for each phase.
Pro tip: Mention that you'd handle Unicode normalization and case folding early to avoid inconsistencies, and that you'd consider using a trie or a count-min sketch for approximate counting if exact counts aren't required, showing awareness of trade-offs.
Ask about file size, available memory, definition of a word (e.g., alphanumeric sequences), case sensitivity, and whether top-K needs to be exact. This ensures the solution fits the context.
Define a tokenizer that splits on non-alphanumeric characters, handles Unicode, and normalizes tokens (lowercasing, stemming/lemmatization if needed). Discuss trade-offs between simple regex and more complex NLP tokenizers.
For files larger than RAM, process the file in chunks, build a frequency map per chunk, and periodically spill to disk. Alternatively, use external sorting: sort chunks of (word, 1) pairs, then merge and count. For distributed systems, hash-partition words across machines.
Use a min-heap of size K while streaming counts, or after counting, use a selection algorithm (quickselect) or sort the frequency map. For distributed counts, merge top-K from each partition.
Time: O(N) for tokenization and counting, plus O(N log K) for heap or O(N) average for quickselect. Space: O(U) for unique words, which may exceed RAM; external sorting uses O(N/B) disk space. Discuss trade-offs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.