This question is basically a full distributed systems exam in one prompt.
Start by clarifying requirements (scale, latency, accuracy, top-K size) and then propose a streaming architecture with partitioned ingestion, sharded token counting, and hierarchical aggregation. Walk through the data flow end-to-end, explicitly addressing fault tolerance, idempotency, and backpressure at each stage.
Pro tip: Emphasize that exact global counts require a shuffle/merge step, but approximate top-K with bounded error (e.g., Count-Min Sketch or Space-Saving) can drastically reduce network and memory costs—discuss the trade-off and when each is appropriate.
Ask about data size, ingestion rate, latency SLA, accuracy (exact vs approximate), top-K size, and failure tolerance. This shapes the entire design.
Describe how text is ingested (e.g., Kafka, distributed file system) and partitioned by token hash to ensure all occurrences of a token go to the same shard. Mention consistent hashing for scalability.
Each shard maintains a local hash map of token counts, periodically flushing partial counts to a downstream aggregation layer. Use combiner-like logic to reduce data movement.
Merge partial counts from all shards, either exactly (via shuffle) or approximately (using sketches). Then compute top-K using a heap or threshold-based approach.
Explain checkpointing, exactly-once semantics via idempotent writes, backpressure handling, and how results are stored (e.g., in a KV store) and served with low latency.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.