← Microsoft Interview Insights

Microsoft·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior
Apr 2026

Summary

Microsoft ML Engineer interview that went deep on infrastructure design. One meaty system design question that took up most of the session, covering a lot of ground around checkpointing, reproducibility, and parallel training correctness.

Questions Asked (1)

Q1

Design a resumable DataLoader for ML training that supports mini-batch iteration over a shuffled dataset, checkpoint/restore of iteration state (including permutation, index, and RNG state), and deterministic replay after resuming. Also discuss correctness in multi-worker and multi-epoch scenarios.

System DesignTechnical Trade-offsAlgorithms & Data Structures
Author's notes

This one is trickier than it looks because saving 'current index' is not enough.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and assumptions, then propose a design that separates the permutation generation, index tracking, and RNG state management. Explain how to checkpoint and restore these components, and discuss multi-worker and multi-epoch correctness with deterministic replay.

Pro tip: Emphasize that determinism requires controlling all sources of randomness (e.g., shuffling, worker seeding) and that checkpointing must capture the exact state of the permutation and RNG, not just the current index. Mention that using a seeded RNG and storing the seed plus the number of random draws is a robust approach.

1. Clarify requirements and assumptions

Ask about dataset size, shuffling strategy, multi-worker setup, and whether exact determinism is required across different hardware. Confirm that checkpointing should be lightweight and portable.

2. Design core components

Propose a DataLoader that maintains a permutation array, a current index, and an RNG state. Use a seeded RNG to generate the permutation and to seed workers. For multi-worker, assign each worker a disjoint subset of indices.

3. Implement checkpoint/restore

Checkpoint the permutation (or the seed and epoch number), current index, and RNG state. On restore, recreate the permutation and RNG, and fast-forward the index. Ensure the checkpoint is serializable and versioned.

4. Ensure deterministic replay

Guarantee that given the same checkpoint, the sequence of batches is identical. This requires deterministic worker seeding, consistent data ordering, and no reliance on non-deterministic operations.

5. Address multi-worker and multi-epoch scenarios

For multi-worker, use a shared seed and worker-specific offsets to avoid overlap. For multi-epoch, reshuffle at each epoch using a deterministic seed derived from the base seed and epoch number. Discuss trade-offs between storing full permutation vs. regenerating from seed.

Key Points to Mention

  • Use a seeded RNG for shuffling and worker seeding to ensure reproducibility.
  • Checkpoint the permutation (or seed and epoch), current index, and RNG state.
  • For multi-worker, partition indices deterministically and avoid data duplication.
  • Reshuffle at each epoch using a deterministic seed derived from base seed and epoch.
  • Consider trade-offs: storing full permutation is memory-heavy but fast; regenerating from seed is lightweight but requires fast-forwarding RNG.
  • Ensure checkpoint compatibility across different numbers of workers by storing logical state, not worker-specific state.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.