Meta·Machine Learning Engineer·Onsite - System Design / Architecture·Staff

StaffPrefer not to say

May 2026

Summary

Meta ML Engineer system design round, one massive question about building an async RL post-training pipeline on top of a live serving system. The scope was brutal and I don't think I got close to covering everything they wanted.

Questions Asked (1)

Design an asynchronous RL-based post-training system (like RLHF or RLAIF) for a chat LLM that is already serving live traffic. Cover the major components such as generators, reward inference, learners, replay buffers, and orchestrators, how data flows between them, batching strategies, off-policy corrections like importance sampling or V-trace, and KL divergence control relative to the base model. Also address safety guardrails, versioning and rollout strategy, online feedback ingestion, credit assignment under delayed or sparse rewards, reward hacking prevention, separation of serving and training clusters, stability monitoring, and cost predictability under async feedback and load spikes.

System DesignTechnical Trade-offsA/B Testing & Experimentation

Author's notes

This was essentially a dissertation prompt compressed into 45 minutes.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining the high-level architecture, emphasizing the separation of serving and training clusters to avoid interference. Then dive into each component (generators, reward inference, learners, replay buffers, orchestrators) and explain data flow, batching, and off-policy corrections. Finally, address safety, versioning, monitoring, and cost predictability, highlighting trade-offs and practical considerations.

Pro tip: Emphasize the importance of a robust orchestrator that manages asynchronous communication and handles failures gracefully, as this is often the most challenging part in production systems. Also, discuss how you would measure and mitigate reward hacking through techniques like reward model ensembles and adversarial training.

1. High-Level Architecture

Describe the separation of serving and training clusters, with the serving cluster handling live traffic and the training cluster running the RL loop. Explain how they interact via a shared model registry and data pipeline.

2. Core Components and Data Flow

Detail the generators (policy model for rollouts), reward inference (reward model scoring), learners (policy updates), replay buffers (storing experiences), and orchestrator (coordinating async tasks). Explain how data flows from serving to training and back.

3. Batching and Off-Policy Corrections

Discuss batching strategies for efficient training, such as prioritized experience replay. Explain off-policy corrections like importance sampling and V-trace to handle stale data from asynchronous generation.

4. KL Control and Stability

Describe how to control KL divergence relative to the base model to prevent drift, using techniques like adaptive KL penalties. Mention stability monitoring with metrics like reward variance and policy entropy.

5. Safety, Versioning, and Cost

Cover safety guardrails (e.g., reward model constraints, output filters), versioning and rollout strategies (canary deployments, A/B tests), online feedback ingestion, credit assignment for delayed rewards, reward hacking prevention, and cost predictability under load spikes.

Key Points to Mention

Asynchronous generation and training with off-policy corrections (importance sampling, V-trace) to handle stale data.
KL divergence control using adaptive penalties or clipping to keep the policy close to the base model.
Replay buffer design with prioritization and staleness handling, and batching strategies for efficient learning.
Safety guardrails including reward model ensembles, output filtering, and human oversight.
Versioning and rollout strategy with canary deployments and A/B testing for safe updates.
Cost predictability via autoscaling, spot instances, and monitoring of resource usage under load spikes.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.