Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Technical screen for a Data Scientist role at Meta, focused entirely on a deep open-ended problem around autonomous driving simulation realism. One question, but it sprawled into five or six sub-problems and easily ate the full hour.

Questions Asked (1)

You have real-world driving logs and simulator outputs for the same set of scenarios. Design an end-to-end approach to evaluate whether the simulator is realistic enough to use for vehicle performance evaluation. Cover what 'realistic' means operationally, which metrics you'd compute across trajectory, interaction, and safety levels, how you'd structure the real-vs-sim comparison, how you'd validate that simulator improvements actually translate to better real-world correlation, and what failure modes you'd watch for.

System DesignProduct Analytics & MetricsA/B Testing & Experimentation

Author's notes

This question is enormous and I think I underestimated that at first.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining 'realistic' operationally as the simulator's ability to reproduce real-world distributions and causal relationships relevant to vehicle performance. Then propose a multi-level evaluation framework comparing real and simulated data across trajectory, interaction, and safety metrics, using statistical tests and correlation analysis. Finally, outline a validation loop where simulator improvements are tested against held-out real data, and discuss failure modes like overfitting to scenarios or ignoring rare events.

Pro tip: Emphasize that realism is not about matching every detail but about preserving the decision-relevant signals for the specific performance evaluation task—this shows you understand the business context and avoid over-engineering.

1. Define Operational Realism

Specify what 'realistic' means for the intended use case: e.g., matching distributions of key performance indicators, preserving interaction dynamics, and reproducing safety-critical events. Align with stakeholders on acceptable tolerances.

2. Select Metrics Across Levels

Choose metrics at trajectory (e.g., position/velocity errors, jerk), interaction (e.g., time-to-collision, gap acceptance), and safety (e.g., collision rate, near-miss frequency) levels. Ensure metrics are computable in both real and simulated data.

3. Structure Real-vs-Sim Comparison

Use paired scenario analysis: for each scenario, compare distributions of metrics using statistical tests (e.g., KS test, Wasserstein distance) and visualize differences. Aggregate across scenarios to assess overall fidelity.

4. Validate Simulator Improvements

When the simulator is updated, re-evaluate on a held-out set of real scenarios. Measure whether improvements in sim realism correlate with better alignment to real-world outcomes, using metrics like correlation of performance rankings.

5. Identify and Monitor Failure Modes

Watch for overfitting to common scenarios, ignoring rare but critical events, mismatched causal relationships, and simulator artifacts that inflate performance. Continuously monitor for distribution shifts.

Key Points to Mention

Operational definition of realism tied to specific performance evaluation goals
Multi-level metrics: trajectory (e.g., RMSE, DTW), interaction (e.g., TTC, headway), safety (e.g., collision rate, near-miss)
Paired scenario comparison using statistical distances and hypothesis testing
Validation of simulator improvements via held-out real data and correlation of rankings
Failure modes: overfitting, rare event neglect, causal mismatches, simulator artifacts
Importance of stakeholder alignment on acceptable realism thresholds

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.