Xometry·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

Xometry ML Engineer interview, system design round focused on productionizing a model under strict latency constraints. Pretty thorough question that covers a lot of ground at once, from profiling to hardware to post-launch monitoring.

Questions Asked (1)

You're handed a trained model and need to deploy it as a real-time inference service with a 200ms latency budget per prediction. Walk through how you'd clarify requirements, measure baseline performance, optimize the model and serving stack, pick hardware, validate accuracy vs. latency tradeoffs, and set up post-launch monitoring.

System DesignTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This one is a full gauntlet disguised as a single question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements and constraints, then establish a baseline latency measurement to identify bottlenecks. Systematically optimize the model, serving stack, and hardware while validating accuracy-latency tradeoffs, and finally set up comprehensive monitoring for post-launch.

Pro tip: Emphasize the importance of end-to-end latency measurement and the use of percentiles (e.g., p95, p99) rather than averages, as real-time systems often have strict tail latency requirements. Also, consider model quantization and hardware acceleration early in the optimization process.

1. Clarify Requirements and Constraints

Ask questions to understand the expected request rate, latency distribution (p50, p95, p99), accuracy requirements, and infrastructure constraints. Determine if the 200ms budget is end-to-end or just model inference.

2. Measure Baseline Performance

Profile the model's inference latency on representative hardware, breaking down time spent in preprocessing, model execution, and postprocessing. Identify bottlenecks and establish a performance baseline.

3. Optimize Model and Serving Stack

Apply model optimizations (quantization, pruning, distillation) and serving optimizations (batching, caching, optimized runtimes like ONNX Runtime or TensorRT). Consider hardware acceleration (GPU, TPU, or specialized inference chips).

4. Validate Accuracy-Latency Tradeoffs

Evaluate the impact of optimizations on model accuracy using a validation set. Ensure that latency improvements do not unacceptably degrade accuracy, and iterate to find the best tradeoff.

5. Set Up Monitoring and Iterate

Deploy with monitoring for latency, throughput, error rates, and accuracy drift. Set up alerts and a feedback loop to continuously optimize and address regressions.

Key Points to Mention

End-to-end latency measurement and percentile analysis (p95, p99)
Model optimization techniques: quantization, pruning, knowledge distillation
Serving stack optimizations: batching, caching, optimized runtimes (ONNX, TensorRT)
Hardware selection: CPU vs GPU vs specialized accelerators, considering cost and scalability
Accuracy-latency tradeoff validation with A/B testing or shadow deployment
Monitoring: latency, throughput, error rates, data drift, and model performance metrics

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.