NVIDIA·Data Scientist·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

System design round at NVIDIA for a Data Scientist role, focused entirely on designing a production-grade HTTP API for serving image model predictions. The question was sprawling and covered basically everything: auth, observability, versioning, security, and even an OpenAPI snippet. A lot to hold in your head at once.

Questions Asked (1)

Design an HTTP API to serve image-based model predictions, covering endpoint design, request/response schemas, auth, rate limiting, security, observability, backward compatibility, and provide an OpenAPI 3.0 snippet for one endpoint.

System DesignAPI & IntegrationsTechnical Trade-offs

Author's notes

This question is basically five questions stapled together.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements (e.g., expected QPS, image size, model type, latency SLA) and then walk through the design in a structured way: endpoints, schemas, auth, rate limiting, security, observability, and backward compatibility. Emphasize NVIDIA-specific considerations like GPU inference optimization, batching, and integration with Triton Inference Server. Conclude with an OpenAPI snippet for one endpoint to demonstrate concreteness.

Pro tip: Show awareness of GPU utilization and cost by discussing dynamic batching and model warm-up, and mention how you'd version the API to avoid breaking clients when models are updated.

1. Clarify Requirements and Constraints

Ask about expected traffic, image size, model type (e.g., classification, detection), latency SLA, and whether the API is internal or public. This shapes decisions on batching, rate limits, and auth.

2. Design Endpoints and Schemas

Define a RESTful endpoint like POST /v1/predict that accepts an image (multipart/form-data or base64) and returns predictions with confidence scores. Specify request/response JSON schemas and error formats.

3. Address Auth, Rate Limiting, and Security

Use API keys or OAuth2 for auth, implement rate limiting per key (e.g., token bucket), and secure the API with HTTPS, input validation, and size limits to prevent abuse.

4. Plan Observability and Backward Compatibility

Instrument with metrics (latency, QPS, error rates), logging, and tracing. Use versioned endpoints (e.g., /v1/) and additive changes to maintain backward compatibility.

5. Provide OpenAPI Snippet and Discuss Trade-offs

Write a concise OpenAPI 3.0 snippet for the predict endpoint, and discuss trade-offs like synchronous vs. asynchronous inference, batching latency vs. throughput, and cost implications.

Key Points to Mention

Use of NVIDIA Triton Inference Server for model serving with dynamic batching and GPU acceleration.
API versioning strategy (e.g., URL path versioning) and deprecation policy to ensure backward compatibility.
Rate limiting implementation (e.g., token bucket, sliding window) and how to communicate limits via headers (X-RateLimit-Limit, X-RateLimit-Remaining).
Security measures: HTTPS, API key rotation, input sanitization, and protection against adversarial images.
Observability: metrics (Prometheus), logging (structured logs), tracing (OpenTelemetry), and alerting on error rates and latency.
OpenAPI 3.0 snippet for POST /v1/predict with request body schema (image file) and response schema (predictions array).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.