NVIDIA·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

System design round at NVIDIA for a software engineering role. The whole thing was one big fraud detection problem and they wanted the full picture: architecture, ML serving, scaling, the works. Dense question, not a lot of room to wing it.

Questions Asked (1)

Design a real-time fraud detection system for a payments company that handles millions of transactions per day. The system needs to approve, flag, or block each transaction within 50ms, sustain over 10k requests per second with headroom for traffic spikes, and support ML model updates without any service downtime. Walk through the full architecture including data flow, model serving, scaling, and how you'd operate it in production.

System DesignTechnical Trade-offsA/B Testing & Experimentation

Author's notes

This is a lot to cover in one question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then present a high-level architecture that separates the low-latency decision path from the model training and management path. Dive into each component—data ingestion, feature computation, model serving, decision engine, and feedback loop—highlighting how you achieve 50ms latency, 10k+ RPS, and zero-downtime model updates. Emphasize trade-offs and operational considerations like monitoring, A/B testing, and rollback strategies.

Pro tip: Quantify the latency budget: e.g., 10ms for feature retrieval, 20ms for model inference, 10ms for decision logic, and 10ms buffer. This shows you understand the tight constraints and can design accordingly.

1. Clarify Requirements and Constraints

Ask questions to confirm transaction volume, latency SLA, accuracy vs. false positive trade-offs, regulatory requirements, and existing infrastructure. Establish the need for real-time decisions and model freshness.

2. High-Level Architecture

Outline the main components: data ingestion (Kafka), stream processing (Flink), feature store (Redis), model serving (Triton), decision engine, and feedback loop. Emphasize separation of online and offline paths.

3. Deep Dive into Critical Path

Detail how a transaction flows: ingestion, feature enrichment, model inference, decision logic, and response. Explain how each step meets the 50ms latency budget and scales to 10k+ RPS.

4. Model Serving and Updates

Describe how to serve ML models with low latency using NVIDIA Triton, and how to update models without downtime via canary deployments, shadow mode, and dynamic model loading.

5. Scaling and Operations

Discuss horizontal scaling, load balancing, auto-scaling, monitoring, alerting, and A/B testing. Cover failure modes, rollback, and continuous training pipelines.

Key Points to Mention

Use of NVIDIA Triton Inference Server for high-performance, multi-framework model serving with dynamic batching and concurrent model execution.
Feature store (e.g., Redis) for low-latency feature retrieval, with both online and offline stores to ensure consistency.
Stream processing with Apache Flink or Kafka Streams for real-time feature computation and complex event processing.
Zero-downtime model updates via canary deployments, shadow mode, and model versioning, leveraging Triton's model repository and dynamic loading.
Latency budget breakdown and techniques to meet 50ms: in-memory feature stores, model quantization, optimized inference, and edge caching.
Operational excellence: monitoring (Prometheus, Grafana), logging, tracing, A/B testing framework, and automated rollback on performance degradation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.