NVIDIA·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Apr 2026

Summary

System design round at NVIDIA for a software engineering role. The question was deep and very GPU-infrastructure-specific, which makes sense given the company but still caught me off guard with how many layers they wanted covered.

Questions Asked (1)

Design a Docker-based CI environment for running automated graphics tests on machines with NVIDIA or AMD GPUs. Cover base image choices, driver and runtime management, image layering and caching, reproducibility, security constraints, debugging inside containers, flaky test handling, and strategies to measure and reduce overall CI runtime.

System DesignTechnical Trade-offsAPI & Integrations

Author's notes

This was a beast of a question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the requirements: which GPU vendors, test types, and CI platform. Then propose a layered architecture: base images with vendor-specific drivers, runtime management via NVIDIA Container Toolkit or AMD ROCm, and a caching strategy for layers. Finally, address reproducibility, security, debugging, flakiness, and runtime optimization with concrete trade-offs.

Pro tip: Emphasize that driver version must match the host, so use a base image with the same driver version as the CI nodes, and avoid bundling drivers in the image. Also, mention using multi-stage builds to keep the final image slim and secure.

1. Clarify Requirements and Constraints

Ask about GPU vendors (NVIDIA/AMD), test types (rendering, compute), CI platform (Jenkins, GitLab, etc.), and security policies. This shapes driver and runtime choices.

2. Design Base Image and Driver Strategy

Choose a minimal base (e.g., Ubuntu) and install only necessary libraries. For NVIDIA, use nvidia/cuda base images; for AMD, use rocm/dev-ubuntu. Drivers should be mounted from host, not baked in.

3. Implement Runtime and Orchestration

Use NVIDIA Container Toolkit (nvidia-docker) or AMD's Docker runtime to expose GPUs. Configure CI jobs to request GPU resources and set environment variables for driver paths.

4. Optimize Layering and Caching

Order Dockerfile instructions from least to most frequently changing. Use BuildKit cache mounts for package managers and test dependencies. Leverage CI cache for Docker layers.

5. Address Reproducibility, Security, and Debugging

Pin versions of base images and dependencies. Run containers as non-root, drop capabilities, and use read-only filesystems. Provide debug tools (e.g., nvidia-smi, rocm-smi) and logging for failures.

6. Handle Flaky Tests and Measure Runtime

Implement retries with backoff for flaky tests, but track flakiness metrics. Profile test stages to identify bottlenecks; parallelize tests across GPUs and use smaller test subsets for quick feedback.

Key Points to Mention

Driver compatibility: host driver version must match container's expected version; use NVIDIA Container Toolkit or AMD ROCm to mount drivers.
Base image selection: use official vendor images (nvidia/cuda, rocm/dev-ubuntu) or minimal OS with runtime libraries; multi-stage builds to reduce size.
Layer caching: order Dockerfile commands to maximize cache hits; use BuildKit cache mounts for apt/yum and pip.
Reproducibility: pin exact versions of base images, drivers, and dependencies; use immutable tags or digests.
Security: run as non-root, drop capabilities, use read-only root filesystem, and avoid privileged mode unless necessary.
Debugging: include diagnostic tools (nvidia-smi, rocm-smi, dmesg) and enable core dumps; use docker exec for interactive debugging.
Flaky tests: implement retries with exponential backoff, quarantine flaky tests, and monitor flakiness rates.
Runtime optimization: parallelize tests across multiple GPUs, use test sharding, and cache test artifacts between runs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.