← NVIDIA Interview Insights

NVIDIA·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

System design round at NVIDIA for a software engineer role. The whole thing basically came down to one deep question about Kubernetes internals and I was not as prepared as I thought I was.

Questions Asked (1)

Q1

Walk me through what Kubernetes Operators and Custom Resource Definitions are, how they work together, and what separates a well-built Operator from a mediocre one.

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

This question is way bigger than it sounds.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining CRDs as the schema for extending the Kubernetes API and Operators as controllers that watch those CRs and reconcile desired state. Explain the reconciliation loop and how the two work together, then contrast well-built Operators (idempotent, level-based, observable, upgrade-safe) with mediocre ones (imperative, status-blind, fragile).

Pro tip: Emphasize that a great Operator is a 'level-based' controller that continuously reconciles actual state to desired state, not an event-driven script—this shows you understand Kubernetes' core design philosophy and avoids common pitfalls.

1. Define CRDs

Explain that Custom Resource Definitions extend the Kubernetes API with new resource types, allowing you to store and manage domain-specific objects (e.g., NVIDIA GPU policies) as first-class citizens.

2. Define Operators

Describe Operators as controllers that encode operational knowledge: they watch custom resources and other resources, then take actions to make the cluster's actual state match the desired state.

3. Explain the reconciliation loop

Walk through how the Operator watches CRs, receives events, and runs a reconcile function that is idempotent and level-based, handling both creation and updates without relying on event order.

4. Contrast well-built vs. mediocre

Highlight qualities of a well-built Operator: idempotent, level-based, handles upgrades and failures gracefully, exposes status and metrics, uses finalizers for cleanup, and follows least privilege. Mediocre ones are imperative, ignore status, lack observability, and break on upgrades.

5. Tie to NVIDIA context

Connect to NVIDIA's domain by mentioning Operators for GPU management (e.g., NVIDIA GPU Operator) that automate driver installation, device plugin deployment, and monitoring, showing relevance to the role.

Key Points to Mention

  • CRDs extend the Kubernetes API and are versioned, with schema validation and conversion webhooks for multi-version support.
  • Operators use the controller pattern and reconcile desired state (spec) with actual state (status) in a continuous loop.
  • Well-built Operators are idempotent, level-based, and handle failures, upgrades, and deletions gracefully via finalizers.
  • Observability is critical: expose metrics, events, and status conditions for debugging and monitoring.
  • Security best practices: use RBAC with least privilege, avoid cluster-admin, and validate inputs.
  • Mediocre Operators are imperative, event-driven, lack status updates, and break on Kubernetes version upgrades.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.