NVIDIA·Software Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

Apr 2026Remote

Summary

NVIDIA software engineer interview that went deep on GPU architecture. One big technical question that sprawled into five or six sub-topics, felt more like a conversation than a Q&A, but I was definitely underprepared on some of the memory hierarchy specifics.

Questions Asked (1)

Compare how a CPU and GPU handle large matrix multiplication, covering core architecture, memory hierarchy, latency-hiding strategies, why matmul maps well to GPUs, and situations where a CPU might still be the better choice.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This was basically five questions dressed up as one and I didn't realize it until I was already three minutes into talking about core counts.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by contrasting the fundamental design philosophies: CPUs optimize for low-latency serial execution with large caches and out-of-order cores, while GPUs optimize for high-throughput parallel execution with thousands of simple cores and massive multithreading. Then explain how matrix multiplication's regular, data-parallel structure maps naturally to GPU architecture, and finish by discussing scenarios where CPU might still win (small matrices, branch-heavy code, low parallelism, or when data transfer overhead dominates).

Pro tip: Mention that for NVIDIA GPUs, libraries like cuBLAS and CUTLASS already achieve near-peak performance, so the real engineering trade-off is often about data movement and kernel fusion rather than raw compute. Also note that modern CPUs with AVX-512 and AMX can be surprisingly competitive for small-to-medium matrices, especially when the data is already in cache.

1. Architecture comparison

Contrast CPU cores (few, complex, out-of-order, deep pipelines, large caches) with GPU cores (many, simple, in-order, high occupancy). Emphasize that CPUs excel at latency-sensitive tasks while GPUs excel at throughput-oriented tasks.

2. Memory hierarchy and data movement

Explain CPU cache hierarchy (L1/L2/L3) and GPU memory hierarchy (registers, shared memory, L1/L2, global memory). Highlight that GPUs rely on shared memory and coalesced global memory access to feed thousands of threads, while CPUs use large caches to hide latency.

3. Latency hiding strategies

Describe how CPUs hide latency via out-of-order execution, speculative execution, and prefetching. For GPUs, explain hardware multithreading (warp scheduling) and how many warps hide memory latency by switching contexts.

4. Why matmul maps well to GPUs

Matrix multiplication is highly data-parallel with regular memory access patterns and high arithmetic intensity (O(N^3) compute vs O(N^2) data). This allows GPUs to use tiling, shared memory, and register blocking to achieve near-peak FLOPs.

5. When CPU is better

Discuss cases: small matrices where kernel launch and data transfer overhead dominate, branch-heavy or irregular computations, low parallelism, or when the data is already in CPU cache and the problem is latency-bound rather than throughput-bound.

Key Points to Mention

SIMT vs SIMD execution models and how they affect parallelism
Arithmetic intensity and the roofline model for matmul
Tiling/blocking techniques to exploit shared memory and registers
Warp scheduling and occupancy as latency-hiding mechanisms on GPUs
Data transfer overhead (PCIe/NVLink) and kernel launch latency
CPU optimizations like AVX-512, AMX, and cache blocking for small matrices

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.