Start by defining the software hierarchy (grid → block → warp → thread) and then map each level to the hardware (GPU → SM → warp scheduler → CUDA core). Use a concrete example like matrix multiplication to illustrate how threads cooperate and how warps execute in lockstep.
Pro tip: Mention that warp size is 32 and that divergence within a warp serializes execution, which is a common performance pitfall. Also note that blocks are scheduled onto SMs, and multiple blocks can reside on one SM, affecting occupancy.
Explain that a CUDA kernel launches a grid of blocks, each block contains threads, and threads are grouped into warps of 32 for execution.
Describe how the grid maps to the GPU, blocks are assigned to Streaming Multiprocessors (SMs), warps are scheduled by warp schedulers, and threads execute on CUDA cores.
Discuss how threads within a block can synchronize and share memory, while warps execute in SIMT fashion; blocks are independent and can execute in any order.
Mention how block size, warp divergence, and occupancy affect performance, and how the model enables massive parallelism.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by categorizing the memory types by scope (per-thread, per-block, per-grid) and then discuss their performance characteristics in terms of latency, bandwidth, and caching. Emphasize how these differences drive optimization strategies like coalescing, tiling, and minimizing global memory traffic.
Pro tip: Relate each memory type to a concrete optimization pattern (e.g., shared memory for tiling, constant memory for broadcast) and mention that on modern GPUs, L1 and shared memory are unified, so tuning the carveout matters.
Group the memory types by their visibility: registers (per-thread), shared (per-block), and global/constant/texture (per-grid). Mention that constant and texture are read-only caches off global memory.
For each type, state latency, bandwidth, and caching behavior. For example, registers are fastest with zero latency, shared memory has low latency and high bandwidth, global memory has high latency but high bandwidth when coalesced.
Discuss how access patterns affect performance: coalesced global accesses, bank conflicts in shared memory, register pressure, and the benefits of constant/texture caches for specific access patterns.
Give examples of how to use each memory type effectively: tiling with shared memory, using constant memory for broadcast reads, texture for spatial locality, and minimizing global memory traffic.
Note that on newer architectures (Volta+), L1 and shared memory are unified, and the carveout can be configured. Also, registers are not directly addressable, and spills hurt performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining occupancy as the ratio of active warps to the maximum supported warps per SM, then explain how it's influenced by register usage, shared memory, and block size. Emphasize that occupancy is a means to an end—maximizing latency hiding—and that optimal occupancy depends on the kernel's characteristics, so reasoning should involve profiling and trade-offs.
Pro tip: Mention that higher occupancy isn't always better; sometimes lower occupancy with more registers per thread yields better performance due to reduced instruction overhead and better data reuse. This shows you understand the nuanced trade-offs beyond textbook definitions.
Explain that occupancy is the ratio of active warps per SM to the maximum number of warps supported by that SM, and it indicates how well the GPU's latency-hiding capabilities are utilized.
List the key hardware resources that limit occupancy: registers per thread, shared memory per block, block size, and the maximum number of blocks per SM. Mention that these are interdependent.
Discuss how increasing occupancy can improve latency hiding but may also increase resource contention or reduce per-thread resources, potentially hurting performance. Give examples like memory-bound vs compute-bound kernels.
Describe how you would use occupancy calculators (e.g., NVIDIA's Occupancy Calculator) and profiling tools (e.g., Nsight Compute) to guide decisions on block size, register usage, and shared memory allocation.
Summarize that the goal is to find the sweet spot for the specific kernel, often by iterating and measuring, rather than blindly maximizing occupancy.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by describing a straightforward triple-loop implementation of GEMM, then systematically apply optimizations such as tiling, memory coalescing, shared memory usage, register blocking, and vectorized loads. Emphasize how each optimization addresses specific bottlenecks (memory bandwidth, latency, compute utilization) and quantify the expected performance gains.
Pro tip: Relate optimizations to the GPU memory hierarchy and roofline model—showing you understand not just what to do but why it works on NVIDIA hardware. Mention that the final optimized kernel approaches the speed of light for the given arithmetic intensity.
Describe a simple triple-loop GEMM where each thread computes one element of C, reading A and B directly from global memory. Highlight that this is memory-bound and suffers from uncoalesced accesses and redundant loads.
Introduce tiling: each thread block computes a tile of C, loading corresponding tiles of A and B into shared memory. This reduces global memory traffic and improves data reuse.
Apply register blocking: each thread computes multiple elements of C, increasing arithmetic intensity and reducing shared memory accesses. Also ensure coalesced global memory accesses and avoid bank conflicts in shared memory.
Discuss further optimizations: vectorized loads (float4), double buffering to overlap computation and memory transfers, and using wider tiles to increase reuse. Mention tuning block size and tile dimensions for specific GPU architectures.
Explain how to measure performance (e.g., using CUDA events, nvprof) and compare against theoretical peak (roofline model). Discuss trade-offs between occupancy, register pressure, and shared memory usage.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Boundary handling always trips me up a little.
Start by explaining that tile size selection balances occupancy, memory bandwidth, and compute throughput, referencing hardware constraints like shared memory and register limits. Then describe how to handle non-multiple sizes using predication or padding, emphasizing correctness and performance trade-offs.
Pro tip: Mention that you profile with tools like Nsight Compute to tune tile sizes for the specific GPU architecture, and that you often use a fallback kernel for edge cases to avoid branch divergence.
Identify the target GPU's shared memory per SM, register file size, and maximum threads per block to bound feasible tile dimensions.
Choose tile sizes that maximize data reuse in shared memory while maintaining enough active warps to hide latency, often using autotuning or heuristics.
Use conditional checks in the kernel to guard loads and stores for out-of-bounds elements, ensuring correctness without separate kernels.
For small remainders, pad matrices to tile multiples if memory allows, or launch a specialized kernel for edge tiles to reduce divergence.
Benchmark different configurations and edge-case handling strategies, using profiling to confirm that the chosen approach meets performance targets.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Short answer: profile with nvprof or Nsight, look at TFLOPS achieved vs theoretical peak, compare to cuBLAS on the same shapes.
Start by clarifying the comparison's scope: specify the matrix dimensions, data types, and hardware (e.g., A100, H100). Then outline a rigorous benchmarking methodology that includes warm-up runs, multiple iterations, and statistical measures like median and 95th percentile. Finally, discuss how to interpret results in terms of achieved TFLOPS and efficiency relative to cuBLAS, and what factors might explain any gaps.
Pro tip: Always compare against cuBLAS with the same problem sizes and data types, and report both absolute throughput and percentage of cuBLAS achieved. This shows you understand that cuBLAS is highly optimized and that context matters more than raw numbers.
Specify the exact matrix dimensions, data types (FP32, FP16, etc.), and GPU architecture. This ensures the comparison is meaningful and reproducible.
Describe how you would measure performance: warm-up iterations, number of runs, synchronization, and metrics like median, mean, and standard deviation. Mention using CUDA events or nvprof/Nsight for timing.
Run both kernels under identical conditions and compute achieved TFLOPS. Compare your kernel's throughput to cuBLAS, reporting both absolute values and relative performance (e.g., 85% of cuBLAS).
Discuss possible reasons for differences: memory bandwidth, compute utilization, tiling strategy, etc. Highlight any trade-offs (e.g., custom kernel may be faster for specific shapes but slower for others).
Summarize when your custom kernel is preferable and what optimizations could close the gap. Emphasize the importance of context and continuous benchmarking.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew the concept: pageable vs pinned memory, async memcpy, double buffering with two streams.
Start by explaining the fundamental bottleneck: host-to-device transfers over PCIe are slow and block the GPU if done synchronously. Then describe how to use pinned (page-locked) memory to enable asynchronous copies via cudaMemcpyAsync, and how CUDA streams allow overlapping these transfers with kernel execution. Finally, discuss trade-offs and best practices for maximizing overlap.
Pro tip: Mention that pinned memory is a scarce resource and over-allocating it can degrade overall system performance; use it judiciously and consider using cudaHostAlloc with cudaHostAllocWriteCombined for write-only data to reduce cache pollution.
Explain that default pageable memory transfers are synchronous and block the GPU, causing idle time. Highlight that PCIe bandwidth is much lower than GPU memory bandwidth, so overlapping is critical.
Describe how to allocate pinned memory with cudaMallocHost or cudaHostAlloc, which allows the GPU to directly access host memory via DMA, enabling true asynchronous copies.
Explain that by issuing cudaMemcpyAsync and kernel launches into different streams, the GPU can overlap data transfers with computation. Use multiple streams to pipeline data chunks.
Divide data into chunks and use a double-buffering scheme: while chunk N is being computed, chunk N+1 is being transferred. This keeps both the copy engine and compute units busy.
Mention that pinned memory is limited and can hurt host performance if overused. Also note that overlapping is only beneficial if compute time is comparable to transfer time; otherwise, the bottleneck shifts.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked through Nsight Compute metrics and the roofline model.
Start by defining each metric in the context of CUDA performance analysis: achieved occupancy as the ratio of active warps to maximum warps per SM, arithmetic intensity as the ratio of floating-point operations to bytes of memory traffic, and memory bandwidth as the rate of data transfer between global memory and SMs. Then explain how to measure each using NVIDIA profiling tools like Nsight Compute or nvprof, and how to interpret the results to identify bottlenecks. Emphasize that these metrics are interrelated and should be analyzed together for optimization.
Pro tip: Mention that achieved occupancy alone is not a goal; it's a means to hide latency. Focus on whether the kernel is memory-bound or compute-bound using the roofline model, and use metrics like achieved occupancy to diagnose why performance is below the roofline.
Clearly define achieved occupancy, arithmetic intensity, and memory bandwidth, and explain their significance in CUDA performance analysis.
Describe the tools available for measurement, such as NVIDIA Nsight Compute, nvprof, or CUPTI, and how they provide these metrics.
Explain how to calculate achieved occupancy using the ratio of active warps per SM to the maximum warps supported, and how to obtain this from profiling tools.
Detail how to compute arithmetic intensity by counting FLOPs and memory traffic, and how to measure memory bandwidth using counters for bytes transferred over time.
Discuss how to use these metrics together to identify bottlenecks (e.g., memory-bound vs compute-bound) and guide optimizations like improving memory coalescing or increasing occupancy.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.