This is the kind of question where knowing the buzzwords isn't enough.
Start by acknowledging that low GPU utilization indicates the GPU is starved for data or blocked on synchronization, so you need to profile the entire pipeline to find the bottleneck. Then systematically isolate whether the issue is in data loading, host-device transfers, kernel execution, or communication, using tools like Nsight Systems and PyTorch Profiler. Finally, benchmark each component and propose targeted optimizations, validating with metrics like SM occupancy and memory throughput.
Pro tip: Always measure before optimizing—use Nsight Systems to get a timeline view first, because intuition about bottlenecks is often wrong. Also, check for common pitfalls like unnecessary host-device synchronizations or small batch sizes that underutilize the GPU.
Measure current GPU utilization, throughput, and step time to quantify the problem. Define what 'good' looks like for this workload (e.g., >80% utilization).
Use Nsight Systems to capture a timeline and identify gaps where the GPU is idle. Look for long data loading, host-device copies, or synchronization points.
Use Nsight Compute or PyTorch Profiler to analyze kernel efficiency, memory bandwidth, and occupancy. Check if kernels are memory-bound, compute-bound, or launch-bound.
Microbenchmark data loading, model forward/backward, and communication separately. Compare against theoretical peaks to identify the bottleneck.
Apply targeted fixes (e.g., increase batch size, use pinned memory, overlap data transfer, optimize kernels) and re-measure to confirm improvement.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.