← Pinterest Interview Insights

Pinterest·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
Jun 2026

Summary

Pinterest ML infra interview, system design round focused entirely on GPU memory and batch size optimization for model serving. Pretty deep technically, more math-heavy than I expected for a design question.

Questions Asked (1)

Q1

You have a GPU with 60 GB of memory. Walk through how you'd figure out the optimal batch size for serving a model, covering parameter memory, activation memory, KV cache for transformer models, framework overhead, and the latency vs throughput trade-off. Also explain how you'd validate your choice empirically.

System DesignTechnical Trade-offs
Author's notes

This is a single question but it's really like five questions stitched together.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by breaking down the memory components: model parameters, activations, KV cache, and framework overhead. Then, calculate the maximum batch size that fits within 60 GB, considering the latency-throughput trade-off, and validate empirically by measuring latency and throughput across batch sizes.

Pro tip: Always measure actual memory usage and latency with profiling tools, as theoretical estimates can be off due to framework-specific optimizations and overheads. Also, consider dynamic batching and sequence length variability in real-world serving.

1. Estimate Model Parameter Memory

Calculate the memory required to store the model weights based on the number of parameters and precision (e.g., FP16, FP32).

2. Estimate Activation and KV Cache Memory

For transformer models, compute activation memory per layer and KV cache size per token, then multiply by batch size and sequence length.

3. Account for Framework Overhead

Add memory overhead from the deep learning framework (e.g., CUDA context, cuDNN workspaces) and any other runtime components.

4. Determine Maximum Batch Size

Subtract fixed costs from total GPU memory and divide remaining memory by per-sample memory (activations + KV cache) to get an upper bound on batch size.

5. Empirically Validate and Tune

Run experiments with different batch sizes, measure latency and throughput, and select the batch size that meets latency SLOs while maximizing throughput.

Key Points to Mention

  • Model parameter memory: size = num_params * bytes_per_param (e.g., 2 bytes for FP16).
  • Activation memory: depends on batch size, sequence length, and model architecture; can be reduced with gradient checkpointing (but not in inference).
  • KV cache: for autoregressive decoding, cache size = 2 * num_layers * num_heads * head_dim * sequence_length * batch_size * bytes_per_element.
  • Framework overhead: typically 1-2 GB for CUDA context and libraries; can vary.
  • Latency vs throughput trade-off: larger batch sizes increase throughput but also latency; find sweet spot based on SLOs.
  • Empirical validation: use profiling tools (e.g., nvidia-smi, PyTorch profiler) and benchmark with realistic input distributions.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.