Uber·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

Interviewed for an MLE role at Uber and got a pretty deep ML theory question about implementing the CLIP contrastive loss from scratch. Not a vibe check, they actually wanted working code and a real explanation of the design choices.

Questions Asked (1)

Given a batch of paired image and text embeddings, implement the symmetric contrastive loss used in CLIP-style training. You need to compute a full similarity matrix, then compute cross-entropy loss in both directions (image-to-text and text-to-image) and average them. Also explain your choices around normalization, temperature, and how you construct the labels.

Technical Trade-offsAlgorithms & Data StructuresSystem Design

Author's notes

I knew the high-level idea of CLIP but writing it out under pressure was a different story.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the problem setup and assumptions, then walk through the implementation step-by-step: normalize embeddings, compute similarity matrix with temperature scaling, create labels as diagonal indices, compute cross-entropy in both directions, and average. Finally, discuss trade-offs and practical considerations like numerical stability and temperature learning.

Pro tip: Mention that you would use logits = (image_embeds @ text_embeds.T) / temperature and that the labels are simply torch.arange(batch_size) on the appropriate device. Also, note that temperature is often a learnable parameter initialized to 0.07 (as in CLIP) and that you should use symmetric loss to ensure both modalities are aligned.

1. Clarify assumptions and setup

Confirm that embeddings are paired (i-th image with i-th text), batch size N, and that we need to compute symmetric loss. Ask about normalization and temperature if not specified.

2. Normalize embeddings

L2-normalize both image and text embeddings to unit length so that dot product equals cosine similarity. This stabilizes training and makes temperature scaling meaningful.

3. Compute similarity matrix and logits

Compute the N x N matrix of dot products between normalized image and text embeddings, then scale by a temperature parameter (often learnable, initialized to 0.07).

4. Compute symmetric cross-entropy loss

Create labels as the diagonal indices (0 to N-1). Compute cross-entropy loss for image-to-text (rows) and text-to-image (columns), then average the two losses.

5. Discuss implementation details and trade-offs

Explain choices: why normalize, why temperature, how to handle numerical stability (e.g., using log-softmax), and whether temperature is fixed or learned. Mention that this is the standard CLIP loss.

Key Points to Mention

L2 normalization of embeddings to unit length ensures cosine similarity and bounded logits.
Temperature parameter scales logits; often learnable and initialized to 0.07 as in CLIP.
Labels are simply the diagonal indices (torch.arange(N)) since positive pairs are on the diagonal.
Symmetric loss averages image-to-text and text-to-image cross-entropy to align both modalities.
Use log-softmax for numerical stability instead of softmax then log.
Batch size matters: larger batches provide more negatives and better contrastive signal.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.