Amazon·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

May 2026

Summary

Amazon ML Engineer system design round, one big open-ended question about building a multimodal retrieval and classification system. Dense topic, lots of surface area to cover.

Questions Asked (1)

Design a multimodal neural network that fuses text and image inputs for retrieval and classification. Walk through your encoder choices for each modality, how you'd fuse them, what training objectives you'd use, how you'd handle missing or noisy modalities, data alignment and augmentation strategies, evaluation metrics, scalability, serving latency, and how you'd adapt the system to a new domain.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This question is basically a whole system design interview compressed into one prompt.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the problem scope and requirements, then propose a modular architecture with separate encoders for each modality, a fusion mechanism, and a multi-task learning setup for retrieval and classification. Discuss trade-offs at each design choice, emphasizing scalability, robustness to missing/noisy data, and adaptation to new domains, aligning with Amazon's production-scale needs.

Pro tip: Demonstrate awareness of real-world constraints by discussing how to handle missing modalities during training and serving, and how to leverage Amazon's existing services (e.g., SageMaker, Elasticsearch) for scalability and latency optimization.

1. Clarify Requirements and Constraints

Ask about data scale, latency requirements, retrieval vs. classification priorities, and domain specifics to tailor the design. This shows you consider business and operational context before diving into technical details.

2. Design Modality-Specific Encoders

Choose appropriate encoders: for text, consider transformer-based models (e.g., BERT) for rich semantics; for images, use CNN (e.g., ResNet) or vision transformers (ViT). Justify choices based on accuracy, latency, and computational budget.

3. Select Fusion Strategy and Training Objectives

Decide on early, late, or hybrid fusion; for retrieval, use contrastive loss (e.g., InfoNCE) to align embeddings; for classification, add a classification head with cross-entropy. Consider multi-task learning to share representations.

4. Address Robustness and Data Challenges

Handle missing modalities via imputation, modality dropout, or separate encoders with fallback. Mitigate noise with data augmentation (e.g., image transforms, text back-translation) and robust loss functions. Ensure alignment via paired data and contrastive learning.

5. Plan Evaluation, Scalability, and Adaptation

Define metrics: retrieval (Recall@K, mAP), classification (accuracy, F1). For scalability, use distributed training, model quantization, and caching. For latency, optimize with ONNX, TensorRT, or model distillation. For new domains, fine-tune with few-shot learning or domain adaptation techniques.

Key Points to Mention

Encoder choices: transformer for text, CNN/ViT for images, with trade-offs between accuracy and latency.
Fusion methods: early (concatenation), late (score averaging), or hybrid (cross-attention); consider computational cost.
Training objectives: contrastive loss for retrieval, cross-entropy for classification, and multi-task learning to balance.
Handling missing/noisy modalities: modality dropout, imputation, robust losses, and data augmentation.
Evaluation metrics: Recall@K, mAP for retrieval; accuracy, F1 for classification; also measure latency and throughput.
Scalability and serving: distributed training, model compression, caching, and use of managed services (e.g., SageMaker, Elasticsearch).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.