Google·Software Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

Google system design screen for a software engineer role, focused entirely on AI inference cost vs latency trade-offs. Pretty meaty single question that branched into a bunch of sub-topics before I even felt like I'd finished my first answer.

Questions Asked (1)

You're building an AI product where each request can be routed to either a high-cost low-latency inference option or a low-cost high-latency one. How do you decide which to use, and how do you think about the trade-offs across user experience, quality, reliability, and cost? What metrics would you track, how would you segment workloads, and would you build a dynamic routing layer or just pick one globally?

System DesignTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This question kept expanding on me.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the product requirements and user expectations, then propose a dynamic routing layer that segments requests based on latency sensitivity, quality needs, and cost budgets. Discuss trade-offs explicitly and define metrics to monitor and iterate on the routing policy.

Pro tip: Emphasize that routing decisions should be driven by business SLAs and user-perceived latency, not just cost—and mention that you'd start with a simple heuristic and evolve to ML-based routing as you gather data.

1. Clarify requirements and constraints

Ask about the product's latency SLAs, quality expectations, cost budget, and traffic patterns to ground the discussion in concrete goals.

2. Define segmentation criteria

Segment workloads by user tier, request type, time sensitivity, and quality requirements (e.g., interactive vs. batch, free vs. paid users).

3. Design routing policy

Propose a dynamic routing layer that uses rules or ML to route each request to the optimal inference option based on segment and real-time signals.

4. Analyze trade-offs

Discuss trade-offs across UX (latency), quality (accuracy), reliability (fallbacks), and cost, and how to balance them via routing thresholds and fallback strategies.

5. Define metrics and iterate

List key metrics (p50/p99 latency, cost per request, quality scores, error rates) and describe how to monitor, A/B test, and refine the routing policy over time.

Key Points to Mention

Dynamic routing based on request context (e.g., user tier, request type, real-time load)
Trade-offs: latency vs. cost vs. quality vs. reliability
Metrics: p50/p99 latency, cost per request, quality metrics (e.g., BLEU, user feedback), error rates, fallback frequency
Segmentation: interactive vs. batch, free vs. paid, simple vs. complex queries
Fallback strategies: if high-cost option fails, fall back to low-cost (and vice versa) with appropriate timeouts
Start with simple heuristics, then evolve to ML-based routing using logged data

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.