Perplexity AI·Software Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

System design round at Perplexity AI for a software engineering role. The whole question was basically one giant design problem about building a multi-provider LLM routing library, and they wanted you to go deep on basically every subsystem you could think of.

Questions Asked (1)

Design a resilient client library that routes requests across multiple external LLM providers. Your design should cover provider registration, capability mapping, rate limiting and quotas per provider, health checks, timeouts, circuit breakers, cost and latency aware load balancing, retry and fallback logic, observability, secure key management, and thread-safe concurrency. Walk through your interfaces, data structures, and how you handle failures.

System DesignAPI & IntegrationsTechnical Trade-offs

Author's notes

This question is basically a full distributed systems design crammed into one prompt.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then present a layered architecture with clear interfaces and data structures. Walk through the request lifecycle, highlighting how each component (registration, health checks, load balancing, retries, circuit breakers) interacts to ensure resilience. Emphasize trade-offs and failure handling at each layer.

Pro tip: Focus on the 'why' behind each design choice—interviewers value reasoning over just listing components. Also, proactively discuss how you'd test and monitor the system to catch issues early.

1. Clarify Requirements and Scope

Ask questions to understand expected scale, latency SLAs, cost sensitivity, and provider diversity. Confirm whether the library is for internal use or external SDK, and what failure modes are most critical.

2. Define Core Interfaces and Data Structures

Outline key abstractions like Provider, Request, Response, and Router. Specify data structures for provider metadata (capabilities, rate limits, health status) and how they are stored and updated thread-safely.

3. Design Request Routing and Load Balancing

Explain how requests are routed based on capabilities, cost, latency, and health. Describe algorithms for cost/latency-aware load balancing and how to incorporate real-time metrics.

4. Implement Resilience Patterns

Detail timeouts, retries with backoff, circuit breakers, and fallback strategies. Discuss how to handle partial failures and avoid cascading failures across providers.

5. Address Observability, Security, and Concurrency

Cover logging, metrics, tracing, and alerting. Explain secure key management (e.g., vaults, rotation) and thread-safe concurrency using locks, atomics, or actor models.

Key Points to Mention

Provider registration and capability mapping: dynamic registration with metadata (supported models, max tokens, cost per token).
Rate limiting and quotas: per-provider token bucket or sliding window, with distributed coordination if needed.
Health checks and circuit breakers: periodic health probes, circuit breaker states (closed, open, half-open) to isolate unhealthy providers.
Cost and latency aware load balancing: weighted routing based on real-time metrics, with fallback to static weights.
Retry and fallback logic: exponential backoff with jitter, idempotency keys, and fallback to alternative providers on failure.
Observability and secure key management: structured logging, metrics (latency, error rates), distributed tracing, and secure storage/rotation of API keys.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.