← Decagon Interview Insights

Decagon·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
May 2026Remote

Summary

System design round at Decagon for a software engineer role, one big question about building an AI gateway layer in front of multiple LLM providers. Pretty dense problem that kept branching into new sub-problems every time I thought I'd covered something.

Questions Asked (2)

Q1

Design an AI gateway that sits between internal services and multiple LLM providers. It should route requests to a chosen provider and fall back to a backup when the primary fails, is rate-limited, or is degraded. What signals drive the failover switch, and how do you prevent the system from flapping back and forth?

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

I started with the obvious stuff: watch for 5xx errors and 429s, track latency p99.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then propose a high-level architecture with a routing layer, health monitors, and a circuit breaker. Explain the signals that trigger failover and how hysteresis, cooldowns, and health checks prevent flapping.

Pro tip: Emphasize that failover should be based on a combination of error rates, latency, and rate-limit headers, not just binary up/down. Use a state machine with half-open probing to safely test recovery and avoid flapping.

1. Clarify Requirements and Constraints

Ask about expected traffic volume, latency SLAs, provider diversity, cost considerations, and whether requests are idempotent. This shapes the design and trade-offs.

2. Design the Gateway Architecture

Outline components: request router, provider adapters, health monitor, circuit breaker, and fallback logic. Consider a control plane for configuration and a data plane for request handling.

3. Define Failover Signals

Specify the signals that trigger failover: error rates (5xx, timeouts), latency percentiles, rate-limit headers (429), and provider health checks. Combine them into a health score.

4. Prevent Flapping with Hysteresis and State Machine

Use a circuit breaker with states (closed, open, half-open). Implement hysteresis: require sustained failure to open and sustained success to close. Add cooldown periods and exponential backoff for probes.

5. Discuss Trade-offs and Operational Concerns

Address trade-offs: latency vs. reliability, cost of fallback, data consistency, and observability. Mention monitoring, alerting, and gradual rollout of changes.

Key Points to Mention

  • Circuit breaker pattern with closed, open, and half-open states
  • Health signals: error rates, latency percentiles, rate-limit headers, and synthetic probes
  • Hysteresis and cooldown periods to prevent flapping
  • Fallback strategies: retry with backoff, queueing, or degraded responses
  • Observability: metrics, logging, and tracing for provider health
  • Trade-offs: cost, latency, and consistency when switching providers

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you design the structured logging system for this gateway? Cover the data model for request and response logs, what metrics surface in dashboards, how you handle retention and PII, and how the fallback policy gets configured (static versus adaptive).

Data ModelingSystem DesignProduct Analytics & Metrics
Author's notes

This part I actually liked.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the gateway's role and logging requirements, then propose a structured data model with fields for request/response, trace context, and metadata. Discuss how metrics are derived from logs for dashboards, and address retention, PII, and fallback policy configuration with trade-offs between static and adaptive approaches.

Pro tip: Emphasize that logging should be designed for both operational debugging and product analytics, and that PII handling must be compliant and configurable per environment. Show awareness of cost and performance implications of high-volume logging.

1. Clarify requirements and constraints

Ask about scale, latency sensitivity, compliance needs, and existing infrastructure. Understand what the gateway does and what insights are needed from logs.

2. Design the structured log data model

Define a schema with fields for request (method, path, headers, body), response (status, latency, body), trace/span IDs, user/session IDs, and timestamps. Use JSON for flexibility and include a version field.

3. Define metrics and dashboards

Derive metrics like request rate, error rate, latency percentiles, and payload sizes. Surface them in dashboards for real-time monitoring and historical analysis, with drill-down to logs.

4. Address retention and PII

Implement tiered retention (hot/warm/cold) and anonymize or redact PII at ingestion. Use configurable policies per data type and comply with regulations like GDPR.

5. Configure fallback policy

Choose between static (fixed thresholds) and adaptive (dynamic based on load/error rates) fallback. Explain trade-offs: static is predictable, adaptive is resilient but complex.

Key Points to Mention

  • Structured logging with JSON schema and versioning for forward compatibility
  • Inclusion of trace context (e.g., OpenTelemetry) for distributed tracing
  • Metrics derivation from logs (e.g., using ELK or Prometheus) and dashboard design
  • PII redaction/anonymization techniques and compliance considerations
  • Retention policies with tiered storage and cost optimization
  • Static vs adaptive fallback: when to use each, and how to configure thresholds

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.