← American Express Interview Insights

American Express·AI Engineer·Onsite - System Design / Architecture·Senior

Senior
May 2026

Summary

System design round at Amex for an AI Engineer role, focused almost entirely on concurrency in multi-user agent systems. Pretty deep technically and covered a lot of ground in one question.

Questions Asked (1)

Q1

How would you design a multi-user AI agent system to handle concurrency safely? Walk through context isolation per user, shared resource management (model servers, vector stores, tool backends), fair queueing, request batching vs. latency SLAs, session memory isolation, and consistency when multiple sessions write to shared agent state. Also cover observability and how you'd test for concurrency bugs.

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

This was a lot to unpack in one question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as a multi-tenant distributed system with strict isolation and fairness requirements, then walk through each layer (context, resources, scheduling, memory, consistency) with concrete design choices and trade-offs. Emphasize how you balance latency SLAs with batching and how you ensure correctness under concurrency through idempotency, versioning, and testing.

Pro tip: At a financial company like American Express, highlight auditability and data isolation as non-negotiable, and propose a 'tenant-aware' scheduler that enforces per-user quotas to prevent noisy-neighbor issues.

1. Define isolation boundaries

Explain how you isolate user context (e.g., separate execution contexts, namespaces, or sandboxes) and session memory (e.g., per-user vector indices or partitioned stores) to prevent cross-user data leakage.

2. Design shared resource management

Describe how model servers, vector stores, and tool backends are shared across users, including connection pooling, rate limiting, and fair queueing (e.g., weighted fair queuing or token buckets per user).

3. Balance batching and latency SLAs

Discuss dynamic batching strategies (e.g., micro-batching with timeouts) and how to prioritize requests to meet SLAs, possibly using priority queues and adaptive batch sizes.

4. Ensure consistency in shared agent state

Address concurrent writes to shared state (e.g., user profiles, tool outputs) using optimistic concurrency control, versioning, or distributed locks, and define conflict resolution policies.

5. Implement observability and concurrency testing

Outline logging, tracing, and metrics for per-user latency, queue depths, and error rates; and describe testing strategies like stress tests, race condition detection, and chaos engineering.

Key Points to Mention

  • Context isolation: per-user execution contexts, session-scoped memory, and data partitioning to prevent leakage.
  • Fair queueing: per-user quotas, weighted fair queuing, and backpressure to avoid noisy neighbors.
  • Batching vs. latency: dynamic batching with timeout, priority queues, and SLA-aware scheduling.
  • Consistency: optimistic concurrency, versioning, idempotent operations, and conflict resolution for shared state.
  • Observability: distributed tracing, per-user metrics, and alerting on queue latency and error rates.
  • Concurrency testing: stress tests, race condition detection tools, and chaos engineering to simulate failures.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.