I started with the happy path, which in hindsight was the wrong move.
Start by clarifying requirements and scale (e.g., QPS, latency, model size), then sketch a high-level architecture with clear separation of concerns: API gateway, conversation service, model inference, safety layer, and tool orchestration. Walk through data flow for a multi-turn streaming request, then dive into storage, scaling, and evaluation, emphasizing trade-offs and Plaid-relevant concerns like PII and reliability.
Pro tip: Anchor your design around the streaming and stateful nature of conversations: treat conversation history as a first-class entity with its own storage and caching strategy, and explicitly discuss how you'd handle partial failures and retries without duplicating or corrupting turns.
Ask about expected QPS, peak load, latency SLOs, model size/hosting, and whether tools are mandatory. Confirm safety and compliance requirements (e.g., PII, toxicity thresholds) and evaluation goals.
Draw the main components: client, API gateway, conversation service (manages history and orchestration), model inference service (with streaming), safety service, tool executor, and storage layers. Explain the request flow for a multi-turn streaming response.
Choose storage for conversation history (e.g., DynamoDB/Cassandra for durability, Redis for hot cache), model artifacts (object store), and tool results. Discuss token limits, summarization, and how to retrieve history efficiently.
Describe horizontal scaling of stateless services, autoscaling for inference (GPU/CPU), load balancing, rate limiting, and graceful degradation. Cover high availability via multi-AZ, retries, idempotency, and fallback models.
Explain toxicity filtering (pre/post), PII detection and redaction, and tool invocation with sandboxing. Outline offline and online evaluation: automated metrics, human review, A/B tests, and monitoring for drift and safety violations.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.