Atlassian·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Jul 2026

Summary

System design round at Atlassian for an ML Engineer role, basically one big question about building an internal enterprise chatbot that integrates with Confluence, Jira, and similar tools. Heavy on architecture and production concerns, less about ML theory than I expected.

Questions Asked (1)

Design an internal enterprise chatbot that lets employees ask questions in natural language, pulling answers from internal tools like Confluence and Jira, while enforcing access control, supporting both document Q&A and action-based workflows, and keeping everything auditable. Walk through the high-level architecture, key components, and how you'd evaluate and monitor it in production.

System DesignTechnical Trade-offsAPI & Integrations

Author's notes

This is a beast of a question and I think I underestimated how many distinct sub-problems are packed into it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and constraints, then present a layered architecture that separates ingestion, retrieval, access control, and action execution. Emphasize how you would evaluate and monitor the system for correctness, security, and performance, and discuss trade-offs between different design choices.

Pro tip: Anchor your design around Atlassian's existing permission models (e.g., Confluence/Jira permissions) and show how you'd enforce them at query time, not just at ingestion. Also, propose a phased rollout starting with read-only Q&A before enabling actions.

1. Clarify Requirements and Constraints

Ask about scale, latency, data freshness, supported actions, and compliance needs. Confirm that access control must mirror source systems and that audit logs are immutable.

2. High-Level Architecture

Outline components: ingestion pipeline (connectors to Confluence/Jira), vector database for retrieval, LLM orchestration layer, access control service, action executor, and audit logging. Explain data flow from query to answer/action.

3. Access Control and Security

Detail how to enforce permissions: propagate user identity, check permissions at query time via source APIs or cached ACLs, and filter retrieved documents. For actions, validate permissions and use OAuth scopes.

4. Evaluation and Monitoring

Define offline metrics (retrieval accuracy, answer correctness) and online metrics (user feedback, latency, error rates). Describe A/B testing, canary deployments, and monitoring for security anomalies and drift.

5. Trade-offs and Iteration

Discuss trade-offs: real-time vs. batch ingestion, fine-tuning vs. RAG, caching vs. freshness. Propose an iterative approach starting with read-only Q&A and expanding to actions.

Key Points to Mention

Retrieval-Augmented Generation (RAG) with vector search over embedded documents
Permission-aware retrieval: filtering results based on user's access rights in Confluence/Jira
Action workflows: using function calling or APIs to execute tasks (e.g., create Jira ticket) with proper authorization
Audit logging: capturing all queries, retrieved documents, actions, and user context for compliance
Evaluation metrics: precision/recall of retrieval, answer relevance, task success rate, and user satisfaction
Monitoring: latency, error rates, drift detection, and security alerts for unauthorized access attempts

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.