← Atlassian Interview Insights
30 minutes is not a lot of time for this scope.
Start by clarifying the product context and scale (e.g., Atlassian's Jira/Confluence, number of users/items, latency SLAs), then walk through the full pipeline: candidate retrieval, ranking, feature engineering, training, and serving. Emphasize trade-offs at each stage (e.g., recall vs. latency, model complexity vs. maintainability) and tie decisions back to business metrics like engagement or task completion.
Pro tip: Anchor your design in Atlassian's B2B collaboration context: recommendations should optimize for team productivity (e.g., suggesting relevant issues, pages, or people), not just clicks. Mention how you'd handle cold-start for new teams and the importance of explainability for enterprise trust.
Ask about scale (users, items), latency budget, offline vs. online metrics, and business goals. Confirm whether it's a real-time or batch system and what data is available (e.g., user interactions, content metadata).
Propose a multi-stage retrieval: candidate generation from multiple sources (e.g., collaborative filtering, content-based, trending) using ANN indexes for scalability. Discuss trade-offs between recall and latency.
Describe a learning-to-rank model (e.g., GBDT or deep neural network) that takes features from user, item, and context. Explain how to balance relevance, diversity, and business rules.
Outline key features (user history, item popularity, co-occurrence, embeddings) and how to compute them offline/online. Discuss training data generation (e.g., negative sampling), model retraining frequency, and evaluation metrics (NDCG, recall@k).
Explain the serving architecture: pre-compute embeddings, use a feature store for low-latency lookups, and deploy models via a model server. Include A/B testing, logging, and monitoring for drift and performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered NDCG and AUC for offline, CTR and engagement metrics for online.
Structure your answer by first distinguishing offline and online evaluation, then detail specific metrics for each phase, and finally explain how they complement each other in a production ML pipeline. Emphasize that offline metrics guide model selection, while online metrics validate real-world impact through experimentation.
Pro tip: Mention that offline metrics are proxies and can mislead due to offline-online gap, so you always validate with online A/B tests and monitor business metrics like engagement and retention. Also, highlight the importance of guardrail metrics to catch negative side effects.
Explain that offline evaluation uses historical data to compare models quickly, while online evaluation uses live traffic to measure real user impact. This sets the stage for metric selection.
List ranking and accuracy metrics such as Precision@K, Recall@K, NDCG, MAP, MRR, and AUC. Mention coverage, diversity, and novelty as additional offline considerations.
Describe engagement metrics (CTR, conversion rate, dwell time), business metrics (revenue, retention), and system metrics (latency, coverage). Include A/B testing methodology and statistical significance.
Discuss how offline metrics inform online experiments and how online results can reveal limitations of offline proxies. Mention techniques like counterfactual evaluation and off-policy evaluation.
Highlight the need for guardrail metrics (e.g., user churn, complaint rate) and long-term metrics (e.g., user satisfaction, lifetime value) to ensure recommendations don't harm the ecosystem.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Talked about separating user features, item features, and context features, and the need for a feature store to avoid training-serving skew.
Start by clarifying the system's requirements and ML use cases, then outline a feature engineering pipeline that covers data sources, transformations, and storage. Emphasize how a feature store enables consistency, reuse, and low-latency serving for both training and inference.
Pro tip: Highlight the importance of feature versioning and monitoring to prevent training-serving skew, and mention how Atlassian's scale and product ecosystem (e.g., Jira, Confluence) would influence design choices like online/offline store separation.
Ask about the specific ML models, data volume, latency requirements, and whether features need to be served in real-time or batch. This ensures the design aligns with business needs.
Describe how raw data is ingested, cleaned, transformed, and aggregated into features. Include both batch and streaming processing, and mention tools like Spark, Flink, or Kafka.
Explain the architecture of a feature store: offline store for training (e.g., S3, BigQuery) and online store for serving (e.g., Redis, DynamoDB). Discuss how it ensures consistency and low-latency access.
Detail how features are versioned, documented, and monitored to avoid training-serving skew. Mention point-in-time correctness for training data.
Cover scalability, cost, security, and how to handle feature updates and backfills. Discuss monitoring for drift and performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the requirements: latency target, throughput, recommendation freshness, and integration with existing Atlassian services. Then propose a high-level architecture that separates offline training from online serving, using a feature store and a low-latency model server. Finally, discuss trade-offs between latency, cost, and accuracy, and how you would monitor and iterate.
Pro tip: Emphasize the importance of pre-computing recommendations and caching them at the edge or in a fast key-value store, as this is often the key to achieving low latency at scale. Also, mention the need for fallback strategies to handle failures gracefully.
Ask about expected QPS, latency SLA (e.g., p99 < 100ms), recommendation freshness (real-time vs batch), and existing infrastructure (e.g., AWS, Kubernetes).
Describe how models are trained and validated offline, and how features and recommendations are pre-computed and stored in a feature store and a low-latency database like Redis or DynamoDB.
Outline the serving architecture: API gateway, model server (e.g., TensorFlow Serving, TorchServe), feature retrieval from online store, and caching layers. Discuss how to handle real-time features if needed.
Explain how to scale horizontally, use load balancing, and implement fallbacks (e.g., popular items) if the model server fails. Mention monitoring and alerting.
Compare latency vs accuracy (e.g., using a simpler model for online serving), batch vs real-time pre-computation, and cost implications. Suggest A/B testing for continuous improvement.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.