Atlassian·Software Engineer·Onsite - System Design / Architecture·Senior

Senior

Jun 2026

Summary

System design round at Atlassian for a software engineer role. The question was a deep dive into building a real-time event filtering pipeline, and it covered basically every layer of the stack from ingestion to fault tolerance.

Questions Asked (1)

Design a streaming data pipeline that ingests high-throughput events and continuously filters out bad data based on configurable rules. Cover ingestion, real-time processing, rule evaluation, scaling, fault tolerance, ordering guarantees, exactly-once delivery, and how rules are versioned and updated.

System DesignTechnical Trade-offsData Modeling

Author's notes

This was a lot to hold in your head at once.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements (throughput, latency, data quality rules, ordering needs) and then propose a high-level architecture using a distributed log (e.g., Kafka) for ingestion, a stream processor (e.g., Flink) for real-time filtering, and a rules engine with versioned rules stored in a database. Walk through each component, emphasizing trade-offs around exactly-once, ordering, and scalability, and conclude with how rules are updated and versioned without downtime.

Pro tip: Demonstrate awareness of operational concerns by discussing how to handle rule updates atomically and how to monitor data quality metrics; mention that you'd use a canary deployment for new rules to avoid widespread data loss.

1. Clarify Requirements and Constraints

Ask questions to understand expected throughput, latency tolerance, data quality rule complexity, ordering requirements, and fault tolerance needs. This ensures the design meets the actual use case.

2. Design Ingestion and Buffering

Propose a scalable ingestion layer using a distributed message queue like Kafka, which handles high throughput, provides durability, and decouples producers from consumers.

3. Design Real-Time Processing and Rule Evaluation

Use a stream processing framework (e.g., Flink, Spark Streaming) to consume events, evaluate configurable rules, and filter out bad data. Discuss how rules are represented (e.g., DSL, SQL) and evaluated efficiently.

4. Address Scaling, Fault Tolerance, and Delivery Guarantees

Explain partitioning for scalability, checkpointing/state management for fault tolerance, and mechanisms for exactly-once processing (e.g., idempotent writes, transactions). Discuss ordering guarantees per partition and how to handle out-of-order events.

5. Manage Rule Versioning and Updates

Describe how rules are stored (e.g., in a versioned database), how updates are propagated to the stream processor (e.g., via a control stream), and how to ensure consistency during rule changes (e.g., atomic swaps, versioned rule sets).

Key Points to Mention

Use of a distributed log (e.g., Kafka) for ingestion to handle high throughput and provide durability.
Stream processing with exactly-once semantics via checkpointing and idempotent operations.
Partitioning strategy to ensure scalability and ordering guarantees per key.
Rule representation and evaluation: possibly using a DSL or SQL-like syntax, with efficient compilation.
Rule versioning: store rules in a versioned repository, use a control topic to broadcast updates, and ensure atomic rule set swaps.
Monitoring and observability: track data quality metrics, rule performance, and system health.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.