Axon·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

System design round at Axon for a software engineer role. The whole thing was one big distributed systems question about logging infrastructure, which sounds straightforward until you're actually in it trying to remember the difference between hot and cold storage tiering while someone watches you draw boxes.

Questions Asked (1)

Design a centralized logging system that collects logs from a massive fleet of hosts, ships them reliably to a cloud backend, and supports interactive querying by keyword, time range, host, service, and severity.

System DesignTechnical Trade-offsData Modeling

Author's notes

I started with the agent layer (Fluent Bit style collectors) and worked toward Kafka for buffering, then into an indexed store for queries.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements such as log volume, retention, query latency, and reliability guarantees. Then design a pipeline with agents on hosts, a scalable ingestion layer, durable storage, and an indexed query service. Discuss trade-offs between consistency, cost, and query performance.

Pro tip: Emphasize backpressure and buffering at the agent level to handle network outages and traffic spikes, and mention using a columnar store like Parquet on S3 with a query engine like Presto for cost-effective interactive queries.

1. Clarify Requirements and Scale

Ask about the number of hosts, log volume per host, retention period, query latency expectations, and reliability needs. This shapes the entire design.

2. Design the Collection Layer

Propose lightweight agents (e.g., Fluentd, Filebeat) that tail logs, add metadata, buffer locally, and forward via a reliable protocol. Discuss batching and compression.

3. Build the Ingestion Pipeline

Use a distributed message queue (e.g., Kafka) to decouple producers and consumers, handle spikes, and ensure at-least-once delivery. Mention partitioning by host or service for scalability.

4. Choose Storage and Indexing

Store raw logs in object storage (e.g., S3) for durability and cost, and build an index (e.g., Elasticsearch) for fast keyword and metadata queries. Discuss trade-offs between hot/warm/cold storage.

5. Enable Querying and Monitoring

Provide a query API and UI that supports filters by time, host, service, severity, and full-text search. Include monitoring of the pipeline itself and alerting on failures.

Key Points to Mention

Use of lightweight agents with local buffering and backpressure to handle network issues.
Leveraging a distributed log (e.g., Kafka) for reliable, scalable ingestion.
Tiered storage: hot index for recent logs, cold object storage for older logs.
Indexing strategy: inverted index for full-text search, columnar storage for analytics.
Query optimization: time-based partitioning, sharding, and caching.
Trade-offs: cost vs. query speed, consistency vs. availability, and operational complexity.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.