Oracle·Software Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

System design round at Oracle for a software engineer role. One big question, lots of follow-ups, and I spent probably too long on the wrong half of it.

Questions Asked (1)

You have 1,000,000 VMs. Design a system that collects the health status of all of them within a 5-minute window. Walk through two approaches: a central monitor polling each VM versus each VM pushing a heartbeat every 10 seconds. What breaks in each, and how do you scale the collection layer?

System DesignTechnical Trade-offs

Author's notes

I jumped straight into the push vs pull comparison which felt natural, but I underestimated how much they wanted me to stress-test the numbers.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements (e.g., health status definition, data retention, failure handling) and then compare the two approaches using back-of-the-envelope calculations for load on the collection layer. For each approach, identify bottlenecks and failure modes, and propose scaling strategies such as sharding, hierarchical aggregation, and load balancing.

Pro tip: Quantify the load: 1M VMs polling every 10 seconds means ~100K requests per second; pushing heartbeats means ~100K messages per second. Show how you'd handle that scale with partitioning and backpressure.

1. Clarify requirements and assumptions

Ask about health status definition, data granularity, retention, and acceptable latency. Assume 1M VMs, 5-minute window, and that each health check is lightweight.

2. Analyze central polling approach

Calculate load: 1M VMs / 300s = ~3,333 polls per second if spread evenly, but polling all within 5 minutes may require bursts. Identify bottlenecks: central monitor becomes a single point of failure and network bottleneck; scaling requires sharding VMs across multiple pollers with a coordinator.

3. Analyze push heartbeat approach

Calculate load: 1M VMs / 10s = 100,000 heartbeats per second. Identify bottlenecks: ingestion layer must handle high throughput; message queue or load balancer may become bottleneck; need partitioning by VM ID and horizontal scaling of collectors.

4. Compare trade-offs and failure modes

Polling: easier to control, but delayed detection and scalability challenges. Push: near real-time, but thundering herd and need for backpressure. Discuss failure handling: missed heartbeats, false positives, and how to detect VM failures.

5. Design scalable collection layer

Propose a multi-tier architecture: for push, use a distributed message queue (e.g., Kafka) with partitioned topics and consumer groups; for polling, use a sharded set of pollers with consistent hashing and a coordination service (e.g., ZooKeeper). Include monitoring and auto-scaling.

Key Points to Mention

Back-of-the-envelope calculations for load (QPS, network bandwidth)
Sharding/partitioning strategies (by VM ID, region, etc.)
Use of message queues (e.g., Kafka) for decoupling and buffering
Load balancing and auto-scaling of collectors
Failure detection and handling (timeouts, retries, dead letter queues)
Trade-offs between polling and push (latency, complexity, scalability)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.