Illumio·Software Engineer·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Kafka deep-dive at Illumio for a software engineer role. The whole session was basically one long scenario question about message processing failures and offset management. Pretty intense if you haven't thought carefully about consumer semantics before.

Questions Asked (1)

A Kafka consumer is processing messages from a partition. Offset 2 fails due to a handler exception, but offset 4 gets processed and its offset is committed. How do you recover and reprocess offset 2 without losing data or reprocessing offsets you've already committed successfully?

System DesignTechnical Trade-offs

Author's notes

This one took me a minute to even fully parse.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, explain the problem: committing offset 4 after offset 2 failed creates a gap, so offset 2 is effectively lost from the consumer's perspective. Then propose a recovery strategy that involves either rewinding the consumer to offset 2 and reprocessing offsets 2-4 (with idempotent handling to avoid duplicate side effects), or using a dead-letter queue to isolate offset 2 while continuing from offset 5. Finally, discuss trade-offs between these approaches and how to prevent similar issues in the future.

Pro tip: Emphasize that exactly-once semantics require idempotent processing or transactional writes; simply rewinding and reprocessing can cause duplicates if offsets 3 and 4 have side effects. Mention that Kafka's consumer API allows seeking to a specific offset, but you must pause the consumer and handle rebalancing carefully.

1. Acknowledge the offset gap and its implications

Explain that committing offset 4 means the consumer will resume from offset 5, so offset 2 is skipped. This creates a data loss scenario unless the failed message is handled separately.

2. Choose a recovery strategy

Decide between rewinding the consumer to offset 2 and reprocessing offsets 2-4 (with idempotency), or routing offset 2 to a dead-letter queue for later manual/automated reprocessing while continuing from offset 5.

3. Implement idempotency or transactional guarantees

If rewinding, ensure that processing offsets 3 and 4 again does not cause duplicate side effects (e.g., use idempotent writes, deduplication keys, or Kafka transactions). If using a DLQ, ensure the DLQ consumer can reprocess offset 2 without affecting the main flow.

4. Handle consumer group rebalancing and offset management

When seeking to a specific offset, pause the consumer, seek, and resume carefully to avoid rebalance issues. Consider committing offsets only after successful processing of all messages up to a point (e.g., using a batch commit after offset 4 is processed).

5. Discuss prevention and monitoring

Suggest improvements like processing messages in order with manual offset commits after each successful message, using a retry topic with delays, or implementing a dead-letter queue pattern to isolate failures without blocking the pipeline.

Key Points to Mention

Kafka consumer offset commit semantics: committing offset 4 implies offsets 0-4 are considered processed, so offset 2 is skipped.
Idempotent processing: ensuring that reprocessing offsets 3 and 4 does not cause duplicate side effects (e.g., using unique message IDs or database upserts).
Consumer seek() API: ability to rewind to a specific offset, but must pause the consumer and handle potential rebalances.
Dead-letter queue (DLQ) pattern: route failed messages to a separate topic for later reprocessing, allowing the main consumer to continue.
Transactional processing: using Kafka transactions to atomically write processed results and commit offsets, ensuring exactly-once semantics.
Trade-offs: rewinding may cause duplicates and block the pipeline; DLQ adds complexity but isolates failures and maintains throughput.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.