I started with the API surface and database schema which felt safe, but the interviewer kept pushing toward the delivery pipeline.
Start by clarifying requirements and scale (1B events/day ≈ 11.6K events/sec average, with peaks). Then propose a decoupled, partitioned architecture using a message queue for buffering and worker pools for delivery, with per-endpoint isolation and retry queues. Finally, discuss trade-offs around delivery guarantees, retry strategies, and failure handling.
Pro tip: Emphasize idempotency and deduplication on the consumer side, since at-least-once delivery means duplicates are inevitable. Also, mention that you'd use a dead-letter queue for poison messages and monitor retry rates to detect systemic issues.
Ask about event size, peak-to-average ratio, latency expectations, and endpoint diversity. Calculate throughput (e.g., 1B/day ≈ 11.6K/sec average, but peaks could be 10x).
Propose an ingestion layer (API/gateway) that writes events to a durable, partitioned log (e.g., Kafka). Then a delivery service consumes from the log, groups by endpoint, and dispatches via HTTP with retries.
Explain how to achieve at-least-once: persist events before acknowledging, use a retry queue with exponential backoff and jitter, and track delivery attempts. Ensure idempotency keys to handle duplicates.
Isolate failures per endpoint using separate queues/partitions and circuit breakers. Scale horizontally by adding consumers; use sharding to distribute load. Implement dead-letter queues for persistent failures.
Discuss monitoring (retry rates, latency, queue depth), and trade-offs like at-least-once vs exactly-once, synchronous vs asynchronous delivery, and cost vs reliability.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying requirements such as event types, delivery guarantees, and expected query patterns. Then design the REST endpoints for webhook registration and delivery history, focusing on resource modeling, filtering, and pagination. Finally, discuss trade-offs and operational considerations like idempotency, security, and scalability.
Pro tip: Emphasize idempotency and security from the start—use idempotency keys for registration and HMAC signatures for payloads. Also, mention cursor-based pagination for delivery history to ensure stable results under high write loads.
Ask about event types, delivery guarantees (at-least-once vs. exactly-once), expected volume, and query patterns (e.g., filtering by status, date range).
Define POST /webhooks for creating a webhook with URL, secret, and event subscriptions. Include GET /webhooks/{id}, PUT/PATCH for updates, and DELETE for removal.
Define GET /webhooks/{id}/deliveries with query parameters for filtering (status, event type, date range) and pagination (cursor-based).
Use query parameters like ?status=failed&event_type=user.created&start_date=...&end_date=...&limit=50&cursor=... for flexible querying.
Discuss idempotency, security (HMAC signatures), rate limiting, retry policies, and monitoring for webhook deliveries.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The unique constraint on user_id and event_id was easy.
Start by outlining the core tables (webhooks and delivery records) and their relationships, then explain how you enforce the one-event-per-user constraint using a unique index on (user_id, event_id). Finally, discuss sharding strategies such as sharding by user_id or event_id, and how you handle cross-shard queries and rebalancing.
Pro tip: Mention that you would use a composite unique index on (user_id, event_id) to enforce the constraint at the database level, and consider using a distributed SQL database like CockroachDB or Vitess for automatic sharding to avoid manual rebalancing complexity.
Describe the webhooks table (id, user_id, event_type, payload, created_at) and delivery_records table (id, webhook_id, status, attempt_count, last_attempt_at, response_code). Explain the one-to-many relationship.
Explain that a unique constraint on (user_id, event_id) in the webhooks table ensures only one webhook per event per user. Mention handling duplicate inserts with upsert or ignoring conflicts.
Discuss sharding by user_id to keep a user's data together, or by event_id for even distribution. Explain trade-offs: user_id simplifies per-user queries but may cause hotspots; event_id distributes load but complicates per-user queries.
Describe how to handle queries that span shards, such as fetching all deliveries for a user. Mention using a global index, scatter-gather, or a separate lookup table.
Explain how to add shards (consistent hashing, range-based sharding) and rebalance data. Mention monitoring shard size and using tools like Vitess or Citus for automation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Cache the webhook config keyed by event_id, TTL of a few minutes, invalidate on write.
Start by clarifying the system's read/write patterns and consistency requirements, then identify cacheable entities like webhook configurations and event payloads. Explain your caching strategy with appropriate TTLs and invalidation mechanisms, emphasizing how you handle updates and deletions to maintain consistency.
Pro tip: Mention that cache invalidation should be event-driven, using the webhook update/delete events themselves to trigger invalidation, and consider using a versioned cache key to avoid stale reads during concurrent updates.
Ask about read/write ratio, consistency needs, and latency requirements to determine what to cache and for how long.
List entities such as webhook configurations, event payloads, and authentication tokens that are frequently read and infrequently changed.
Decide between write-through, write-behind, or cache-aside, and set TTLs based on data volatility and consistency requirements.
Use event-driven invalidation: when a webhook is updated or deleted, publish an event to invalidate or update the cache entry, possibly using a versioned key.
Address race conditions, cache stampedes, and failure modes like cache unavailability, ensuring graceful degradation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This was the real meat of the interview and where I spent the most time.
Structure your answer around a reliable, scalable webhook delivery system: start by acknowledging the need for retries with exponential backoff and jitter, then explain how to classify errors (transient vs. permanent) to decide retry eligibility, and finally describe an asynchronous, decoupled architecture (e.g., message queues with per-subscriber isolation) to prevent slow subscribers from blocking others. Emphasize trade-offs like retry limits, dead-letter queues, and monitoring.
Pro tip: Show maturity by discussing idempotency and delivery guarantees—mention that retries can cause duplicate deliveries, so you'd include a unique event ID and encourage subscribers to deduplicate. Also, highlight the importance of observability (metrics, logs, alerts) to detect systemic issues and tune retry policies.
Ask about expected delivery guarantees (at-least-once vs. exactly-once), scale (events per second, number of subscribers), and latency tolerance. This shows you tailor solutions to context.
Propose exponential backoff with jitter to avoid thundering herd, set a maximum retry count or time window, and use a dead-letter queue for persistent failures. Mention that retries should be asynchronous.
Distinguish transient errors (e.g., 5xx, timeouts, network issues) from permanent ones (e.g., 4xx like 400, 401, 404). Only retry transient errors; for permanent errors, log and alert without retrying.
Use a message queue per subscriber or a worker pool with concurrency limits, so a slow subscriber doesn't consume shared resources. Implement circuit breakers to pause delivery to failing subscribers.
Include a unique event ID in payloads and document that subscribers should deduplicate. Add metrics (success/failure rates, retry counts), logging, and alerting to monitor system health and adjust policies.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
SSRF came up and I gave the standard answer about blocklisting internal IP ranges and validating URLs before dispatch.
Start by outlining a defense-in-depth strategy for SSRF: validate and resolve URLs, enforce network-level restrictions, and monitor for anomalies. Then explain how to implement payload signing using HMAC with a shared secret, including timestamping and signature verification on the subscriber side.
Pro tip: Emphasize that SSRF prevention is not just about input validation but also about network segmentation and egress filtering, and that payload signing should include a timestamp to prevent replay attacks.
Ensure the URL uses HTTPS, has a valid domain, and does not point to internal IPs or metadata endpoints. Use a whitelist of allowed domains if possible.
Resolve the hostname to an IP and check it against a blocklist of private, loopback, link-local, and reserved IP ranges. Re-resolve after redirects to prevent DNS rebinding.
Use a dedicated egress proxy or firewall to restrict outbound requests to only necessary ports and external IPs. Isolate the service making requests to limit blast radius.
Generate a signature using HMAC-SHA256 over the payload and a timestamp, using a shared secret unique to each subscriber. Include the signature and timestamp in headers.
Document how subscribers should verify the signature and reject requests with invalid signatures or expired timestamps. Log and alert on verification failures.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.