Start by clarifying requirements and scale constraints, then walk through each layer of the pipeline (ingestion, storage, processing, querying) systematically while explicitly calling out trade-offs at each decision point. Anchor every design choice to the 1M+ events/sec scale requirement and Amazon's operational context, demonstrating awareness of cost, reliability, and compliance concerns throughout.
Pro tip: Amazon interviewers reward candidates who proactively surface operational concerns like cost attribution, data skew in partitions, and compaction strategies — mentioning tools like AWS Glue for schema registry or Apache Iceberg as a table format upgrade path over raw Parquet signals real-world production experience beyond textbook designs.
Confirm the event volume (1M+ events/sec), latency SLAs (real-time dashboards vs. batch reports), retention windows, and compliance requirements (GDPR/CCPA for PII). Establish whether the primary use case is operational monitoring, billing, or ML feature generation, as this drives storage and query design.
Propose a Kafka cluster with topic partitioning keyed on advertiser_id or campaign_id to ensure ordering and enable parallel consumption, sizing partitions to handle 1M events/sec with appropriate replication factor (3) for fault tolerance. Introduce a schema registry (e.g., AWS Glue Schema Registry with Avro/Protobuf) to enforce schema contracts at the producer level and enable safe schema evolution.
Organize S3 data in a Hive-compatible partition hierarchy such as s3://bucket/clickstream/year=YYYY/month=MM/day=DD/hour=HH/ using columnar Parquet format with Snappy compression to optimize Presto scan efficiency. Use a compaction job (Spark or AWS Glue) to merge small files from streaming writes into larger files (128MB–1GB), preventing the 'small file problem' that degrades Presto query performance.
Mask or tokenize PII fields (user IDs, IPs) at the Kafka consumer layer before writing to S3, storing raw PII only in an access-controlled encrypted S3 prefix with short retention. For schema evolution, use backward/forward-compatible Avro schemas with the registry enforcing compatibility rules, and control costs via S3 Intelligent-Tiering, partition pruning in queries, and result caching in Presto.
Optimize Presto queries by leveraging partition pruning, predicate pushdown, and ORC/Parquet statistics, and consider materialized aggregations (e.g., hourly rollup tables) for common dashboard queries. Explicitly discuss the Lambda/Kappa architecture trade-off: a real-time path (Kafka → Flink → serving store like DynamoDB) for sub-minute latency vs. a batch path (S3 → Presto) for historical accuracy and cost efficiency, recommending a hybrid approach based on SLA tiers.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.