Summary
Salesforce system design round focused entirely on building out a recommendation service end to end. Pretty dense session covering everything from API shape to storage schemas to how batch training and online serving share the same infrastructure without stepping on each other.
Questions Asked(5)
Started with the API contract and worked outward, which felt right.
Suggested Approach
Start by clarifying requirements and scale expectations, then systematically walk through the serving stack from client to backend — covering API design, load balancing, and stateless server architecture. Ground your design decisions in concrete trade-offs rather than listing buzzwords, showing you understand why each component exists.
Clarify Requirements & Scale
Ask about expected QPS, latency SLAs (e.g., p99 < 100ms), number of users, and whether recommendations are real-time or pre-computed. Establish whether the client is a web app, mobile app, or internal service, as this shapes API and caching strategy.
Design the Client-Facing API
Define a clean REST or gRPC API (e.g., GET /v1/recommendations?userId=&context=&limit=) with versioning, pagination, and clear request/response contracts. Discuss authentication (OAuth/JWT), rate limiting headers, and how clients should handle partial failures or fallback content.
Load Balancing & Traffic Distribution
Describe a multi-layer load balancing approach: a global DNS/anycast layer (e.g., AWS Route 53 or Cloudflare) for geo-routing, followed by an L7 application load balancer (e.g., NGINX, AWS ALB) for path-based routing and health checks. Explain why stateless servers enable simple round-robin or least-connections algorithms without sticky sessions.
Stateless Application Server Design
Explain that application servers hold no user session state — all context comes from the request or is fetched from shared stores (Redis for sessions, a feature store for user embeddings). This allows horizontal auto-scaling based on CPU/latency metrics and seamless rolling deployments with zero downtime.
Caching, Resilience & Observability
Layer in a distributed cache (Redis or Memcached) to serve pre-computed recommendations for high-traffic users, and define a TTL strategy balancing freshness vs. load. Address resilience with circuit breakers, fallback recommendations (e.g., trending items), and observability via distributed tracing, latency histograms, and cache hit-rate dashboards.
Key Points to Mention
This part actually went well.
Suggested Approach
Start by identifying the core entities and their relationships, then design each schema with scalability and query patterns in mind. Explicitly discuss trade-offs between normalization and denormalization, and tie your decisions back to real-world access patterns like lookups by user, item, or time range.
Define Core Entities and Relationships
Identify the three primary entities — Users, Items, and Interaction Logs — and articulate how they relate (one user has many interactions, one item has many interactions). Clarify cardinality and whether the relationship is many-to-many.
Design the Users Schema
Define the Users table with fields like user_id (UUID or auto-increment primary key), profile attributes (name, email, created_at, updated_at), and any segmentation metadata. Discuss indexing on user_id and potentially email for fast lookups.
Design the Items Schema
Define the Items table with item_id as the primary key, descriptive fields (title, category, metadata as JSON), and timestamps. Discuss whether item metadata should be stored inline or in a separate attribute table depending on variability.
Design the Interaction Logs Schema
Design the Interaction Logs table with fields: interaction_id, user_id (FK), item_id (FK), signal_type (enum: click, purchase, view, like), timestamp, and optional context fields (session_id, device_type). Discuss partitioning by timestamp or user_id and choosing between RDBMS vs. columnar/NoSQL stores for high write throughput.
Address Indexing, Partitioning, and Trade-offs
Discuss composite indexes (e.g., user_id + timestamp for user history queries, item_id + signal_type for item analytics) and trade-offs like write amplification vs. read speed. Mention archival strategies for old interaction logs and whether a time-series database or data warehouse (e.g., BigQuery, Redshift) is more appropriate at scale.
Key Points to Mention
This is where it got tricky.
Suggested Approach
Start by distinguishing the two access patterns — batch training jobs that favor high-throughput sequential scans and online serving that demands low-latency point lookups and writes — then design a layered storage and indexing strategy that serves both without compromise. Propose a Lambda or Kappa-style architecture where a hot path (e.g., Kafka + a low-latency store like Cassandra or DynamoDB) handles real-time reads/writes, while a cold path (e.g., partitioned Parquet on S3 or Delta Lake) serves batch training efficiently.
Clarify Access Patterns & SLAs
Define the read/write patterns for each workload: batch jobs need full table scans over time ranges with high throughput, while online serving needs sub-10ms point reads by user/session ID and high-write throughput for logging interactions. Nail down data volume, retention period, and acceptable latency before proposing any design.
Design the Partitioning Scheme
Propose a composite partitioning strategy — partition by tenant/org ID first (for multi-tenancy isolation and query pruning), then by time (e.g., date or hour) to support efficient time-range scans in batch jobs and TTL-based data expiry. Explain how this avoids hot partitions and enables parallel batch processing.
Choose Storage Layers & Indexing
Recommend a dual-store approach: a low-latency store (Cassandra or DynamoDB) with a primary key of (tenant_id, user_id, timestamp) for online serving, and a columnar format (Parquet/ORC on S3 or Delta Lake) with partition pruning for batch training. Add secondary indexes or materialized views sparingly, only where query patterns justify the write overhead.
Address Write Path & Consistency
Describe how interaction logs are written — likely via a message queue (Kafka) that fans out to both the hot store for serving and the cold store for batch, ensuring durability without blocking the write path. Discuss consistency trade-offs: eventual consistency is acceptable for training data, but serving reads may need read-your-writes guarantees.
Discuss Trade-offs & Operational Concerns
Acknowledge trade-offs such as storage cost duplication across hot and cold layers, index maintenance overhead, and compaction strategies for the columnar store. Mention monitoring partition skew, handling schema evolution (e.g., adding new interaction types), and data retention/archival policies to show operational maturity.
Key Points to Mention
Went with a TTL-based cache in front of the serving layer.
Suggested Approach
Start by acknowledging that caching in a recommendation system is a multi-layered problem, then walk through where caches live (user-level, item-level, model output), and explicitly discuss the freshness vs. latency tension with concrete strategies. Ground your answer in real trade-offs rather than abstract theory, showing you understand that stale recommendations can hurt user experience while cold computation hurts performance.
Define What Gets Cached
Identify the distinct cacheable artifacts: pre-computed recommendation lists per user, item feature vectors, model embeddings, and collaborative filtering scores. Clarify that different artifacts have different staleness tolerances — item metadata changes rarely, while user behavior signals change frequently.
Describe the Caching Architecture
Outline a tiered approach: an L1 in-process cache for ultra-low latency on hot users, an L2 distributed cache (e.g., Redis or Memcached) for shared access across service instances, and an optional CDN layer for anonymous or segment-level recommendations. Explain how TTLs differ per tier.
Address Cache Invalidation Strategies
Discuss TTL-based expiration, event-driven invalidation (e.g., invalidate a user's cache when they make a purchase or explicit rating), and write-through vs. write-behind patterns. Acknowledge that event-driven invalidation is more accurate but adds system complexity.
Quantify the Freshness vs. Latency Trade-off
Explicitly frame the trade-off: shorter TTLs improve recommendation relevance but increase cache miss rates and backend load, while longer TTLs reduce latency and cost but risk serving outdated or irrelevant recommendations. Propose using user activity signals to dynamically adjust TTLs — highly active users get shorter TTLs.
Discuss Failure Modes and Mitigations
Cover cache stampede (many simultaneous misses on expiry) and how to mitigate it with probabilistic early expiration or a mutex/lock pattern. Also mention graceful degradation — if the cache is unavailable, fall back to a simpler heuristic (e.g., popularity-based recommendations) rather than blocking the request.
Key Points to Mention
Blanked for a second on the consistency angle specifically.
Suggested Approach
Frame your answer around the core tension between batch training (high throughput, periodic) and online serving (low latency, continuous), and explain how you would architect the system to isolate failures and maintain data consistency across both pipelines. Use concrete patterns like blue-green deployments, versioned model artifacts, and distributed transaction strategies to demonstrate depth. Conclude by acknowledging trade-offs, showing you understand that perfect consistency often conflicts with availability and performance.
Define the Failure Domains
Start by identifying the distinct failure domains: the batch training pipeline, the model registry/artifact store, the feature store, and the online serving layer. Clearly separating these domains prevents cascading failures and makes fault isolation tractable.
Address Model Versioning & Safe Deployment
Explain how versioned model artifacts and blue-green or canary deployments allow new models from batch training to be promoted to serving without downtime or inconsistency. This ensures the serving layer always references a stable, validated model version while a new one is being tested.
Ensure Feature Consistency
Describe how a unified feature store (e.g., Feast or an internal equivalent) with point-in-time correct lookups prevents training-serving skew and ensures that features used during batch training match those available at inference time. Discuss read/write isolation so batch writes do not corrupt live serving reads.
Handle Distributed Consistency & Idempotency
Discuss using idempotent operations, distributed locks or optimistic concurrency control, and event-driven architectures (e.g., Kafka) with at-least-once delivery plus deduplication to handle partial failures gracefully. Mention checkpointing in batch jobs so they can resume without reprocessing data from scratch.
Observability, Circuit Breakers & Fallbacks
Explain how real-time monitoring, alerting, and circuit breakers allow the serving layer to fall back to a previous stable model or a rule-based default if a newly deployed model degrades. Emphasize that fault tolerance is incomplete without the ability to detect and recover from failures automatically.
Key Points to Mention
Discussion(5)
Sign in to join the discussion.
Anchoring a latency budget at the start is one of those things that feels like throat-clearing until you realize it actually structures the whole conversation. I made the same mistake once, spent ten minutes on API shape before anyone had agreed on what "fast enough" meant, and then the interviewer started asking questions that implicitly assumed sub-100ms p99 when I'd been designing for something looser. For a recommendation service at Salesforce scale you're probably talking 150-200ms p99 end to end as a reasonable opening stake, which immediately tells you whether you need a cache layer, whether you can afford a synchronous model call, and how many fallback tiers you need before you just return popular items. The stateless server approach you described is correct and the interviewer likely knew that, so the push on latency was probably their way of seeing if you'd made conscious tradeoffs or just listed components. One thing worth adding if you revisit this: mention consistent hashing at the load balancer layer if you're planning any request-level caching on the app servers, because naive round-robin kills your cache hit rate when the same user's requests scatter across ten boxes.
Tying the freshness tolerance back to domain specifics was the right move. A lot of candidates give the generic TTL tradeoff answer and leave it floating in the abstract. The out-of-stock invalidation hook is a good concrete edge case because it shows you've thought about what makes recommendations wrong in a way that actually matters to the business, not just wrong in a theoretical consistency sense.
The versioned snapshot framing for feature stores is solid and probably saved you on this one. The failure mode I'd have wanted to explore more, if time allowed, is what happens when the batch job produces a bad model and you need to roll back while serving is already using the new snapshot. Circuit breakers on the serving path help with downstream failures but they don't help you when the model itself is the problem. Shadow deployments or canary rollouts at the model layer, where you serve the new model to a small traffic slice before full promotion, are the cleaner answer to concurrent batch-and-serve risk. The consistency question for recommendations is almost always "eventual is fine" but the interesting follow-up is how you detect when eventual has drifted far enough to cause real quality degradation, and whether your monitoring catches that before users do.
The two-store answer you landed on is basically the right architecture, and I'd push back on any uncertainty about whether you convinced them. The reasoning is sound because the access patterns are genuinely incompatible at scale: a columnar store or data warehouse handles sequential batch scans well, a row-oriented store or feature store handles point lookups, and trying to serve both from one physical table means you're constantly tuning indexes that fight each other. The part I'd have added is the replication lag question, because the interviewer's "same physical table" probe was probably fishing for that. If your write-optimized store feeds a read replica or a feature store via CDC or scheduled export, you need to be explicit about how stale that read path can get before a training job produces a model that's effectively trained on yesterday's data but served against today's user state. For batch training that lag is usually fine. For online feature freshness it might not be, depending on how fast user behavior shifts. Separating the stores also means your batch jobs can do full table scans without taking out locks or burning IOPS that the serving path needs, which is the practical reason this pattern exists beyond just theoretical access pattern purity.
The enum point is underrated and you were right to lean into it. Free-text signal fields become a nightmare the moment two teams log the same event with slightly different strings and your aggregation pipeline silently splits what should be one cohort.