Start by clarifying requirements and scale, then walk through the system architecture from job submission to scheduling, execution, and monitoring. Emphasize trade-offs in scheduling algorithms, failure handling, and observability, and conclude with how you'd validate and iterate on the design.
Pro tip: Anchor your design around Kubernetes primitives (CRDs, controllers, scheduler framework) to show domain expertise, and explicitly discuss how you'd handle multi-tenancy and resource fairness—common pain points in production schedulers.
Ask about scale (jobs/sec, cluster size), job types (batch, long-running), resource types (CPU, GPU, memory), and SLAs. Establish whether this is an extension of Kubernetes or a standalone system.
Define a REST/gRPC API for job submission, validation, and status queries. Use a CRD-based approach for Kubernetes-native integration, with admission controllers for policy enforcement.
Describe a two-phase scheduling (filtering and scoring) with pluggable policies. Discuss resource constraints, affinity/anti-affinity, priority, preemption, and fairness (e.g., DRF, quotas).
Cover retries, idempotency, job checkpointing, node failures, and scheduler high availability. Explain how to detect and recover from partial failures without data loss.
Define key metrics (job latency, throughput, resource utilization, queue depth, failure rates). Use Prometheus for collection, Grafana for dashboards, and Alertmanager for alerting with SLO-based thresholds.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.