Freddie Mac·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Apr 2026

Summary

System design round at Freddie Mac for an ML Engineer role, focused entirely on a single massive RAG question covering basically every dimension of a production AI system. No coding, no behavioral stuff, just this one sprawling prompt and however long you could hold the thread.

Questions Asked (1)

You run an enterprise RAG system for high-stakes document QA like mortgage underwriting or legal review. Users don't trust the answers because they can't verify sources, gauge uncertainty, or know when the model might hallucinate. How do you redesign the system to increase user confidence? Cover the full stack: product UX, retrieval and generation architecture, confidence scoring, calibration, fallback behavior, human feedback loops, monitoring, privacy, compliance, and adversarial testing.

System DesignTechnical Trade-offsProduct Sense & Ideation

Author's notes

This question is essentially a whole system design interview compressed into one prompt.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the redesign around a trust-first architecture that treats verifiability, calibrated uncertainty, and graceful fallback as first-class system properties rather than afterthoughts. Walk through the full stack in layers—UX, retrieval/generation, confidence/calibration, human-in-the-loop, and governance—tying each to concrete mechanisms like span-level citations, conformal prediction, and abstention policies. Anchor everything in the high-stakes regulatory context (e.g., Freddie Mac's model risk management and fair lending requirements).

Pro tip: Emphasize that confidence must be calibrated and actionable—users need to know not just 'how sure' the model is, but what to do when it's unsure (abstain, escalate, or request more context). Mention that you'd validate calibration with reliability diagrams and expected calibration error (ECE) on held-out, domain-shifted data, and tie abstention thresholds to business cost asymmetries (false confident answer vs. unnecessary escalation).

1. Design for verifiability and transparency in the UX

Make every answer traceable: inline citations to source spans, highlighted evidence, and a 'show your work' view. Surface uncertainty visually (e.g., confidence bands, color-coded risk) and provide clear affordances to inspect, challenge, or escalate.

2. Harden retrieval and generation for grounded, attributable answers

Use hybrid retrieval (dense + sparse) with reranking, enforce strict grounding via constrained decoding or post-hoc attribution, and require span-level citations. Implement abstention when evidence is weak or conflicting.

3. Implement calibrated confidence scoring and fallback policies

Train a separate calibration model (e.g., temperature scaling, conformal prediction) on domain data to produce reliable confidence scores. Define tiered fallback: high confidence → auto-answer; medium → answer with warning; low → abstain and route to human or request clarification.

4. Build human feedback loops and continuous monitoring

Capture user corrections, ratings, and escalation outcomes to fine-tune retrieval, generation, and calibration. Monitor drift, hallucination rates, citation accuracy, and calibration metrics in production with alerts and dashboards.

5. Embed privacy, compliance, and adversarial testing throughout

Apply PII redaction, access controls, and audit logging; align with regulations (e.g., ECOA, FCRA, GDPR). Red-team with adversarial queries, prompt injections, and distribution shifts to stress-test robustness and fairness before deployment.

Key Points to Mention

Span-level citations and evidence highlighting to enable source verification
Calibrated confidence scores (e.g., conformal prediction, temperature scaling) with reliability diagrams and ECE
Tiered fallback and abstention policies tied to business cost asymmetries
Human-in-the-loop feedback for continuous improvement and escalation workflows
Production monitoring for hallucination, drift, citation accuracy, and calibration
Privacy/compliance controls (PII redaction, audit logs, model risk management) and adversarial red-teaming

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.