← Bank of America Interview Insights

Bank of America·Machine Learning Engineer·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

ML engineer interview question about RAG systems, specifically around building user trust. One question, pretty deep technically, covered everything from calibration to adversarial testing.

Questions Asked (1)

Q1

How would you improve user confidence in a RAG-based system, both from a design perspective and in terms of actual engineering implementation?

System DesignTechnical Trade-offsProduct Analytics & Metrics
Author's notes

This one sprawled in a direction I didn't expect.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around the two dimensions the question asks for: design (UX, transparency, trust signals) and engineering (retrieval quality, grounding, evaluation). Use a layered trust framework—from data ingestion to user interface—and tie each layer to measurable confidence metrics. Emphasize that in a regulated banking context, confidence is not just UX but also auditability and compliance.

Pro tip: Anchor your answer in measurable outcomes: propose A/B tests or offline metrics (e.g., citation accuracy, hallucination rate) to quantify confidence improvements, and mention how you'd instrument the system to monitor trust over time. This shows you think like a product-minded ML engineer, not just a model builder.

1. Clarify what 'confidence' means

Define user confidence as a combination of answer accuracy, transparency, and consistency. Distinguish between model confidence (e.g., logprobs) and user-perceived confidence, and note that both matter.

2. Design-level improvements

Propose UI/UX features like inline citations, confidence indicators, source previews, and fallback messages when the system is unsure. Emphasize transparency and user control (e.g., ability to see retrieved documents).

3. Engineering-level improvements

Discuss retrieval enhancements (hybrid search, re-ranking, query expansion), grounding techniques (constrained generation, citation enforcement), and hallucination mitigation (self-check, verification).

4. Evaluation and monitoring

Outline how to measure confidence: offline metrics (faithfulness, answer relevance, citation precision) and online metrics (user feedback, task success, escalation rate). Propose continuous monitoring and alerting.

5. Iterate with feedback loops

Describe how to close the loop: use user feedback to fine-tune retrieval and generation, and A/B test design changes. Highlight the importance of governance and compliance in a bank.

Key Points to Mention

  • Retrieval quality: hybrid search (BM25 + embeddings), re-ranking, and query understanding to ensure relevant context.
  • Grounding and citation: enforce that answers are supported by retrieved sources, with inline citations and confidence scores.
  • Hallucination mitigation: techniques like self-consistency, verification against sources, and abstention when uncertain.
  • Transparency and UX: show sources, allow users to inspect retrieved documents, and provide clear uncertainty indicators.
  • Evaluation metrics: faithfulness, answer relevance, citation accuracy, and user trust surveys; A/B testing for design changes.
  • Regulatory and compliance considerations: audit trails, explainability, and alignment with banking regulations (e.g., fair lending, data privacy).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.