Plaid·Machine Learning Engineer·Onsite - System Design / Architecture
May 2026
Plaid ML engineer interview focused entirely on designing an evaluation and debugging framework for a production RAG semantic search system in fintech. Five-part deep dive, very systems-thinking heavy, less about coding and more about whether you can reason about quality measurement end to end.
- You own a production RAG-based semantic search feature for a fintech product. Design a lightweight but production-ready evaluation and debugging framework covering retrieval quality, answer quality, faithfulness, latency, and cost as offline metrics. What do you measure at each stage and what thresholds would you hold?
- What online signals would you track for this system, and how would you run a controlled experiment for a retriever or prompt change without contaminating results?
- Where does your benchmark data come from, how do you create labels, and how do you keep the eval dataset fresh without breaking historical comparisons?
- How do you integrate evaluation into CI/CD, what are your model-release gates, and what would you alert on in production dashboards?
- Search relevance suddenly drops in production. Walk through exactly how you'd localize the failure to a specific pipeline stage, confirm the root cause, apply a fix, and prove the fix actually worked.
- Your LLM-as-judge for faithfulness is itself a model. How do you evaluate and trust the judge, and what do you do when the judge disagrees with human labels?
- A new retriever improves your offline ranking metric but loses the online A/B test on answer-acceptance. How do you reconcile that, and which signal do you believe?
- A regulator asks you to prove that a specific user could not have retrieved a document they weren't entitled to. What in your logging and eval system lets you answer that, and how do you proactively test for permission leaks?
- The corpus churns 10% daily but is 90% stable week over week. How do you keep retrieval quality stable across re-indexing, and how would you detect a silent indexing failure on the churning slice?
“This was the core of the whole interview and I spent probably too long on the retrieval side before getting to faithfulness.” The rest of the author's notes on Machine Learning Engineer interview at Plaid, Onsite - System Design / Architecture round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post