← American Express Interview Insights
I explained direct as a user crafting malicious input themselves vs indirect as the attack arriving through retrieved content or tool output.
Start by clearly defining direct and indirect prompt injection, emphasizing the source of the malicious input (user vs. external data). Then explain how the distinction affects threat modeling, focusing on trust boundaries, attack surface, and mitigation strategies. Use a concrete example from a financial services context to illustrate the risks.
Pro tip: Highlight that indirect prompt injection is often more dangerous in production systems because it can bypass user-facing input filters and exploit trusted data sources, making it a critical consideration for AI systems that ingest external content.
Explain that direct prompt injection occurs when a user intentionally provides malicious input directly to the model, attempting to override instructions or extract sensitive information.
Describe indirect prompt injection as when malicious content is embedded in external data sources (e.g., documents, emails, web pages) that the model processes, leading to unintended actions.
Compare how direct injection exploits the user-model interface, while indirect injection exploits the model's trust in external data, crossing different trust boundaries.
Discuss why the distinction matters: direct injection is often mitigated by input sanitization and user authentication, whereas indirect injection requires securing data pipelines, validating external content, and limiting model permissions.
Give a concrete example, such as an AI assistant summarizing customer emails that contain hidden instructions to exfiltrate data, to illustrate the real-world impact in a financial institution.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered all four but my answer on privilege escalation was thin.
Structure your answer around the four threat categories, explaining each with a concrete example and its impact on an LLM agent system. Then discuss how these threats interrelate and propose layered mitigations, emphasizing defense-in-depth and secure design principles relevant to a financial institution like American Express.
Pro tip: Tie each threat to a real-world consequence in financial services—such as unauthorized transactions or data breaches—and mention how you would validate mitigations through red-teaming and continuous monitoring, showing you think beyond just model-level fixes.
Briefly outline the components of an LLM agent system (model, tools, memory, external APIs) and how each expands the attack surface. This sets the stage for categorizing threats.
Describe how an agent might leak sensitive data through prompts, tool outputs, or side channels, and give an example like extracting customer PII via crafted queries. Mention mitigation such as output filtering and data access controls.
Discuss how an attacker could trick the agent into performing actions beyond its intended permissions, such as accessing admin tools or escalating via tool chaining. Highlight the need for least-privilege design and runtime authorization checks.
Cover jailbreaks as bypassing safety guardrails via adversarial prompts, and tool misuse as abusing legitimate tools for malicious ends (e.g., sending unauthorized emails). Emphasize input sanitization, tool allowlisting, and behavioral monitoring.
Summarize a defense-in-depth strategy combining model alignment, input/output validation, sandboxing, and human oversight. Discuss trade-offs between security, usability, and latency, showing balanced engineering judgment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer by first categorizing prompt injection defenses into layers (input, model, output, and system-level), then for each layer describe the mitigation and its limitations. Emphasize a defense-in-depth strategy, acknowledging that no single solution is foolproof, especially in high-stakes domains like finance.
Pro tip: Highlight that prompt injection is an evolving threat and that continuous monitoring and red-teaming are essential; also mention that some mitigations may trade off with model performance or user experience, so balancing is key.
Describe techniques like input validation, escaping, and using allowlists to block malicious patterns. Mention limitations such as evasion via obfuscation or novel attack vectors.
Discuss prompt engineering (e.g., clear instructions, delimiters), fine-tuning for robustness, and using smaller specialized models to detect injections. Note limitations like model susceptibility to adversarial examples and the difficulty of covering all attack surfaces.
Explain post-processing checks, anomaly detection, and human-in-the-loop for sensitive actions. Limitations include false positives, latency, and the challenge of defining normal behavior.
Cover architectural safeguards like sandboxing, least privilege, and separating trusted and untrusted components. Limitations include complexity, cost, and potential single points of failure.
Emphasize ongoing testing, updating defenses, and learning from incidents. Limitation: resource-intensive and reactive to new threats.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by framing the problem around the unique risks of LLM agents in finance: data leakage, prompt injection, and regulatory compliance. Then walk through a layered defense-in-depth architecture covering input, model, output, and runtime, emphasizing trade-offs between security, latency, and user experience. Conclude with monitoring and governance to ensure continuous protection.
Pro tip: Tie every layer to a specific financial regulation (e.g., GLBA, SOX, PCI DSS) and show how you'd measure its effectiveness with metrics like false positive rate and containment time—this demonstrates you understand both security and business impact.
Identify assets (PII, transaction data), threat actors (external attackers, malicious insiders), and attack vectors (prompt injection, data poisoning, model inversion). Prioritize risks based on regulatory and financial impact.
Implement input validation, sanitization, and prompt hardening to block injection attacks. Use allowlists for user intents and detect anomalous patterns with ML-based filters.
Secure the model with access controls, encryption, and differential privacy. Deploy runtime guards like rate limiting, sandboxing, and real-time anomaly detection to contain breaches.
Filter outputs for sensitive data leakage, enforce compliance policies, and apply human-in-the-loop for high-risk actions. Use output watermarking and audit logs for traceability.
Set up centralized logging, alerting, and regular red-teaming. Establish governance with clear incident response and update defenses based on emerging threats and audit findings.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on 'provenance' as a formal term even though I knew the concept.
Start by defining indirect prompt injection and content provenance, then explain how provenance acts as a trust signal to filter or sanitize untrusted content before it reaches the model. Next, discuss the limitations of provenance, such as spoofing, incomplete coverage, and the challenge of maintaining provenance across transformations, and conclude with complementary defenses like input validation and output monitoring.
Pro tip: Emphasize that provenance is a necessary but not sufficient defense; it shifts the problem to trust management and requires a defense-in-depth strategy. Mention that in high-stakes domains like finance, you must also consider adversarial attacks on the provenance system itself.
Briefly explain indirect prompt injection: malicious instructions embedded in external content that the model processes. Then define content provenance as metadata about the origin and integrity of content, which can be used to assess trustworthiness.
Describe how provenance enables systems to distinguish trusted sources (e.g., internal documents) from untrusted ones (e.g., user-generated content). This allows for policies like ignoring or sanitizing untrusted content before it reaches the model.
Discuss limitations: provenance can be spoofed if not cryptographically secured; it may not cover all content (e.g., dynamically generated text); and it can be lost during transformations like summarization or translation.
Suggest additional measures such as input sanitization, output monitoring, and model-level defenses (e.g., instruction hierarchy) to create a defense-in-depth approach.
Summarize that provenance is valuable but not a silver bullet, and that a layered security strategy is essential for robust protection.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.