American Express·AI Engineer·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Technical interview for an AI Engineer role at American Express focused entirely on how you evaluate LLM and agent systems in production. One big meaty question that branched into a lot of sub-topics. Dense but interesting.

Questions Asked (1)

How would you design an end-to-end evaluation framework for an AI system that uses LLMs or agents? Walk through both offline and online approaches, how you handle agent-specific metrics, regression testing across model versions, and where human eval fits in. Also discuss what can go wrong with these setups.

System DesignProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was basically the whole interview packed into one question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing evaluation as a layered system: offline benchmarks for fast iteration, online metrics for real-world impact, and human eval for nuanced quality. Then walk through agent-specific metrics (task success, tool use, trajectory), regression testing via golden datasets and canary deployments, and failure modes like data leakage or metric gaming. Close with trade-offs between cost, latency, and coverage, tailored to American Express's risk and compliance context.

Pro tip: Emphasize that evaluation is not a one-time task but a continuous pipeline with versioned datasets and automated alerts; mention that at a regulated company like Amex, you'd also track fairness, explainability, and audit trails.

1. Define objectives and metrics

Clarify what the AI system is supposed to achieve (e.g., customer support resolution, fraud detection) and map to measurable metrics: task success, latency, cost, safety, and business KPIs. For agents, include tool-call accuracy, step efficiency, and goal completion.

2. Offline evaluation

Use curated golden datasets, synthetic data, and unit tests for components. For LLMs, measure perplexity, BLEU/ROUGE, and LLM-as-judge; for agents, simulate environments and score trajectories. Ensure datasets are versioned and cover edge cases.

3. Online evaluation

Deploy with A/B tests, canary releases, and shadow mode. Track live metrics: user engagement, conversion, error rates, and guardrail violations. Use interleaving and counterfactual logging to compare model versions without full deployment.

4. Human evaluation

Incorporate human raters for subjective quality, safety, and bias. Use structured rubrics, calibration sessions, and spot-checking. For agents, review full interaction traces to assess reasoning and tool use.

5. Regression testing and monitoring

Automate regression suites on new model versions using the golden datasets. Set up continuous monitoring with alerts for drift, degradation, and anomalies. Maintain a feedback loop to update datasets and metrics.

Key Points to Mention

Agent-specific metrics: task success rate, tool call accuracy, number of steps, and trajectory quality.
Regression testing: golden datasets, versioned prompts, and canary deployments to catch performance drops.
Human eval: essential for subjective tasks, safety, and bias; use rubrics and calibration to reduce variance.
Failure modes: data leakage, metric gaming, distribution shift, and overfitting to offline benchmarks.
Trade-offs: cost vs. coverage, latency vs. accuracy, and automation vs. human oversight.
Compliance and fairness: audit trails, explainability, and bias detection, especially in finance.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.