Summary
Attentive DS interview that was basically a live incident triage case. They dropped a real-sounding SMS delivery degradation scenario on me and wanted the whole nine yards: metrics, slicing, hypotheses, experiment design, rollback. Pretty intense for a single session.
Questions Asked(5)
This was the whole interview basically.
Suggested Approach
Treat this as a structured root cause analysis by first confirming the scope and timing of the drop, then systematically ruling out hypotheses using the geographic and carrier-specific constraints as key discriminators. Since two recent changes (link-shortener domain and carrier registrations) coincide with the incident, prioritize those as primary suspects while keeping external carrier-side filtering as a parallel hypothesis.
Confirm & Scope the Signal
Validate the 6.5pp drop is statistically significant and not a data pipeline artifact by checking raw send vs. delivered counts, not just rates. Segment by carrier, region, message type (transactional vs. promotional), and time to precisely bound the blast radius.
Isolate Variables Using Natural Experiments
Use Canada and UK as control groups — since they are unaffected, the issue is US-specific and likely regulatory or carrier-policy-driven rather than a global infrastructure failure. The fact that only two carriers are affected further narrows the cause to something those carriers share, such as a filtering rule or registration validation logic.
Correlate with Recent Changes
Map the exact deployment timestamps of the new link-shortener domain and carrier registration updates against the onset of the drop to establish temporal causality. Check whether the new domain is on any carrier spam blocklists (e.g., via Cloudmark or similar reputation services) and whether the registration update introduced any field mismatches that could trigger carrier filtering.
Analyze Failure Modes in Delivery Logs
Pull carrier-returned error codes for failed messages to distinguish between soft failures (e.g., unreachable handset) and hard rejections (e.g., carrier content filtering or registration mismatch). A spike in a specific error code like 'message blocked' or 'spam detected' would strongly implicate the link-shortener domain or registration issue.
Remediate & Monitor
If the link-shortener domain is implicated, roll back to the old domain or submit the new one for carrier whitelisting while monitoring recovery in real time. If carrier registration is the cause, work with the carrier partner to correct the registration record and implement a canary rollout process for future registration changes to prevent recurrence.
Key Points to Mention
Went through carrier, sender type (short code vs toll-free vs 10DLC), campaign template, message length, URL domain, vertical, send time, and new vs existing senders.
Suggested Approach
Start by clarifying what 'delivery drop' means in Attentive's context (SMS/email delivery rates), then systematically slice the data across multiple dimensions to isolate whether the issue is platform-wide or localized. Use a top-down funnel decomposition approach, narrowing from broad segments to specific root causes before proposing hypotheses.
Define & Confirm the Metric
Clarify the exact definition of 'delivery rate' (e.g., messages delivered / messages sent) and confirm the time window, baseline, and magnitude of the drop to ensure you're solving the right problem.
Check for Data & Instrumentation Issues
Before drawing conclusions, verify there are no logging errors, pipeline delays, or tracking changes that could create a false signal in the delivery data.
Slice by Key Dimensions
Segment the data across critical dimensions such as carrier/network, geography (country, region), client/brand, message type (SMS vs. MMS vs. email), campaign type, device OS, and time of day to identify where the drop is concentrated.
Isolate the Segment & Correlate with Changes
Once the affected segment is identified, cross-reference with recent changes — carrier policy updates, new client onboarding, code deployments, or regulatory shifts — to find correlated events that explain the drop.
Formulate & Prioritize Hypotheses
Rank hypotheses by likelihood and impact, then propose quick validation tests (e.g., A/B comparison of affected vs. unaffected segments, or pulling carrier error codes) to confirm the root cause before recommending a fix.
Key Points to Mention
Duplicate sends, clock skew, logging gaps, retry policy changes, throughput caps.
Suggested Approach
Frame your answer around a systematic pre-analysis checklist that covers data integrity, pipeline health, and system-level anomalies before touching any statistical methods. Demonstrate that you understand spurious results often stem from upstream issues rather than genuine signal, and that catching these early saves significant time and avoids misleading conclusions. Structure your response as a layered investigation moving from infrastructure to data quality to business context.
Check for Recent System or Pipeline Changes
Review deployment logs, ETL job histories, and infrastructure change records around the time the anomaly appeared. A code release, schema migration, or data pipeline update is often the root cause masquerading as a statistical signal.
Validate Data Completeness and Freshness
Confirm that data ingestion is complete — check for missing partitions, delayed batch jobs, or partial loads that could create artificial drops or spikes. Verify row counts, null rates, and timestamp ranges against historical baselines.
Inspect Logging and Instrumentation Integrity
Audit the event tracking or logging layer for SDK changes, client-side bugs, or sampling rate shifts that could distort metrics. For a company like Attentive (SMS/push marketing), check if event firing logic or deduplication rules changed.
Segment and Stratify the Data
Break the metric down by key dimensions — platform, region, cohort, data source — to isolate whether the anomaly is universal or confined to a specific slice. A localized issue in one segment often points to a system artifact rather than a real phenomenon.
Cross-Validate with Independent Data Sources
Compare the metric against corroborating signals from independent systems (e.g., server-side logs vs. client-side events, internal data vs. third-party attribution). Divergence between sources is a strong indicator of an instrumentation or pipeline artifact.
Key Points to Mention
A/B old vs new domain, stratified by carrier and sender type.
Suggested Approach
Frame the experiment as a controlled A/B test where you isolate the domain variable by randomly assigning users or messages to either the old or new link-shortener domain, then measure click-through rates and downstream conversion metrics. Emphasize the importance of ruling out confounders (timing, audience segment, message content) before attributing causality to the domain change. Conclude by discussing how you'd validate results with statistical significance tests and sanity checks.
Define the Hypothesis & Metric
Clearly state the null hypothesis (e.g., 'The new domain has no effect on click-through rate') and identify the primary metric (CTR) plus guardrail metrics like deliverability rate, unsubscribe rate, and conversion rate. This scopes the experiment and prevents metric fishing.
Design the Experiment Structure
Randomly assign outgoing messages or user cohorts to a control group (old domain) and a treatment group (new domain), ensuring all other variables — message content, send time, audience segment — remain identical. Use stratified randomization if segments differ meaningfully in baseline CTR.
Address Confounders & Validity Threats
Check for pre-experiment imbalance with an A/A test, and account for potential confounders such as domain reputation warm-up period, ISP blacklisting of the new domain, or seasonal traffic shifts. Consider running the experiment long enough to capture at least one full weekly cycle.
Determine Sample Size & Duration
Use a power analysis to calculate the minimum detectable effect (MDE) given the observed drop magnitude, desired statistical power (80-90%), and significance level (α=0.05). Ensure the experiment runs long enough to achieve the required sample size without peeking-induced false positives.
Analyze Results & Draw Conclusions
Apply the appropriate statistical test (e.g., z-test for proportions) and check both primary and guardrail metrics; if the new domain shows significantly lower CTR with no offsetting gains, conclude causality and recommend rollback or domain remediation. Document findings and propose follow-up experiments if results are inconclusive.
Key Points to Mention
Rollback to old domain, revert 10DLC registration changes if feasible, escalate to carriers with error code evidence.
Suggested Approach
Frame your answer around a structured, data-driven incident response process that balances speed with rigor, demonstrating both technical depth and cross-functional communication skills. Show that you think proactively about rollback triggers before deployment, not just reactively during an incident. Emphasize how you align stakeholders on evidence thresholds upfront to avoid ambiguity under pressure.
Define Rollback Triggers Pre-Deployment
Before any model or system change goes live, establish explicit, quantitative thresholds (e.g., >5% drop in click-through rate, >2x increase in prediction latency) that automatically signal a rollback. Document these in a shared runbook so all stakeholders — engineering, product, and business — are aligned before the incident occurs.
Identify the Rollback Mechanism
Describe the technical rollback path, such as reverting to a previous model version via a feature flag, re-routing traffic to a shadow model, or restoring a prior pipeline snapshot. Clarify the estimated time-to-rollback and any data integrity considerations, such as whether predictions already served need to be corrected downstream.
Monitor and Triage with Real-Time Evidence
During an incident, pull real-time metrics from monitoring dashboards (e.g., model performance KPIs, data drift signals, business metrics like revenue or engagement) to assess severity and confirm the root cause hypothesis. Distinguish between a model issue, a data pipeline issue, or an upstream system failure before executing the rollback.
Execute Rollback and Communicate Cross-Functionally
Initiate the rollback once thresholds are breached and notify all relevant stakeholders — engineering, product, and business leads — with a clear status update including what was rolled back, why, and the expected timeline for resolution. Avoid jargon and translate technical findings into business impact language.
Establish Resolution Evidence Threshold
Define 'resolved' as a sustained return to baseline metrics over a statistically meaningful observation window (e.g., key KPIs within normal range for 24–48 hours with no anomalies), not just a momentary recovery. Require sign-off from both technical and business stakeholders before officially closing the incident and scheduling a post-mortem.
Key Points to Mention
Discussion(5)
Sign in to join the discussion.
48 hours is the right call. Carrier routing table updates and spam filter recalibrations can take a full business cycle to propagate, so a 24-hour window can catch you in a false recovery. The 1 percentage point threshold is reasonable but I'd also want to see the failure code distribution return to baseline, not just the top-line rate, because you could hit your delivery rate target while still having an elevated share of filtered messages that are being retried successfully. That's a fragile recovery.
Blanking on carrier queue backoffs is so relatable. It's one of those things where the moment they nudge you, you feel like an idiot because of course carriers throttle and backoff during high-volume windows, and that can look like a delivery rate drop when it's actually a delay. The other one I'd add that sometimes gets missed: check whether the logging pipeline itself changed around the same time. At a previous job we spent two hours on an "incident" that turned out to be a Kafka consumer lag issue making it look like events weren't happening.
Vertical being last is a reasonable critique. SHAFT (sex, hate, alcohol, firearms, tobacco) content categories get flagged more aggressively by carriers, and if the new domain or registration metadata somehow changed how messages were classified, SHAFT-adjacent verticals would surface the problem first and hardest. That's a high-prior slice, not an afterthought. The sender type dimension (short code vs toll-free vs 10DLC) is probably your most diagnostic cut after carrier, because 10DLC registration changes would affect that bucket specifically and leave short codes untouched. If the drop is concentrated in 10DLC senders on two specific carriers, you've basically confirmed the hypothesis in one pivot.
The retroactive randomization problem is genuinely hard and I think a lot of people freeze on it. The cleanest answer is probably a difference-in-differences framing: use Canada and UK as your control group since they weren't affected, compare the pre/post delivery rate change in the US against the pre/post change in those markets. If the gap widens exactly at the rollout timestamp and the control markets are flat, that's strong observational evidence even without a clean experiment. You can also look at any senders who weren't migrated to the new domain yet as a natural control within the US, though you'd need to be careful about selection effects since who gets migrated first is probably not random. The hierarchical logistic regression approach you described is right for the prospective experiment, but for the post-hoc analysis the DiD or interrupted time series framing is more defensible. Stratifying by carrier in either case is non-negotiable given the US-specific pattern.
The geographic isolation is doing so much work in this prompt and you're right that anchoring on it faster would've helped. US-only, two specific carriers, exact timing aligned with a domain and registration change: that's not a product bug, that's almost certainly a carrier filtering event or a registration propagation failure. I'd have framed the triage in two tracks from the start rather than treating all hypotheses equally. Track one is carrier-side: did the new link-shortener domain hit a spam filter, did the 10DLC registration changes cause a mismatch between what carriers expect and what's actually sending? Track two is instrumentation: is the drop real or are we missing acks? The error code question is where a lot of people stumble. The distinction that matters is between a 30x (carrier rejected, message never attempted delivery) versus a 5xx or timeout (attempted but failed in transit) versus a carrier-side spam classification code like some T-Mobile and AT&T filtering codes that look like soft failures but are actually hard blocks. If you're seeing rejection codes at the acceptance layer, that's registration or filtering. If you're seeing delivery timeouts, that's more infrastructure. Fumbling that distinction is understandable because SMSC error taxonomies are genuinely messy and vary by carrier, but it's worth knowing the broad categories before an SMS-adjacent interview.