Start by defining the funnel stages from spend to revenue, then compute stage-to-stage conversion rates and cumulative CAC at each milestone. Emphasize that CAC should be calculated as cumulative spend divided by the number of customers reaching each stage, and discuss how to handle channel-level granularity and time-based aggregation.
Pro tip: Always clarify whether CAC should be based on spend up to that stage or total spend, and consider using incremental CAC to isolate the cost of moving from one stage to the next—this shows you understand the business nuance beyond basic funnel math.
Identify the sequential stages: spend → visits → form starts → form completions → loans funded → revenue. Clarify that spend is a cost input, not a funnel stage, and decide how to handle revenue as the final output.
For each consecutive pair of stages, compute the conversion rate as (next stage count / current stage count) * 100. Aggregate across channels or keep channel-level depending on the question.
Cumulative CAC at stage X = total spend / number of customers at stage X. Incremental CAC from stage A to B = (spend up to B - spend up to A) / (customers at B - customers at A).
Create a funnel chart showing counts and conversion rates, and a line or bar chart showing CAC at each milestone over time or by channel.
Check for data quality issues (e.g., missing values, inconsistent definitions), and interpret which stages have the highest drop-off or cost inefficiency.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the attribution windows and the conversion event, then compute ROAS and payback period under each model using the appropriate revenue and cost allocations. For re-attribution from raw logs, explain how to reconstruct user journeys by joining clicks and impressions to conversions, then apply the attribution rules programmatically.
Pro tip: When re-attributing, be explicit about how you handle multiple touches within the window (e.g., first-touch means the earliest touch in the 7-day window, not the first ever) and how you treat conversions with no touches. Also, mention that payback period should be calculated on a cohort basis to account for time to recoup CAC.
Clarify ROAS (revenue / ad spend) and payback period (time to recover CAC) and specify the attribution models: last-click (100% credit to last touch) and 7-day first-touch (100% credit to first touch within 7 days before conversion).
For each conversion, identify the last click before conversion (within a reasonable window, e.g., 30 days) and assign full revenue to that campaign. Then aggregate revenue and cost per campaign to compute ROAS and payback period.
For each conversion, find the first touch (click or impression) within 7 days prior to conversion and assign full revenue to that touch. Aggregate similarly to compute ROAS and payback period.
Join raw click and impression logs with conversion logs on user ID, sort by timestamp, and for each conversion apply the attribution rule (last-click or first-touch within window) to assign credit. Handle edge cases like multiple conversions per user and missing user IDs.
Check for discrepancies, ensure total attributed revenue equals total revenue, and compare ROAS and payback across models to discuss implications for budget allocation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is the part I wish I'd practiced more.
Start by framing the problem as a budget allocation task using response curves to model diminishing returns. Explain that you would fit a log or Hill function to historical channel spend and ROAS data, then compute marginal ROAS at current spend levels. Finally, recommend reallocating budget toward channels with the highest marginal ROAS until they equalize, providing an estimate of the expected lift.
Pro tip: Emphasize that marginal ROAS, not average ROAS, should guide budget shifts, and mention that you would validate the model with a holdout test or incrementality experiment before fully committing the budget.
Clarify the goal (e.g., maximize total ROAS or revenue) and identify available data: historical spend, revenue, and other channel-specific factors. Ensure data is at the right granularity (e.g., daily or weekly).
For each channel, fit a log or Hill function to model the relationship between spend and ROAS (or revenue). Use regression techniques and validate the fit with metrics like R-squared or by checking residuals.
Calculate the derivative of the fitted curve at the current spend level for each channel to estimate marginal ROAS. This represents the incremental return from the next dollar spent.
Reallocate budget from channels with low marginal ROAS to those with high marginal ROAS until marginal ROAS equalizes across channels (subject to constraints like minimum spend). This maximizes total return.
Estimate the expected change in total ROAS or revenue from the reallocation. Recommend a validation plan, such as a geo holdout test, to confirm the model's predictions before full rollout.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Geo holdout felt like the right answer here since time-based has too much confounding with seasonality for a lending product.
Start by clarifying the business objective and the budget recommendation being tested, then propose a geo-based holdout design with randomization at the geographic level (e.g., DMA or state) to avoid contamination. Outline sample size calculation using historical variance and minimum detectable effect, and define success criteria that tie directly to the budget decision (e.g., incremental profit or ROI).
Pro tip: Emphasize that geo experiments require careful matching or synthetic control to account for regional differences, and that time-based holdouts are better for measuring long-term effects but risk seasonality confounds. Mention that you'd pre-register the analysis plan to avoid p-hacking.
Clarify the budget recommendation (e.g., increase spend in channel X by Y%) and state the null and alternative hypotheses. Specify the primary metric (e.g., incremental conversions or profit) and guardrail metrics.
Select geo-based randomization (e.g., DMAs) for marketing budget tests to minimize spillover, or time-based holdout if geo is not feasible. Justify the choice based on contamination risks and practical constraints.
Use historical data to estimate baseline variance and determine the minimum detectable effect (MDE) that is business-relevant. Calculate required number of geos or time periods for adequate power (e.g., 80%) and significance level (e.g., 5%).
Identify potential contamination (e.g., users crossing geos, ad spillover) and mitigate via buffer zones, intent-to-treat analysis, or using synthetic control. Discuss seasonality and external factors for time-based designs.
Pre-specify success criteria: statistically significant lift in primary metric with no degradation in guardrails, and practical significance (e.g., ROI > threshold). Outline analysis method (e.g., difference-in-differences, CausalImpact) and sensitivity checks.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Retargeting is the obvious one since you're mostly capturing people who were already going to convert.
Start by identifying channels that typically appear effective in attribution but are prone to non-incrementality, such as retargeting and branded search. Then, outline a diagnostic strategy that combines experimental designs (e.g., geo holdouts, ghost ads) with causal inference methods to isolate incremental lift. Emphasize the importance of validating attribution models against these experiments to quantify and correct for non-incremental effects.
Pro tip: When diagnosing non-incrementality, focus on channels where users have already expressed intent (e.g., retargeting) or where brand equity drives conversions (e.g., branded search). These are classic cases where attribution overstates impact, and experiments often reveal minimal true lift.
List channels likely to be non-incremental, such as retargeting, branded search, and email to existing customers. These often capture demand rather than create it.
Use randomized controlled trials (e.g., ghost ads, PSA controls) or geo-based holdout experiments to measure true incremental lift for each channel.
Contrast experimental lift with attribution-reported conversions to quantify the gap and identify over-attribution.
When experiments are infeasible, use techniques like synthetic control, difference-in-differences, or instrumental variables to estimate incrementality.
Incorporate experimental results to calibrate or de-bias attribution models, ensuring future budget allocation reflects true incremental value.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.