Attribution models only tell part of the story. They show touchpoints but not true incrementality.
How do you actually measure if a channel is driving new users vs just taking credit for users who would have converted anyway?
Testing holdout groups seems like the obvious answer but curious what methods others are using.
Turn off channels for a week and watch what happens.
Split testing with different promo codes per channel works.
Check your monthly spend against conversions. Not perfect, but you’ll spot trends.
Geo holdouts work great for this. Pick similar regions, shut off the channel in half for 4-6 weeks, then compare.
I tested Facebook campaigns across 20 metro areas last year. Killed ads in 10 cities and conversions dropped 23% - way less than FB’s attribution claimed.
When geo tests don’t work, I use the platforms’ own incrementality tools. Google and Meta both have built-in options that create synthetic control groups.
The hard part? Running tests long enough to catch delayed conversions. Most people check results too early and miss half the story.
Surprising finding - channels that looked awful in last-click attribution actually had solid lift when tested right. Search gets especially undervalued in traditional models.
Check your baseline conversion rates before launching new channels. Compare them once the channels are running.
Watch organic traffic patterns closely. If a channel brings in new users, your baseline should remain stable while total conversions increase.
For example, if attribution shows 100 conversions but organic drops by 60, that channel likely only brought in 40 new users.
Use a combination of methods. Geo tests give the best insights, but they need a lot of data. If you’re smaller, try switching off your main channel for a couple of weeks to see how others perform. You might find they cover 40-60% of the lost traffic if there’s real overlap. For tracking, create weekly cohorts. New users from one channel should show different behaviors compared to your baseline if they are truly new. Look into retention rates and LTV patterns to identify channels that might just be stealing credit.