If you haven't read Rand Fishkin's What if Performance Advertising Is Just an Analytics Scam? then I'd highly recommend you do.

It's a near-perfect archetype of the sort of issues that get raised by someone who knows just enough about performance marketing to know the contentious areas, but not enough to understand their solutions.

Many performance marketers will be familiar with the criticisms raised, having heard them from CMOs, clients, and other stakeholders over the years. For those less familiar, I'm going to run through them, and some of the issues with Fishkin's line of argument.

Selection bias

Fishkin starts his article with a quote from Brian Chesky, Airbnb's CEO, which amounts to:

• We cut our ad spend to $0 and our overall sales dropped by only 5%.

The article later references similar case studies from Uber, Bloomberg and Ebay, amongst others. In each of these cases, the brand in question drops their ad spend by some seemingly drastic amount, and sees little to no change in overall demand.

As impressive as these examples might seem, we have to consider the fact that no company would ever publicise their results had they observed the opposite. No CMO is going to go on Medium and write an article called We dropped our spend by X%, and saw our volume drop by X%. Even if they did write such a case study, it's unimaginable that it should become part of modern marketing folklore in the way that Airbnb and Ebay's experiments have.

What this means is that it's incredibly easy to find case studies where large drops in spend have led to little or no drops in growth. This isn't to say that this is the norm though, and that any brand could (as in the case of Airbnb) drop their paid spend by 100% and expect only a 5% drop in volume.

Fishkin's examples sit at one end of a spectrum. It's the only end of the spectrum worth writing and reading about however, which is why it's so easy to mistake them as sitting in the middle of the spectrum.

For what it's worth, I've run a number of lift tests before where I've seen that a brand's attributed conversion numbers line up near-perfectly with the incremental conversion numbers produced from a 50-50 lift test. Sadly, writing about these sorts of results hardly makes for a good blog post.

Some things take time

Selection bias aside, there's another issue with the sort of examples that Fishkin references.

Let's focus on Airbnb as an example. Airbnb is a household name. It's become a household name over the years as a result of sustained brand marketing efforts, and word of mouth.

Brand marketing takes a significant amount of time to have a positive effect on volume, and so we should expect it to take an equally significant amount of time for a cut in brand marketing spend to have a negative effect on volume.

In the sort of case studies that Fishkin references, it's rarely clear what timescale the cut in spend is being evaluated over. The timescale looks fairly short in Airbnb's case, where Chesky references yearly volume numbers, and it's not clear to me whether it's reasonable to expect a cut in brand spending to show itself over such a short timescale. It's entirely possible that Airbnb's volumes are benefitting from years of prior brand advertising spend.

Counterfactuals

The last issue with these sorts of case studies is that they never seem to take seriously the counterfactual, of what would volume have been if spend wasn't cut.

The Chesky quote in Fishkin's article implies that only 5% of volume was lost after Airbnb pulled it's marketing spend, but this plays on an assumption that volumes would have stayed level if marketing spend hadn't changed year on year. There's nothing to say that Airbnb wouldn't have increased its traffic significantly if marketing spend had stayed the same.

Let's imagine that Airbnb's traffic volume would have increased by 20% year-on-year, had all variables (including paid marketing spend) stayed constant. If this were the case, then it's clear we shouldn't be thinking of paid marketing as contributing only 5% of Airbnb's traffic volume (100% - 95%), but rather we should be thinking of it as contributing the (far larger) difference between a 20% growth, and a 5% decline.

My 20% number is purely illustrative. It does go to show that calculating the impact of paid marketing by simply turning everything off, without any control, is an inappropriate way of measuring the impact of paid.

Straw men

A broader issue with Fishkin's article concerns its scope.

Much of the article takes aim at specific types of lower-funnel marketing channels, such as retargeting and branded search. Fishkin is completely right to question the incrementality of these channels, however I've never met a performance marketer who takes attributed results from these channels at face value.

No performance marketer will ever work these channels to the same cost per acquisition (CPA) goal as their higher-funnel channels. Every paid search advertiser, for example, will work to a lower target CPA on brand search than non-brand search. The reason they do this is because they implicitly or explicitly believe that brand search conversions are less incremental than those that come through non-brand search.

• A brand search term is one that contains an advertiser's brand name, and a non-brand search term is a search term that doesn't. Nike would consider nike running shoes to be a brand search term, and running shoes to be a non-brand search term.

The implicit maths that goes on in the search advertiser's head is something like:

I think that a non-brand search conversion is 20x more likely to be incremental than a brand search conversion. Therefore if my non-brand CPA target is $20, my brand CPA target should be $1.

What the advertiser is doing here is trying to balance out their attributed CPA targets, in order to bring in the same cost per incremental conversion on each of their channels. In doing so, they're hardly being fooled by their analytics telling them that their brand campaigns appear to have a phenomenal CPA.

Don't get me wrong, the example as I illustrated it above doesn't always happen in practice. Many advertisers won't know their true incrementality ratio between channels like brand & non-brand search. Even if they do, they might not choose to use this information correctly, or might choose to work to an unbalanced CPA on some channels for reasons other than performance.

At the same time though, what this shows is that the average performance marketer has a far better grasp of incrementality than Fishkin implies. They're not simply being scammed by their analytics software.

Moreover, most of Fishkin's article levels criticisms precisely at channels where it is obvious that not all attributed conversions are incremental. There's nothing wrong in pointing these channels out, but to write off the entirety of performance marketing as an analytics scam as a result is a conclusion far larger than what Fishkin has argued for.

Misunderstandings of attribution

One topic which Fishkin brings up (through a quote from Avinash Kaushik) is the difference between attribution and incrementality.

• Attribution is simply the science (or, too often, the art) of distributing credit for conversions across your various marketing channels. When we do attribution modeling in tools like Adobe or Google Analytics, we are essentially saying that if there were four interactions with our owned, earned, and paid media channels before a last-click conversion from paid search, then the credit for that conversion should be distributed to all the interactions.
  
  The question incrementality asks is “How many of those conversions would have happened anyway, without any advertising spend?”
  
  Incrementality is incredibly hard to measure. Just think of all the data you need to collect to figure out the answer. It’s also incredibly complex to measure — in part because when senior leaders ask the incrementality question, they often mean something different.

This idea isn't central to Fishkin's piece, but I've come across it before and thought it would be useful to spend a minute debunking it.

Attribution and incrementality are the same thing, so long as we're considering all channels.

What do I mean by so long as we're considering all channels?

Imagine you're an ad network, and you're trying to distribute credit for a conversion amongst the campaigns of a particular advertiser. All you know is that 1 conversion happened, and that it happened from a user who clicked on multiple campaigns from the advertiser in question. However you divide that credit, be it linearly, time-dependently, or using a DDA model, the sum of credits allocated will add up to 1.

You're not going to add all the credits up to 1 because you genuinely think that you drove 1 incremental conversion; you're doing so because  you don't have visibility on all channels that a user could have interacted with prior to conversion. If it did have visibility, then you could in theory allocate that 1 conversion's credit across all the relevant channels, attempting to model the incremental effect that each channel had in the user's conversion journey.

This of course isn't easy to do; I'm not suggesting for a moment it is. My point is that what we're doing above looks like a lot like what Kaushik calls attribution, and yet we're also trying to answer the question that he poses under the label of incrementality. This is because when we have visibility on all the channels involved in a conversion journey, the two become the same thing.

A better way to understand the relationship between attribution and incrementality measurement is as follows. Attribution is simply an edge case for incrementality, which appears only when we don't have visibility on all channels that a user interacts with prior to conversion.

It's not just over-attribution

One of the central ideas of Fishkin's article is that ad networks constantly over-attribute credit to themselves, but it's worth considering that it's common for them to under-attribute too.

One way of thinking about over- and under-attribution is through the ratio of incremental/attributed conversions that an advertiser measures on a given ad network. Incremental conversions can be measured through a lift test or quasi experiment, attributed conversions can be pulled straight from the ad network's tracking, and dividing the former by the latter gives us a handy ratio to measure how closely the network's tracking matches incremental results.

If the ratio is near 1, then this suggests that the ad network is doing a good job on reporting genuine, incremental conversions. It might be doing this by accident, but it still means that attributed conversion numbers provide a good estimate of how many incremental conversions you're driving.

If the ratio is far below or above 1, then this means that the network in question is over-reporting or under-reporting conversions respectively (at least compared to true incremental results).

Fishkin's article implies that this ratio is frequently less than 1, and doesn't even consider the possibility that this number is ever greater than 1. And yet this is frequently the case; it's common for ad networks to generate more incremental conversions than they attribute.

This is typically the case for more upper-funnel campaigns, on upper-funnel networks. Here the conversion cycle may be too long for the network's attribution windows to capture all incremental conversions, and so methods like lift testing and quasi-experiments may reveal that the network generated more incremental conversions than it reported.

I'm not for a moment suggesting that this is the norm. At the same time though, it's not uncommon, and the possibility of it happening shouldn't be ignored.

Comparing apples & oranges

At one point in his article, Fishkin cites research into TV elasticity, and asks why we should expect performance channels to be any different.

Elasticity is the relationship between marketing spend and incremental revenue generated by that spend. Fishkin references a Freakonomics episode, which in turn references a study that found a 1% elasticity on a brand's TV advertising (meaning a doubling in marketing spend contributes a 1% increase in sales).

Fishkin then raises the question: if TV is so inelastic, why should we expect performance marketing to be different? Two possible reasons why are:

• TV and digital are different beasts
  ‍The journey to conversion from a typical performance ad is much quicker than that from a TV ad; simply click on the ad and you're already on your way. Fishkin saying that the two channels should have the same elasticity makes no more sense than saying Tortoises can live for a hundred years, so why can't my cat?
• Back to selection bias
  ‍The sorts of brands that advertise on TV are typically large, well-established brands. As a result, many will have hit saturation points in their advertising, leading to diminishing returns on marketing spend. It's no surprise therefore that TV advertising should have such low elasticity, as the brands which engage in it are already saturating their markets. This is less likely to apply to your typical performance-first/digital advertiser.

Setting the bar low

The very last point I'll raise on Fishkin's article isn't a criticism of any of his points per se, but rather with the general attitude he exemplifies. At one point he writes (referencing Avinash Kaushik's blog post):

But, when you get to the bottom of his post, you’ll see a methodology for measuring incrementality that’s probably effective, but so painfully challenging to execute that almost no one will bother.

I feel like this sets the bar far too low for performance marketers, as if our job should just be to set up campaigns, build reports, and not worry about the big, hard questions like incrementality. As more of the operational tasks involved in performance marketing become automated and productised, the value in a good performance marketer will shift heavily towards an ability to ask & answer questions on topics like incrementality.

Fishkin's admitting that methods like marketing mix modelling are painfully challenging is reflective of the gradually changing demographics of performance marketers today. Whereas performance marketing roles were previously filled by career marketers, newer entrants to the industry are increasingly coming from more quantitative backgrounds.

I think that many of these newer entrants will see questions of incrementality as intellectually stimulating, rather than painfully challenging.