What are some challenges with incrementality tests?

Incrementality tests, while valuable for measuring the true impact of marketing efforts, come with some challenges:

Sufficient Sample Size: There is a minimum sample size needed to run incrementality tests. The needed amount and for which variable will depend on the test details, most importantly the methodology used.
Controlled Audience Integrity: The ability to create control and expose audiences is the foundation for incrementality testing. Running incrementality tests on a channel or tactic is dependent on being to create and maintain the integrity of these audiences through the duration of the test.
Complexity and Resource Intensity: Conducting incrementality tests can be time-consuming and resource-intensive. These tests require data science and data engineering resources to be executed properly. This complexity can be a barrier for some organizations, especially those with limited resources
Potential Lost Short-term Revenue: In an incrementality test a control group is held out from seeing the marketing being tested. The holdout group is usually 10-30% of the total audience of the channel being tested. At a whole business level, this normally only equals about 1-3% of overall revenue, with the decrease in profit being even less as testing reduces your ad spend cost. While there may be lost revenue in the short term from decreasing your marketing spend, the long-term impact of measuring and optimizing to incrementality will greatly outweigh it.

Luckily for marketers, we built Incrementality-based Attribution to help address these challenges, making running incrementality tests simple.