What are some common incrementality test designs?

Common incrementality test designs include various experimental approaches to measure the incremental impact of marketing campaigns. The best method to use is dependent on the marketing tactic being measured. Here are some of the most frequently used designs:

1. A/B Testing: This is a straightforward method where the audience is split into two groups: a test group that is exposed to the marketing intervention and a control group that is not. The difference in outcomes between these groups indicates the incremental impact of the campaign. This is the best method for testing variables of your website, like landing pages.

   
   

2. Holdout Testing: A portion of the audience is randomly excluded from the marketing campaign to serve as a control group. This method helps in understanding the true impact of the campaign by comparing the exposed group with the holdout group. This approach is best used for channels such as email.

   
   

3. Geo Matched Market Testing: Individual geos are grouped together to make two similar geo groupings to be used as the test and control groups. This design is the most practical and accurate methodology for incrementality testing for ad channels.

   
   

4. Time-Based Testing: This involves running the campaign during specific time periods and comparing the results to non-campaign periods. It helps in understanding the temporal effects of marketing activities. This methodology is prone to biases due to seasonal impact.

   
   

5. Synthetic Controls: This approach uses statistical models to create a synthetic control group from historical data. It is less resource-intensive than live experiments and helps estimate the incremental impact of marketing activities, but is less accurate than other methods.

   
   

Each of these designs has its own strengths and limitations, and the choice of design depends on the specific requirements of the analysis, available data, and the desired insights.