What is incrementality in mobile app marketing?

Incrementality is a method of measurement used to determine the impact of an activity on an app’s key performance indicators (KPIs) such as installs or in-app events for a set period of time. Using incrementality, you can see what would happen if you did not do this activity.

With incrementality, you can answer questions like:

• Did optimizing my app title using a high-volume keyword lead to actual organic installs? Or did I simply redistribute traffic from other branded search terms?
• Are my paid campaigns driving new users or simply being credited with users who would have come to my app organically anyway?
• Does targeting competitive keywords lead to actual incremental growth or not?
• Did an event drive actual growth, or was it part of a normal seasonal pattern?
• For a marketing activity, was there a spike in downloads, engagement, or revenue beyond expected trends?

In short, incrementality will help you decide if an activity was successful in influencing your marketing growth. Doing so helps you discover which marketing activities are worth your time and budget.

The importance of incrementality for ASO and paid campaigns

Determining a marketing effort’s incremental value is key to successfully optimizing your marketing strategy for both the short and long term. As user data has become more aggregated for privacy-conscious purposes, marketers have found measuring campaign impact a bit more difficult and have turned to incrementality testing in marketing for greater insights.

The benefits of incrementality for ASO

Some of your ASO efforts have greater influence on your ROI than others and it’s paramount you know the difference.

Applying incrementality to app store optimization (ASO) can help you:

• Better understand the influence of updating your app stores’ creatives on your app’s performance.
• See which in-app events or promotional content drove user engagement or revenue.
• Assign value to the impact of being featured in the app stores.
• Measure the impact of a metadata update on your app’s visibility and visitor traffic in the app stores.

The advantages of incrementality for paid UA efforts

And guess what? Incrementality goes well beyond ASO to help you to determine uplift in your paid user acquisition efforts as well.

Here are four ways to use incrementality to gain paid UA insights:

1. Identify by how much you can improve your app’s conversion rate by linking a Custom Product Page to an Apple Search Ads campaign.
2. Discover the specific amount by which you can decrease your cost per install by connecting an Apple Search Ads to a specific Custom Product Page.
3. Assess if a TV campaign should receive credit for driving an increase in branded search traffic or not.
4. Look for potential organic uplifts or cannibalization effects from a paid media campaign.

Incrementality helps you connect the dots from your marketing efforts to your budget and overall growth so that you can best optimize your ASO efforts and paid campaigns for maximum success.

How to measure incrementality in marketing?

There are several different approaches to measuring incrementality in marketing, with the most common being A/B testing, holdout groups, matched market testing, and time series analysis.

A/B testing

This older approach is likely the most familiar in marketing. To perform an A/B test, you must split your audience into two groups. Group A, acts as the control group, with no change, while Group B is exposed to your marketing campaign or variant. A/B testing helps you determine which element of an app or an ad performed better in terms of user engagement, and conversion rates.

Example: You want to measure the incremental impact of changing your app store screenshots on conversion rates. Group A sees the existing screenshots, while Group B sees a new design that highlights key features of your app. If the test group has a significantly higher conversion rate, then we know that the new screenshots drove incremental installs.

Limitations of A/B testing

While A/B testing is a great way to monitor micro-level changes such as ad creatives of app store screenshots, it doesn’t account for external factors such as competitor actions, seasonal trends, or algorithm updates. Additionally, an A/B test may exhibit a temporary uplift, but it doesn’t always predict long-term success.

Holdout groups

A holdout group is a subset of your targeted audience that is excluded from a marketing campaign or change being tested. It’s used to determine the effect of the campaign or update by comparing outcomes of those who were excluded and those who weren’t. This helps to establish causality by providing a baseline for comparison.

Example: In a marketing campaign, the exposed group sees ads on Google, while the holdout group does not see any ads. If the exposed group buys whatever the ad was selling at a higher rate than the holdout group, then the ads are driving proper sales. If not, then the ads can be determined to be a waste of money.

Limitations of holdout groups

In terms of ASO updates on the App Store or Google Play Store, holdout groups can’t actually prevent a segment of users from seeing them, rendering them impractical for monitoring metadata updates. Also, a group of users may be excluded but still hear about an app by word-of-mouth, browsing their app store, or competitor ads and so it’s possible that the excluded segment is not truly excluded.

Matched market testing

This approach is favored when it’s difficult to create a control group. Matched market testing is when you compare the performance of your target market with a similar market that did not receive the same marketing effort. Note that these markets should be comparable in terms of app store dynamics, user behavior and competitive landscapes.

If there is a difference in outcomes, then this indicates an incremental effect of the marketing effort. If there is no significant difference, then the incremental impact is negligible.

Example: You want to test the incremental impact of running Apple Search Ads in a specific country, so you select two similar markets, such as Canada and the U.S. Canada acts as the test group and is shown Apple Search Ads. The U.S. serves as the control group and does not receive any Apple Search Ads.

After four weeks, the test group Canada shows significantly higher lift in both organic and paid installs compared to the U.S. control group. This reveals that the Apple Search Ads drive incremental growth and that it is not cannibalizing organic traffic.

Limitations of matched market testing

In reality, no two markets are identical when considering app store dynamics, regional user behavior, and local marketing conditions. These subtle differences between two markets can skew results. Similar to A/B testing, matched market testing lacks the ability to provide long-term insights such as retention or lifetime value (LTV).

Time series analysis

A time series analysis analyzes historical data to identify patterns and trends that occur both before and after a marketing campaign. By examining deviations from these patterns using a statistical approach, you can estimate the incremental impact of the campaign. As this method adjusts for external factors, it’s highly useful for real-world ASO and user acquisition measurement.

Example: A gaming app runs a Halloween-based event, offering in-game rewards for the holiday. A time series analysis looks at the daily downloads, engagement, and revenue before, during, and after this event. If we look at the historical data and compare it to the most recent data and note that downloads rose quite high beyond the normal Halloween trends, then we can assume that the event likely drove incremental growth.

Why time series analysis is the optimal incrementality measurement approach

While the traditional experimental methods of A/B testing, holdout groups, and matched market testings offer useful insights regarding marketing performance, they all are hampered by significant limitations. These include being unable to account for external factors, long-term impact, and real-world market dynamics.

This is where time series analysis shines as it allows for controls regarding seasonality, competitor updates, and algorithm changes. As it identifies trends before, during and after a change, it provides both short and long-term performance insights. Therefore, time series analysis works the best for monitoring ASO and paid user acquisition efforts. It helps you see the impact over time.

AppTweak’s Reporting Studio measures incrementality using a time series analysis because our experience with clients has taught us it’s imperative to measure uplifts both during and after an event. As impact analysis allows us to see incremental lifts during a marketing campaign and also monitor the residual effects post-event. To get insightful use cases on using AppTweak’s Incrementality Analysis check out how to Measure the true impact of ASO and paid UA.

Conclusion

Incrementality allows you to make data-driven decisions by isolating true growth from existing trends, seasonality, and cannibalization.

In terms of ASO, incrementality shines a light on whether or not your ASO efforts are driving true organic growth or simply redistributing existing demand.

For paid campaigns, incrementality helps to ensure your budget is allocated correctly. This prevents you from spending unnecessarily on users who would have been captured without ads.

If you’re not leveraging incrementality to fully understand your marketing efforts, we can guarantee you that your competitors are. Improve your ROI by ensuring every cent you spend and every action you take translates into growth. Incrementality in marketing is here to stay. Try it now.