Common questions about CPP A/B Experiments in SplitMetrics Acquire.
Setup
- Can I run two experiments on the same ad group at the same time?
No. An ad group can only be included in one active or scheduled experiment at a time. Before starting a new experiment on the same ad group, you’ll need to either wait for the current experiment to finish or stop it manually.
- Can I change the primary metric or testing method after the experiment starts?
No. The primary metric and testing method settings are locked once the experiment is created. If you need to change any of these settings, stop the current experiment and create a new one.
- Can I add or remove variations after launch?
No. Variations are fixed once the experiment starts. If you want to test a different set of product pages, create a new experiment.
- Which testing method should I choose?
Each testing method is designed for a different use case:
Dayparting — Recommended for your first experiment. It provides balanced daytime and evening coverage and avoids traffic imbalance. Supports up to 2 product pages.
Parallel — Best for getting results faster. Recommended for high-traffic ad groups where speed is important. Supports up to 4 product pages.
Switch — Best for clean sequential comparisons. Each variation receives full traffic during its assigned period. Supports up to 4 product pages.
See 📖 Plan and Set Up Your Experiment for a detailed comparison table.
- My estimated duration is too long. What can I do?
Experiment duration depends on your ad group traffic, number of variations, and selected primary metric. To shorten the estimated duration:
Use a higher-traffic ad group to collect data faster
Test fewer variations (for example, 2 instead of 4)
Choose Parallel testing, which collects data from all variations simultaneously and has a shorter minimum duration
Select a primary metric that requires less data volume, such as TTR instead of CPA
- If the test is set to run for 2 weeks, can I extend it and continue running it for another few weeks if needed?
Yes. You can do this by selecting the Run Until Winner Is Found mode when setting up the experiment. In this mode, the estimated duration is only a prediction based on historical data, not a fixed end date. The experiment will continue running until the system collects enough data to confidently determine a winner.
- What are the minimum requirements for an ad group?
Your ad group must:
Have an active delivery status
Include at least 30 days of historical data with impressions, taps, and downloads
Not already belong to another active or scheduled experiment
Be part of a Search Results campaign
- Can I test multiple ad groups with one CPP?
Not yet. Currently, each experiment supports testing multiple CPPs within a single ad group. Support for testing across multiple ad groups is planned for a future release.
Running Experiments
- Can I make changes to my campaign while the experiment is running?
It’s recommended to avoid manual changes to campaigns, ad groups, keywords, or bids while the experiment is running. Changes during the test can affect traffic distribution and make results less reliable. The Health monitoring system automatically detects and flags external changes.
- Should I change my CPP screenshots or promotional text during the experiment?
No. CPP performance is directly tied to your creative assets, including screenshots and promotional text. Updating creatives during the experiment invalidates the comparison between variations.
- What happens to my automations during the experiment?
All automations connected to experiment objects are automatically paused during the experiment, including Rules, Bid Strategies, Budget Allocation, Keyword Expansion Strategies, and CPP Scheduler. Bulk updates in Ads Manager also skip experiment objects automatically. Once the experiment ends, all automations are restored.
- I noticed a traffic drop after launching the experiment. Is this normal?
Yes. When the experiment starts, the system creates new test ad groups and pauses the original ad group. Apple Ads may need some time to adjust to the new setup, so a temporary traffic dip during the first day is expected.
- Why is one variation receiving more traffic than another?
In Parallel experiments, traffic distribution is controlled by the Apple Ads auction. Even though the system creates identical test ad groups, Apple Ads may naturally send more traffic to one variation if it predicts stronger conversion performance. If the imbalance becomes too large, the Health monitoring system will alert you.
Results and Decisions
- How long should I wait before making a decision?
Whenever possible, let the experiment run until completion. The system uses minimum time and data requirements to improve result reliability. The estimated duration shown during setup provides a realistic expectation based on your ad group’s historical performance.
- What does “Inconclusive” mean?
An Inconclusive result means the system collected enough data for analysis, but could not confidently confirm a winner.
There are two possible outcomes:
A variation is leading — One variation performed better, but the statistical confidence was not high enough to confirm it as the winner
No meaningful difference — Variations performed similarly on the selected primary metric
Both outcomes are valid. You can either keep your current CPP, make a business decision based on supporting metrics, or run another experiment with more traffic.
- Can I trust an early signal?
Yes. Early signals reflect real trends in the data and are designed to help you understand experiment direction before completion.
In Parallel experiments, early signals reflect real-time side-by-side performance
In Switch and Dayparting experiments, early signals become more reliable once each variation has collected enough comparable time periods
An early signal is not a confirmed winner, but it can help you anticipate the likely outcome.
- Why didn’t the system declare a winner even though one variation looks better?
The system requires several conditions before confirming a winner, including:
Enough data volume
Enough experiment runtime
High statistical confidence
Stable performance over time
If any of these conditions are not met, the system will not declare a winner, even if one variation appears to be ahead. This helps reduce false positives caused by short-term fluctuations.
- What if the experiment ended with “Not enough data”?
This usually means the ad group did not generate enough traffic during the experiment. For your next test, consider:
Using a higher-traffic ad group
Testing fewer variations
Running the experiment longer
Choosing a primary metric that requires less data volume, such as TTR instead of CPA
- How can I track specific in-app events, such as FTD or Cost per FTD, in the experiment dashboard?
All full-funnel metrics, including events such as FTD and Cost per FTD, are available in the Detailed View of the experiment after clicking Show Analytics.
After the Experiment
- What happens to my ad groups when the experiment ends?
Once the experiment finishes, the system automatically restores the original setup:
All test ad groups are paused
The original ad group is re-enabled with its original settings
All automations are restored
This happens automatically regardless of the experiment outcome.
- If I run many CPP experiments, will duplicated ad groups build up over time, and can they be removed?
Right now, duplicated ad groups created for CPP experiments can only be removed directly in Apple Ads. We plan to add the ability to remove them from our platform in a future release. We’re also working on a CPP rotation approach within the original ad group, which will reduce the need to create duplicate ad groups for each experiment.
- Can I run another experiment on the same ad group?
Yes. Once the previous experiment reaches Completed or Stopped status, the ad group becomes available again for future experiments.
General
- What campaign types are supported?
Currently, only Search Results campaigns are supported.
- Is there a minimum traffic requirement?
Your ad group must include at least 30 days of historical data with impressions, taps, and downloads. There is no fixed traffic minimum, but low-traffic ad groups may require significantly longer experiment durations and may not collect enough data for reliable results.
- How is experiment duration estimated?
Estimated duration is calculated using your ad group’s historical 30-day performance data together with your experiment configuration, including:
Number of variations
Testing method
Selected primary metric
The estimate is shown during setup so you can understand the expected runtime before launching the experiment.