Test multiple mobile messages with a small group of contacts to see which message is most effective.
Test mobile app messages
- Open your message draft in the Content area and set up your message.
- After you enter the message details, turn on the A/B testing option.
- Select an app message type (push, inbox-only, in-app) and a group that contains only this app type.
- On the Enter content page, create 2-4 variant messages:
- (Required) On the A tab (opened by default), enter content for the mobile app message.
- (Required) Click the B tab. Enter alternative content for the second variant message.
- (Optional) Click tabs C and D if you want to test three or four message variants.
- Click Continue to advance to the Test your message page where you will define the test settings.
- Select the percentage or number of contacts from the contact source that you want to use in testing. Use the toggle bar or enter a number.
- Alternatively, type the number of contacts you want to include in the test. Click the Calculate icon to show how many contacts in the contact source selected the mobile app. You may need to recalculate if you change the app group or contact source on the Set up your message page.
-
Select the metric that determines the winning message:
- Notification Click Rate
- Inbox Click Rate
- Inbox Effective Rate
- App Open Rate
- URL Click Rate
- Dialer Click Rate
- Click Continue to open the Confirm and send page.
-
Choose how you would like to proceed once the winning message is calculated.
- Send notification to a specific email address if you want to choose the message to be sent to the rest of the contacts manually. This email shows high-level information about the A/B test.
- Send winning notification to remainder of contacts if you want the system to automatically send the winning message to the remaining contacts in the contact source.
-
Schedule the start time of the A/B test by choosing Send now or Schedule, the end time, and test duration (optional).
- If the winning message requires an expiration date, go to the Schedule this message page to configure the Set message expiration date setting.
When you complete the test and send to the remaining contacts, you can review the metrics in the Push notification report.
Review the A/B test results
After the A/B test message is either sent to the remaining contacts or scheduled, go to the A/B Test Status page to view the content sent to remaining contacts and see the A/B test results for the test message.
- In the Content menu, click Mobile messages and open the Sent or Schedule tab to view the Sent summary or Scheduled summary page .
- Click a variant message's name. At the end of an A/B test message is variant 'A', 'B', 'C', or 'D'. Then, view the message content.
- Click A/B Test Results to see the status and results of the test. The Winning Metric that is chosen by the marketer when the A/B Test was defined is highlighted in a dark blue color.
- Status shows where the test is in the process:
- In progress
- Completed
- Canceled by user – manually cancelled when the marketer selected the End test without sending remainder link
- Winner determined
- Started
- Completed
- Winning metric: the metric that is chosen by the marketer when defining the A/B test.
- What's next identifies the required user action.
- Ongoing metrics always appears and shows metrics from the test start time until the current date. When this tab is open, a message displays at the top of the page The results are for the current date.
- Complete metric appears when the test is completed/winner is determined and metrics reflect a snapshot of results from test start time to the time the test end. When this tab is open, a message displays at the top of the page The results are for the date when the campaign was completed.
The push notification report shows the metrics for mobile app messages and individual variant messages. When you select a campaign from the list, you can see each variant message (message names include an A, B, C, or D at the end of the name).
Response window for scheduled A/B test using recipient time zone
When you select Use recipient time zone to schedule the A/B test start time, the A/B test end time uses the time zone configured in the user profile. This can reduce the response window for recipients in earlier time zones. In the following scenario, Evan determines the response window for A/B test recipients when he schedules the test using each recipient's time zone.
Evan, the Mobile App channel owner at CityCool, wants to schedule the A/B test start time using each recipient's local time zone. His recipients are located in the United States in the following time zones: Pacific, Mountain, Central, and Eastern. He is located in Chicago where it is currently 2:00 PM Central time. Evan wants to schedule the send for 3:00 PM Central time.
Before he schedules his test, Evan creates a few tables to determine the time recipients can respond to the test if he selects a specific end time (10:00 PM) and if he schedules the test duration for 4 hours.
| Recipient time zone | Current time | Scheduled time | Send time (recipient time zone) | End time | Response window |
|---|---|---|---|---|---|
| Pacific | 12:00 PM | 3:00 PM | 3:00 PM | 8:00 PM | 5 hours |
| Mountain | 1:00 PM | 3:00 PM | 3:00 PM | 9:00 PM | 6 hours |
| Central | 2:00 PM | 3:00 PM | 3:00 PM | 10:00 PM | 7 hours |
| Eastern | 3:00 PM | 3:00 PM | 3:00 PM | 11:00 PM | 8 hours |
| Recipient time zone | Current time | Scheduled time | Send time (recipient time zone) | End time | Response window |
|---|---|---|---|---|---|
| Pacific | 12:00 PM | 3:00 PM | 3:00 PM | 5:00 PM | 2 hours |
| Mountain | 1:00 PM | 3:00 PM | 3:00 PM | 6:00 PM | 3 hours |
| Central | 2:00 PM | 3:00 PM | 3:00 PM | 7:00 PM | 4 hours |
| Eastern | 3:00 PM | 3:00 PM | 3:00 PM | 8:00 PM | 5 hours |