Test multiple Simple, Inbox Only, or In-App mobile app messages with a small group of contacts to see which message is most effective.
- Enter the A/B test message details into your message.
- Enable the A/B Testing setting.
- Create at least two A/B test variant messages.
- Define the A/B test by configuring the test settings. This includes choosing the number or percentage of contacts included in your test, defining the winning metrics, scheduling the test, and more.
- Confirm the message variants and settings before you send it.
- Review the test results.
When you complete the test and send to the remaining contacts, you can review the metrics in the Push notification report.
Entering A/B test message details
Open your message draft in the Content area and set up your message.
- Select one app group. If the app group contains more than one app type, A/B testing cannot be enabled. You can choose a different app group or edit the existing app group in the Development menu.
- Select a message type. You can only test messages for the same mobile app message type. A/B Testing is available only for Simple, Inbox, and In-App messages.
Enabling A/B testing
After you enter the message details, turn on the A/B Testing option.
Creating the A/B test variant messages
- (Required) On the A tab (opened by default), enter content for the mobile app message.
- (Required) Click the B tab. Enter alternative content for the second variant message.
- (Optional) Click tabs C and D if you want to test three or four message variants.
Defining A/B test settings
- Use the toggle bar or enter a number to choose the number or percentage of contacts who will receive the test.
- Alternatively, type the number of contacts you want to include in the test. Click the Calculate icon to show the number of contacts in the contact source that selected the mobile app. You may need to recalculate this number if you change the app group or contact source on the Set Up Your Message page.
Select the metric that determines the winning message. How do you want to determine the winner? Decide if you want to base the winning test message on Notification Click Rate, Inbox Click Rate, Inbox Effective Rate, App Open Rate, URL Click Rate, or Dialer Click Rate.
- Send notification to a specific email address: Do you (or someone on your team) want to receive an email notification about the winning test message and then manually send the winning message to the remaining contacts in the contact source? This email shows high-level information about the A/B Test. The email recipient can also select the A/B Status Page link at the bottom of the email to go to the A/B Test Results Status page where they can decide what to do next with the A/B test.
- Send winning notification to remainder of contacts: Do you want the system to automatically send the winning message to the remaining contacts in the contact source?
- Send Now: Send the test messages as soon as you submit the A/B Test request.
- Schedule: Use the calendar to send the test messages at a later date/time. You can determine when you want to process the contact source and schedule the message to send using a different time zone. If you schedule the message using a different time zone and the scheduled time has passed in the alternate or recipient time zone, the message is immediately sent to the recipients.
- Use alternate time zone: Schedule the A/B test start time using a time zone different from the time that is configured in your user profile settings.
- Use recipient time zone: Schedule the A/B test start time using each recipient's local time zone. When you select this option, click Calculate to see a count of the recipients and time zones that are included in your contact source and app group. You can also View Details to see the location of your recipients in their respective time zones on an interactive map.
- Schedule an End Time: Use the calendar to schedule an end time.
- Schedule Test Duration: Select this option if you want to enter the number of days or hours the test will run.
(Optional) If the winning message requires an expiration date, go to the Schedule This Message page to configure the Set Message Expiration Date setting.
Confirming the message variants before sending
- Actions: Each variant is organized by the app content, for example,'iOS - A' or 'iOS - B'.
- In-app and Inbox Content: Templates are displayed by variant message.
- Settings Applied: View the A/B Testing information at the top and scroll down to review the settings for each variant.
- Schedule: See the schedule settings for the entire message block.
Reviewing the A/B test results
After the A/B test message is either sent to the remaining contacts or scheduled, go to the A/B Test Status page to view the content of the message sent to remaining contacts and see the A/B test results for the test message.
In the Content menu, click Mobile Messages and open the Sent or Schedule tab to view the Sent Summary or Scheduled Summarypage .
Click a variant message's name. At the end of an A/B test message is variant 'A', 'B', 'C', or 'D'. Then, view the message content.
Click A/B Test Results to see the status and results of the test. The Winning Metric that is chosen by the marketer when the A/B Test was defined is highlighted in a dark blue color.
Status: Shows where the test is in the process.
- In Progress: Test is in progress. the winning message is not determined.
- Completed: Test is complete.
- Canceled by user: Test was manually canceled when the marketer selected the End Test Without Sending Remainder link on this page.
- Winner Determined: The winning message is determined.
Started: When the test began.
Completed: When the test ended.
Winning Metric: The metric that is chosen by the marketer when defining the A/B test.
What's Next: Identifies the user action that is needed.
- Ongoing Metrics: Always appears and shows metrics from the test start time until the current date. These metric results can change every time that a marketer views the page, as additional contacts that were sent an A/B test message might have opened the messaged or performed more actions. When this tab is open, a message displays at the top of the page The results are for the current date.
- Complete Metric: Appears when the test is completed/winner is determined and metrics reflect a snapshot of results from test start time to the time the test was completed/winner was determined. When this tab is open, a message displays at the top of the page The results are for the date when the campaign was completed.
- Tool Tip: Provides helpful information about how each metric is calculated.
- Buttons on variant messages: Click to send any one of the variant messages. These buttons do not appear after the remaining contacts receive the winning message or if the marketer manually cancels the test.
- End Test Without Sending Remainder link: Click to manually cancel the test.
- Content links: Each message variant has a link under the Content column, "View [X] message details". Select this link to review the content for that message variant on the Sent Summary page in a new tab.
Viewing reporting metrics
The Push Notification Report shows the metrics for mobile app messages and individual variant messages. When you select a campaign from the list, you can see each variant message (message names include an A, B, C, or D at the end of the name).