With A/B testing, you can test up to 4 different email versions to see which one generates the most response, based on the variables you are comparing. The actions of your contacts can tell you which email they prefer based on content, subject line, and from domains.
When you perform an A/B test, recipients on an enabled seed list receive both versions of the email.
Configure A/B testing
Configuring A/B testing is part of setting up your email for sending.
- Open your email and click Review and send.
- On the A/B testing page, configure your test, including what areas of the email you want to test, and how many versions to test.
Summary and status
You can view the summary and status of the email after you send the A/B test. The A/B testing report doesn't display until all emails that were sent from the test are completed.
After you configure and send the email, click Sent Email at the bottom of the Confirm and Send page. A list of sent emails are displayed on the Email and templates page.
Click Scheduled to see the emails that will be sent. You can also stop and unschedule emails on this page.
The A/B Test Status page provides data and metrics for your test. On the Sent tab, click the A/B test email name link. For A/B test status, click View.
If you configured the test to notify you when the test is complete, click A/B Test Status page in the email notification.
The following table describes the states of the A/B test:
|Test not started||The A/B test is scheduled for a time in the future, and it has not yet started.|
|In progress||The A/B test started but has not yet determined a winner.
You see this even if you manually selected a segment to send to before the configured test completion time is reached.
|Winner determined||The A/B test determined a winner.
You see this regardless of whether the remainder segment has been sent to.
|Canceled by user||The A/B test was canceled.
To cancel a test, click End Test Without Sending Remainder at the bottom of the A/B Test Status page.
|Failed - system error||If the A/B test fails for some reason, it doesn't become scheduled or get started.
It cleans itself up, and the only way you will know about the failure is by looking at the status on the A/B Test data job.
Error: Status failed: Error initializing A/B Test
If you have invalid data in restricted field types of your database. For example, text values in a numeric field type, you may receive an error initializing your A/B test
To resolve this issue, perform a Validate Data action on the database. (Open your database and find the Validate Data button.) When the data job is complete, download the results log. The log indicates each contact with invalid data, the affected fields, and the problem with the fields. You can either update the contact information within the database or perform an import update for those specific contacts. To make the updates in the database, click the contact email address and update the values.
After you correct your data, rerun the Validate Data job to make sure that there are no more errors. When the data job runs clean, run your A/B split test again. If you still receive an error, contact Client Support and provide:
- Pod number
- Organization name
- User name
- Database name
- Database location (folder path)
- Query name and location, if applicable
- A/B split test data job ID that provides the failure notice
- Validate data job ID
Cancel an A/B test
You can cancel an A/B test from the sent list. However, if the test is scheduled, you must cancel the data job that initializes the A/B test.
- Open the email.
- Click A/B Test Results to open the A/B Test Status page.
- Click Cancel Test. Your A/B test is now canceled.
Scheduled A/B tests do not show in the scheduled list. You must cancel the data job that initializes the A/B test. To do this:
- Navigate to Resources > Data Jobs.
- Select the appropriate Initializing A/B test data job.
- Click the Cancel button.
Emails in the A/B test use the same cancel functionality as regular emails. Any email that is before the stop point is not sent. Any emails after the stop point is sent.
Think of it like this:
- There's a line of cars at a red light.
- When the light then turns green, the cars start to move.
- When the light turns red again, the cars stop but any cars that got past the red light, keep moving.
This is how canceled emails work. Any emails that went before the cancellation, keep going to their destinations. Any emails that are waiting in line, are canceled.
When you cancel an A/B test in Settings > Activity Reports > System Activity > Active Mailings, it does not affect the remainder. The remainder must be canceled in the Sent tab of the A/B tests. Check the data jobs to make sure it is canceled.
In the Sent tab of the A/B test status, verify that the status is Test Cancelled By User, No Remainder Sent. In the data job, you will see something like
47376871 Determining winner for A/B test 'September 2015 - Fashion 1 - Male (1)' with id of 100848 Determine A/B test winner Cancelled
The remainder does not go to the Sent tab because no Mailing ID is generated.
A/B testing validation errors
A/B testing uses field-level validations. If you do not configure a field correctly, an error displays.
The following table describes common validation errors:
|All email bodies must match the ones in the delivery options tab.||Email body settings for all templates in the A/B test must be the same. It is based on template A's draft email bodies (what's selected in delivery options, not A's template-level email body.)|
|The tracking level of your template is set to no tracking.||Occurs when the template tracking is off (Tracking level = No tracking) in any of the templates' settings and when A/B testing is on.|
|This information entered in Test Panel [A] is identical to Test Panel [C].||Occurs when any two variants in the A/B test tabs are identical.|
|Post to social settings are not applied to your A/B test||Occurs during the health check validation process. Review your post to social settings.|
|Create CRM campaign setting is not applied to your A/B test||Occurs during the health check validation process. Review your CRM settings.|
|Maximum messages per contact settings are not applied to your A/B test||Occurs during the health check validation process. Review your contact source settings.|
|A/B test schedule buttons grayed out||
You can't edit this page if A/B test is on. You must click Change A/B Test Schedule to change the schedule for the A/B test.
|Please provide at least 2 subject lines||Occurs when only one subject line is created.|
|Please select a metric to determine the winner||Occurs when the user does not select a metric from the drop-down list.|
|This time has already passed. Your email will be sent immediately.||Occurs if the user does not change the start date and returns to the A/B testing page an hour later.|
|This end date occurs before the start date. Select a date in the future.||Occurs if a scheduled end date happens before the scheduled start date.|
|You cannot apply an A/B test to a contact list. Please select a database or query.||Occurs when a contact list is attached to the email.|
|Amount of time to determine winner must be set for a longer period than your Send Time Optimization (STO).||Occurs if the STO is set to a pre-defined value for the contact source and the time to determine the winner is less than the STO value.|
Errors, warnings, and spam scores
View the errors, warning, and spam scores for each email. You must return to your email draft to address errors and warnings.
- Errors show the items for each email that you must fix before you can send the A/B test email.
- Warnings show the items for each email that you might want to fix before you send the A/B test email.
- Click each tab to see the spam score for each email.
Do A/B tests split at the time of send or schedule?
For a regular, scheduled email to be sent to a query, the query calculates automatically at send time regardless when the email is scheduled. This process is ideal because it ensures that the contacts who receive the email reflect the last query calculation results at send time.
If you set up an A/B test with a query as contact source and send it immediately, the A/B split is based on who is in the query at send time.
However, if you schedule the A/B test, the A/B split is based on who meets the query criteria at the time you schedule the email, not at the time of send. The initializing A/B test data job, which is responsible for segmenting the query into A, B, C, and D, is generated when you complete the setup of the A/B test.
A/B testing and duplicate contacts
Contacts might receive multiple emails. When an A/B test is sent, each send is considered a separate email.
For example, you perform an A/B test that sends to two 10% segments of your list and the remainder is sent to the other 80% of the contacts. The contact
email@example.com exists three times in this list. Each instance is placed in one of the segments, so that he is in the first two 10% segments and the remaining 80%. In this case, the contact receives three separate emails: two for the 10% tests, and one for the winning email to the remaining 80%.
However, if two instances of
firstname.lastname@example.org are in the same segment, and the third instance is in another, he would receive two emails. In the segment where he exists twice, the system sends to the record with the smallest recipient ID (oldest database contact) for that email.
If all three instances of
email@example.com exist in the same segment, then he would receive one email.
A/B Testing does not support the following features:
- Email-level frequency control
- Post to social
- Create CRM settings
- Automated messages or program email