With A/B testing, you can test up to 4 different email versions to see which one generates the most response, based on the variables you are comparing. The actions of your contacts can tell you which email they prefer based on content, subject line, and from domains.

When you perform an A/B test, recipients on an enabled seed list receive both versions of the email.

Configure A/B testing

Configuring A/B testing is part of setting up your email for sending. 

1. Open your email and click Review and send.
2. On the A/B testing page, configure your test, including what areas of the email you want to test, and how many versions to test.

Summary and status

You can view the summary and status of the email after you send the A/B test. The A/B testing report doesn't display until all emails that were sent from the test are completed.

After you configure and send the email, click Sent Email at the bottom of the Confirm and Send page. A list of sent emails are displayed on the Email and templates page.

Note:
Click Scheduled to see the emails that will be sent. You can also stop and unschedule emails on this page.

The A/B Test Status page provides data and metrics for your test. On the Sent tab, click the A/B test email name link. For A/B test status, click View.

If you configured the test to notify you when the test is complete, click A/B Test Status page in the email notification.

The following table describes the states of the A/B test:

Error: Status failed: Error initializing A/B Test

If you have invalid data in restricted field types of your database. For example, text values in a numeric field type, you may receive an error initializing your A/B test

To resolve this issue, perform a Validate Data action on the database. (Open your database and find the Validate Data button.) When the data job is complete, download the results log. The log indicates each contact with invalid data, the affected fields, and the problem with the fields. You can either update the contact information within the database or perform an import update for those specific contacts. To make the updates in the database, click the contact email address and update the values.

After you correct your data, rerun the Validate Data job to make sure that there are no more errors. When the data job runs clean, run your A/B split test again. If you still receive an error, contact Client Support and provide:

• Pod number
• Organization name
• User name
• Database name
• Database location (folder path)
• Query name and location, if applicable
• A/B split test data job ID that provides the failure notice
• Validate data job ID

Cancel an A/B test

You can cancel an A/B test from the sent list. However, if the test is scheduled, you must cancel the data job that initializes the A/B test.

1. Open the email.
2. Click A/B Test Results to open the A/B Test Status page.
3. Click Cancel Test. Your A/B test is now canceled.

Scheduled A/B tests do not show in the scheduled list. You must cancel the data job that initializes the A/B test. To do this:

1. Navigate to Resources > Data Jobs.
2. Select the appropriate Initializing A/B test data job.
3. Click the Cancel button.

Emails in the A/B test use the same cancel functionality as regular emails. Any email that is before the stop point is not sent. Any emails after the stop point is sent.

Think of it like this:

• There's a line of cars at a red light.
• When the light then turns green, the cars start to move.
• When the light turns red again, the cars stop but any cars that got past the red light, keep moving.

This is how canceled emails work. Any emails that went before the cancellation, keep going to their destinations. Any emails that are waiting in line, are canceled.

When you cancel an A/B test in Settings > Activity Reports > System Activity > Active Mailings, it does not affect the remainder. The remainder must be canceled in the Sent tab of the A/B tests. Check the data jobs to make sure it is canceled.

In the Sent tab of the A/B test status, verify that the status is Test Cancelled By User, No Remainder Sent. In the data job, you will see something like

47376871 Determining winner for A/B test
'September 2015 - Fashion 1 - Male (1)' with id of 100848 Determine A/B test winner
Cancelled

The remainder does not go to the Sent tab because no Mailing ID is generated. 

A/B testing validation errors

A/B testing uses field-level validations. If you do not configure a field correctly, an error displays.

The following table describes common validation errors:

Errors, warnings, and spam scores

View the errors, warning, and spam scores for each email. You must return to your email draft to address errors and warnings.

• Errors show the items for each email that you must fix before you can send the A/B test email.
• Warnings show the items for each email that you might want to fix before you send the A/B test email.
• Click each tab to see the spam score for each email.

Do A/B tests split at the time of send or schedule? 

For a regular, scheduled email to be sent to a query, the query calculates automatically at send time regardless when the email is scheduled. This process is ideal because it ensures that the contacts who receive the email reflect the last query calculation results at send time.

If you set up an A/B test with a query as contact source and send it immediately, the A/B split is based on who is in the query at send time.

However, if you schedule the A/B test, the A/B split is based on who meets the query criteria at the time you schedule the email, not at the time of send. The initializing A/B test data job, which is responsible for segmenting the query into A, B, C, and D, is generated when you complete the setup of the A/B test.

A/B testing and duplicate contacts

Contacts might receive multiple emails. When an A/B test is sent, each send is considered a separate email.

For example, you perform an A/B test that sends to two 10% segments of your list and the remainder is sent to the other 80% of the contacts. The contact joe@email.com exists three times in this list. Each instance is placed in one of the segments, so that he is in the first two 10% segments and the remaining 80%. In this case, the contact receives three separate emails: two for the 10% tests, and one for the winning email to the remaining 80%.

However, if two instances of joe@email.com are in the same segment, and the third instance is in another, he would receive two emails. In the segment where he exists twice, the system sends to the record with the smallest recipient ID (oldest database contact) for that email.

If all three instances of joe@email.com exist in the same segment, then he would receive one email.

Not supported

A/B Testing does not support the following features:

• Email-level frequency control
• Post to social
• Create CRM settings
• Automated messages or program email