With A/B testing, you can test up to four different email versions to see which one generates the most response, based on the variables you are comparing. The actions of your contacts can tell you which email they prefer based on content, subject line, and from domains.

When you perform an A/B test, recipients on an enabled seed list receive both versions of the email.

Configure A/B testing

Configuring A/B testing is part of setting up your email for sending.

1. Open your email and click Review and send.
2. On the A/B testing page, configure your test, including what areas of the email you want to test, and how many versions to test.

Contact sources for A/B testing

Only databases and queries (excluding legacy queries) can be used for A/B testing.

You can create a query that includes a contact list that contains your A/B test contacts, here's how:

1. After you create a contact list, set it up by using the following criteria in a query:
   • Go to Profile > Contact List.
   • Specify Is in Contact List and select the contact list that you want to use in the query.
   • To include multiple contact lists in the A/B split test, change the AND to OR for each new set of criteria.
2. In the email template, select the query as the contact source for your A/B split test email.
3. Save the email and click On to Send to go to the send email draft.

When you use new queries for A/B testing, the parent database from which the query was made must use the org default VMTA.

To ensure that the parent database is using the default VMTA, complete the following steps.

1. Select the database that you used to make the query, which is called the parent database.
2. Click Settings.
3. Make sure that the VMTA Name is set to Use Parent Contact Source VMTA.

When you choose an email for an A/B test, the system creates queries of the same name that are used to send all emails, including the winning subject line and from name.

When you schedule an A/B test, contact lists are created at the scheduled time. If the contact source you used is a query, the query is run and the contact lists are created at the scheduled time, not at send time.

Send the test leader to the remainder

Use the Send Leader to Remainder button to send the winning email to the remainder group. The remainder group includes everyone who was not placed into one of the test lists.

Email changes and A/B test remainders

Changes to an email used in an A/B test can impact the remainder send. For example, let's say you create an A/B test. The A and B versions are sent, but the winner isn't chosen yet. You change the content of the email that was used in the A and B versions. Those changes will affect the remainder send after the winner is chosen.

Here's what happens:

When you schedule an A/B test, Campaign records various aspects between each version (A/B/C/D) that can be different based on the settings you apply:

• Template
• Subject Line
• From Name
• From Address
• Reply-to Address

If you choose a different template between versions to test content, Campaign records the ID for each.

When the A/B test is scheduled, a data job ("Initializing A/B test X with ID of X") sorts the contacts in your contact source into contact lists and sends the email based on the settings you applied for each version.

When you choose the winning version or the winning version automatically sends, or schedules a new email based on the settings that are used to send the winning version, one setting that is always included is the ID, so when the winning version is sent to the remainder, the email that is sent is based on what content exists in the ID.

You can modify a template that is used for A/B testing between the time the initial A and B versions are sent and when the winner is chosen. The changes that are made to the template are reflected in the remainder send. This works as designed because when Campaign sends the winner, it uses the settings from that version (which includes the ID) and a template can be modified at any time.

Send to the remainder of a canceled test

You can send an email to the remainder of the contacts when you cancel your A/B test after the A/B test was sent but before the testing period is complete.

The easiest way to send to the remainder of a canceled A/B test email is to locate the remainder query that is created as part of an A/B split test. Then, calculate the query and assign this query as your contact source to your email.

Note: When you create an A/B test, a contact list is created for each split of your primary contact source you specify for the test.

• AB test shared test segments
• AB test private test segments

These contact lists are named similarly to template names and an iteration, such as (1). The name also has a letter to identify the test segment, for example B.

It is important to note that the remainder is a query. The query is named in a similar manner to the contact lists, but the query name contains "remainder" at the end. You might find that this has 0 contacts. This would be expected if you canceled while the test was still running.

1. Go to Data > View Queries.
2. Find the following folder in your shared or private folder (depending on where your database is located and the send is from): AB Test Shared (or Private) Remainder.
3. Select the query name with the word remainder at the end.
4. Click Calculate Size or open the query and then calculate the query.

A/B testing and approval groups

If a member of an approval group reviews and rejects an A/B test email, the A/B test still determines a winner and sends the email to the remainder list.

The rejection of the A/B email does not equate to a canceled A/B test. To prevent the winning email from sending, you have to cancel the A/B test entirely.

Cancel an A/B test

You can cancel an A/B test from the sent list. However, if the test is scheduled, you must cancel the data job that initializes the A/B test.

1. Open the email.
2. Click A/B Test Results to open the A/B Test Status page.
3. Click Cancel Test. Your A/B test is now canceled.

Scheduled A/B tests do not show in the scheduled list. You must cancel the data job that initializes the A/B test. To do this:

1. Navigate to Resources > Data Jobs.
2. Select the appropriate Initializing A/B test data job.
3. Click the Cancel button.

Emails in the A/B test use the same cancel functionality as regular emails. Any email that is before the stop point is not sent. Any emails after the stop point is sent.

Think of it like this:

• There's a line of cars at a red light.
• When the light then turns green, the cars start to move.
• When the light turns red again, the cars stop but any cars that got past the red light, keep moving.

This is how canceled emails work. Any emails that went before the cancellation, keep going to their destinations. Any emails that are waiting in line, are canceled.

When you cancel an A/B test in Settings > Activity Reports > System Activity > Active Mailings, it does not affect the remainder. The remainder must be canceled in the Sent tab of the A/B tests. Check the data jobs to make sure it is canceled.

In the Sent tab of the A/B test status, verify that the status is Test Cancelled By User, No Remainder Sent. In the data job, you will see something like

47376871 Determining winner for A/B test
'September 2015 - Fashion 1 - Male (1)' with id of 100848 Determine A/B test winner
Cancelled

The remainder does not go to the Sent tab because no Mailing ID is generated.

Summary and status

You can view the summary and status of the email after you send the A/B test. The A/B testing report doesn't display until all emails that were sent from the test are completed.

After you configure and send the email, click Sent Email at the bottom of the Confirm and Send page. A list of sent emails are displayed on the Email and templates page.

Note: Click Scheduled to see the emails that will be sent. You can also stop and unschedule emails on this page.

The A/B Test Status page provides data and metrics for your test. On the Sent tab, click the A/B test email name link. For A/B test status, click View.

If you configured the test to notify you when the test is complete, click A/B Test Status page in the email notification.

The following table describes the states of the A/B test:

State	Description
Test not started	The A/B test is scheduled for a time in the future, and it has not yet started.
In progress	The A/B test started but has not yet determined a winner. You see this even if you manually selected a segment to send to before the configured test completion time is reached.
Winner determined	The A/B test determined a winner. You see this regardless of whether the remainder segment has been sent to.
Canceled by user	The A/B test was canceled. To cancel a test, click End Test Without Sending Remainder at the bottom of the A/B Test Status page.
Failed - system error	If the A/B test fails for some reason, it doesn't become scheduled or get started. It cleans itself up, and the only way you will know about the failure is by looking at the status on the A/B Test data job.

Error: Status failed: Error initializing A/B Test

If you have invalid data in restricted field types of your database. For example, text values in a numeric field type, you may receive an error initializing your A/B test

To resolve this issue, perform a Validate Data action on the database. (Open your database and find the Validate Data button.) When the data job is complete, download the results log. The log indicates each contact with invalid data, the affected fields, and the problem with the fields. You can either update the contact information within the database or perform an import update for those specific contacts. To make the updates in the database, click the contact email address and update the values.

Do no opt-out these contacts or remove them from the database. Although the contacts won't be mentioned in the data validation process, they could still have an impact on the A/B split test data job. After you update them with valid data, you can then opt out or remove them.

After you correct your data, rerun the Validate Data job to make sure that there are no more errors. When the data job runs clean, run your A/B split test again. If you still receive an error, contact Client Support and provide:

• Pod number
• Organization name
• User name
• Database name
• Database location (folder path)
• Query name and location, if applicable
• A/B split test data job ID that provides the failure notice
• Validate data job ID

A/B testing validation errors

A/B testing uses field-level validations. If you do not configure a field correctly, an error displays.

The following table describes common validation errors:

Error	Details
All email bodies must match the ones in the delivery options tab.	Email body settings for all templates in the A/B test must be the same. It is based on template A's draft email bodies (what's selected in delivery options, not A's template-level email body.)
The tracking level of your template is set to no tracking.	Occurs when the template tracking is off (Tracking level = No tracking) in any of the templates' settings and when A/B testing is on.
This information entered in Test Panel [A] is identical to Test Panel [C].	Occurs when any two variants in the A/B test tabs are identical.
Post to social settings are not applied to your A/B test	Occurs during the health check validation process. Review your post to social settings.
Create CRM campaign setting is not applied to your A/B test	Occurs during the health check validation process. Review your CRM settings.
Maximum messages per contact settings are not applied to your A/B test	Occurs during the health check validation process. Review your contact source settings.
A/B test schedule buttons grayed out	You can't edit this page if A/B test is on. You must click Change A/B Test Schedule to change the schedule for the A/B test.
Please provide at least 2 subject lines	Occurs when only one subject line is created.
Please select a metric to determine the winner	Occurs when the user does not select a metric from the drop-down list.
This time has already passed. Your email will be sent immediately.	Occurs if the user does not change the start date and returns to the A/B testing page an hour later.
This end date occurs before the start date. Select a date in the future.	Occurs if a scheduled end date happens before the scheduled start date.
You cannot apply an A/B test to a contact list. Please select a database or query.	Occurs when a contact list is attached to the email.
Amount of time to determine winner must be set for a longer period than your Send Time Optimization (STO).	Occurs if the STO is set to a pre-defined value for the contact source and the time to determine the winner is less than the STO value.

Errors, warnings, and spam scores

View the errors, warning, and spam scores for each email. You must return to your email draft to address errors and warnings.

• Errors show the items for each email that you must fix before you can send the A/B test email.
• Warnings show the items for each email that you might want to fix before you send the A/B test email.
• Click each tab to see the spam score for each email.

Timing of A/B tests split

For a regular, scheduled email to be sent to a query, the query calculates automatically at send time regardless when the email is scheduled. This process is ideal because it ensures that the contacts who receive the email reflect the last query calculation results at send time.

If you set up an A/B test with a query as contact source and send it immediately, the A/B split is based on who is in the query at send time.

However, if you schedule the A/B test, the A/B split is based on who meets the query criteria at the time you schedule the email, not at the time of send. The initializing A/B test data job, which is responsible for segmenting the query into A, B, C, and D, is generated when you complete the setup of the A/B test.

A/B testing and duplicate contacts

Contacts might receive multiple emails. When an A/B test is sent, each send is considered a separate email.

For example, you perform an A/B test that sends to two 10% segments of your list and the remainder is sent to the other 80% of the contacts. The contact joe@email.com exists three times in this list. Each instance is placed in one of the segments, so that he is in the first two 10% segments and the remaining 80%. In this case, the contact receives three separate emails: two for the 10% tests, and one for the winning email to the remaining 80%.

However, if two instances of joe@email.com are in the same segment, and the third instance is in another, he would receive two emails. In the segment where he exists twice, the system sends to the record with the smallest recipient ID (oldest database contact) for that email.

If all three instances of joe@email.com exist in the same segment, then he would receive one email.

Not supported with A/B testing

A/B Testing does not support the following features:

• Email-level frequency control
• Post to social
• Create CRM settings
• Automated messages or program email