A/B Test Campaigns

Learn how to create and send an A/B test campaign

Ira avatar
Written by Ira
Updated over a week ago

We all should agree that the Content of the message doesn't determine whether the customer opens your email. Each customer will first see the Subject line, the Sender's name, and the Sender email address. If you want to understand how they can impact your open rates, you may set up an A/B test Campaign.

Omnisend provides you with the ability to test your campaigns in different ways:

  • Testing the email content along with the Subject line and Sender's name;

  • Testing each separately or combining them the way you'd like.

This article explains how A/B testing works and how you can set up and launch an A/B test campaign on Omnisend.

Contents

Before You Begin

  • Although A/B testing can only be performed with segments containing ten contacts or more, we suggest having at least 2000 subscribers and splitting it to 25% for A, 25% for B, and 50% for the winner. This way, you would be sure about the accuracy of your results.

  • It is up to you to choose which metric decides the winner: opens or clicks. However, remember that opens will only be a reliable source of truth once iOS 15 is released. More about this here.

  • If you change your mind about the test, you can either pause it or select the winner manually and let it through. Once paused, the resume option is available for up to 7 days from the start of the Campaign.

  • A/B test campaigns cannot be boosted.

  • Users on all Omnisend plans can use A/B test campaigns.

How It Works

A/B testing allows you to choose the best subject line, Sender's name, Sender email address, or Email Content for your Campaign. While setting up the Campaign, you can select the percentage of recipients A and B versions sent to, and the winner will be chosen automatically.

As the clicks and opens do not happen immediately, we suggest postponing your winning Campaign for 24 hours. Your data will then be accurate enough to choose the winner correctly, and your Campaign will go out at the same time the A and B versions went out without impacting the results. If that is too long, postponing for at least a couple of hours is suggested.

Creating A/B Test Campaign

To start setting up your A/B Subject line or Sender's name and email test campaign, click on New Campaign under the Campaigns tab, locate the Subject line A/B test, and click Create this test button.

Once you click it, you'll be brought to the setup wizard. Here, you can change all settings for your A/B Campaign. In the Email A/B test, you can simultaneously test all the parts of your email campaign: Subject line, Sender's name, Sender email address, and Content.

However, if you want, you can test either of these parts separately (e.g., test the sender's name and Content but use the same subject line and Sender email address).

Once you're all set with both versions' subject lines, Sender names, and email addresses, click the 'Create email' button for Version A.

Template

Select one of the templates and edit the email content. When version A is ready, click the 'Next step' button.

The template step allows you to choose the template you will work with on your Campaign. It is the same step you encounter with regular campaigns, so feel free to select your favorite template or one of your saved ones. You can change the template chosen before you send your Campaign.

Content

This step allows you to edit all the Content in your newsletter. Once you complete everything within the Content setup, click Finish Editing.

The email content for Version A will be saved, and you'll be taken back to the Setup Wizard. If needed, you can get back to the version A editor, preview it, or send the test email.

Proceed with the same steps for Version B. When both Versions are ready, click the 'Next Step' button.

Recipients

Once you have your Content ready, the selection of your recipients follows. You can choose to send it to all subscribers or particular segments. Regardless of which option you select, the A and B versions will be sent to randomly chosen contacts from the chosen segments. Make sure your chosen segment has ten contacts or more. Otherwise, A/B testing is not available. Once you complete everything within it, click Next Step.

A/B settings

The A/B settings step allows you to choose your preferences for your testing campaign.

Here, you can decide how many recipients should receive the winning version. In the following example, the campaign's recipients will be divided into two groups: 20% will receive version A, and another 20% will receive version B. The selection of the winner will be based on the actions taken by these recipients, such as opening or clicking (depending on the chosen winning version metric).

Once you have decided on the test group size, you can choose how the winning combination should be determined: by open rate or click rate. We suggest using open rates for a more accurate outcome. You can also choose how long we should wait to decide the winner. We recommend waiting 24 hours, but you can go as low as 1 hour or as high as seven days. Once you set the A/B settings, click Next Step to move forward.

Send

It is the last step of your campaign setup. You can now either schedule this Campaign for later or send it immediately. The process will be initiated once you click the send button, and the A and B versions will go out to your test groups.

The system may inform you if you are testing a campaign with a sender's email address that still needs verification. Please verify it before you can start an A/B testing campaign. Verification is as simple as clicking the VERIFY button and checking your inbox for the email. Click a verification link there, and you are good to go.

Once everything is settled with your Sender email address, please double-check the whole setup: subject lines, Sender's name, chosen segments, and A/B testing preferences.

You can now either schedule the Campaign for later or send it immediately.

Changing Winning Settings

After sending your A and B tests, you will have time before the winning Campaign gets sent. If you wish to change its settings, you may either Stop A/B Testing or Manually select the winner.

You can do this from Reports → Campaigns → open your Campaign:

You can also do this directly from the Campaigns tab → open more options:

If you stop testing, the email will be paused and available for Resume for seven days. You won't be able to Segment your customers based on the results of the Email Campaign sending, but you can check the list of each batch recipient in the Activity tab of the Reports section.

A/B Test Report

The A/B Test Campaign Report provides comprehensive data on various metrics, allowing you to analyze the performance of your email campaign. Understanding the report and its metrics allows you to make data-driven decisions to optimize your future marketing efforts.

To access the report, navigate to the Reports tab → Campaigns → select your campaign.

In the Overview section, you will see an analysis of your A/B Test Campaign performance:

  • Emails Sent: This metric represents the total number of emails sent during the A/B test campaign.

  • View Rate: View Rate indicates the percentage of recipients who viewed the email. A higher view rate signifies better engagement and interest in your email.

  • Click Rate: The Click Rate represents the percentage of recipients who clicked on any links within the email.

  • Unsubscribe Rate: The Unsubscribe Rate signifies the percentage of recipients who unsubscribed from further communications.

  • Spam Rate: Spam Rate indicates the percentage of emails marked as spam by the recipients.

  • Revenue: Revenue reflects the total monetary value generated from the campaign. It helps you understand the direct impact of your email campaign on sales and conversions.

You will be able to analyze A/B Test Results. This section compares the performance of different versions (A and B) of your email content. It provides insights into each version's open rate, click rate, unsubscribe rate, spam rate, and revenue. By comparing the metrics, you can identify which version performed better and learn from the differences.

You will also see the first 24-hour performance data. This section displays the performance of the A/B test campaign during the first 24 hours. It includes views, clicks, and device reports. By analyzing the early performance, you can gain initial insights into the effectiveness of your campaign and make any necessary adjustments.

In the contact activity tab, you can see detailed information on actions performed by your contacts for this campaign.

FAQ

Can I A/B test an email at a different time? Unfortunately, it is not possible. Both A and B batches should be sent simultaneously. However, you may use the A/B test Segment to split contacts and schedule emails sent at different times.

Does Omnisend have A/B testing for automated sequences? You can add the A/B test block anywhere in the automation workflow structure.

What if I need more contacts? To do A/B testing, you need at least ten subscribed contacts, which means those who opted in to receive email marketing from you. A subscribed contact will appear in green in your Audience.

Read about the differences between subscribed and non-subscribed for a more in-depth look here.

Did this answer your question?