We all should agree that the content of the message doesn't determine whether the customer opens your email or not. The first thing, each customer will see is the Subject line, Sender's name, and Sender's email addresses. If you want to understand how each of them can impact your open rates, you may set up an A/B test Campaign.

Omnisend provides you with the ability to test your campaigns in three different ways:

  • Testing subject line (Default Email Editor only)

  • Testing the Sender's name and Sender's email address (Default Email Editor only)

  • Testing the email content along with the Subject line and Sender's name (New Email Editor only)

This article explains how A/B testing works and how you can set up and launch an A/B test campaign on Omnisend.

Contents

Before you start your A/B test
How A/B test campaigns work
Setting up an A/B Subject line or Sender's name test campaign Setting up an A/B Subject line, Sender's name, and Content test campaign
Changing winning Campaign settings
FAQ

Before you start your A/B test

  • Although A/B testing can only perform with segments containing 10 contacts or more, we suggest having at least 2000 subscribers and splitting it to 25% for A, 25% for B, and 50% for the winner. This way, you would be sure about the accuracy of your results.

  • It is up to you to choose which metric decides the winner: opens or clicks. However, keep in mind that opens won't be a reliable source of truth once iOS 15 is released. More about this here.

  • If you change your mind about the test, you can either pause it or select the winner manually and let it through. Once paused, the resume option is available for up to 7 days from the start of the Campaign.

  • A/B test campaigns cannot be boosted.

  • Users on all Omnisend plans can use A/B test campaigns.

How A/B test campaigns work

A/B testing allows you to choose the best subject line, Sender's name, Sender's email address, or Email Content for your Campaign. While setting up the Campaign, you can select the percentage of recipients A and B versions will be sent to, and the winner will be chosen automatically.

As the clicks and opens do not happen immediately, we suggest postponing your winning Campaign for 24 hours. Your data will then be accurate enough to choose the winner correctly, and your Campaign will go out at the same time the A and B versions went out, without making any impact on the results. If that is too long of a wait, postponing for at least a couple of hours is suggested.

Setting up an A/B Subject line or Sender's name and email address test campaign

To start setting up your A/B Subject line or Sender's name and email test campaign, click on New Campaign under the Campaigns tab, locate Subject line A/B test and click on Create this test button.

Once you click it, you'll be brought to the setup wizard. Here you can change all settings for your A/B campaign.

Settings

To start the setup, choose what you would like to test: Email Subject line or Sender's name (along with the Sender's email address).

If you choose to test your subject lines, the system provides you with a form with two subject lines for each of the versions. See the image below for the Settings step when the subject line testing is chosen.

If you choose to test your Sender's name, the system provides you with a form with two Sender's names and two Sender's email fields for each of the versions. See the image below for the Settings step when the Sender's name testing is chosen.

The fields for the A/B test should not be matching, and the better-performing one is a winner. You will be able to set up the preferred portions of the A/B and winner campaign later on.

Once you complete everything within the Settings tab, click NEXT STEP to move forward.

Template

The template step allows you to choose the template you are going to be working with on your Campaign. It is the same step you encounter with regular campaigns, so feel free to select your favorite template or one of your saved ones. You can change the template chosen before you send your Campaign.

Content

This step allows you to edit all the content on your newsletter. As both A and B versions will be using the same content in the Subject line A/B test campaign, you only need to add it once. This step does not differ from the regular Campaign, and you can use all the same settings you find there. Once you complete everything within the content set up, click Next to move forward.

Further steps will be described here, as they're the same for both A/B test campaigns.

Setting up an A/B Subject line, Sender's name, and Content test campaign

To start setting up your A/B Subject line, Sender's name, and Content test campaign, click on New Campaign under the Campaigns tab, locate the Email A/B test and click on Create this test button. Keep in mind that this A/B test campaign is available for the New email builder only.

Once you click it, you'll be brought to the setup wizard. Here you can change all settings for your A/B campaign. In the Email A/B test, you can test all the parts of your email campaign simultaneously: Subject line, Sender's name, Sender's email address, and Content. However, if you want, you can test either of these parts separately (e.g. test sender's name and content, but use the same subject line and sender's email address).

Once you're all set with the Subject lines, Sender names, and email addresses for both versions, click the 'Create email' button for Version A. Select one of the templates and proceed with editing the email content. When version A is ready, click the 'Next step' button:

The email content for Version A will be saved and you'll be taken back to the Setup Wizard. You'll be able to get back to the Version A editor if needed, preview it or send the test email.

Proceed with the same steps for Version B. When both Versions are ready to go, click the 'Continue' button:

You'll be taken to the Recipients step, then.

Recipients

Once you have your content ready, the selection of your recipients follows. You can choose to send to all subscribers or particular segments. Regardless of which option you select, the A and B versions will be sent to randomly chosen contacts from the chosen segments. Make sure your chosen segment has ten contacts or more. Otherwise, A/B testing is not available. Once you complete everything within it, click NEXT STEP.

A/B settings

A/B settings step allows you to choose your preferences for your testing campaign. First, you can select the size of your test group. We suggest going with 25/25/50, but you can go as low as 10/10/80 or as high as 50/50. In such a case (50/50), there will be no winner campaign, and both versions will be sent to an equal amount of recipients. To change these options, click and drag the handle between B and Winner (see image below).

Once you have decided on the sizes of the test group, you can choose how the winning combination should be determined: by open rate or by click rate. We suggest using open rates for a more accurate outcome. You can also choose how long we should wait to decide the winner. We recommend waiting for 24 hours, but you can go as low as 1 hour or as high as seven days.

Once you set the A/B settings, click NEXT STEP to move forward.

Send

It is the last step of your campaign setup. You can now either schedule this Campaign for later or send it immediately. Once you click the send button, the process will be initiated, and A and B versions will go out to your test groups.

If you are testing a campaign with a sender's email address that is not verified yet, the system will inform you about it. It would be best if you verified it before you can start an A/B testing campaign. The verification process is as simple as clicking on the VERIFY button and checking your inbox for the email. Click a verification link there, and you are good to go.

Once everything is settled with your Sender's email address, feel free to double-check the whole setup: subject lines, Sender's name, chosen segments, and A/B testing preferences.

You can now either schedule the Campaign for later or send it immediately.

Changing winning Campaign settings

After your A and B tests are sent, you will have some time before the winning Campaign will get sent. If you wish to change its settings, you may either Stop A/B Testing, either Manually select the winner.

If you choose to stop testing, the email will be paused and available for Resume for 7 days. You won't be able to Segment your customers based on the results of the Email Campaign sending, but will be able to check the list of each of the batches recipients in the Activity tab of the Reports section.

FAQ

Can I A/B test an email at different time? Unfortunately, it is not possible. Both A and B batches should be sent simultaneously. However, you may use the A/B test Segment for splitting contacts and scheduling emails to be sent at different times.

Does Omnisend have A/B testing for automated sequences? Yes, you can add the A/B test block anywhere in the automation workflow structure.

What if I don’t have enough contacts? To do A/B testing, you need to have at least 10 subscribed contacts. Meaning those contacts, that opted-in to receive email marketing from you. A subscribed contact will appear green in All contacts.

For a more in-depth look, read about the differences of subscribed and non-subscribed here.

Did this answer your question?