Any marketer wants to understand how different variations in the automation sequence can help to gain a better result. The metric can vary from the open or click rate to the number of sales the workflow reports.
This article explains how you can add an A/B test split to the automation workflow and which metrics you can check.
Contents
Setup process
Choosing the condition for the A/B testing
Estimating the result of the test
Use cases
Setup process
To add the A/B test split to the automation workflow, drag the A/B test block to the sequence.
A/B testing block has the following properties:
Only two paths can be added to the block settings, but you can split the Split;
unlike Split condition, the A/B testing block may have a single resolution pass;
all blocks, except for the Conditional Split, can be added to the A/B testing paths. You can add another A/B test block to one of the flow paths.
Note, if you're adding an A/B testing block to already enabled automation, it won't work for customers who are already in the workflow, only for those who'll enter this workflow later.
A/B test path percentage
In the block settings, you will need to specify the percentage of the customers passing through the A and B paths of the flow.
The default value is set to 50/50, but you can change it to any other fraction. In case the goal of the automation is to test different instances, a 50 to 50 ratio is the best.
However, if you want to distribute the discount randomly, it might be a good decision to change the value to a smaller fraction, for example, 30 to 70, or vice versa.
Deleting the A/B test block
If you choose to delete the A/B test block, you must select what part of the block should be removed: the whole block, the A, or the B part.
Note, in the A/B test block, you can set the settings to 0 and 100%, therefore disabling one of the paths instead of removing it completely.
Choosing the condition for the A/B testing
One of the most critical aspects of A/B testing is choosing the subject of the test and the metric determining the winner. Although you may want to test several parameters, it is crucial only to check one thing at a time to get accurate results. Some of the assets you may want to test include:
Subject lines, Pre-headers, and sender's email addresses;
Different channels and their combinations;
The time when the message gets most of the attention;
The content of the message, including the image-to-text ratio, headers, discounts, etc.
💬 Check out the Blog article to learn how you can Make Your Automated Messages More Relevant with A/B Testing.
Estimating the result of the test
Finally, you will also want to understand what impact each of the paths had on the automation performance. After choosing the metric, you should also understand what parameters can influence the result. For example, the open rate is mainly determined by the subject line, pre-header, or sender's name. In contrast, the content of the email message determines your customers' engagement and the number of orders they place.
After clicking the Show stats button, you will get your workflow key conversion metrics like open, click, and sales data collected within the last 30 days directly to the automation editor.
That will help you quickly decide on the best path performer and switch your automation accordingly. To see the lifetime performance, you must switch to the Reports section.
Sent — number of messages sent;
Opened — open rate;
Clicked — click-through rate;
Sales — sales attributed to the messages;
Completed — number of customers who passed through the workflow stage or received the message.
Skipped — number of customers that didn't receive the message due to the channel opt-in status.
Once you have enough data to decide on the best performer, you can click on the button to set one of the paths to 100%:
On the other hand, you can also add other split conditions and tailor your targeting based on the test results.
When you start the A/B testing, make sure you provide enough time to collect the metric, but take your time with the final decision. Once you understand what is performing better, adjust your workflow settings accordingly.
Use cases
Although the A/B test block logic seems to be simple, it is a powerful solution, giving you a new level of visibility into your contacts' behavior. To help you build your strategy, we also want to share a few ideas and recommendations.
Splitting flow into three or more paths with equal probability
Although changing the number of the A/B test branches is impossible, you may reach that goal by adding another Split to one of the paths. As you may see from the screenshot, we first split the flow into two unequal parts and then add a 50/50 Split.
That's it! We've got three branch-split. Applying the same logic, you can reach even more divergency 🚀
Test different delays with A/B testing block
If you are looking into the ability to A/B test different delays, we recommend adding the same message to each workflow branch. It will provide better visibility into your workflow performance, separately showing opens, clicks, and sales.
Note! You can clone your message block and drag it between your Split branches.
💬 You can find even more examples and recommendations for A/B testing in our Blog article: Make Your Automated Messages More Relevant with A/B Testing.
We've already shared a few ideas but will be glad to review any of your concerns and look for the solution together 💪 Feel free to jump on a chat or email us at [email protected].