Any marketer wants to understand how different variations in the automation sequence can help to gain a better result. The metric can vary from the open or click rate to the number of sales the workflow reports.
In this article, we explain how you can add an A/B test split to the automation workflow and which metrics you can check.
Contents
Set-up process
To add the A/B test split to the automation workflow, simply drag the A/B test block to the sequence.
A/B testing block has the following properties:
only two paths can be added to the block settings, but you can split the Split;
unlike Split condition, the A/B testing block may have a single resolution pass;
all of the blocks, except for the Conditional Split, can be added to the A/B testing paths. You can add another A/B test block to one of the flow paths.
Note, if you're adding an A/B testing block to enabled automation, it won't work for customers that are already in the workflow, only for those who'll enter this workflow later.
A/B test paths percentage
In the block settings, you will need to specify the percentage of the customers passing through the A and B path of the flow.
The default value is set to 50/50, but you can change it to any other fraction. In case the goal of the automation is to test different instances, a 50 to 50 ratio is the best. However, if you want to distribute the discount in a random manner, it might be a good decision to change the value to a smaller fraction, for example, 30 to 70 or vise versa.
Deleting the A/B test block
If you choose to delete the A/B test block, you will need to select what part of the block should be removed: the whole block, the A, or the B part.
Note, in the A/B test block you can set the settings to 0 and 100%, therefore disabling one of the paths instead of removing it completely.
Choosing the condition for the A/B testing
One of the most critical aspects of A/B testing is choosing the subject of the test and metric determining the winner. Although you may want to test several parameters, it is crucial only to check one thing at a time to get accurate results. Some of the assets you may want to test include:
Subject lines, Pre-headers, and sender's email addresses;
Different channels, and their combinations;
The time when the message gets most of the attention;
The content of the message, including the image-to-text ratio, headers, discounts, etc.
💬 Check out the Blog article to learn how you can Make Your Automated Messages More Relevant with A/B Testing.
Estimating the result of the test
Finally, you will also want to understand what impact each of the paths had on the automation performance. After choosing the metric, you should also understand what parameters can influence the result. For example, the open rate is mostly determined by the subject line, pre-header, or sender's name. In contrast, the content of the email message determines your customers' engagement and the number of orders they place.
After clicking on the Show stats button, you will get your workflow key conversion metrics like open, click, and sales data collected within the last 30 days directly to the automation editor. That will help you to make a quick decision on the best path performer and switch your automation accordingly. To see the lifetime performance, you will need to switch to the Reports section.
Sent — number of messages sent;
Opened — open rate;
Clicked — click-through rate;
Sales — sales attributed to the messages;
Completed — number of customers who passed through the workflow stage or received the message from it.
Skipped — number of customers that didn't receive the message due to the channel opt-in status.
Once you have enough data to decide on the best performer, you can click on the button to set one of the paths to 100%:
Or you can also add another Split conditions and tailor your targeting based on the test results. For example, in the B path, we see that 200 customers have skipped the message, which means that they didn't subscribe to the SMS channel. But the performance of the path is much better in terms of sales. Based on this data, instead of choosing one of the paths explicitly, we may add Conditional Split checking if the customer subscribed to the SMS channel.
When you start the A/B testing, make sure you provide enough time to collect the metric, but do not delay the final decision too much. Once you understand what is performing better, try to adjust your workflow settings accordingly.
Use cases
Although the A/B test block logic seems to be simple, it is a powerful solution giving you a new level of visibility onto your contacts' behavior. To help you built your strategy, we also want to share a few ideas and recommendations.
Splitting flow into three or more paths with equal probability
Although it is not possible to change the number of the A/B test branches, you may reach that goal by adding another Split to one of the paths. As you may see from the screenshot, we first split the flow into two unequal parts, and then add a 50/50 Split.
That's it! We've got three branch-split. Applying the same logic, you can reach even more divergency 🚀
Test different delays with A/B testing block
If you are looking into the ability to A/B test different delays, we would recommend adding the same message to each of the workflow branches. It will provide better visibility into your workflow performance, showing opens, clicks, and sales separately.
Note! You can clone your message block and drag it between your Split branches.
💬 You can also find even more examples and recommendations for A/B testing in our Blog article: Make Your Automated Messages More Relevant with A/B Testing.
We've already shared a few ideas but will be glad to review any of your concerns and look for the solution together 💪 Feel free to jump on a chat or email us at [email protected].