A good email marketing automation tool is a complex beast. We want lots of features from something so essential for running our business, but it's also easy to get lost in all the complexity.
For today's review, I want to look at one of the most pivotal features in email marketing tools: A/B testing.
A/B testing can make a huge difference to your business, especially if you do it consistently, over time. So let's take a look at what split testing features are available in different email marketing tools and how they work.
Let's start with an overview of the tools reviewed and the A/B testing broken down into sub-features:
A/B Testing for Broadcast Emails
A/B Test Emails in Automation Flows
A/B Test Entire Automation Flows
Optimize for Product Purchase
Note: every solution that has A/B testing features allows optimizing for opens or clicks, so I didn't include that in the comparison table.
- Solid A/B testing for subject line, from name and email content on broadcast emails.
- A/B testing can be applied to individual emails inside an automation flow, including determining the winner automatically.
- A/B testing of entirely different flows possible.
- A/B tests in automation flows can be optimized for almost any conversion goal imaginable.
- No revenue tracking, no optimizing for revenue goals.
- Not many codeless integrations with payment processors, shopping carts etc, available.
As you'll see further below, the cons of ActiveCampaign are really first world problems, compared to how lacking some of the other A/B testing feature sets are. As far as I'm concerned, all ActiveCampaign needs to be a perfect solution is revenue tracking and more integrations. As it is, you'll have to do some custom coding or integrate via Zapier, to be able to optimize for product purchase related goals.
Having said that: ActiveCampaign clearly boasts the most extensive and most powerful A/B testing features of any of the tools I've tested so far.
Not using ActiveCampaign yet? Click here to get it.
- The "journey" builder for automation flows is visual and intuitive.
- It's possible to test entirely different flows... but this is more useful in theory than in practice, with Autopilot.
- Split tests can't be tied to conversion goals.
- Too many steps and too much manual work involved in setting up a simple A/B test.
- Because of the lack of goals and need for manual intervention, the feature is impractical for automation flows.
Autopilot's A/B testing feature is the perfect example of "so close and yet so far". Having a split feature as a step to insert in an automation flow is great, but the feature's potential is ruined by a lack of conversion goals. Without goals, you can't automatically determine a winner and that makes using the feature slightly impractical for broadcast emails and very impractical for ongoing automations.
- A/B testing for broadcast subject lines is very quick and easy.
- The only option is to test 2 subject lines in a broadcast email. There are no other A/B testing features.
- The criteria for determining a winner aren't suitable for broadcasts to small lists and can't be edited.
Convertkit takes the extremely minimal approach to A/B testing. There's only one feature: testing two subject lines against each other. And it's only available in broadcast emails. The upside of this is that it's very quick and easy to use. And it is a feature you'll benefit from: make a habit of always testing subject lines and over time, the benefits really accumulate. It's too bad that there are literally no other ways to incorporate A/B testing in your Convertkit marketing campaigns.
- A/B testing is possible for emails within follow-up sequences.
- The feature is not user friendly.
- There's no A/B testing for automation flows or broadcast emails.
Drip really broke my heart on this one. In my testing so far, I'd really warmed up to it, because it's wonderfully geeky and power-user oriented. But on A/B testing, it really drops the ball. The feature that exists involves many steps to set up and is limited to emails within follow-up sequences. You can get some benefit out of the feature as it is, but it just pales in comparison to the ease of use offered in other tools or the power offered in ActiveCampaign.
- (Mostly) intuitive and quick A/B test setup for emails.
- Options for testing subject lines, email content, sender address and more.
- A/B test emails (with automatically determined winner) can be used within automation flows.
- In the first step, the feature is strangely hidden in an easy-to-miss link.
- Automation flows can't be split tested.
GetResponse is a solid 2nd rank solution, regarding A/B testing. The testing feature is well designed and easy to use (once you found the first step) and you can insert emails that are automatically optimized for conversions in your automation flows. You can't, however, test entirely separate flows against each other.
- Decent A/B testing for broadcast emails.
- No automatic winner.
- A/B testing is decent, but limited.
iContact receives just about receives a passing grade, in this comparison. The A/B testing feature isn't so bad or so limited that it's a total deal breaker, but it's also just not very good. There's nothing exciting about it and it's certainly not a reason to switch to iContact.
- Easy to use A/B testing for broadcast emails.
- Good choice of testing options (subject line, email content etc.)
- You can connect MailChimp to certain ecommerce solutions and optimize your emails for revenue(!)
- No automation flows like in many other tools.
- Hence, no testing within automation flows.
MailChimp was a very pleasant surprise in this roundup review. The standout feature is that you can optimize your emails for revenue! OMG, MailChimp, what is this? I'm starting to like you!
This is the one feature I wish other solutions would implement as well. In many cases, it's perfectly reasonable to optimize for email opens or link clicks, but for direct response type promotional emails, optimizing for revenue is awesome.
- Decent, easy to use A/B testing for broadcast emails.
- No A/B testing for automation workflows.
MailerLite offers another mid-tier entry in this roundup. There's a good enough testing feature available for broadcast emails, but that's where it ends. Better than iContact or the tools that don't offer any testing at all, but a far cry from something like ActiveCampaign.
- A/B testing available for both broadcast emails and emails within automation flows.
- No A/B testing for entirely different automation workflows.
- Feature isn't user friendly or easy to understand.
Mautic is another candidate that misses the mark. It is possible to run split tests with this tool, but setting them up and taking action on the results is a bit of a convoluted mess. User friendliness is something I take seriously because of how us humans behave. We may have good intentions and say to ourselves "testing is important and I'll do it even if the feature is complicated" but in practice, in the real world, the more complicated a feature is, the less likely we are to actually use it.
I selected the tools to test here based on what's popular and what I get asked about a lot. Among the candidates, Aweber, SendinBlue and Sendy are basically instantly disqualified because they don't have any split testing features at all.
In terms of choosing a tool to use, a lack of A/B testing is a deal breaker for me. It's a reason to not start using any of these tools (even if they're cheaper or have other good features) and it's a reason to switch from these tools to something else, if you're currently using them.
What's With Me and ActiveCampaign?
I've been recommending ActiveCampaign for a long time and as you can see, it comes off as a clear winner, here. Plus, it's the first one in the list!
This may make you suspicious. Am I just fluffing ActiveCampaign, here?
I was actually surprised at the results from this test. I'd been warming up to Drip over the course of my testing (it has some features I really, really love) and I was expecting it to have a much stronger showing here. I was also surprised that Autopilot didn't fair better and looking at the comparison table at the top: going into the review, I would not have guessed that there'd be so many red crosses and so few green checkmarks in the end.
So, no: this review wasn't set up to make ActiveCampaign look good. And it's first in the list because I sorted the tools alphabetically.
Needless to say, now that I've investigated all these tools more closely, it further reinforces my recommendation of ActiveCampaign. I do hope that other tools will catch up, though.
Want to Learn the Ninja Tricks? Want to Switch?
I'd like to know from you: are you interested in learning some advanced marketing tricks that can be done with ActiveCampaign (or a different tool)? Would you be willing to pay for a course on the topic?
Also, if you're using a different solution, do you need help switching?
Let me know your thoughts on this by leaving a comment below.
And of course, if there are other tools you think I should include in a future update to this review, please let me know!