November 27, 2017 - 50 Comments
A good email marketing automation tool is a complex beast. We want lots of features from something so essential for running our business, but it's also easy to get lost in all the complexity.
For today's review, I want to look at one of the most pivotal features in email marketing tools: A/B testing.
A/B testing can make a huge difference to your business, especially if you do it consistently, over time. So let's take a look at what split testing features are available in different email marketing tools and how they work.
More...
Let's start with an overview of the tools reviewed and the A/B testing broken down into sub-features:
A/B Testing for Broadcast Emails | Automatic Winner | A/B Test Emails in Automation Flows | A/B Test Entire Automation Flows | Optimize for Product Purchase | |
|---|---|---|---|---|---|
Note: every solution that has A/B testing features allows optimizing for opens or clicks, so I didn't include that in the comparison table.
Pros:
Cons:
As you'll see further below, the cons of ActiveCampaign are really first world problems, compared to how lacking some of the other A/B testing feature sets are. As far as I'm concerned, all ActiveCampaign needs to be a perfect solution is revenue tracking and more integrations. As it is, you'll have to do some custom coding or integrate via Zapier, to be able to optimize for product purchase related goals.
Having said that: ActiveCampaign clearly boasts the most extensive and most powerful A/B testing features of any of the tools I've tested so far.
Not using ActiveCampaign yet? Click here to get it.
Pros:
Cons:
Autopilot's A/B testing feature is the perfect example of "so close and yet so far". Having a split feature as a step to insert in an automation flow is great, but the feature's potential is ruined by a lack of conversion goals. Without goals, you can't automatically determine a winner and that makes using the feature slightly impractical for broadcast emails and very impractical for ongoing automations.
Pros:
Cons:
Convertkit takes the extremely minimal approach to A/B testing. There's only one feature: testing two subject lines against each other. And it's only available in broadcast emails. The upside of this is that it's very quick and easy to use. And it is a feature you'll benefit from: make a habit of always testing subject lines and over time, the benefits really accumulate. It's too bad that there are literally no other ways to incorporate A/B testing in your Convertkit marketing campaigns.
Pros:
Cons:
Drip really broke my heart on this one. In my testing so far, I'd really warmed up to it, because it's wonderfully geeky and power-user oriented. But on A/B testing, it really drops the ball. The feature that exists involves many steps to set up and is limited to emails within follow-up sequences. You can get some benefit out of the feature as it is, but it just pales in comparison to the ease of use offered in other tools or the power offered in ActiveCampaign.
Pros:
Cons:
GetResponse is a solid 2nd rank solution, regarding A/B testing. The testing feature is well designed and easy to use (once you found the first step) and you can insert emails that are automatically optimized for conversions in your automation flows. You can't, however, test entirely separate flows against each other.
Pros:
Cons:
iContact receives just about receives a passing grade, in this comparison. The A/B testing feature isn't so bad or so limited that it's a total deal breaker, but it's also just not very good. There's nothing exciting about it and it's certainly not a reason to switch to iContact.
Pros:
Cons:
MailChimp was a very pleasant surprise in this roundup review. The standout feature is that you can optimize your emails for revenue! OMG, MailChimp, what is this? I'm starting to like you!
This is the one feature I wish other solutions would implement as well. In many cases, it's perfectly reasonable to optimize for email opens or link clicks, but for direct response type promotional emails, optimizing for revenue is awesome.
Pros:
Cons:
MailerLite offers another mid-tier entry in this roundup. There's a good enough testing feature available for broadcast emails, but that's where it ends. Better than iContact or the tools that don't offer any testing at all, but a far cry from something like ActiveCampaign.
Pros:
Cons:
Mautic is another candidate that misses the mark. It is possible to run split tests with this tool, but setting them up and taking action on the results is a bit of a convoluted mess. User friendliness is something I take seriously because of how us humans behave. We may have good intentions and say to ourselves "testing is important and I'll do it even if the feature is complicated" but in practice, in the real world, the more complicated a feature is, the less likely we are to actually use it.
I selected the tools to test here based on what's popular and what I get asked about a lot. Among the candidates, Aweber, SendinBlue and Sendy are basically instantly disqualified because they don't have any split testing features at all.
In terms of choosing a tool to use, a lack of A/B testing is a deal breaker for me. It's a reason to not start using any of these tools (even if they're cheaper or have other good features) and it's a reason to switch from these tools to something else, if you're currently using them.
I've been recommending ActiveCampaign for a long time and as you can see, it comes off as a clear winner, here. Plus, it's the first one in the list!
This may make you suspicious. Am I just fluffing ActiveCampaign, here?
I was actually surprised at the results from this test. I'd been warming up to Drip over the course of my testing (it has some features I really, really love) and I was expecting it to have a much stronger showing here. I was also surprised that Autopilot didn't fair better and looking at the comparison table at the top: going into the review, I would not have guessed that there'd be so many red crosses and so few green checkmarks in the end.
So, no: this review wasn't set up to make ActiveCampaign look good. And it's first in the list because I sorted the tools alphabetically.
Needless to say, now that I've investigated all these tools more closely, it further reinforces my recommendation of ActiveCampaign. I do hope that other tools will catch up, though.
I'd like to know from you: are you interested in learning some advanced marketing tricks that can be done with ActiveCampaign (or a different tool)? Would you be willing to pay for a course on the topic?
Also, if you're using a different solution, do you need help switching?
Let me know your thoughts on this by leaving a comment below.
And of course, if there are other tools you think I should include in a future update to this review, please let me know!
What’s the best webinar software out there? This seemingly simple question can be frustratingly difficult to answer, as I found out in the course of this review project. My goal was to field test every major platform in the webinar space to find out how they all stack up against each other and finally find something
Read MoreMany webinar tools have added a directory feature in the recent past. The promise is intriguing: list your webinar in a public directory and you could get more traffic and attendees! But does it work in practice? Should you consider webinar directories for your webinar marketing? In this post, I’ll explain why these directories are (currently)
Read MoreRunning an online business always involves some degree of creative work. That can be anything from graphic design and photo editing for content on your website to video and audio editing for video content and podcasts. The default tool suite for this kind of creative work is Adobe’s Creative Cloud. Many users are frustrated by the
Read MoreWhat’s the best setup & gear for hosting live webinars? What do you need, to ensure high quality audio, video and an overall positive experience for your attendees? In this post, I’ll show you a behind-the-scenes look at the webinar streaming setup that I currently use, after doing countless webinars and tweaking my setup continuously, over
Read MoreIn my extensive testing of webinar tools, there’s an issue I complained about a couple of times: limited runtimes for webinars. For example, Crowdcast and Webinar Ninja events are limited to a maximum of 2 hours. Now, 2 hours seems like a pretty long time… and these days, everything’s instant and social and people’s attention spans are
Read More