A good email marketing automation tool is a complex beast. We want lots of features from something so essential for running our business, but it's also easy to get lost in all the complexity.
For today's review, I want to look at one of the most pivotal features in email marketing tools: A/B testing.
A/B testing can make a huge difference to your business, especially if you do it consistently, over time. So let's take a look at what split testing features are available in different email marketing tools and how they work.
More...
Comparison Table
Let's start with an overview of the tools reviewed and the A/B testing broken down into sub-features:
A/B Testing for Broadcast Emails | Automatic Winner | A/B Test Emails in Automation Flows | A/B Test Entire Automation Flows | Optimize for Product Purchase | |
---|---|---|---|---|---|
Note: every solution that has A/B testing features allows optimizing for opens or clicks, so I didn't include that in the comparison table.
ActiveCampaign
Pros:
- Solid A/B testing for subject line, from name and email content on broadcast emails.
- A/B testing can be applied to individual emails inside an automation flow, including determining the winner automatically.
- A/B testing of entirely different flows possible.
- A/B tests in automation flows can be optimized for almost any conversion goal imaginable.
Cons:
- No revenue tracking, no optimizing for revenue goals.
- Not many codeless integrations with payment processors, shopping carts etc, available.
As you'll see further below, the cons of ActiveCampaign are really first world problems, compared to how lacking some of the other A/B testing feature sets are. As far as I'm concerned, all ActiveCampaign needs to be a perfect solution is revenue tracking and more integrations. As it is, you'll have to do some custom coding or integrate via Zapier, to be able to optimize for product purchase related goals.
Having said that: ActiveCampaign clearly boasts the most extensive and most powerful A/B testing features of any of the tools I've tested so far.
Not using ActiveCampaign yet? Click here to get it.
Pros:
- The "journey" builder for automation flows is visual and intuitive.
- It's possible to test entirely different flows... but this is more useful in theory than in practice, with Autopilot.
Cons:
- Split tests can't be tied to conversion goals.
- Too many steps and too much manual work involved in setting up a simple A/B test.
- Because of the lack of goals and need for manual intervention, the feature is impractical for automation flows.
Autopilot's A/B testing feature is the perfect example of "so close and yet so far". Having a split feature as a step to insert in an automation flow is great, but the feature's potential is ruined by a lack of conversion goals. Without goals, you can't automatically determine a winner and that makes using the feature slightly impractical for broadcast emails and very impractical for ongoing automations.
Convertkit
Pros:
- A/B testing for broadcast subject lines is very quick and easy.
Cons:
- The only option is to test 2 subject lines in a broadcast email. There are no other A/B testing features.
- The criteria for determining a winner aren't suitable for broadcasts to small lists and can't be edited.
Convertkit takes the extremely minimal approach to A/B testing. There's only one feature: testing two subject lines against each other. And it's only available in broadcast emails. The upside of this is that it's very quick and easy to use. And it is a feature you'll benefit from: make a habit of always testing subject lines and over time, the benefits really accumulate. It's too bad that there are literally no other ways to incorporate A/B testing in your Convertkit marketing campaigns.
Pros:
- A/B testing is possible for emails within follow-up sequences.
Cons:
- The feature is not user friendly.
- There's no A/B testing for automation flows or broadcast emails.
Drip really broke my heart on this one. In my testing so far, I'd really warmed up to it, because it's wonderfully geeky and power-user oriented. But on A/B testing, it really drops the ball. The feature that exists involves many steps to set up and is limited to emails within follow-up sequences. You can get some benefit out of the feature as it is, but it just pales in comparison to the ease of use offered in other tools or the power offered in ActiveCampaign.
GetResponse
Pros:
- (Mostly) intuitive and quick A/B test setup for emails.
- Options for testing subject lines, email content, sender address and more.
- A/B test emails (with automatically determined winner) can be used within automation flows.
Cons:
- In the first step, the feature is strangely hidden in an easy-to-miss link.
- Automation flows can't be split tested.
GetResponse is a solid 2nd rank solution, regarding A/B testing. The testing feature is well designed and easy to use (once you found the first step) and you can insert emails that are automatically optimized for conversions in your automation flows. You can't, however, test entirely separate flows against each other.
iContact
Pros:
- Decent A/B testing for broadcast emails.
Cons:
- No automatic winner.
- A/B testing is decent, but limited.
iContact receives just about receives a passing grade, in this comparison. The A/B testing feature isn't so bad or so limited that it's a total deal breaker, but it's also just not very good. There's nothing exciting about it and it's certainly not a reason to switch to iContact.
MailChimp
Pros:
- Easy to use A/B testing for broadcast emails.
- Good choice of testing options (subject line, email content etc.)
- You can connect MailChimp to certain ecommerce solutions and optimize your emails for revenue(!)
Cons:
- No automation flows like in many other tools.
- Hence, no testing within automation flows.
MailChimp was a very pleasant surprise in this roundup review. The standout feature is that you can optimize your emails for revenue! OMG, MailChimp, what is this? I'm starting to like you!
This is the one feature I wish other solutions would implement as well. In many cases, it's perfectly reasonable to optimize for email opens or link clicks, but for direct response type promotional emails, optimizing for revenue is awesome.
MailerLite
Pros:
- Decent, easy to use A/B testing for broadcast emails.
Cons:
- No A/B testing for automation workflows.
MailerLite offers another mid-tier entry in this roundup. There's a good enough testing feature available for broadcast emails, but that's where it ends. Better than iContact or the tools that don't offer any testing at all, but a far cry from something like ActiveCampaign.
Mautic
Pros:
- A/B testing available for both broadcast emails and emails within automation flows.
Cons:
- No A/B testing for entirely different automation workflows.
- Feature isn't user friendly or easy to understand.
Mautic is another candidate that misses the mark. It is possible to run split tests with this tool, but setting them up and taking action on the results is a bit of a convoluted mess. User friendliness is something I take seriously because of how us humans behave. We may have good intentions and say to ourselves "testing is important and I'll do it even if the feature is complicated" but in practice, in the real world, the more complicated a feature is, the less likely we are to actually use it.
Disqualified
I selected the tools to test here based on what's popular and what I get asked about a lot. Among the candidates, Aweber, SendinBlue and Sendy are basically instantly disqualified because they don't have any split testing features at all.
In terms of choosing a tool to use, a lack of A/B testing is a deal breaker for me. It's a reason to not start using any of these tools (even if they're cheaper or have other good features) and it's a reason to switch from these tools to something else, if you're currently using them.
What's With Me and ActiveCampaign?
I've been recommending ActiveCampaign for a long time and as you can see, it comes off as a clear winner, here. Plus, it's the first one in the list!
This may make you suspicious. Am I just fluffing ActiveCampaign, here?
I was actually surprised at the results from this test. I'd been warming up to Drip over the course of my testing (it has some features I really, really love) and I was expecting it to have a much stronger showing here. I was also surprised that Autopilot didn't fair better and looking at the comparison table at the top: going into the review, I would not have guessed that there'd be so many red crosses and so few green checkmarks in the end.
So, no: this review wasn't set up to make ActiveCampaign look good. And it's first in the list because I sorted the tools alphabetically.
Needless to say, now that I've investigated all these tools more closely, it further reinforces my recommendation of ActiveCampaign. I do hope that other tools will catch up, though.
Want to Learn the Ninja Tricks? Want to Switch?
I'd like to know from you: are you interested in learning some advanced marketing tricks that can be done with ActiveCampaign (or a different tool)? Would you be willing to pay for a course on the topic?
Also, if you're using a different solution, do you need help switching?
Let me know your thoughts on this by leaving a comment below.
And of course, if there are other tools you think I should include in a future update to this review, please let me know!
Ontraport… ;)
What are you trying to tell us with this comment, David?
Ontraport is prohibitively expensive and it doesn’t have automatic winner selection in its A/B testing, according to their knowledge base. Do you think it would fare well in this comparison?
I understand Ontraport is even more expensive than InfusionSoft, and has fewer features. Both are way to expensive for me, anyway. Personally, I think there are 3 uses for the email softwares: 1. Broadcast (of information or offers), 2., Nurture type emails workflows (teaching about something), and 3. Contingency emails (why didn’t you buy? or upsells, perhaps). It’s too easy to get involved with (and waste time with) #3 complicated workflows like why did you not buy, etc.
I keep hearing a voice in the back of my head saying it’ll be easier for me to make money teaching this stuff rather than applying it to selling a service like SEO or PPC services? I moved away from the idea of making websites for people because it’s somewhat “1-off” and done, with a small monthly maintenance fee. I am targeting running businesses with websites that need more marketing on a monthly basis ($700/month and up). In 2 months, had 4 bites, but no customers.
I would say that the value of the pure broadcast email is declining quite steeply. Without some targeting and behavior-based stuff, email messages are just so likely to get lost in the fray, these days.
Regarding doing this as a service: for sure, there’s a lot of complexity here and I bet it’s something that 90%+ of businesses don’t do very well. So there’s opportunity in this, for sure.
So you say Email messages are on the decline. Where do you see “marketing” gravitating to in the future?
Shane said pure broadcast. That means what most people call newsletter, right?
I guess (and I desperately hope!) email based targeted communication with clients is not declining but the future of marketing.
Save me, Shane.
Response rates to emails in general have been going down for years and that’s not going to change.
One thing you can do to counter this is to use targeting and segmentation. However, you can’t escape the decline. Highly targeted, segmented and relevant emails will get greater engagement rates than untargeted broadcasts, yes. But had you sent the same, highly targeted messages a few years ago, engagement rates would have been far higher than they are now.
Ok, that’s kind of bad news. So matt’s question is not only legitimate but essential.
We temporally focus on automated email marketing, using Thrive Themes and Active Campaign as suggested.
If that’s not the future, we really need to know, where to turn to.
Shane, can’t we have some kind of strategy course where people can learn how to enter a market / build a niche using the great tools we have here?
Even if they start with no client / subscriber at all?
That’s what we learn from our clients daily. Although they have good products or concepts to sell, they are totally lost in marting hell, knowing not where to put their effort
Facebook, Google Ads and all the stuff…
We tend to recommend automated email marketing to them.
But now…
We really need your expertise to build a future-based strategy.
Hi Shane, Thanks for the info and YES! Active Campaign Rocks! but, the email creator could be improved. From what I understand, Drip has a much better email editor? Finally, yes I wish there was a decent course for AC. Emails, Sequences, and it would be nice if they had a feature like AWEBER that allows sharing emails or just plain uploading a written series of emails.
Thank you for your comment, Sherwood! I agree that the email composer in AC isn’t that thrilling. The one in Drip is just a lot simpler, which I prefer. I send text-only emails anyway.
What do you mean about sharing emails? Like templates?
I always look for someone who has gone farther down the path to be my guide. Plus, you’re a great teacher! I’m interested.
Thank you, Steven!
Yes. a course on Ac would be good.
Thanks for your comment, Rory.
I’ve gone pretty deep into ActiveCampaign, so I think I could put together a decent course on that. :)
Scrolling through that table, it was interesting to see that ActiveCampaign had received green checkmarks across the 5 criteria. I was also surprised that Drip didn’t have more green checkmarks.
I’m in the middle of deciding which email marketing solution I want to use, and this has caused me to pause and really consider whether I should go with Drip, or choose something else.
For affiliate marketers, the thing about Active Campaign is that, allegedly, you can’t use affiliate links in Active Campaign emails. As for me, personally, I’m an author, so I only see myself using it to build a readers’ list and send links to non-affiliate book pages or my site itself.
Yeah, I was surprised about Drip as well. It’s such a good tool in almost every other way. But the A/B testing leaves a lot to be desired.
Regarding the affiliate marketing, I’m working on a post that clarifies the various policies of the different email marketing tools, since that’s a question that comes up a lot.
Shane and D.K., there are a number of reports of Drip email sequences going out late, and thus missing their mark, not correlating with your offer, and resulting in a loss of revenue ($15 – $20K in one guy that puts together master mind meeting’s case. He left Drip for that reason.
This kind of thing is really difficult to test for, unfortunately. Technical problems like this can be a total deal breaker (and were the reason I left Ontraport back when it was still OfficeAutopilot), but I don’t think there’s any way to reliably test and include something like this in a review. :/
Great overview! thx- and I’m a happy AC and Thrive user…..would be great of one day the API integration between AC and Thrive would be “improved”- Any news on that?
Thank you for your comment, Peter.
What would you like to see changed or improved, in the integration?
I just picked up a couple licenses to ConvertFox during their lifetime deal. Could you take a look at it and see if it will be compatible with Thrive? I haven’t even had a chance to dig in just yet. I am curious where it stacks up against the competition?
I had a brief look at their website and from what I saw, it seems highly unlikely that they can compete with any of the tools presented here…
Hey Shane, thanks for testing all these services. I was inclined to chose MailerLite, because of there great plans for starting out. (First 1.000 subscribers free) Because of this and earlier recommendation of you I am now seriously looking into ActiveCampaign.
I would love it if you would write a post / make a podcast, about how to write good email campaigns for people starting of with email campaigns.
– What to put in your emails
– How many emails
– How to build great email sequens
– How to engage your readers
– etc
Thanks for al the great tutorials!
Ah yes, that’s a whole other can of worms as well. :)
I do think I could put something together, since I have a decent amount of experience in email marketing. I’ll have a think about it.
Yes a course on active campaign would be great.
Thank you, Jerry!
Yes to paid course on AC. Especially from someone actively using it for online business.
Thanks a bunch Shane!
Thank you, Tommy!
Did anyone come across any self-hosted email service provider that are half decent? Apparently, Sendy cuts the cost by a lot, but it’s not exactly intuitive to use. The other two I came across are Mailtrain and Interspire, but not sure whether they have any A/B testing feature if at all.
Would love to know about the possibility of developing a WordPress plugin for delivering email using one’s own server (Shane! If that sounds like a good product idea that can beat the hell out of the current market I would love to hear more!)
If not, a course on Active Campaign (with affiliate link of course) would be nice. I’m currently with Mailchimp for its easiness to craft well-design emails but am preparing to do the switch and stepping up the game.
Hello Thomas,
Personally, I would avoid self-hosted email marketing solutions because for me, it’s too expensive and too complicated. And yes, I really mean “too expensive” because sure, the initial costs are lower, but I’m not going to be messing with email servers, delivery issues and so on, so I will have to pay for a specialized service and for a person to handle the technical details and that’s going to get really expensive at scale. So, self hosted solutions are something I wouldn’t personally use and I also can’t recommend for those reasons.
Wow. Thanks for the insight Shane. I’m pretty new to this self-hosted thingy so I’m not very well versed on the hidden cost associated with it. I did find a trend of companies jumping into this Amazon SES bandwagon (basically leveraging Amazon’s servers to send emails at low cost), but I don’t think there’s a Web application of ActiveCampaign’s simplicity and caliber that is built based on Amazon SES yet.
InfusionSoft.
For the simple sake of clearly showing how much their platform sucks!
+1 for an advanced course on AC.
Thank you, Nick! I agree that Infusionsoft is a bit of a train wreck. I don’t have an account with them anymore, because they insisted that I had to pay $300/month even for a test account that would only be used for our integration testing for Thrive Leads, with no contacts or email sending activity…
Shane, awesome stuff like always. Thank you for the great deeds you are doing here. Sure I’m willing to pay for a course in Active Campaign. I’d be glad to end the search and the email marketing tool hopping.
If we Thrivers would agree to one tool and learn all the tricks in using it to the fullest, I’d be extremely thankful.
With this said, I’d love to see Klick-Tipp being involved in your research.
They appear very straight to the subject of clean conversion focused email marketing and it feels to me that they have a spirit very similar to Thrives Themes.
However, they have at least 1 downside for they are currently in German language only.
On the other hand, they seem to be flexible enough in many things. Maybe we can discuss some kind of joint venture with them.
Thank you, Chris!
Klick-Tipp I didn’t include because they are Germany-only and that’s only a very small percentage of my audience.
I would really like a course that uses Thrive Themes and AC. I have both, and I’m learning so much lately. It’s been driven home (through your podcasts, blogs, and a tech marketing online coach) that I need to have control of everything personally, but it’s a lot. My site was built by a designer using TT. She is now a coach, and I’m in her class. I’m not a tech person, nor a marketing expert. I do bookkeeping and accounting things. So, a course that starts with ‘click here’ and do this… Then watch the results… And, it teaches me what I’m doing, would be great.
People are frequently trying to sell me things like Click Funnels, which my coach tells me I can do via TT, and Contactually, which has functions that are available in AC. I’m trying to go slow enough to not break the bank with SaaS. I’d like to use both TT and AC to their fullest abilities.
I appreciate how comprehensive both AC and TT are, but it’s simply a lot to absorb. Thanks for branching out an offering these sorts of lessons.
Thank you for your comment, Barb!
It is a lot, yes. This is one of the difficult things about being an entrepreneur: it requires so many different skills and knowledge in so many areas.
I am using Zaxaa BEATS (soon to be rebranded as BirdSend for its upcoming version 2). It has decent A/B testing options and if I compare it with others according to the table above, then I think it will get first 3 ticks and last 2 cross, just like GetResponse above. I really like Zaxaa (their cart solution as well as their email autoresponder), but it seems Zaxaa is not that known amongst the internet marketing community.
Thank you for your comment.
Zaxaa is indeed not as popular as it perhaps deserves to be. I didn’t include it here because it’s not primarily and email marketing platform.
Thanks for the post man, that’s very helpful. It’s actually a good rundown of these tools in general, besides A/B testing.
I’ll just add my vote for the AC course. Would buy. Once I could afford AC, that is.
‘Cause the thing about AC – besides affiliate links, which I didn’t know about but would like you to clarify in another post – is the hefty price tag, even for a beginner LITE account.
It’s obviously the best tool in this review, but it would be nice to know which tool would give you the most value for a beginner’s budget.
As far as I can tell, the winner among the most affordable tools in your second category (below AC, but useful) would be MailerLite. It seems to be the only cheap solution that has a proper workflow builder. What it lacks is tagging. I’ve never tried it though, so I’d like to know your opinion.
Has any of you guys worked with it? Problems?
Yes, of course, a paid course on this great tool. Now migrating from Aweber to AC. Thanks Shane for your great work!!
Great review! I just miss Mailchimp’s multivariate testing from the article.
Hi Shane, another very relevant and useful analysis! You included several services I hadn’t heard of. However, I am a bit surprised and disappointed that Ontraport was not included. With the latest version, it has graphical workflows, revenue tracking, very granular analysis of how various parts of a campaign are working, and more (that I don’t quite understand). I’m also a huge fan of AC and at one point was very eager to switch over, but ultimately, one the first complex workflow I tried on it, it just didn’t have the capability. AC is still great, but OP is certainly worthy of a look and a comparison here. It has a lot of power under the hood that the other services don’t have, and in that way is very underpriced compared to HubSpot and others that become very pricey for similar advanced capability.
Thanks,
Bhavesh
p.s. I would note there’s no reason not to use 2 different services depending on needs and complexity of what needs to be accomplished.
Thank you for your comment! I’ll give Ontraport another look, then.
I don’t agree that using 2 systems is a good idea, though. You have to take extra steps to synchronize them with each other, you have to set up a system by which people who unsubscribe from one are also unsubscribed from the other and so on. A single system should be capable of doing all the follow-up marketing you need to do, so you can have good targeting, segmentation and personalization across the board.
Why have you totally ignored Bhavesh Patel post above? Seems like it totally effects you testing above? Do you need to amend what you stated?
In what way have I ignored it?
I’m afraid you got it wrong with GetResponse.
At 28:10 you say: “we can choose to A/B test an email within this flow”.
In the post section, you write, “you can insert emails that are automatically optimized for conversions in your automation flows”.
NO, you can’t.
Choosing an A/B test from that dropdown only COPIES the text of the A/B message to create another message that is then used in the workflow. I just spent 20 minutes on the live chat with the support guys over this, because I simply couldn’t believe it.
But this is the answer I got: “The message content only gets copied into the workflow. It becomes a separate message that appears like one of the messages in your A/B test. It doesn’t utilize the A/B test feature in workflows.”
GetResponse can only send A/B tests to a specific group of recipients, at a specific date and time, and THEN, when the A/B test is done, you can use the winner as a normal email message.
In other words, the automations in GR don’t have any A/B testing capabilities whatsoever. There is no “dynamic” testing such as in Active Campaign.
When I pointed this out to the support guy, he said that no one in development ever mentioned any plans to add this in the future but he would add it as a feature request.
You are 100% correct about a/b testing not working in automations within Getresponse. Shane, I just spent 20 minutes with getresponse support and they confirmed that a/b testing works only doe 1 off newsletters and not for automations. Your video and post was incorrect – unfortunately it got my hopes up, as I was happy with getresponse. Perhaps an update is in order to prevent other folk getting theirs up too? Now I am looking at other autoresponders that have this functionality… Pity – I do like getresponse.
Thank you for bringing this up. In my testing, I was convinced that A/B testing works in follow-ups as well. It’s a while ago, so my memory of it isn’t totally clear anymore, but I probably misinterpreted something. I apologize for that. I will re-test this and update the post.
There are three things I really dislike AC. The fonts on mobile is so small that you cannot view, even if you enlarge the fonts. It looks better on desktop. The AC editor really sucks and makes you hate writing email. There’s no easy way to create nicer footer. Last thing, their support is not always prompt and personal. Other than these, AC is affordable, and packed with features. Thanks Shane.