By Kate Priestman – Head Of Marketing, Global App Testing
For a long time, A/B testing has been the gold standard when it comes to testing things like content marketing performance and marketing campaign elements. A good A/B test gives clear, unambiguous results, is easy to understand, and (depending on the variables!) isn’t usually complicated to run.
However, A/B tests do have significant limitations, and some wonder if they’re still relevant in 2022. Surely, as technology advances, we can find a better, more accurate test?
Here, we’ll take you through what A/B testing is, how to run an A/B test, what A/B testing is useful for, the limitations of A/B testing, and whether or not A/B testing is still relevant in 2022.
What is A/B testing?
An A/B test is essentially a ‘competition’ which pits two variables against each other under test conditions.
Think of a trip to the opticians. When you’re wearing those weird frames, the optician will cover one eye and swap out a series of lenses in the uncovered frame while asking you if they make your vision better or worse. This process continues until they have found the perfect lens strength for your vision.
This, in a nutshell, is A/B testing. The optician is testing similar variables against one another to discover which performs best.
What is A/B testing used for?
A/B testing is used to test similar variables to see which performs best.
Let’s say, for example, that you’re running an email campaign. You want this campaign to achieve a high open rate, and you know that subject lines are important for this.
But what is more likely to encourage opens? Is a subject line with emojis (‘🌞😍Check out our great SUMMER SALE! 😍🌞’) eye-catching, or do the emojis distract from the actual message?
To find out, you can run an A/B test.
How do I run an A/B test?
First, pick your variables. A/B tests work best when you’re testing like against like. This ensures that your results will indicate a clear winner. In our example case, you are testing two different versions of a subject line.
If your variables are too far apart (for example, testing a subject line against a sender name) the results will be too ambiguous to be useful.
Second, pick a test group. In this case, choose a segment of your email marketing list and split it into two. Again, to keep the test as controlled as possible, make sure that each text segment is as similar as possible.
Thirdly, send one half of your test segment one variable (in this case, the emoji subject line) and the other (the emoji-free subject line) to the other.
Run the test for a statistically significant amount of time (ideally over a week).
Finally, check the results. If the subject line with emojis gets more opens then the subject line without, it’s fair to assume that subject lines with emojis are more eye-catching and engaging than those without.
What are the benefits of A/B testing?
A/B testing has a lot going for it. Here are just a few of its benefits:
- It gives a very clear result. Assuming you’ve set up your test properly, an A/B test leaves no room for ambiguity. It’s incredibly rare for there to be a ‘draw’ in an A/B test.
- It pinpoints specifics. The clarity and specificity of an A/B test means it’s very useful for things like QA in agile process testing. You can use an A/B test to pinpoint exact performance factors, enabling developers and creative teams to work faster and with more accuracy.
- It’s a popular, well-known, and widely available form of testing. Pretty much every marketing and development platform will provide A/B testing facilities.
- The results are easy to understand. This can be useful when it comes to presenting results to managers, team leaders, and higher-ups.
- You can get really granular with A/B testing. It’s a great way to test out each tiny detail and make sure that you’re using the right variable at every level.
What are the limitations of A/B testing?
Nothing is perfect, and A/B testing is no different. Here are some of the limitations of A/B testing:
- It only tests two variables. You can add extra variables if you want, but the result won’t be as clear and unambiguous as an A/B test. Similarly, if you have a more ambiguous problem involving multiple differing variables, an A/B test won’t be much help. For example, an A/B test would not be much use for beta app testing, because there are simply too many variables at play to give a useful result.
- It doesn’t take context into account. For example, if you are using A/B testing to boost social media engagement, a variable which gets a lot of negative engagement will show as the winner against a post with a smaller level of positive engagement.
- It doesn’t reveal flaws. Instead, it reveals the least-worst option. All A/B testing can do is tell you which of two variables performs better. It can’t tell you why they perform better, or if your campaign strategy as a whole needs improvement.
- It can be a time and resource drain. Testing everything two at a time is a long process. Of course, automations can take on a lot of this burden, but it’s still an inefficient way to test an entire campaign.
Is A/B testing still relevant in 2022?
A/B testing is brilliant at what it does. But it’s not always the right choice of test.
The best approach is a hybrid one, using A/B testing where appropriate and other forms of testing and evaluation elsewhere.
For example, you might use focus groups to get a big-picture understanding of your app’s performance, and then drill down into the details with A/B testing.
In the future, it is possible that things like AI and neural networks will play an increased role in testing. However, right now neural networks are complex and often expensive bits of kit. When asking yourself when to use a neural network, the task of picking between two slightly differing subject lines probably isn’t the answer.
The bottom line is that, yes, A/B testing is still relevant. It may be old, but sometimes the old marketing techniques are the best. To determine performance between two similar variables, it still can’t be beaten.
The trick is knowing when to use it!
