Ab Testing

How should we analyse A/B Testing results?

It has been a month since the first Growth Hacking İstanbul event took place. Right after the event, I shared my presentation titled “How to make the growth process a part of your business” on SlideShare. I wanted to individually address the topics we discussed that day and present them in a format that everyone can access. 

My first article will be about the misconceptions one might have while evaluating the results of A/B testing. When we want to increase the performance of a given page or mobile app, we also want to be able to measure the effect it has produced clearly, to verify if our ideas really do work. In order to do so, instead of permanently implementing the changes we have in mind, we prefer to experiment with them to see how well they perform. That is where A/B testing comes into play.

As there are plenty of examples about how A/B testing should be done, some common misconceptions also come with it. I previously wrote about what needs to be done for a successful and issue free A/B testing process. Another thing that has captured my attention lately is that these misconceptions are not only present in the A/B testing process, but also in the evaulation stage of that process. In this article, I will be discussing how these test results should be analyzed by presenting an exemplary analysis myself.

First of all, let’s take a look at the results of this Google Analytics A/B test I just did.

The results look good, don’t they? The test displays an improvement of 43% in comparison to the regular page itself. But is that really the case? Let’s do a little digging.

See, the results changed in a heartbeat! The test did not actually succeed. The test displays an actually 4.84% lower rate of conversion in comparison to the regular version of the page. If that’s the case, why did we get such results the first time? That’s because we restricted the audiences for our test during the evaluation. What are the “audiences” that we should be concerned with here? They represent a synthesis of the users who are browsing our page on their computers, users who landed on it via Google ads and the new visitors to the page. If we continue digging in, who knows what else might come up! Let’s take another look.

The results are different again. Our conversion rate is 79.44% higher than the previous time! This time the parameters are set for mobile users, Google ads visits and first time visitors. Let’s try something else.

Now the results are even better, with a conversion rate of 96.84%! Why so high? Because the audience now includes mobile users, Google ads visits, and not new visitors, but repeat visitors this time.

What are the key points?

We hear a new optimisation story each day. And it goes without saying that we need to be more careful about these stories, as the A/B testing data can easily be manipulated, which is the case with all types of tests, and it does get manipulated. The results of the test vary greatly depending on the audience. For a test to be deemed successful, the audience in question needs to be experiencing the problem at hand. Normally, the ratio between the users that are new visitors and the repeated visitors is not the same. The same goes for users who are browsing from desktop versus mobile. Even the sources you are acquiring users from and the campaigns you are using can make a big difference in terms of conversion rates.

While one test audience can display an increase of 100%, whereas the same amount could be a decrease of 20% for a different audience. The success of your test is just as important as the solution that it brings to the problem you addressed in the beginning. In conducting your research, to get a correct reading of your results, always remember to consider the problem you started out with and look into the audience that the problem itself is inherently related to.

Leave a Reply

Your email address will not be published. Required fields are marked *