A/B Testing Method

By Aliu A. Onifade

The internet provides developers of connected software an unprecedented opportunity to accelerate innovation by evaluating ideas and accurately using trustworthy controlled experiments (Kohavi, 2015). In particular, in marketing and web analytics, A/B testing is a controlled experiment with two variants A and B. More specifically, it is a form of statistical hypothesis testing or two-sample hypothesis testing as used in the field of statistics. Also, in online settings such as web design, A/B testing identifies the changes of web pages that increase the outcome of interest e.g., click-through rate for a banner advertisement (Kohavi, 2015).


A/B testing is a way to compare two versions of a single variable by testing a subject’s response variable A against variable B and determining which of the two variables is more effective (Kohavi, 2015).

According to Martin and Hanington (2012), A/B testing is an optimisation technique. It is used to compare two versions of a design. Furthermore, this method is changed from the traditional mail practice in which two different versions of the same mail are sent out to different people to ascertain which one gets a better response rate.

For instance, users are introduced to the control and treatment version, after which, the performance runtime, user behaviors, and surveys are collated and analyzed to ascertain the better version (Kohavi R., Longbotham R, Sommerfield D., et al., 2009).

Moreover, if the treatment version (B) proves to be better than the existing one (A) in terms of likes, clicks, and purchases pattern, etc, tends to displace the original. By using A/B tests, one can predict which change on the website will influence better performance because it allows the designer to avoid certain pitfalls when making changes to a website’s design.

A/B Testing Measures

Rabhan (2013) highlighted that A/B tests should be used anytime a new idea or a change is implemented on a webpage. Besides, according to conversion expert, Smith (2017), the most suitable situations for the use of A/B testing are:

  • Designing a new website
  • Changing a service, a plugin, or a feature
  • At the time of price change
  • When there’s a need to improve the conversion rate
  • Immediately there’s a problem with the user engagement on the website.

Users of A/B Testing Method

The process of A/B testing varies significantly, from programmers and designers who develop the control version of the web interface to managers who plan the project to the testers that are actual users in the context of A/B testing, and thus are representatives of the current and the potential user group. This test method can be implemented without developers because of the relatively ease-to-use (Cardello, 2014).

Undoubtedly, the test method provides guidelines for designers to create products or services that meet current user expectations (Young, 2014). The A/B testing is similar to some other user experience methodology such as usability testing. Specifically, one has to keep in mind that the characteristic that differentiates this testing from others is the ability to provide quantitative user feedback for known user-experience (UX). Undoubtedly,A/B testing is an optimal solution method from among a set of alternatives for an already-known UX problem (Young, 2014).

Steps to Implement A/B Testing method

Firstly, is to identify the problem, followed by experiment planning (tester sample), the definition of the right metrics, and tracking the right sample of users (Kohavi, 2015).

The identification depends on the organisational approach. After defining a tester sample, the metrics of the experiment should be concluded. Hence, the two common metrics of A/B tests are the percentage of users who take the desired action (Nielsen, 2005), and the click-through-rate (CTR), which relates the probability of success in a trial (Regelson and Fain 2006), metrics like these can be a part of the overall evaluation criteria, which represent a higher-level single metric, this incorporates the trade-off among other organizational metrics like cost, revenue, etc. (Kohavi, 2015). In general, these tests evaluate if the statistical distribution of the treatment is different from that of the control.

To sum up, A/B testing is used to compare two versions of the same design (control A and treatment B). However, the test method provides insight into user behaviours and expectations and to identify and implement changes to web page designs that would increase or maximise an outcome of interest (Young, 2014).


Cardello, J. (2014), Define stronger A/B test variations through UX research. NN Group. https://www.nngroup.com/articles/ab-testing-and-ux-research/ {Accessed Online on 31.10.2020}

Kohavi, R. (2015), KDD2015 keynote. ExP Platform. https://exp-platform.com/kdd2015keynotekohavi/ on 2 November 2020.

Martin, B., & Hanington, B. M. (2012), Universal methods of design: 100 ways to research complex problems, develop innovative ideas, and design effective solutions. Beverly, MA: Rockport Publishers.

Nielsen, J. (2005), Putting A/B Testing in its place. NN Group. Retrieved from https://www.nngroup.com/articles/putting-ab-testing-in-its-place/ on 31 October 2020.

Rabhan, B. (2013), Convert every click: Make more money online with holistic conversion rate.

Regelson, M., & Fain, D.C. (2006), Predicting click-through rate using keyword clusters.

Smith, J. (2017), The 5 times when you absolutely must do A/B testing. Crazy Egg. Retrieved from https://www.crazyegg.com/blog/when-must-test/ on 3 October 2020.

Wang, J. (Ed.). (2014), Encyclopedia of business analytics and optimization, IGI Global.

Young, S. W. (2014), Improving library user experience with A/B testing: Principles and process. Weave: Journal of Library User Experience1(1).