How to do an A/B test?
Posted: Mon Jan 06, 2025 8:32 am
Before performing an A/B test, two requirements must be met: the first is to monitor the site's performance (visits, conversions, clicks, etc.).
Compare results from different periods and analyze if there is a drop in your key metrics. A good way to do this is through traffic analysis tools such as Google Analytics .
The second requirement is to review the business goals, and then define a measurable objective , such as: increase conversion by X%, get more leads per day or increase the number of interactions through chat.
Learn more:
Article
SMART goals: what they are and how to define them in your company
Image from: SMART goals: what they phone database are and how to define them in your company
Next, you should follow these steps to do an A/B test :
Define a channel (online store, email, landing page or ad).
Formulate a hypothesis and the variations that will be part of the test.
Configure variations in the testing tool.
Wait the necessary time to collect the results.
Document the type of test, start and end date, and the results obtained.
Implement the winning variation.
Please note that the experiment can be performed in two ways:
Through tools that divide the sample in the following way: 50% of the audience sees version A and the other half sees version B of an element.
Manually , by observing the metrics generated by the original model and, subsequently, its variant, for the same time period.
Nowadays, A/B testing platforms are quite developed. Many of them automatically determine the split of the tests randomly and, in the end, even implement the most successful variation.
We will talk about these tools later, but first, we want to share some tips to apply when doing A/B testing.
Best practices when doing an A/B test
While A/B testing is relatively easy to perform, attention to detail is required to ensure accuracy and effectiveness. Keep these things in mind when running your experiments:
Consider seasonality
On special dates or times, such as Hot Sale or Christmas, the results tend to be inflated and are not necessarily consistent with the normal habits of users.
Set a reasonable duration for the test
The ideal duration of an A/B test will depend on the objective you have defined. However, the cycle usually ranges from one to three months .
An A/B test for an email can be shorter and generate results in a few hours , depending on the sample.
A test for a website design, for example, can take up to a month to generate meaningful conclusions, especially if the site has little traffic.
We bring you more content on defining objectives:
Article
OKR: what it is and how to apply this methodology
Image from: OKR: what it is and how to apply this methodology
Focus on a few elements
Modifications to an A/B test should be made in parts to differentiate the experiment that actually yielded results.
In this sense, focus on one element at a time . If you test with many variables at the same time, it becomes more difficult to understand which change was responsible for the results.
There are other multivariate methodologies that test more than one element simultaneously. However, they require specialized tools and professionals to carry out this complete analysis.
Compare results from different periods and analyze if there is a drop in your key metrics. A good way to do this is through traffic analysis tools such as Google Analytics .
The second requirement is to review the business goals, and then define a measurable objective , such as: increase conversion by X%, get more leads per day or increase the number of interactions through chat.
Learn more:
Article
SMART goals: what they are and how to define them in your company
Image from: SMART goals: what they phone database are and how to define them in your company
Next, you should follow these steps to do an A/B test :
Define a channel (online store, email, landing page or ad).
Formulate a hypothesis and the variations that will be part of the test.
Configure variations in the testing tool.
Wait the necessary time to collect the results.
Document the type of test, start and end date, and the results obtained.
Implement the winning variation.
Please note that the experiment can be performed in two ways:
Through tools that divide the sample in the following way: 50% of the audience sees version A and the other half sees version B of an element.
Manually , by observing the metrics generated by the original model and, subsequently, its variant, for the same time period.
Nowadays, A/B testing platforms are quite developed. Many of them automatically determine the split of the tests randomly and, in the end, even implement the most successful variation.
We will talk about these tools later, but first, we want to share some tips to apply when doing A/B testing.
Best practices when doing an A/B test
While A/B testing is relatively easy to perform, attention to detail is required to ensure accuracy and effectiveness. Keep these things in mind when running your experiments:
Consider seasonality
On special dates or times, such as Hot Sale or Christmas, the results tend to be inflated and are not necessarily consistent with the normal habits of users.
Set a reasonable duration for the test
The ideal duration of an A/B test will depend on the objective you have defined. However, the cycle usually ranges from one to three months .
An A/B test for an email can be shorter and generate results in a few hours , depending on the sample.
A test for a website design, for example, can take up to a month to generate meaningful conclusions, especially if the site has little traffic.
We bring you more content on defining objectives:
Article
OKR: what it is and how to apply this methodology
Image from: OKR: what it is and how to apply this methodology
Focus on a few elements
Modifications to an A/B test should be made in parts to differentiate the experiment that actually yielded results.
In this sense, focus on one element at a time . If you test with many variables at the same time, it becomes more difficult to understand which change was responsible for the results.
There are other multivariate methodologies that test more than one element simultaneously. However, they require specialized tools and professionals to carry out this complete analysis.