Almost all A/B testing tools, including Google Optimize, calculate the conversion rate per variant based on sessions. This means they determine the conversion rate by dividing the number of conversions per variant by the number of sessions in that variant. This is also how the conversion rate is displayed in Universal Analytics.
However, in Google Analytics 4, two conversion rates are available: a session-based conversion rate and a user-based conversion rate. Why does Google Analytics 4 show two different conversion rates? And which rate is the right one? In this article, we discuss the pros and cons of both conversion rates and provide advice on which is the most reliable.
Scenario 1: When measuring based on users might be better
Imagine you're about to book a city trip to London. You start your browser and find some nice hotels through a comparison site. You unknowingly end up in the variant of an A/B test running on the category page of the comparison site. Before booking, you want to discuss the options with your partner. You close your browser and revisit the options together over the weekend. In the new session, you're placed in the control version of the A/B test and book the hotel in this version.
In the above scenario, a user needs two sessions to achieve a conversion. When you calculate the conversion rate of this A/B test based on users, it comes out to 100%. However, when you calculate the conversion rate based on sessions, the conversion rate is only 50%. Quite a difference. In reality, the truth will likely lie somewhere in between. But for the conclusions of your A/B test, you will need to choose between these conversion rates. In this case, it makes sense to determine the conversion rate based on users. There is, after all, one user who converted once.
An additional complexity is that the same user, due to cookie expiration, can be placed in different variants during multiple sessions within the same A/B test. This means the method used also affects the variant to which the conversion is attributed. Read more here about the challenges of A/B testing and cookies.
Below, we describe a second scenario where it might be better to determine your conversion rate based on sessions.
Scenario 2: When measuring based on sessions might be better
Imagine you're looking for a new winter coat. You start your browser (new session) and visit various websites to view their winter coats. After a short search, you find a webshop with a winter coat that meets your needs. You unknowingly land in the control of an A/B test running on the product page of this webshop. You also see a nice scarf in this webshop. You hesitate to buy it but ultimately decide not to. One day later, you decide to buy the scarf after all. You return to the webshop. You again end up in the same A/B test and purchase the scarf.
In the above scenario, the same user converted twice within the same A/B test. When you calculate the conversion rate of this A/B test based on users, it is 200%. However, when you calculate the conversion rate based on sessions, the percentage is 100%. Again, in this scenario, a significant difference in conversion rate can arise based on your choice of statistic. In this scenario, it makes more sense to choose a session-based conversion rate. There were two separate purchase processes at two different times.
There are multiple scenarios where it makes sense to use a different conversion rate. Below, we discuss the pros and cons of using each conversion rate.
Users get the statistical advantage
Statistically, your conclusions become more reliable when you calculate the conversion rate based on users. Within statistical principles, you want to use random samples in A/B tests, known as “random samples.” A “random sample” is a random selection of individuals from the entire population, where each group of individuals has an equal chance of being chosen. This sampling method ensures that you examine your hypothesis without bias. It provides the purest data in your A/B test, leading to the most representative and reliable results.
Before a sample can truly be called random, certain conditions must be met. All events within the sample must be independent. In statistics, events are independent when there is no connection between them. When you calculate your conversion rate based on sessions, this can, statistically speaking, cause problems.
Think back to the first scenario. In this case, there is a connection between the different events. The sessions are initiated by the same user. This violates a condition for the statistic. When you determine the conversion rate based on sessions in this scenario, you can draw statistically less reliable conclusions. When you determine the conversion rate based on users, you don't have this problem. In this scenario, all individual users are independent of each other.
In practice, using sessions is still better
However, in practice, it is almost impossible to reliably determine your conversion rate based on users. This is due to the different definitions of a user in different browsers. Whether someone is seen as the same user is determined by the cookie in your browser. When someone visits your website, Google Analytics looks for a tracking cookie to determine if the person is a new or returning user. If this cookie is present, Google Analytics registers this person as a returning user. If the cookie is not present, Google Analytics registers the person as a new user.
It depends on the presence of cookies whether a user is considered the same user over time. Because browsers do not store cookies for the same duration, significant differences can arise in the definition of a user. In Safari, a first-party cookie is stored for only 7 days (if the user does not return to the website within a week), while in Google Chrome, it is stored much longer. So if the same user visits your website via Safari and returns two days later, this user is still registered as two different users. Whereas, if this user did the same via Google Chrome, the user would be registered as the same user.
When you calculate your conversion rate based on users, the conversion rate depends on the browsers used by the people in the A/B test. This means you cannot reliably determine your conversion rate if you measure it based on users. When you measure the conversion rate based on sessions, you are not affected by external factors causing differences in the definition of a session. You can set the definition of a session yourself in Google Analytics. This ensures you always have a consistent definition of a session and can reliably determine your conversion rate.
Practice beats theory
There are pros and cons to both conversion rates. However, to draw conclusions about your A/B test, you will need to choose between the two. Despite Google Analytics 4 offering the option to calculate two different conversion rates, we would choose to use one type of conversion rate. This way, you always base your A/B tests on the same statistic, making it easier to compare them.
The choice you make can greatly influence the outcome of your A/B test. In our view, at this moment, it is always better to choose a session-based conversion rate. Although it is theoretically better to determine the conversion rate based on users, it is not yet practical due to differences between browsers. As always, you must consider the limitations of the real world. In theory, it's nice that Google Analytics 4 has made a user-based conversion rate available. However, in practice, this conversion rate will yield less reliable conclusions than the session-based conversion rate.
```
