Unleash

A/B/n experiments in 3 simple steps

Feature toggles, such as Unleash, are a great tool to enable A/B/n experiments.
Talking to customers, we often find that many are really eager to start experimenting but they often find it a bit difficult to get started. Blockers may be many, one that we repeatedly hear is inter-team dependency to get up and running.

At Unleash we truly believe that development teams shall be as autonomous as absolutely possible.
What is important is to empower the team with the tools and mandate to be as independent as possible from other teams and their priorities. Our experience is that it is common to put the ownership and responsibility of the shared analytics tool into one dedicated team. The result is that this analytics tool team gets a bottle-neck for the teams that are to enable A/B testing.

This guide will show you how you can get started with A/B/n experiments in 3 simple steps using Unleash – all handled within the team.
Before we get started, there are some basics that you should have in place:

  • Unleash – this is to create and manage the A/B test in your application
  • A preferred analytics tool. In this guide, we provide a step-by-step guide using Google Analytics
  • Clear understanding of the business goals of the development team – e.g. the development team need to understand the KPIs they want to improve through their A/B testing

 

Our reference case is built on a joint effort we did together with the Norwegian Labor and Welfare Organization (NAV). NAV is using unleash self-hosted part of their software toolbox.

Step 1: Define an alternative to the targeted KPI

The first enabling step for any A/B/n experiments is to define an alternative that we believe will perform better on the decided target KPI. For simplicity – let’s assume that the experiment will test the color of the call-to-action button, the team expects a green button will perform better than the existing blue.
The team then needs to decide on what part of the traffic they want to perform the experiment on. In this case, the team decides that they want to run the experiment on 50% of the traffic.

1) Log into your Unleash instance
2) Create a new feature toggle by pressing the “+”

Unleash Feature Toggle

3) Give the feature toggle a name according to your naming convention. In this example, we have chosen “AB_experiment”
4) We also recommend that you give the feature toggle a short description – this is for your reference and documentation
5) Now you need to choose an activation strategy. For this example we have chosen to run the experiment on 50% of the traffic – hence we choose the “gradualRolloutUserId”. This activation strategy means that the Unleash engine picks 50% of the users to be part of the experiment. We know this – is the percentage is set to “50%. We also want to make the experiment sticky – which means that if a user has seen the experiment once, we want this user to have a consistent experiment. In this example, we are using the UID to create stickiness. UID is part of the user context in the Super Insurance web application.

Creating a feature toggle in Unleash

6) Now we want to create the variants that contain the actual experiment. Here we define a new variant as well as a control variant. It is worth noticing that all the variants will be weighted equally. You are able to define many variants – in this example, we have created 2. “BlueButton” is the control variant, while “GreenButton” is the alternative we want to test.

Screen from Unleash product - ab experiment

Step 2 – Set up the experiment in Google Analytics

To visualize your A/B/n experiment results in Google Analytics, we choose to use “Custom dimensions”.

You will find this by logging onto the Google Analytics admin console. Click “Custom definitions”, “Custom dimensions” and then “New custom definition”.

Edit custom dimension

In this example, we have chosen to call the Custom dimension “featureToggleName”. The name itself is irrelevant to the Unleash SDK, but the user needs to know the id of the dimension reserved for the Unleash data. We have also chosen to use the “Session” scope on the dimension. Usually, for a user that returns to the site after a day or two, the experiment might not be that relevant to that specific user. This article provides additional details on the available scopes for Custom dimensions.

Step 3 – Connecting the dots

The last step in setting up the A/B/n experiments is to connect the experiment with the metrics. The developer in the team now needs to enter a few lines of code to get the magic to happen.
In the source code, where both the Unleash SDK and Google analytics SDK is running.

const toggleName = ‘AB_feature_toggle’;

const variant = unleash.getVariant(toggleName);

console.log(variant.name);// Maps ‘dimension4’ to ‘unleash’.

gtag(‘config’, ‘UA-YOUR_ID-1’, { ‘custom_map’: {‘dimension4’: ‘unleash’}

});// Sends an event that passes ‘variant’ as a parameter.

gtag(‘event’, ‘unleash_dimension’, {‘unleash’:`${toggleName}:${variant.name}`});

 

Your experiment is now up and running. You are now able to explore the results of the experiment using your already existing reports and views in Google Analytics. The Custom dimensions are available as a secondary dimension that you can easily access.

If you are using Google Tag Manager you can pass the data to the data layer and onwards to your analytics tool such as Google Analytics.
First, create your event JSON and push it to the data layer like so:

dataLayer.push({ //send data to GTM

‘event’: ‘unleash’, //name your event

‘feature’: ‘new-logo’, //name of the feature

‘variant’: ‘control’ //name of the variant the user sees

})

 

Then, in Google Tag Manager, create 4 tags

  • One custom event to trigger when your Unleash event is passed
  • One data layer variable to store the name of your feature
  • One data layer variable to store the name of your feature variant
  • One Google Analytics event tag to send the data to Google Analytics when the custom event triggers

Want to try Unleash?

GET STARTED TRY OUR DEMO

Share this article

Explore further