A/B Testing

A/B tests are statistical experiments that help you decide whether a change is actually making a significant impact on your product.

To give a real example for a SaaS product, let's say you want to test the difference between two welcome emails that get sent out when a customer signs up for a trial.

Example

Here's how it could play out:

1. For new trial signups going forward, give half of your customers a long, detailed welcome email, and give the other half a shorter welcome email.
2. Record which version they saw.
3. Record whether these people end up paying for the full version. This is the key performance metric you're trying to improve with the test. The test answers: "Did the email version influence more people to convert?"
4. Report on the results, comparing how the two conditions performed. Make sure enough people have run through the test to show that the results are actually due to the different emails, and not a result of chance.

What Every A/B Test Needs

You can run an A/B test using Kissmetrics' JavaScript Library, or set up the test through a homebrew solution on your servers, or use a service like Optimizely, Visual Website Optimizer or Google Website Optimizer.

Regardless of how the test is run, every A/B test needs the same three things:

1. Each visitor is randomly assigned to a variation, either A or B.
2. Make sure that the visitor always sees either only A or only B. Otherwise, your results are tainted if someone ends up seeing more than one variation.
3. Record which variation this visitor saw, so that you can refer to it when checking the results of the A/B test.

That covers the test itself. There's one last thing:

• Record the end goal that you are interested in. Are you interested in increasing signups with this test? Measure Signups. Are you interested in whether people continue to return to your site? Measure the number of Site Visits.

I get it. How do I get started with running a test?

The three articles below will help you pick the right solution for you, whether that's using the Kissmetrics JavaScript Library, or using another testing platform, or adding Kissmetrics to your in-house testing code.

Viewing Results

If you've followed the articles below to get your test started, then whichever variation each visitor is assigned to will be saved to a Kissmetrics property. You can then segment any report by that property to see the test's performance. You can even use our dedicated A/B Test Report tool to determine if the results are significant:

• Using the JavaScript Library to Run an A/B Test

  The Kissmetrics JavaScript Library provides a JS function to help you set up the A/B test. It does the three things every A/B test needs, all in one fell swoop. However, it requires you to edit some JavaScript on your site, though. Read here for some examples.

• Tracking Whole Page A/B Tests

  Learn how to track two completely different page designs not just changes in individual elements.

• Using Weighted Variants

  Learn how to use weighted variants to more finely control experiments.

• Using Your Own Server-Side Code

  If you already have existing server-side code that drives your A/B tests, here is how to pass your results into Kissmetrics.

• Using an A/B Testing Platform

  If you don't want to write any code, you can try a platform like Convert, Optimizely, Unbounce, or Visual Website Optimizer to set up the A/B test.