What other A/B testing experts do when there is no clear winner

When there is no clear winner (results are very close, significance reached), what do you do? We asked A/B testing experts what they thought and how they operated at their companies.

Tyler Roehmholdt, Web Marketing Manager at Campaign Monitor

For an inconclusive test, oftentimes we’ll stick to the control if it’s a change to something that already exists on the page. If we went with the new variation, at best it will perform the same as the original over a longer period of time, at worst we’d see a decrease in conversions over time. If it’s a new addition to the page, we’ll be much more open to promoting the variation to production.

Will Kurt, Data Scientist at Kissmetrics

The big thing to keep in mind with A/B testing is you’ll get better results in the long run if you stick with only decision that are clear wins. I would recommend sticking with the original since the uncertainty in the results means you always risk that the other variant is worse. It’s a much better use of your time to wait until you’ve found a really great variant, and not risk too much on variants that are likely to bring you little if any gain.

Jake Peterson, Growth at Segment.io

After 2 weeks I stop the experiment, then I wait another 2 weeks to see the long term effects on metrics like revenue or what plans they are on. Trials for software as a service businesses are anywhere from 2 weeks to 30 days, and I can see what happened as a result of a particular test. I’ll also ask qualitative questions to my team like “what resonated with customers more?” Our sales team has a good beat on this as they are talking with customers every day. Bottom line, I think that when there is no clear winner that you look at how your long term metrics are affected, not just what the experiment was trying to optimize.

Is anything on this page unclear? Suggest edits on Github!