Talk Abstract: A/B testing: Setting up for success & choosing the right Statistical Framework. We are all familiar with the benefit that A/B testing can have on businesses’ success, whether that’s improving customer experience or increasing revenue on websites and apps. In this session we’ll talk through tips we’ve learned from running a multitude of A/B tests at Cleo, from design through to implementation and analysis. The concepts of ‘holdouts’ in testing will be introduced, with a focus on how they can help teams track long term impacts to KPIs. Lastly we’ll look at Bayesian vs Frequentist frameworks, the benefits and pitfalls of each, and how we’re exploiting Bayesian statistics to get the most out of our A/B tests at Cleo.
Bio: Leigh recently joined Cleo as a Product Analyst, having worked in experimentation roles for over 3 years. He’s previously worked on some of the largest websites in the UK, including Sainsbury’s & Argos, having run hundreds of experiments across products such as search engines, recommender systems and conversion funnels. Prior to this, he obtained his Masters in Chemistry from the University of Cambridge.