Talk abstract: A/B testing: Setting up for success & choosing the right Statistical Framework. We are all familiar with the benefit that A/B testing can have on businesses’ success, whether that’s improving customer experience or increasing revenue on websites and apps. In this session we’ll talk through tips we’ve learned from running a multitude of A/B tests at Cleo, from design through to implementation and analysis. The concepts of ‘holdouts’ in testing will be introduced, with a focus on how they can help teams track long term impacts to KPIs. Lastly we’ll look at Bayesian vs Frequentist frameworks, the benefits and pitfalls of each, and how we’re exploiting Bayesian statistics to get the most out of our A/B tests at Cleo.
Bio: Benj did his PhD at Oxford modelling the flocking behaviour of birds, before turning his attention to how digital products can learn from masses of data on human behaviour. His career as a data scientist has involved developing recommender systems, classification models, and network analytics tools for several London startups. He has been a senior data scientist at Cleo for the past two years, working on machine learning models used in the Cleo app.