A/B testing is a powerful tool in any Data Scientist’s arsenal. It enables us to construct hypotheses and make careful changes to our users’ experiences while collecting results. This allows us to learn why certain elements of our experiments impact user behavior. In online A/B testing, A refers to ‘control’ or the original testing variable, and B refers to ‘variation’ or a new version. Both control and variant are shown to users at random, and statistical analysis determines which variation performs better for a given KPI or business goal. As visitors are served either the control or variant, their engagement with each experience is measured and collected in a dashboard and analyzed through a statistical engine. We can then determine whether changing the experience had a positive, negative, or neutral effect on visitor behavior. But A/B testing can also be complex. If you’re not careful, you can make incorrect assumptions regarding the distribution of traffic, the statistical test, or even the hypothesis rejection. We want to use this roundtable as an opportunity to have an open discussion on the different perspectives and ways to tackle the problems of online A/B testing. What works for your specific use case? How often do you run A/B tests? When do you decide to stop? Etc.