Etsy, Inc.
Identifying direct and indirect effects in A/B tests

Abstract:

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing an A/B test on a target element of an online platform. In one aspect, a method comprises: conducting an A/B test on a target element of an online platform, comprising: for each user in a population of users, measuring: (i) an outcome of the user interacting with the online platform, and (ii) an interaction of the user with a mediator element of the online platform; and determining, based on the A/B test, a direct effect value that estimates an expected change in user outcomes when the test version of the target element is presented instead of the control version of the target element that is caused independently of induced changes in user interaction with the mediator element.