Microsoft Corporation
Estimating treatment effect of user interface changes using a state-space model

Abstract:

Examples described herein generally relate to a computer device including a memory, and at least one processor configured to evaluate a change to a user interface. The computer device monitor user interactions with the user interface prior to and after a change to the user interface. The monitoring includes collecting result metric data per user. The computer device divides users into a treated group and a control group based on whether each user engages in a particular interaction. The computer device generates a result metric time series for the treated group and a partitioned result metric time series for the control group. The computer device estimates a conditional distribution of the result metric and a counterfactual behavior using a Bayesian machine learning model based on the result metric time series. The computer device determines a treatment effect of the change to the user interface on the result metric data.