Netflix, Inc.
Stratified sampling applied to A/B tests

Abstract:

Embodiments provide techniques for testing a plurality of variations of a user experience, where each of the plurality of variations is distinct from other variations in the plurality of variations. For each permutation of a plurality of stratification factors that characterize users of the online site, embodiments generate a respective set of pseudorandom numbers. Additionally, user requests are assigned to variations in the plurality of variations, based on the generated sets of pseudorandom numbers. Embodiments also track user retention information relating to the user requests. At least one of the plurality of variations is rated, based on the user retention information.