eBay Inc.
TECHNIQUES FOR ENHANCING THE QUALITY OF HUMAN ANNOTATION

Abstract:

Methods for enhancing or automating a review process of annotation tags for a set of tokens is described. A system may receive a list of tokens with associated tags for each token for a data set and may output any identified inconsistencies where a token is assigned at least two different tags. For example, instead of a human looking at each token individually or taking a sample set of the tags for review, the described techniques may look at all tokens with the associated tags in a set of data and may leverage reorganizing the tokens and associated tags to highlight errors to be fixed. Accordingly, the system may look across all tokens within an entire data set, while a review (e.g., by a human) of possible errors of the data set is limited to the highlighted errors flagged by the system.