International Business Machines Corporation
Ranking Candidate Documents for Human Annotation Task in Real-Time

Abstract:

A system configured to rank and display candidate documents for human annotation task. The system executes instructions to receive a human annotation of a first unannotated document in a list of documents from a document set; update an annotated entities and corresponding entity types set based on the human annotation of the document from the document set; perform auto-mapping of annotated entities to corresponding entity types on a remaining set of documents in the document set based on the updated annotated entities and corresponding entity types set; calculate a score for each document in the remaining set of documents based on the auto-mapping of annotated entities; and update an order of the remaining set of documents being displayed for human annotation based on the calculated score for each document in the remaining set of documents in the document set.