International Business Machines Corporation
Crowdsourced Proactive Testing System for Named Entity Recognition Models in IT Support

Abstract:

A system configured to proactively test a Named Entity Recognition model using crowdsourcing, the system comprising memory for storing instructions, and a processor configured to execute the instructions to receive the NER model as input; provide an explanation for predictions made by the NER model to a crowd; receive a test sample from a first crowd worker, the test sample is intended to generate an error for the NER model; generate a prediction using the NER model based on the test sample as input; receiving validation data corresponding to the prediction of the NER model; receive categorization data of the error corresponding to the test sample; and improve the NER model based on the test sample.