International Business Machines Corporation
Routine evaluation of accuracy of a factoid pipeline and staleness of associated training data

Abstract:

A mechanism is provided for routinely evaluating an accuracy of a request processing pipeline. A set of questions is executed through the request processing pipeline, producing a list of answers, supporting documents, and accuracy metrics. A determination is made as to whether a document contribution value of each document associated with the answer is equal to or above a document contribution threshold value. For those documents equal to or above the document contribution threshold value, a snapshot is stored in a training-data data structure. Based on a clustering of questions, for each question cluster, a determination is made of an average accuracy metric. A comparison is performed and a determination is made as to whether an accuracy metric delta exceeds the accuracy metric threshold value. If so, a differential report is generated indicating a review is needed of a training of the request processing pipeline.