International Business Machines Corporation
AUTOMATED EVALUATION OF INFORMATION RETRIEVAL

Abstract:

A method, system, and computer program product for automated evaluation of information retrieval systems are provided. The method accesses a natural language query from a set of natural language queries. The natural language query is associated with a query difficulty level. The method generates one or more natural language responses to the natural language natural language query. Each natural language response is associated with at least one facet of the plurality of facets. The method generates a set of feedback cues. A set of search results for the natural language query are returned. The set of search results include a highest ranked natural language response of the one or more natural language responses. The method generates an evaluation result for the HCIR system for the query difficulty level based on the one or more natural language responses, the set of search results, and the set of feedback cues.