International Business Machines Corporation
Detecting and removing bias in subjective judging

Abstract:

A method, system, and computer program product algorithmically analyzes scoring data from a competitive event where the scoring, determined by a plurality of evaluators, is based on subjective criteria. The method receives, and/or determines a scale factor associated with each evaluator. The method adjusts scores awarded by each evaluator, based on respectively corresponding scale factors, to arrive at normalized scores. The method, thereby minimizes influences of biases associated with the evaluators.