Oracle Corporation
REMOVING UNDESIRABLE SIGNALS FROM LANGUAGE MODELS USING NEGATIVE DATA

Abstract:

A method for training a language model using negative data may include accessing a first training corpus comprising positive training data and accessing a second training corpus comprising negative training data. The method may further include training a first language model using at least the first training corpus, the second training corpus, and a maximum likelihood function. The maximum likelihood function may maximize the likelihood of the first language model predicting the positive training data while minimizing the likelihood of the first language model predicting the negative training data.