ServiceNow, Inc.
LOOKUP SOURCE FRAMEWORK FOR A NATURAL LANGUAGE UNDERSTANDING (NLU) FRAMEWORK

Abstract:

A natural language understanding (NLU) framework includes a lookup source framework, which enables a lookup source system to be defined having one or more lookup sources. Each lookup source of the lookup source system includes a respective source data representation that is compiled from respective source data. For example, a source data representation may include source data arranged in a finite state transducer (IFST) structure as a set of finite-state automata (FSA) states, wherein each state is associated with a token that represents underlying source data. Different producers can be applied during compilation of a source data representation to derive additional states within the source data representation from the source data. Certain states of the source data representation that contain sensitive data can be selectively protected through encryption and/or obfuscation, while other portions of the source data representation that are not sensitive may remain in clear-text form.