Apple Inc.
CONTEXTUAL SENTENCE EMBEDDINGS FOR NATURAL LANGUAGE PROCESSING APPLICATIONS

Abstract:

Methods and systems for embedding natural language sentences within a highly-dimensional vector space are provided. Additionally, various applications relating to natural language processing, are provided. Such applications include digital assistants and search engines, as well as systems for classifying, sorting, organizing, and/or pairing content that are associated with natural language objects. The sentence vector embeddings encode various semantic features of the sentence. Two separate language models, arranged in a serial architecture are employed to generate a sentence vector. The first language model generates token vectors for each of the tokens included in the sentence. The token vectors are employed as inputs to the second language model. The second language model generates the sentence vector for the sentence. A sentence vector embeds the semantic context of the corresponding natural language object within the vector space. The second language model may be trained via supervised learning on multiple semantic-related tasks.