Wipro Limited
System and method for annotation of tokens for natural language processing

Abstract:

This disclosure relates to method and system for annotating tokens for natural language processing (NLP). In one embodiment, the method may include segmenting a plurality of corpus based on each of a plurality of instances, deriving a plurality of entities for each of the plurality of instances based on at least one of a machine learning technique or a deep learning technique, determining a word vector for each of the plurality of entities associated with each of the plurality of instances, and labelling a plurality of tokens for each of the plurality of instances. It should be noted that the plurality of tokens associated with the plurality of entities may be identified based on a frequency of each of the plurality of entities.