Microsoft Corporation
INTEGRATED BROWSER EXPERIENCE FOR LEARNING AND AUTOMATING TASKS

Abstract:

In non-limiting examples of the present disclosure, systems, methods and devices for automating web browser task actions are presented. An indication to record a new action may be received. One or more steps associated with the action may be performed during the recording. Each step may comprise interaction with a different webpage element corresponding to an HTML node. The HTML node, and one or more additional HTML nodes may be extracted and/or tagged, and a machine learning model may be applied to the extracted/tagged nodes. The machine learning model may have been trained to create templates for identifying interacted-with web elements. The automated action may be performed by applying the machine learning model to one or more websites. The machine learning model may identify the correct web elements to interact with and move through the action steps in an automated manner to perform the action.