Microsoft Corporation
SEMANTICALLY TAGGED VIRTUAL AND PHYSICAL OBJECTS

Abstract:

A head mounted display device is provided that includes a display device, a camera device, an input device, and a processor. The processor is configured to store a database of physical objects and virtual objects that have been associated with one or more semantic tags. The processor is further configured to receive a natural language input from a user via the input device and perform semantic processing on the natural language input to determine a user specified operation and identify one or more semantic tags indicated by the natural language input. The processor is further configured to select a target virtual object and a target physical object based on the identified one or more semantic tags, perform the determined user specified operation on the target virtual object based on the target physical object, and display the target virtual object at a physical location associated with the target physical object.