Adobe Inc.
Structure-based transformers with localization and encoding for chart question answering

Abstract:

Embodiments are disclosed for determining an answer to a query associated with a graphical representation of data. In particular, in one or more embodiments, the disclosed systems and methods comprise obtaining a visual embedding for a graphical representation of data, the visual embedding representing a plurality of graphical elements. The one or more embodiment further include obtaining a query embedding for a query associated with the graphical representation of data, the query embedding representing a plurality of textual elements of the query with at least one textual element substituted with an identifier for at least one graphical element of the set of graphical elements. The one or more embodiment further include generating a chart sequence from the visual embedding and a query sequence from the query embedding, generating an output sequence based on the graph and the query sequences, and determining an answer to the query from the output sequence.