Oracle Corporation
PARAMETER SHARING DECODER PAIR FOR AUTO COMPOSING

Abstract:

Techniques for auto composing using a transformer-based language model having a parameter sharing decoder pair (PSDP) that reduces a number of parameters of the model and maintains the capability of generating understandable and reasonable compositions. In one particular aspect, a method is provided that includes obtaining a full encoder sequence, and inputting the full encoder sequence into a transformer model having a PSDP. The PSDP includes: a first decoder having parameters that are shared across all N layers of the first decoder; and a second decoder having parameters that are shared across all N layers of the first decoder. The parameters of the first decoder are different from the parameters of the second decoder. The method further includes using the transformer model to predict sequence elements based on the full encoder sequence, generate an output sequence comprising the sequence elements, and output an output sequence different from the full encoder sequence.