Microsoft Corporation
FACTORIZED NEURAL NETWORK

Abstract:

Aspects of the present disclosure relate to factorized neural network techniques. In examples, a layer of a machine learning model is factorized and initialized using spectral initialization. For example, an initial layer parameterized using an initial matrix is processed such that it is instead parameterized by the product of two or more matrices, thereby resulting in a factorized machine learning model. An optimizer associated with the machine learning model may also be processed to adapt a regularizer accordingly. For example, a regularizer using a weight decay function may be adapted to instead use a Frobenius decay function with respect to the factorized model layer. The factorized machine learning model may be trained using the processed optimizer and subsequently used to generate inferences.