Alibaba Group Holding Limited
MULTI-LEVEL SPARSE NEURAL NETWORKS WITH DYNAMIC REROUTING

Abstract:

Systems and methods for providing a neural network with multiple sparsity levels include sparsifying a matrix associated with the neural network to form a first sparse matrix; training the neural network using the first sparse matrix to form a second sparse matrix by fixing values and locations of non-zero elements of the first sparse matrix and updating a zero-value element of the first sparse matrix to be a non-zero value, wherein non-zero elements of the second sparse matrix includes the non-zero elements of the first sparse matrix; and outputting the second sparse matrix for executing the neural network.