NVIDIA Corporation
PARALLELIZATION STRATEGIES FOR TRAINING A NEURAL NETWORK

Abstract:

System and methods to train a neural network to systematically find a cross-over point, given the number of devices (e.g., Graphical Processing Units) used to train a deep learning (DL) model, that indicates which parallelization strategy to implement when optimizing the training of the DL model on a particular system to achieve maximum efficiency gains.