International Business Machines Corporation
DECENTRALIZED PARALLEL MIN/MAX OPTIMIZATION

Abstract:

Techniques are provided for decentralized parallel min/max optimizations. In one embodiment, the techniques involve generating gradients based on a first set of weights associated with a first node of a neural network, exchanging the first set of weights with a second set of weights associated with a second node, generating an average weight based on the first set of weights and the second set of weights, and updating the first set of weights and the second set of weights via a decentralized parallel optimistic stochastic gradient (DPOSG) algorithm based on the gradients and the average weight.