Intel Corporation
COMPUTATIONAL OBJECT STORAGE FOR OFFLOAD OF DATA AUGMENTATION AND PREPROCESSING

Abstract:

A system that executes a distributed application, such as a machine learning model, can have a processor node among a system of nodes to generate a request for data and a storage node among a system of nodes that stores the requested data. The processor node will use the data for iterative processing to train a machine learning model. The storage node receives the request for the data, reads the data, preprocess the data to perform requested data transformation on the data on demand, and provides the preprocessed data to the processor node for the iterative processing. The processor node can request storage system nodes to store data in a manner suitable for preprocessing. In response to receiving a request, the storage node can interpret hints or metadata associated with the storage operation and perform the requested data store operation.