International Business Machines Corporation
OPTIMIZING RESOURCE ALLOCATION FOR DISTRIBUTED STREAM PROCESSING SYSTEMS

Abstract:

Computer software executing on computer hardware that performs the following operations: (i) training a machine learning model to determine allocations of computing resources to processing elements of a stream processing job according to a specified objective; and (ii) allocating a set of computing resources to the processing elements by: allocating to the processing elements a first subset of the set of computing resources based, at least in part, on a minimum resource requirement for the processing elements, and allocating to the processing elements a second subset of the set of computing resources based, at least in part, on an allocation determined using the trained machine learning model.