International Business Machines Corporation
DYNAMICALLY GENERATING AN OPTIMIZED PROCESSING PIPELINE FOR TASKS

Abstract:

A method for dynamically generating an optimized processing pipeline for tasks is provided. The method identifies one or more tasks to be executed from defined tasks that are defined declaratively as a number of stages of input data, data transformations, and output data. The method processes the identified tasks to determine dependencies between the tasks based on their defined stages and creates one or more optimized data processing pipelines by performing a dependency resolution procedure on stages of all tasks in parallel using the task dependencies to determine the order of the stages and removing duplication of stages between tasks.