Meta Platforms, Inc.
Machine learning system flow processing

Abstract:

Some embodiments include a method of machine learner workflow processing. For example, a workflow execution engine can receive an interdependency graph of operator instances for a workflow run. The operator instances can be associated with one or more operator types. The workflow execution engine can assign one or more computing environments from a candidate pool to execute the operator instances based on the interdependency graph. The workflow execution engine can generate a schedule plan of one or more execution requests associated with the operator instances. The workflow execution engine can distribute code packages associated the operator instances to the assigned computing environments. The workflow execution engine can maintain a memoization repository to cache one or more outputs of the operator instances upon completion of the execution requests.