Microsoft Corporation
Coordinating distributed task execution

Abstract:

The disclosed embodiments provide a system for coordinating distributed task execution. During operation, the system retrieves a first batch of tasks from multiple message queues in a distributed messaging system. Next, the system assigns the first batch of tasks to multiple task executors in a thread pool based on availabilities of the multiple task executors. The system also tracks statuses associated with processing the first batch of tasks based on communications from the multiple task executors. The system further periodically commits, based on the tracked statuses, offsets of completed tasks in the multiple message queues to the distributed messaging system.