Microsoft Corporation
Dynamic scaling for data processing streaming system

Abstract:

Described herein is a system and method for dynamically scaling a stream processing system (e.g., "exactly once" data stream processing system). Various parameter(s) (e.g., user-configurable capacity, real-time load metrics, and/or performance counters) can be used to dynamically scale in and/or scale out the "exactly once" stream processing system without system restart. Delay introduced by this scaling operation can be minimized by utilizing a combination of mutable process topology (which can dynamically assign certain parts of the system to a new host machine) and controllable streaming processor movement with checkpoints and the streaming protocol controlled recovery which still enforces the "exactly once" delivery metric.