VMware, Inc.
Impartial buffering in stream processing

Abstract:

Techniques for generating a stream processing pipeline are provided. In one embodiment, a method includes generating a plurality of pipeline stages of a stream processing pipeline in accordance with a configuration file. The plurality of pipeline stages includes a first buffer stage designated for a first data service and a second buffer stage designated for a second data service. The method further includes collecting data items; processing the collected data items; and storing at least a portion of the processed data items in the first buffer stage and at least a portion of the processed data items in the second buffer stage. The method further includes transmitting the data items stored in the first buffer stage to the first data service at a first transmission rate; and transmitting the data items stored in the second buffer stage to the second data service at a second transmission rate.