Expedia Group, Inc.
Architecture for large payload handling in event pipeline

Abstract:

Systems and methods are provided for automatically orchestrating the handling of events through a processing pipeline without limitation (or without a substantial limitation) as to the size of the event payload associated with the event. The event pipeline system stores event payloads in data stores and generates notifications regarding the events. The notifications may be placed into event streams for handling by various processing components of the event pipeline system. The processing components may receive notifications or events that they are to process, and may separately access event payloads from the data stores. The processing components may generate and save processed event payloads to the data stores in a streaming fashion such that the computing resources of the processing components do not limit (or substantially limit) the size of the event payloads that the processing components may handle.