International Business Machines Corporation
ADAPTIVE DATA INGESTION RATES

Abstract:

Described are techniques for data ingestion including determining a respective moving average streaming rate for each of a plurality of incoming data streams to a cluster-computing framework. The techniques further include determining a respective ingestion frequency for each of the plurality of incoming data streams by dividing a platform-preferred ingestion rate of the cluster-computing framework by a respective moving average streaming rate of each of the plurality of incoming data streams. The techniques further include ingesting each of the plurality of incoming data streams to the cluster-computing framework at the respective ingestion frequency.