International Business Machines Corporation
AUTOSCALING A DISTRIBUTED STREAMING PLATFORM IN AN EVENT-DRIVEN MICROSERVICE ARCHITECTURE

Abstract:

A computer-implemented method and computer program product for autoscaling a streaming platform in an event-driven microservice architecture. Upon a new consumer being added to a consumer group, the streaming platform triggers a rebalance phase if the number of the consumers in the consumer group is less than the number of the partitions and assign a new partition if the number of the consumers in the consumer group is greater than the number of the partitions. Upon a consumer being removed from a consumer group, in response to determining that the number of partitions is greater than the number of consumers, the streaming platform stops message publishing to a partition that is consumed by the consumer, allows remaining messages in the partition to be consumed, and deletes the partition in response to determining that all the remaining messages are consumed.