International Business Machines Corporation
AUTO-SCALING A QUERY ENGINE FOR ENTERPRISE-LEVEL BIG DATA WORKLOADS

Abstract:

Aspects of the present invention disclose a method, computer program product, and system for auto-scaling a query engine. The method includes one or more processors monitoring query traffic at the query engine. The method further includes one or more processors classifying queries by a plurality of service classes based on a level of complexity of a query. The method further includes one or more processors comparing query traffic for each service class with a concurrency threshold of a maximum number of queries of the service class allowed to be concurrently processed. The method further includes one or more processors instructing auto-scaling of a cluster of worker nodes to change a number of worker nodes available in the cluster based on the comparison, over a defined period of time, of the query traffic relative to a defined upscaling threshold and a defined downscaling threshold.