Oracle Corporation
FAST AND SCALABLE MULTI-TENANT SERVE POOL FOR CHATBOTS

Abstract:

Techniques are disclosed for providing a scalable multi-tenant serve pool for chatbot systems. A query serving system (QSS) receives a request to serve a query for a skillbot. The QSS includes: (i) a plurality of deployments in a serving pool, and (ii) a plurality of deployments in a free pool. The QSS determines whether a first deployment from the plurality of deployments in the serving pool can serve the query based on an identifier of the skillbot. In response to determining that the first deployment cannot serve the query, the QSS selects a second deployment from the plurality of deployments in the free pool to be assigned to the skillbot, and loads a machine-learning model associated with the skillbot into the second deployment, wherein the machine-learning model is trained to serve the query for the skillbot. The query is served using the machine-learning model loaded into the second deployment.