Nutanix, Inc.
AI INFERENCE HARDWARE RESOURCE SCHEDULING

Abstract:

Systems and methods described herein generally relate to compute node resource scheduling. AI inference services described herein may receive a request to execute a machine learning model in a clustered edge system. To determine which hardware resource comprising computing nodes of the clustered edge system on which to execute the machine learning model, AI inference services may compare the computational workload of the machine learning model, with the computational abilities and functions of the hardware resources. In examples, the comparison is based on a scheduling algorithm, including an identification stage to identify candidate hardware resources capable of executing the machine learning model, and a scoring stage to select the best candidate hardware resource for executing the machine learning model. A scheduler may assign the machine learning model to the selected hardware resource for execution by the AI inference services.