Twitter, Inc. (delisted)
Resource use and operational load of performing computing and storage tasks in distributed systems

Abstract:

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing computing and storage tasks in distributed systems. One of the methods includes obtaining multiple resource use information items associated with performance of a group of tasks on a distributed computing framework having multiple clusters; for each task of the group of tasks having a corresponding task type, determining a measure of operational load of performing the task on the distributed computing framework based on the selected resource use information items associated with the task and one or more unit measures associated with the task; combining each measure of operational load to compute a combined measure of operational load for performing the group of tasks on the distributed computing framework; identifying one or more properties of the group of tasks that indicate a need to adjust the combined measure; and adjusting the combined measure of operational load.