Amazon.com, Inc.
Multi-layer network for metric aggregation

Abstract:

A multi-layered parallel aggregation can be performed on large-scale metric streams using layers of independent host server computers that perform partial aggregations on results of a previous layer and pass a result of the partial aggregation to a next layer of host server computers, until a single host server computer in a last layer can calculate a final output, which is a combination of the partial aggregations. Because every layer is aggregating input streams, the quantity of data exchange lowers as the layer number increases. The number of layers is chosen so as to ensure that data ingested by a last layer is sufficiently manageable that a single host server computer in the last layer can aggregate all of the partial aggregates into a final result.