Roblox Corporation
Using map-reduce to increase processing efficiency of small files

Abstract:

A system and method includes increasing processing efficiency of small files. The method includes generating a file including a plurality of file identifiers that identify a plurality of files in a file system. The method includes causing an execution of a map operation based on the file. The map operation distributes the plurality of file identifiers in a plurality of groups across a plurality of nodes of a computer cluster. The method includes causing an execution of a reduce operation on the plurality of groups of the plurality of file identifiers by the plurality of nodes of the computer cluster. The reduce operation for a first group of the plurality of groups includes reading data of first files of the plurality of files from the file system based on the first group of file identifiers and merging the data of the first files into an output file.