Roblox Corporation
ADDRESSING DATA SKEW USING MAP-REDUCE
Last updated:
Abstract:
A system and method includes using a queue with reduce operations. A method can include, responsive to generation of one or more markers by a first node or a second node, causing a submission of one or more markers to a queue associated with a computing cluster. Additionally, responsive to a determination that the first node has completed a first reduce operation, directing the first node to perform a first copy operation to copy first data identified by a first marker of the one or more markers in the queue, where the first copy operation is performed concurrently with the second reduce operation.
Status:
Application
Type:
Utility
Filling date:
6 May 2021
Issue date:
19 Aug 2021