Alibaba Group Holding Limited
System and method for facilitating reduction of complexity and data movement in erasure coding merging on journal and data storage drive

Abstract:

A system is provided for performing erasure coding (EC) in a distributed storage system. During operation, the system can perform a partial encoding of a received first set of data fragments and second set of data fragments using EC to generate a first and a second EC codeword, respectively. The system can then distribute the first and the second set of data fragments among a set of storage nodes within the distributed storage system. The system can also distribute a first and the second set of intermediate parity fragments in the first and second EC codeword, respectively, among a subset of the storage nodes with alignments. The system can then merge the first and the second set of intermediate parity fragments to generate an overall parity for both the first and the second set of data fragments. The system can store, based on the alignments, each overall parity fragment in the overall parity in the corresponding subset of storage nodes.