Walmart Inc.
Systems and methods for distributed data validation

Abstract:

Embodiments of the present disclosure include systems and methods for validating a target data table based on a source data table. A distributed memory comprises a plurality of computing systems, each storing at least a portion of the source data table and the target data table in local memory. Processing engines can be efficiently executed on each of the plurality of computing systems to perform comparison functions based on in-memory data. A checksum comparison engine is configured to compare source and target checksums. A data aggregation engine is configured to produce column-based aggregation summaries. A rule generation engine is configured to generate validation rules for checking by a validation engine.