Meta Platforms, Inc.
Systems and methods for updating data pipelines

Abstract:

A computer-implemented method for updating data pipelines may include (i) identifying at least one file that defines a series of transformations for data that passes through a data pipeline, for an application, that reads from at least one input table and writes to at least one output table, (ii) designating as a development version of the data pipeline at least one additional file that comprises a modified version of the file (iii) creating an additional version of the output table, (iv) detecting an execution of the additional file, and (v) directing output data written by the file to the additional version of the output table instead of the output table in response to both detecting the execution of the additional file and determining that the additional file is designated as the development version. Various other methods, systems, and computer-readable media are also disclosed.