Palantir Technologies Inc.
CONTINUOUS BUILDS OF DERIVED DATASETS IN RESPONSE TO OTHER DATASET UPDATES

Abstract:

A data processing method comprises creating and storing a dependency graph representing at least one derived dataset and one or more raw datasets or intermediate derived datasets on which the at least one derived dataset depends; reading configuration data specifying one or more periods for one or more datasets in the dependency graph; detecting a first update to a first dataset; initiating a first build of a first intermediate derived dataset only when a then-current time is within a first period of the one or more periods or a previous build of the first intermediate derived dataset occurred earlier than a then-current time less a second period of the one or more periods; asynchronously detecting a second update to a second dataset; initiating, in response to the second update, a second build of a second intermediate derived dataset that depends on the second dataset.