Palantir Technologies Inc.
Processing streaming data in a transaction-based distributed database system

Abstract:

Techniques for implementing a streaming transaction in a transaction based data storage system is disclosed. In an embodiment, a server computer system stores a dataset comprising a plurality of files where the dataset comprises a corresponding version number. The server computer system receives a command to modify the dataset and, in response, starts a streaming transaction for the dataset. During the streaming transaction, the server computer system receives a plurality of updates to the dataset, wherein each of the plurality of updates modifies one or more files of the plurality of files. The server computer system executes the plurality of updates in the dataset without modifying the corresponding version number of the dataset. When the server computer system commits the streaming transaction to the dataset, the server computer system increments the corresponding version number of the dataset.