Palantir Technologies Inc.
ORCHESTRATION SYSTEM FOR STREAM STORAGE AND PROCESSING

Abstract:

A method, system and computer program product for orchestrating stream storage and processing is disclosed. The method, performed by one or more processors, may comprise receiving a query specifying a change to a first transformer of a data processing pipeline, the first transformer receiving an existing data stream of data records from a first data source and providing transformed data records to a first data sink. The method may also disclose identifying at least a first and second type of change to be implemented by the query and implementing the change specified in the query dependent on the identified first or second change type. The implementing may comprise one of: changing the first data transformer in accordance with the query and providing transformed output to the first data sink; and deploying a second transformer, being a changed version of the first transformer as specified in the query, for operating in parallel with the first transformer, and providing its transformed output to a second data sink.