Humana Inc.
Data model driven design of data pipelines configured on a cloud platform

Abstract:

A system performs continuous delivery of a data pipeline on a cloud platform. The system receives a specification of the data pipeline comprising data pipeline units. The system generates a deployment package for each data pipeline unit for a cloud platform. The system provisions computing infrastructure on the cloud platform according to the system configuration of the data pipeline unit. The data pipeline may be implemented as a data mesh. The data pipeline generates one or more data models. The system receives a schema representing a modification in a data model based on a change in the requirements of a consumer system. The system determines the changes to the data pipeline based on the received schema and reconfigures the data pipeline to generate the modified data model. The system manages access control of data to minimize the exposure to data in case of accidental or malicious data breach.