DataOps Architecture

Conceptual architecture

DataOps is logically and physically separated into three distinct planes of operation, as illustrated in the diagram below:

Management plane

The DataOps.live data product platform, hosted on our infrastructure, is the system's Management Plane, providing all testing, observability, release, and deployment through the DataOps pipelines.

Control plane

Pipelines physically run on runners, remote agents hosted on customer infrastructure. Therefore, this execution environment is isolated from the public cloud and provides a secure runtime for the orchestration and preparation of data.

Data plane

Separated again from the DataOps.live data product platform's management and control planes, data source systems and target data platforms (particularly Snowflake), are located in the data plane. This can include any external data transformation or governance platforms, such as ELT/ETL, data catalogs, data sharing, etc., all of which can potentially be orchestrated via DataOps.