DataOps Orchestration
All DataOps jobs must use an orchestrator image that provides the most appropriate applications and tools to run the job.
The most commonly used orchestrations are:
- Snowflake Object Lifecycle (SOLE) Orchestrator for executing SOLE configurations against Snowflake
- Modelling and Transformation (MATE) Orchestrator for building and testing MATE (dbt) models
- Secrets Manager Orchestrator for loading sensitive configuration from secure storage
- Utilities Orchestrator for executing scripts and running ad-hoc commands
- Python3 Orchestrator for executing Python scripts and apps
Orchestration Types
DataOps orchestrators fall into two types: pre-set or flexible.
Pre-Set Orchestrators
Most DataOps orchestrators support a pre-set operation, meaning they perform a single main action, configured using job variables. Therefore, the job's script block only needs to call the /dataops entry point, e.g.:
My Job:
...
script: /dataops
It is possible to insert additional orchestrator scripts into the /dataops execution sequence to perform additional setup or other actions.
Flexible Orchestrators
Some orchestrators do not have a pre-set action and instead provide a set of utilities that
support whatever actions the job developer requires. Jobs using flexible orchestrators do not
typically call the /dataops
entrypoint (although it is still available if needed),
but instead define their own sequence of script actions.
List of Orchestrators
- API Orchestrator
- Azure Data Factory Orchestrator
- Azure Orchestrator
- DataPrep Orchestrator
- Fivetran Orchestrator
- Git Orchestrator
- Matillion Orchestrator
- Montecarlo Orchestrator
- Python2 Orchestrator
- Python3 Orchestrator
- R Orchestrator
- Secrets Manager Orchestrator
- SOLE (Snowflake Object Lifecycle Engine) Orchestrator
- Stitch Orchestrator
- Talend (TAC) Orchestrator
- Talend (TMC) Orchestrator
- Utils Orchestrator