Skip to main content

DataOps Environments

By default, DataOps is set up to use the following environments, configured from the project variables in the DataOps Reference Project.

environments-diagram _shadow_

Production

Associated with the project's production branch, master/main (value of variable DATAOPS_BRANCH_NAME_PROD), the production environment is always the first pipeline to run. It will set up the primary production database and other key resources in Snowflake. As production is a persistent, long-running branch, its resources are likewise almost always long-running resources. Therefore, these resources, especially the production database, are not usually recreated in DataOps.

Careful!

For all new projects, the first-ever pipeline to run must be in production (the main/master branch) so that when feature branch pipelines run, they can clone the production database.

By default, all production Snowflake objects will have the suffix PROD (value of variable DATAOPS_ENV_NAME_PROD). For instance, the primary production database is typically named DATAOPS_PROD.

Quality Assurance

Like production, the QA environment is a long-running branch with persistent resources in Snowflake. By using the standard branch name qa (value of variable DATAOPS_BRANCH_NAME_QA), this environment will, unless configured otherwise, use the suffix QA (value of variable DATAOPS_ENV_NAME_QA) for its database and other account-level Snowflake resources. For instance, the primary quality assurance database is typically named DATAOPS_QA.

Development

Whereas production and QA are long-running environments, DataOps treats Development as more transient, refreshing all resources and recreating the primary development database as a clone of the production database every time a pipeline runs.

Therefore, pipelines that run in the dev branch (value of variable DATAOPS_BRANCH_NAME_DEV) look slightly different from those in the higher branches (like prod and qa), as ingestion jobs are generally automatically omitted due to the available cloned production data.

Feature Branches

DataOps will designate any branch named differently from those detailed above as a feature branch. There is no mandated naming convention, but customers are encouraged to implement a naming policy and standard that fits their processes.

Databases and other account-level objects in Snowflake managed by a feature branch will take a naming suffix from a simplified version of the branch name. For example, a branch named DWH-12345 will result in a suffix of FB_DWH12345, which will, in turn, name the feature branch database DATAOPS_FB_DWH12345.

Resources

To learn more about namespacing for the given environment refer to the SOLE Fundamentals Guide and the SOLE Namespace and Environment Management Reference.