Skip to main content

Upgrading from DataOps v4 to DataOps v5

caution

Major releases may not be backward compatible. The changes required to move a project between major versions are listed below.

Project changes

Snowflake Object Lifecycle Engine (SOLE) Changes

Snowflake is now managed by SOLE. The project should be updated to use a SOLE configuration, and the jobs for "clone, mask and grant" should be removed.

The new template project should be used as a guide for project changes.

Reference Project

A wide set of changes are included in the v5 reference project. To update your project simply modify your pipelines/includes/bootstrap.yml file to change from

include: 
- project: "reference-template-projects/dataops-template/dataops-reference"
ref: v4.0
file: "/pipelines/includes/base_bootstrap.yml"

to

include: 
- project: "reference-template-projects/dataops-template/dataops-reference"
ref: 5-stable
file: "/pipelines/includes/base_bootstrap.yml"

Orchestrator Image Changes

The v5 orchestrators are all tagged with 5-stable and are published in the dataopslive namespace. So, all image names in the project should be updated to the form dataopslive/<image-name>:5-stable. We recommend using image names from the reference project, as described above.

Modelling and Transformation changes

5-stable introduces dbt version 1.0.1, which is an upgrade from dbt version 0.17 in 4-stable. There are several changes required in dataops/modelling/dbt_project.yml

  • 'config-version': 2 should be added to the top of the YAML file.

  • The 'source-paths' key should be renamed to 'model-paths'

  • The 'data-paths' key should be renamed to 'seed-paths'

When completed the top of your dataops/modelling/dbt_project.yml file may look like:

## Project
name: "TrueDataOpsDemo"
version: "0.1"
config-version: 2
profile: "snowflake_operations"
## Source paths
model-paths: ["models", "sources"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
seed-paths: ["data"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

Additionally, the name of the directory that dbt packages into which packages are installed has changed from 'dbt_modules' to 'dbt_packages'. This will only affect projects that are manually installing new dbt packages.

To use the new Security and Governance features, a post hooks is required. This post hook should be defined once per project, at the top level of the models in the dbt_project.yml file. For example,

models:
my_model:
+post-hook:
- "{{ dataops.apply_snowflake_governance() }}"

The + before post_hook ensures that dbt will process this as a directive, rather than a model directory.

Tests have been redesigned, and now return failing rows. Custom test SQL will need updating - please see the dbt documentation.

The dlx_modelling_and_transformation_libs package has been renamed to dataops. Any references to the dlx_modelling_and_transformation_libs package should be updated to refer to dataops in your project.