Skip to main content

Upgrading from DataOps v4 to DataOps v5

warning

Major releases may not be backward compatible. The changes required to move a project between major versions are listed below.

Project changes​

Snowflake Object Lifecycle Engine (SOLE) changes​

Snowflake is now managed by SOLE. You must update your project to use a SOLE configuration and remove the "clone, mask, and grant" jobs.

The new template project should be used as a guide for project changes.

Reference project​

A wide set of changes are included in the v5 reference project. To update your project simply modify your pipelines/includes/bootstrap.yml file to change from

include:
  - project: "reference-template-projects/dataops-template/dataops-reference"
    ref: v4.0
    file: "/pipelines/includes/base_bootstrap.yml"

to

include:
  - project: "reference-template-projects/dataops-template/dataops-reference"
    ref: 5-stable
    file: "/pipelines/includes/base_bootstrap.yml"

Orchestrator image changes​

The v5 orchestrators are all tagged with 5-stable and are published in the dataopslive namespace. So, all image names in the project should be updated to the form dataopslive/<image-name>:5-stable. We recommend using image names from the reference project, as described above.

Modeling and transformation changes​

5-stable introduces dbt version 1.0.1, which is an upgrade from dbt version 0.17 in 4-stable. There are several changes required in dataops/modelling/dbt_project.yml

  • 'config-version': 2 should be added to the top of the YAML file.

  • The 'source-paths' key should be renamed to 'model-paths'

  • The 'data-paths' key should be renamed to 'seed-paths'

When completed the top of your dataops/modelling/dbt_project.yml file may look like:

## Project
name: "TrueDataOpsDemo"
version: "0.1"
config-version: 2
profile: "snowflake_operations"
## Source paths
model-paths: ["models", "sources"]
analysis-paths: ["analysis"]
test-paths: ["tests"]
seed-paths: ["data"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

Additionally, the name of the directory that dbt packages into which packages are installed has changed from 'dbt_modules' to 'dbt_packages'. This will only affect projects that are manually installing new dbt packages.

To use the new Security and Governance features, a post hook is required. This post hook should be defined once per project, at the top level of the models in the dbt_project.yml file. For example,

models:
  my_model:
    +post-hook:
      - "{{ dataops.apply_snowflake_governance() }}"

The + before post_hook ensures that dbt will process this as a directive, rather than a model directory.

Tests have been redesigned, and now return failing rows. Custom test SQL will need updating - please see the dbt documentation.

The dlx_modelling_and_transformation_libs package has been renamed to dataops. Any references to the dlx_modelling_and_transformation_libs package should be updated to refer to dataops in your project.

Last updated on