Release Notes September 2023

Orchestrators

Feature

Collibra orchestrator

The Collibra orchestrator, which interacts with Collibra to publish metadata about the data transformed in your DataOps pipeline, is now officially released on DataOps.live.

This means it is fully supported and accessible to all our customers. For more information, see the Collibra Orchestrator documentation.

PubPrev

VaultSpeed orchestrator

The VaultSpeed orchestrator, which helps deploy the data vault you have designed in VaultSpeed into a Snowflake database, is now in public preview.

This means it has broadly finished development and will not change significantly when fully released to all customers. For more information, see the VautSpeed Orchestrator documentation.

Feature

Bundled Feature Flag

Bundled feature flags provide a more efficient way to adopt and test new features while minimizing risks. For more information, see Bundled Feature Flags.

In this release, we're introducing a DataOps.live bundle, which includes several behavioral adjustments. These changes are incorporated within a bundled feature flag, which is optional, offering flexibility to our users.

You can activate it by adding the value 2023_08 to the variable DATAOPS_ENABLE_BEHAVIOR_CHANGE_BUNDLE in your project settings.

Details about the changes in this bundle are outlined below:

• Currently, empty YAML files or empty lists in YAML files in your configuration cause compile errors. With the feature flag activated, the Snowflake Object Lifecycle Engine (SOLE) handles empty lists and null values identically.
• Currently, SOLE does not revoke grants on the privileges of Snowflake objects when an empty list is provided. With the feature flag activated, you can revoke all privileges with an empty grants: list.
• Currently, when SOLE runs in a feature branch with DATAOPS_FEATURE_BRANCH_NONDB_ENV_NAME set to DEV, account-level resources are NOT reused from the DEV environment and are created for each feature branch. And when the feature branch teardown job runs, all account-level resources (apart from the main DB) are dropped from the DEV environment. With the feature flag activated, SOLE doesn't create DEV resources in a feature branch, and, more importantly, it doesn't teardown DEV resources in a feature branch.

Bug Fix

Bug fixes in SOLE

• All grants on the privileges of Snowflake objects are revoked if an empty grants: list is provided in the configuration file.
• DEV environment resources aren't removed when DATAOPS_FEATURE_BRANCH_NONDB_ENV_NAME is set to DEV in a feature branch.
• Masking policies defined in a shared database and shared between branches aren't removed from the shared tags when running a destroy job in a pipeline.
• With the SHOW <objects> command, both wildcard characters _ and % are escaped now.

DataOps development environment

Improvement

New DDE setup wizard

We have added a DDE setup wizard that handles authentication and setup for you, ensuring a swift start with the development environment. For additional details, see Giving DevReady access to Snowflake or Giving DevPod access to Snowflake, based on the deployment model you use.

Platform

Improvement

Enhanced DataOps reference project

We've added more optional jobs to the reference project for managing Snowflake development environments, making setting up and removing such environments effortless. We've also introduced execution rules to handle Snowflake in different setups.

See Reference Project Jobs for more information.

Improvement

Updated DataOps template project

We've deprecated the DataOps Standard Template V1 and introduced a more advanced and enhanced version, the DataOps Standard Template V2. This template project helps you start quickly with DataOps as it initializes your project with the recommended project structure and the default project settings. See Creating a Project from a Template for more information.

note

While the deprecated V1 template remains available for use, it will no longer have the latest updates.

Improvement

Bug Fix

Improvements and bug fixes

• Data product platform: We've introduced general performance improvements and bug fixes to the data product platform.
• DataOps template project bugfix - 230916: We've resolved an issue with the two DataOps Standard Templates released earlier this month. Previously, projects created with these templates contained legacy artifacts (pipelines and merge requests). This has now been fixed, and any projects created with the template will have a clean history.