Feature Collibra orchestrator
The Collibra orchestrator, which interacts with Collibra to publish metadata about the data transformed in your DataOps pipeline, is now officially released on DataOps.live.
This means it is fully supported and accessible to all our customers. For more information, see the Collibra Orchestrator documentation.
PubPrev VaultSpeed orchestrator
The VaultSpeed orchestrator, which helps deploy the data vault you have designed in VaultSpeed into a Snowflake database, is now in public preview.
This means it has broadly finished development and will not change significantly when fully released to all customers. For more information, see the VautSpeed Orchestrator documentation.
Feature Bundled Feature Flag
Bundled feature flags provide a more efficient way to adopt and test new features while minimizing risks. For more information, see Bundled Feature Flags.
In this release, we're introducing a DataOps.live bundle, which includes several behavioral adjustments. These changes are incorporated within a bundled feature flag, which is optional, offering flexibility to our users.
You can activate it by adding the value
2023_08 to the variable
DATAOPS_ENABLE_BEHAVIOR_CHANGE_BUNDLE in your project settings.
Details about the changes in this bundle are outlined below:
- Currently, empty YAML files or empty lists in YAML files in your configuration cause compile errors. With the feature flag activated, the Snowflake Object Lifecycle Engine (SOLE) handles empty lists and null values identically.
- Currently, SOLE does not revoke grants on the privileges of Snowflake objects when an empty list is provided. With the feature flag activated, you can revoke all privileges with an empty
- Currently, when SOLE runs in a feature branch with
DEV, account-level resources are NOT reused from the DEV environment and are created for each feature branch. And when the feature branch teardown job runs, all account-level resources (apart from the main DB) are dropped from the DEV environment. With the feature flag activated, SOLE doesn't create DEV resources in a feature branch, and, more importantly, it doesn't teardown DEV resources in a feature branch.
Bug Fix Bug fixes in SOLE
- All grants on the privileges of Snowflake objects are revoked if an empty
grants:list is provided in the configuration file.
- DEV environment resources aren't removed when
DATAOPS_FEATURE_BRANCH_NONDB_ENV_NAMEis set to
DEVin a feature branch.
- Masking policies defined in a shared database and shared between branches aren't removed from the shared tags when running a destroy job in a pipeline.
- With the
SHOW <objects>command, both wildcard characters
%are escaped now.
DataOps development environment
Improvement New DDE setup wizard
We have added a DDE setup wizard that handles authentication and setup for you, ensuring a swift start with the development environment. For additional details, see Giving DevReady access to Snowflake or Giving DevPod access to Snowflake, based on the deployment model you use.
Improvement Enhanced DataOps reference project
We've added more optional jobs to the reference project for managing Snowflake development environments, making setting up and removing such environments effortless. We've also introduced execution rules to handle Snowflake in different setups.
See Reference Project Jobs for more information.
Improvement Updated DataOps template project
We've deprecated the DataOps Standard Template V1 and introduced a more advanced and enhanced version, the DataOps Standard Template V2. This template project helps you start quickly with DataOps as it initializes your project with the recommended project structure and the default project settings. See Creating a Project from a Template for more information.
While the deprecated V1 template remains available for use, it will no longer have the latest updates.
Improvement Bug Fix Improvements and bug fixes
- Data product platform: We've introduced general performance improvements and bug fixes to the data product platform.
- DataOps template project bugfix - 230916: We've resolved an issue with the two DataOps Standard Templates released earlier this month. Previously, projects created with these templates contained legacy artifacts (pipelines and merge requests). This has now been fixed, and any projects created with the template will have a clean history.