Release Notes August 2024

Feature

DataOps Runner for Snowpark Container Services reaches Public Preview

We are making a new version of the DataOps Runner available starting with the August release. So far, we have supported deploying a runner with Docker as a single instance and Kubernetes as part of an auto-scaling group.

Now, you can also deploy the DataOps Runner on Snowpark Container Services as a service native to Snowflake. No data leaves Snowflake now; you don't need to use an external service as a dependency.

• Keep everything inside Snowflake – even credentials
• No additional deployment is needed at AWS/Azure/Google

Learn how to set up the DataOps Runner for Snowpark Container Services.

Operate

Improvement

DataOps project templates

• We have released an update to both the Professional edition template and Enterprise edition template. The main change is using SOLE for Data Products as the default.

Bug Fix

Bugfixes in Operate

We have fixed a visual bug when viewing a project's issues. Now, all users with permission can view project issues.

Orchestrators

Feature

Enabling Deletion Prevention of Snowflake objects by default

Starting with the August release, deletion prevention of Snowflake objects is now enabled by default. Previously, you had to activate this feature manually, but we have made it the default setting to enhance data protection.

What this means for you:

• Default Activation: Deletion Prevention is automatically enabled for all users, and you need to explicitly mark objects to be deleted in your SOLE configuration
• Opt-Out: If you prefer not to use this feature, you can easily disable it by setting the DATAOPS_PREVENT_OBJECT_DELETION variable to 0 in your project settings file pipelines/includes/config/variables.yml.

We encourage you to review the updated SOLE documentation and adjust your project settings accordingly. If you have any questions or need further assistance, please don't hesitate to contact our support team.

Feature

Informatica CDGC orchestrator

We've introduced a new Informatica CDGC orchestrator to help you publish metadata about the data transformed in a DataOps pipeline.

See the new Informatica CDGC orchestrator documentation for up-to-date information.

Feature

Allow support for multiple arguments in rel() in SOLE for Data Products

The rel() function in SOLE for Data Products now supports multiple arguments. This allows you to specify the object's schema and database name to allow for more flexibility in referencing managed objects in the YAML configuration.

See the Procedure documentation for more details and examples.

Feature

Add Support for ENDPOINT argument in External Stage

SOLE adds support for a new parameter endpoint when defining an External Stage object. This parameter allows you to create an external stage that uses AWS S3-compatible storage like Wasabi.

See the Stage documentation for details and examples.

Improvement

Improvements in SOLE and Orchestrators

• SOLE now enables Object Deletion Prevention by default for all objects. For more information, see the Object Deletion Prevention documentation.
• dbt runtime and third-party packages have been updated to the latest versions in the MATE and Snowflake orchestrator. Check out the updated dbt packages list for dbt 1.4, 1.5, and 1.7.

Bug Fix

Bug Fixes in SOLE and Orchestrators

• Fixed a bug in SOLE where tags could not be applied to Hybrid Tables.
• Fixed a bug in SOLE where the specified Warehouse specified in DATAOPS_SOLE_WAREHOUSE was not being used in SOLE hooks.