Release Notes June 2023

Orchestrators

Feature

Managing Account parameters in SOLE

SOLE (Snowflake Object Lifecycle Engine) now supports managing the parameters of the current Account. You can define values for the Account parameters such as MIN_DATA_RETENTION_TIME_IN_DAYS, PERIODIC_DATA_REKEYING etc., in SOLE, and SOLE will automatically apply the changes to the Account object.

See Account for more information about supported parameters and examples of usage.

Feature

Using MATE with different dbt versions

MATE (Modeling and Transformation Engine) has introduced support for dbt-core 1.5. To enable this version, set the variable DATAOPS_FEATURE_MATE_DBT_VERSION to 1.5 in the file pipelines/includes/config/variables.yml.

Once set, MATE jobs will use dbt-core 1.5 instead of the default version 1.0.

With this update, MATE now supports the following versions of dbt-core:

• 1.0 (default and deprecated)
• 1.4
• 1.5

warning

As of October 2023 onward, the data product platform and its components will no longer support dbt 1.0. We strongly recommend upgrading to a later supported version, such as 1.4 or 1.5, for better performance and access to new features.

Improvement

Comma-separated secrets in the Secret Manager orchestrator

We have enhanced the secrets selection process in the Secrets Manager orchestrator. Now, you can specify a comma-separated list of secrets.

Improvement

Job fails on error in the DataPrep orchestrator

We have added the parameter DATAOPS_DATAPREP_FAIL_ON_ERROR to the DataPrep orchestrator, introducing a new behavior when processing and synchronizing files between two locations. This variable is disabled by default. Once enabled, any encountered errors during file processing will cause the job to fail.

We also improved the job logs to include a summary that indicates the number of files successfully processed and the files that encountered errors during processing. This provides a clear overview of the job's execution and enables easier identification of files with issues.

Improvement

Improved logging in DataOps render

In dataops-render, the render engine that runs in jobs within a pipeline to render the .template files, we have enhanced the logging functionality to provide comprehensive information regarding the files that encountered issues while rendering.

With the enhanced logging, you have access to detailed information that will aid in identifying and resolving rendering issues.

Improvement

Enhancements in SOLE

• Reduced the minimum value allowed for auto_suspend in Warehouse from 60 to 40 seconds.
• Improved the usage of the flag DONT_USE_SOLE_GRANT_MANAGEMENT.
• Added support for the query acceleration parameters enable_query_acceleration and query_acceleration_max_scale_factor in Warehouse.
• Added support for the comment parameter in API Integration.
• Added support for the notify_users parameter in the Resource Monitor orchestrator.
• Added support for the REFERENCES privilege on Materialized Views in SOLE.

Bug Fix

Bug fixes in orchestrators

• Fixed metadata column quoting in the Stage Ingestion orchestrator.
• Fixed an issue in MATE where auto-generation of sources failed if MATE metadata was missing in some cases.

Bug Fix

Bug fixes in SOLE

• Fixed an issue while performing revokes on future grants in SOLE for objects like external tables, materialized views, etc.
• Fixed parsing of the empty string for the environment variable DONT_USE_SOLE_GRANT_MANAGEMENT to be considered false now. See SOLE Grant Management for more information.
• Fixed granting IMPORTED PRIVILEGES on databases that are not created from Share. IMPORTED PRIVILEGES applies only to databases created from Share.