Release Notes May 2024

Orchestrators

Feature

Include statement from an external file in the Function or Procedure objects

Snowflake Object Lifecycle Engine (SOLE) now extends the statement: key for functions and stored procedures by allowing you to include the statement from an external file using the directive !dataops.include. In particular, for very long statements, you can now benefit from proper syntax highlighting and other language support in our development environment, e.g., SQL, Python, or JavaScript.

See the updated documentation for more information:

• for procedures
• for functions

Feature

dbt Cloud orchestrator

We've introduced a new dbt Cloud orchestrator to help you easily interact with dbt Cloud to trigger jobs from a DataOps pipeline.

See the new dbt Cloud orchestrator documentation for more information.

Feature

data.world catalog orchestrator

The data.world catalog orchestrator is now available to help you interact with the data.world data catalog to publish metadata about the data transformed in a DataOps pipeline.

See the updated catalog orchestrator documentation for more information.

Improvement

New parameter in the Table object

Now, with the introduction of the always_drop_and_create parameter in the table definition for ingestion tables, you can organize all columns in the table within Snowflake in the same order as they are defined in the YAML file.

This parameter ensures that new columns are not appended at the end. Be aware though, that using the parameter means you will lose all data in the target table. Thus, only use the parameter for tables that see a complete data refresh as part of running a pipeline.

Read more at SOLE table reference guide.

Bug Fix

Bug fixes in SOLE

• If a default database in the User config is defined, it leads to an error
• Fixed table relation in Pipe configuration
• Fixed the Publish Docker Images jobs of the Orchestrators' CI/CD pipeline
• Enforced strict validation for table_grants and schema_grants in SOLE: This change is incorporated within the bundled feature flag 2024_05 that you need to activate by adding the value 2024_05 to the variable DATAOPS_ENABLE_BEHAVIOR_CHANGE_BUNDLE in your project settings. For more information about all Dataops.live bundles, see Bundled Feature Flags.

Improvement

Improved YAML validation

We've improved the validation of the YAML templates present in your project. Any invalid usage of tab characters (\t) in your YAML files will lead to pipeline failure.

For more information, check the Template Rendering page.

DataOps CLI (SOLE Generator)

Feature

New command to generate YAML configuration for SOLE for Data Products

SOLE Generator now supports generating YAML configuration compatible with SOLE for Data Products through the command dataops sole-dp gen command.

For more information, see the SOLE Generator documentation.