Release Notes February 2023

Orchestrators

Feature

Grants and overloading function in SOLE

• Grant option on roles: Allowing specific roles to grant the same privileges to other roles is now possible in SOLE using with_grant_option. See updated SOLE documentation.

  note

  With this release, SOLE would start revoking privileges from roles that have the grant_option present in Snowflake but not in the SOLE configuration, which was previously ignored.

• Future grants: Future grants can now be specified in a database or a schema when objects aren't managed by SOLE. See Database and Schema.

• Overloading function: SOLE now supports adding multiple functions with the same name but different arguments with the Snowflake objects: Function, External Function, and Procedure.

• Support grants on SAML Integration: SAML Integration now supports granting the USAGE, USE_ANY_ROLE, and OWNERSHIP privileges in SOLE.

Feature

MATE with dbt 1.3

MATE with dbt 1.3 is now available in public preview.

Feature

New Informatica Cloud Orchestrator

A new orchestrator that allows integrating all existing Informatica cloud taskflows into a DataOps pipeline is now available. See updated Orchestrator documentation.

Feature

Collibra Orchestrator

The Collibra orchestrator is now available in public preview. See updated Orchestrator documentation.

Improvement

Enhancements in Orchestrators and MATE

• Added support for defining Matillion custom parameters using MATILLION_VARIABLES in the Matillion orchestrator. See updated Orchestrator documentation.
• Added database and schema support in the dataops-snowsql command-line arguments.
• Added support for additional materialization when granting privileges to roles in MATE.
• Added the new parameter TRANSFORM_MODEL_INDIRECT_SELECTION_MODE to control indirect selection in tests in MATE. See updated Orchestrator documentation.

Bug Fix

Bug Fixes in SOLE and MATE

• Fixed credential masking in state file in SOLE.
• Fixed error while configuring the with_tags functionality with different databases in SOLE.
• Fixed error where default data retention for schema and table was set to 1 in SOLE.
• Fixed the usage of FULL_REFRESH_STRING variable in MATE.

DataOps CLI (SOLE Generator)

Feature

Infer schema from a parent directory

SOLE Generator now supports the -s option to inherit the schema name from the subdirectory if the schema isn't defined. See updated SOLE documentation.

Feature

Support for expressions in column defaults

SOLE Generator now supports functions and other expressions as default values for columns.

Improvement

Enhancements

We've improved the parsing of inline and out-of-line primary keys.

Bug Fix

Bug fixes

• Fixed parsing error, which occurs when a column is named Month.
• Fixed invalid parsing of constraints as columns.
• Fixed invalid parsing for tables containing CLUSTER BY.
• Fixed invalid parsing for queries containing IF NOT EXISTS.

Feature

Masking of Secrets in Pipeline Job Logs

Vault secrets now appear as [MASKED] in the pipeline job logs. See updated documentation.

EoL

Orchestrators Reaching End of Life

The Autoingestion and Python2 orchestrators have reached their end of life.

Platform

Improvement

Running multiple pipeline configuration files

We have improved our internal logic with how pipeline configuration files are processed and updated the Running Pipelines documentation for increased transparency.

Improvement

Pipeline trigger API

We have improved the DataOps.live support for triggering pipelines via API. See updated API Documentation.