Skip to main content

Orchestrators Release Notes (May 11, 2022)

v5.4.7 (5-latest | 5-stable)

We have added new features to the existing DataOps Orchestrators' functionality. They are divided into the following categories:

SOLE Special Character Support

Role names containing spaces can now be managed through SOLE. This streamlines third-party integrations, allowing you to use the same name convention across your ecosystem.

If you are integrating Microsoft Azure Directory with Snowflake and you have a role such as DEVELOPMENT TEAM, you can manage the role permissions in SOLE without needing to make any additional changes to the role name.

For example, if you want to apply the DEVELOPMENT TEAM role to a TEST_GROUP_ROLE, use the following:

dataops/snowflake/roles.yml
roles:
TEST_GROUP_ROLE:
roles:
- DATAOPS_ADMIN
- "DEVELOPMENT TEAM"

For more information, see the SOLE Special Character Support reference document.

Enhanced Logging

We continue to develop richer log files to enable faster debugging and troubleshooting.

Richer log files

  • The SOLE log files will now explicitly highlight the action that SOLE was trying to perform, alongside the SQL statement that failed.

    richer-log-files _shadow_

  • In the case of multiple errors, the parser will report all the errors it can find, together with context, to help you identify the issues and fix as many as possible at once. richer-log-files _shadow_

Real Time Logging

SOLE logs now stream as the process is running, rather than updating at the end of each section

Other Features

  • We added parameter defaults for the Parquet file formats. It is now easier to configure SOLE to manage Parquet file formats, especially useful when ingesting Parquet files into Snowflake.

  • We fixed an issue with the TRANSFORM_EXTRA_PARAMETERS_BEFORE configuration parameter in MATE. Previously, this parameter was inserted in the incorrect place in the MATE Transform action, which stopped it from working as expected.