Skip to main content

Orchestrators Release Notes (April 25, 2022)

v5.4.6 (5-latest | 5-stable)

We have added new features to the existing DataOps Orchestrators' functionality. They are divided into the following categories:

Extended Snowflake Object Management

Our expanded support for the following managed objects and grants have been added to the DataOps Snowflake Object Lifecycle Engine (SOLE):

Enhanced Logging

We have developed richer log files to enable faster debugging and troubleshooting. The SOLE log files now include information about:

Hooks Logging

A hook is a placeholder or interface that allows you to insert a custom function before or after key stages in the SOLE process. This upgrade now provides enhanced detail in the SOLE log files to capture relevant information about the hooks you choose to use in your pipeline architecture to allow for better visibility and debugging.

Here are the Before and After Hook logging examples:

Before

hooks-before _shadow_

After

hooks-after _shadow_

Destroy Job Logging

Running the SOLE Destroy Job in debug mode will log all objects being deleted as well as a summary of the objects not deleted, together with a reason why each one hasn't been deleted.

Here are the Before and After Destroy Job logging examples:

Before

destroy-job-before _shadow_

After

destroy-job-after _shadow_

Missing Warehouse Warning

SOLE will fail if the user it is connecting to in Snowflake does not have an associated warehouse (default or specified). From this upgrade onwards, SOLE will log a warning to flag this configuration detail ahead of time or before the pipeline fails.

missing-warehouse-warning _shadow_

SOLE-Specific Debugging

When debugging, you can use the new DATAOPS_SOLE_DEBUG parameter; thus, preventing SOLE from revealing credentials in its log files.

dataops-sole-debug _shadow_

Increased Flexibility

This upgrade includes the following points that fall within SOLE's increased capacity to be flexible:

Externally-Defined Namespacing

Managing namespaces is hard. We have solved this and made namespacing easier, so you can now opt to use custom namespacing formats by including namespace: external in your SOLE configuration files. No prefixes or suffixes will be added to your Snowflake objects, and SOLE will continue to manage the complete object lifecycle.

For more information, see the DataOps Namespace and Environment Management reference document.

Here are several examples of SOLE log files, logging namespacing details:

External Namespace Configuration

namespace-4 _shadow_

SOLE Log Entry for Role Creation

namespace-5 _shadow_

SOLE Log Entry for Role Removal

namespace-6 _shadow_

Choosing a Warehouse

SOLE will use the warehouse linked to the associated Snowflake user credentials by default. SOLE will fail if the user does not have a default warehouse attached to its account.

From this upgrade onwards, you can specify the warehouse you want SOLE to use, providing more options around how you choose to manage your Snowflake ecosystem. Therefore, the default warehouse attached to your user account is no longer the only option.

choose-warehouse1 _shadow_

choose-warehouse2 _shadow_