Skip to main content

Orchestrators Release Notes (April 25, 2022)

v5.4.6 (5-latest | 5-stable)

We have added new features to the existing DataOps Orchestrators' functionality. They are divided into the following categories:

Feature release status badge: Feature
Feature
Extended Snowflake Object Management

Our expanded support for the following managed objects and grants have been added to the DataOps Snowflake Object Lifecycle Engine (SOLE):

Feature release status badge: Feature
Feature
Enhanced Logging

We have developed richer log files to enable faster debugging and troubleshooting. The SOLE log files now include information about:

Hooks Logging

A hook is a placeholder or interface that allows you to insert a custom function before or after key stages in the SOLE process. This upgrade now provides enhanced detail in the SOLE log files to capture relevant information about the hooks you choose to use in your pipeline architecture to allow for better visibility and debugging.

Here are the Before and After Hook logging examples:

Before

hooks-before !!shadow!!

After

hooks-after !!shadow!!

Destroy Job Logging

Running the SOLE Destroy Job in debug mode will log all objects being deleted as well as a summary of the objects not deleted, together with a reason why each one hasn't been deleted.

Here are the Before and After Destroy Job logging examples:

Before

destroy-job-before !!shadow!!

After

destroy-job-after !!shadow!!

Missing Warehouse Warning

SOLE will fail if the user it is connecting to in Snowflake does not have an associated warehouse (default or specified). From this upgrade onwards, SOLE will log a warning to flag this configuration detail ahead of time or before the pipeline fails.

missing-warehouse-warning !!shadow!!

SOLE-Specific Debugging

When debugging, you can use the new DATAOPS_SOLE_DEBUG parameter; thus, preventing SOLE from revealing credentials in its log files.

dataops-sole-debug !!shadow!!

Feature release status badge: Feature
Feature
Increased Flexibility

This upgrade includes the following points that fall within SOLE's increased capacity to be flexible:

Externally-Defined Namespacing

Managing namespaces is hard. We have solved this and made namespacing easier, so you can now opt to use custom namespacing formats by including namespace: external in your SOLE configuration files. No prefixes or suffixes will be added to your Snowflake objects, and SOLE will continue to manage the complete object lifecycle.

For more information, see the DataOps Namespace and Environment Management reference document.

Here are several examples of SOLE log files, logging namespacing details:

External Namespace Configuration

namespace-4 !!shadow!!

SOLE Log Entry for Role Creation

namespace-5 !!shadow!!

SOLE Log Entry for Role Removal

namespace-6 !!shadow!!

Choosing a Warehouse

SOLE will use the warehouse linked to the associated Snowflake user credentials by default. SOLE will fail if the user does not have a default warehouse attached to its account.

From this upgrade onwards, you can specify the warehouse you want SOLE to use, providing more options around how you choose to manage your Snowflake ecosystem. Therefore, the default warehouse attached to your user account is no longer the only option.

choose-warehouse1 !!shadow!!

choose-warehouse2 !!shadow!!