Orchestrators Release Notes (September 22, 2022)

v5.7.0 (5-latest)

Feature

SOLE

• You can now easily apply tags on the Snowflake objects Database, Schema, and Table by using the new keyword with_tags. For example:

  databases:
    DATABASE_1:
      schemas:
        SCHEMA_1:
          tables:
            TABLE_1:
              with_tags:
                DEPARTMENT: Sales
              columns:
                COLUMN_1:
                  type: TEXT
                  with_tags:
                    DEPARTMENT: Sales

  Based on these configurations, SOLE generates and executes a set of statements to update or apply the tags to the related object. Refer to the SOLE tag management for a complete discussion of how to use the new capabilities.

• Better validation techniques for the SOLE configuration are implemented:

  • Schema validation fails if an invalid or unsupported argument is specified
  • SOLE configuration key validation throws an error if the object identifier is not a string
  • Stage validation is added for the encryption parameter of an internal stage

Feature

Orchestrators

• The SecretsManger orchestrator now supports integration with Hashicorp Vault
• A new Snowpark Python orchestrator has been introduced to help you query and process data during pipeline execution in private preview.

Improvement

In SOLE and Orchestrators

• Improved the usability of SOLE logging when it runs the first time and discovers all existing objects. Instead of logging many errors for objects it didn't know about yet, it now logs only warnings and limits the number of warnings to better indicate that this is expected behavior.
• Updated logging to use the DataOps Logger in the Reporting orchestrator and improve the log info and layout
• Supported attaching Network Policy to User and Account
• TMC orchestrator:
  • Added seamless retries for the HTTP status codes 502, 503, and 504
  • Updated logging to warn you when expected errors happen, e.g., network issue

Bug Fix

In MATE, SOLE, and Orchestrators

• Fixed the macro dataops.apply_dynamic_policy in MATE so that ALTER query takes the model's materialization in consideration
• Fixed a bug in SOLE and StageIngestion where no ingestion was performed if another object with same name exists in same schema
• Fixed a bug in the Reporting orchestrator, where not specifying a RENDER script led to the job failing
• Fixed the name of the file uploaded for metadata profile in the data.world orchestrator