Skip to content

SOLE Orchestrator

Enterprise

Image$DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE

The Snowflake Object Lifecycle Engine (SOLE) orchestrator is responsible for engaging with SOLE to process the Snowflake object configuration stored in the /dataops/snowflake project directory.

For more information, refer to the SOLE User Guide and the SOLE Reference Guide.

Usage

SOLE uses lifecycle actions to manage its processes and operations. As described in the SOLE Lifecycle Actions documentation, there are different lifecycle actions grouped into the following groups:

Warning

PLAN-ALL and APPLY-ALL are coupled and apply only to non-cloned Snowflake databases.

Let's look at the code for each group as found in the DataOps Reference Project:

Setting up Snowflake with AGGREGATE

pipelines/includes/local_overrides/snowflake_lifecycle.yml
"Set Up Snowflake":
  extends:
    - .agent_tag
  stage: Snowflake Setup
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  resource_group: $CI_JOB_NAME
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    - if: '$DATAOPS_SOLE_ENABLE_PLAN_APPROVAL == "1" || $DATAOPS_SOLE_ENABLE_PLAN_APPROVAL == "true"'
      when: never
    - when: on_success

Cleaning up with AGGREGATE-DESTROY

pipelines/includes/local_overrides/snowflake_lifecycle.yml
"Tear Down Snowflake":
  extends:
    - .agent_tag
  stage: Clean Up
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE-DESTROY
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  resource_group: $CI_JOB_NAME
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    ## Block this job from main (PROD) and qa (QA) branches
    - if: "$CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD || $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA"
      when: never
    ## For all other branches, enable this job to be run manually
    - when: manual

Setting up Snowflake with PLAN-ALL

pipelines/includes/local_overrides/snowflake_lifecycle.yml
"Plan Snowflake Setup":
  extends:
    - .agent_tag
  stage: Snowflake Setup
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: PLAN-ALL
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  resource_group: $CI_JOB_NAME
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
      - dataops/report/plan_viewer/
    name: "SOLE Plan Viewer"
    expose_as: "dataopsreport"
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    - if: '$DATAOPS_SOLE_ENABLE_PLAN_APPROVAL == "1" || $DATAOPS_SOLE_ENABLE_PLAN_APPROVAL == "true"'
      when: on_success
    - when: never

Applying actions with APPLY-ALL

pipelines/includes/local_overrides/snowflake_lifecycle.yml
"Apply Plan to Snowflake":
  extends:
    - .agent_tag
  stage: Snowflake Setup
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: APPLY-ALL
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  resource_group: $CI_JOB_NAME
  needs:
    - job: Plan Snowflake Setup
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
      - dataops/report/plan_viewer/
    name: "SOLE Plan Viewer"
    expose_as: "dataopsreport"
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    - if: '$DATAOPS_SOLE_ENABLE_PLAN_APPROVAL == "1" || $DATAOPS_SOLE_ENABLE_PLAN_APPROVAL == "true"'
      when: manual
    - when: never

Supported parameters

Parameter Required/Default Description
CONFIGURATION_DIR REQUIRED. Defaults to $CI_PROJECT_DIR/dataops/snowflake The project directory where the Snowflake configuration files are located.
DATAOPS_SOLE_ACCOUNT REQUIRED—from connection parameters The Snowflake account. If the account is a region-based account or the organization name is not used, the region must be added to this variable.
DATAOPS_SOLE_USERNAME REQUIRED—for password-based authentication, from connection parameters The Snowflake account username
DATAOPS_SOLE_PASSWORD REQUIRED—for password-based authentication, from connection parameters The Snowflake account password
DATAOPS_SOLE_ROLE REQUIRED—from connection parameters The Snowflake role that can run queries.
DATAOPS_PREFIX REQUIRED—from project settings This prefix is added to all account-level objects and databases (except the default database).
DATAOPS_DATABASE REQUIRED—from project settings The default database name - required if the default database is specified in the configuration as {{env.DATAOPS_DATABASE}}.
DATAOPS_ENV_NAME REQUIRED—from project settings This suffix is added to all account-level objects and databases (except the default database).
LIFECYCLE_ACTION REQUIRED See below for this parameter's valid values.
ARTIFACT_DIRECTORY Optional. Defaults to $CI_PROJECT_DIR/snowflake-provider The path to where the artifacts such as the resource files, import files, and state files are uploaded. It is essential to specify the ARTIFACT_DIRECTORY as an artifact in the related DataOps job.
DISABLE_PERSISTENT_CACHE Optional If set, it disables persistent cache usage by the orchestrator. The user must handle the generated file transfers between jobs.
DATAOPS_DATABASE_MASTER Optional—from project settings The PROD or main database - required if the production database is referenced in the configuration by the variable {{env.DATAOPS_DATABASE_MASTER}}.
DATAOPS_ENV_NAME_PROD Optional. Defaults to main Specifies the branch used as the production environment
DATAOPS_NONDB_ENV_NAME Optional The suffix override value - if specified, it overrides all branch-specific suffixes with its value.
DATAOPS_PRESERVE_OBJECT_NAME Optional If set, it enables special character support.
DATAOPS_PREVENT_OBJECT_DELETION Optional. Default to 1 If set to 0, it disables SOLE's Object Deletion Prevention funtionality. Read more about it here.
DATAOPS_SOLE_DEBUG Optional If set, it enables the debug logs for SOLE without exposing any credentials when writing to these logs.
DATAOPS_SOLE_DISABLE_IMPLICIT_GRANTS Optional. Defaults to false If set, it prevents SOLE from granting USAGE privilege on parent objects implicitly.
DATAOPS_SOLE_ENABLE_PLAN_APPROVAL Optional If set, it enables a coupled plan and apply process that allows you to review and approve modifications to a Snowflake database before applying them.
DATAOPS_SOLE_MANAGE_REFERENCES Optional If set, you can grant the REFERENCES privilege to the roles for materialized views.
DATAOPS_SOLE_OVERRIDE Optional If set, it overrides the object's delete/update prevention feature.
DATAOPS_SOLE_WAREHOUSE Optional—from connection parameters The Snowflake warehouse used for SOLE grant management.
SNOWFLAKE_WAREHOUSE Optional—from connection parameters The Snowflake warehouse used by SOLE to run hooks.
DONT_USE_SOLE_GRANT_MANAGEMENT Optional If set, it disables SOLE grant management for managing the privilege/role grants and revokes to use the Terraform Provider.
LIFECYCLE_IMPORT_FILE Optional. Defaults to resource_imports.json The file name containing the resource import statements.
LIFECYCLE_MANAGE_OBJECT Optional See below for this parameter's valid values. If no value is specified, all object groups are managed.
LIFECYCLE_RESOURCE_FILE Optional. Defaults to resources.tf.json The file name containing the resource definitions.
LIFECYCLE_STATE_RESET Optional If set, the local state is reset and all defined and managed objects are re-initialized.
GENERATE_HOOKS_ARTIFACTS Optional If set, it adds pre- and post-hooks to the generated report section of your pipeline. For more information about hooks, see Object Group Hooks.
PROVIDER_DIR Optional. Defaults to /snowflake-provider The provider configuration directory path that must contain a file named provider.tf with a list of Snowflake providers.
SET_TERRAFORM_KEYS_TO_ENV Optional If set, it exports SOLE credentials from the DataOps Vault to the pipeline environment.
SOLE_LOG_CLEANUP Optional If set, it removes all the log files from the host system's cache.
SOLE_CACHE_CLEANUP Optional If set, it removes the host system's cache.

CONFIGURATION_DIR

The CONFIGURATION_DIR parameter specifies your project directory where the Snowflake configuration files are located. Refer to the SOLE Managed Object Reference for supported configurations.

LIFECYCLE_ACTION

The valid values for this variable are as follows:

  • AGGREGATE - execute compile, validate, plan, and apply
  • COMPILE - only compile the Snowflake configuration found in the CONFIGURATION_DIR
  • VALIDATE - only validate the compiled resources
  • PLAN - only plan the validated configuration
  • APPLY - only apply the planned configuration
  • PLAN-ALL - execute compile, validate, and plan — but without "apply" contrary to AGGREGATE
  • APPLY-ALL - only apply the planned configuration by PLAN-ALL
  • AGGREGATE-DESTROY - execute compile, validate, plan-destroy, and destroy
  • PLAN-DESTROY - only plan how to tear down the Snowflake configuration
  • DESTROY - only execute the teardown plan

LIFECYCLE_MANAGE_OBJECT

The valid values for this variable are as follows:

If no value is specified for this parameter, all object groups are managed in sequence.

DATAOPS_SOLE_DEBUG

Setting DATAOPS_SOLE_DEBUG to a value enables SOLE'S debug mode. This generates more log-output related to SOLE'S internal processes and is useful for debugging if a job fails or behaves unexpectedly. No credentials are logged when using DATAOPS_SOLE_DEBUG. The SOLE credentials are masked by a fixed 16-length character X.

Key-pair authentication

SOLE Orchestrator supports using key-pair authentication. To learn how to configure it, see key-pair authentication.

Example jobs

As described in the Usage section above, the SOLE orchestrator facilitates the implementation of the two aggregate lifecycle_actions, AGGREGATE and AGGREGATE-DESTROY. At the risk of duplicating content, providing context to the following example jobs is a good idea. Therefore, let's assume you need to create a new feature branch to develop a new report. Once the code has been written, it must be tested. To test this report, you need to run a DataOps pipeline, including the setup and tear-down Snowflake jobs.

Setting up Snowflake

The default from the DataOps Reference Project for setting up Snowflake uses the AGGREGATE lifecycle action.

pipelines/includes/local_overrides/snowflake_lifecycle.yml
"Set Up Snowflake":
  extends:
    - .agent_tag
  stage: "Snowflake Setup"
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  resource_group: $CI_JOB_NAME
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

Tearing down Snowflake

The default from the DataOps Reference Project for tearing down Snowflake for a feature branch uses the AGGREGATE-DESTROY lifecycle action.

pipelines/includes/local_overrides/snowflake_lifecycle.yml
"Tear Down Snowflake":
  extends:
    - .agent_tag
  stage: Clean Up
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE-DESTROY
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  resource_group: $CI_JOB_NAME
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    ## Block this job from main (PROD) and qa (QA) branches
    - if: "$CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD || $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA"
      when: never
    ## For all other branches, enable this job to be run manually
    - when: manual

Individual jobs

For further examples on how to use the lifecycle actions and the object hierarchies with the SOLE orchestrator, refer to the lifecycle action jobs user guide.