Snowflake Object Lifecycle Orchestrator
Type | Pre-Set |
---|---|
Image | $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE |
The Snowflake Object Lifecycle Engine (SOLE) is a pre-set orchestrator. And it is responsible for engaging with the Snowflake Object Lifecycle Engine (or SOLE) to process the Snowflake object configuration stored in the /dataops/snowflake
project directory.
For more information, refer to the SOLE User Guide and the SOLE Reference Guide.
Usage
SOLE uses lifecycle-actions to manage its processes and operations. As described in the lifecycle actions doc, there are eight different lifecycle actions, grouped into two groups:
Let's look at the code for each group as found in the DataOps Reference Project:
Snowflake Setup with AGGREGATE
"Set Up Snowflake":
extends:
- .agent_tag
stage: Snowflake Setup
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE
ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
resource_group: $CI_JOB_NAME
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
Cleanup with AGGREGATE-DESTROY
"Tear Down Snowflake":
extends:
- .agent_tag
stage: Clean Up
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE-DESTROY
ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
resource_group: $CI_JOB_NAME
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
rules:
## Block this job from master (PROD) and qa (QA) branches
- if: '$CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD || $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA'
when: never
## For all other branches, enable this job to be run manually
- when: manual
Supported Parameters
Parameter | Required/Default | Description |
---|---|---|
CONFIGURATION_DIR | REQUIRED, defaults to $CI_PROJECT_DIR/dataops/snowflake | The project directory where the Snowflake configuration files are located. |
ARTIFACT_DIRECTORY | Optional, defaults to $CI_PROJECT_DIR/snowflake-provider | The path to where the artifacts such as the resource files, import files, and state files are uploaded. It is essential to specify the ARTIFACT_DIRECTORY as an artifact in the related DataOps job. |
LIFECYCLE_ACTION | REQUIRED | See below for this parameter's valid values. |
LIFECYCLE_MANAGE_OBJECT | Optional | See below for this parameter's valid values. If no value is specified, all object groups are managed. |
DATAOPS_SOLE_ACCOUNT | REQUIRED, from connection parameters | The Snowflake account. If the account is a region-based account or the organization name is not used, the region must be added to this variable. |
DATAOPS_SOLE_USERNAME | REQUIRED , from connection parameters | The Snowflake account username |
DATAOPS_SOLE_PASSWORD | REQUIRED , from connection parameters | The Snowflake account password |
DATAOPS_SOLE_ROLE | REQUIRED , from connection parameters | The Snowflake role that can run queries. |
DATAOPS_SOLE_WAREHOUSE | Optional , from connection parameters | The Snowflake warehouse used for SOLE Grant Management. |
DATAOPS_PREFIX | REQUIRED, from project settings | This prefix is added to all account-level objects and databases (except the default database). |
DATAOPS_DATABASE | REQUIRED, from project settings | The default database name - required if the default database is specified in the configuration as {{env.DATAOPS_DATABASE}} . |
DATAOPS_DATABASE_MASTER | Optional, from project settings | The PROD or main/master database - required if the production database is referenced in the configuration by the variable {{env.DATAOPS_DATABASE_MASTER}} . |
DATAOPS_ENV_NAME | REQUIRED, from project settings | This suffix is added to all account-level objects and databases (except the default database). |
DATAOPS_ENV_NAME_PROD | Optional, defaults to master | Specifies the branch used as the production environment |
DATAOPS_NONDB_ENV_NAME | Optional | The suffix override value - if specified, it overrides all branch-specific suffixes with its value. |
LIFECYCLE_RESOURCE_FILE | Optional, defaults to resources.tf.json | The file name containing the resource definitions. |
LIFECYCLE_IMPORT_FILE | Optional, defaults to resource_imports.json | The file name containing the resource import statements. |
LIFECYCLE_STATE_RESET | Optional | If set, the local state is reset and all defined and managed objects are re-initialized. |
PROVIDER_DIR | Optional, defaults to /snowflake-provider | The provider configuration directory path - it must contain a file named provider.tf with a list of Snowflake providers. |
DISABLE_PERSISTENT_CACHE | Optional | If set, it disables persistent cache usage by the orchestrator. The user must handle the generated file transfers between jobs. |
DONT_USE_SOLE_GRANT_MANAGEMENT | Optional | If set, it disables SOLE Grant Management for managing the privilege/role grants and revokes and uses the Terraform Provider. |
DATAOPS_SOLE_OVERRIDE | Optional | If set, it overrides the object's delete/update prevention feature. |
DATAOPS_SOLE_DEBUG | Optional | If set, it enables the debug logs for SOLE without exposing any credentials when writing to these logs. |
SOLE_LOG_CLEANUP | Optional | If set, it removes all the log files from the host system's cache. |
SOLE_CACHE_CLEANUP | Optional | If set, it removes the host system's cache. |
DATAOPS_PRESERVE_OBJECT_NAME | Optional | If set, it enables special character support. |
SET_TERRAFORM_KEYS_TO_ENV | Optional | If set, it exports SOLE credentials from the DataOps Vault to the pipeline environment. |
CONFIGURATION_DIR
The CONFIGURATION_DIR
parameter specifies your project directory where the Snowflake configuration files are located. Refer to the SOLE Managed Object Reference for supported configurations.
LIFECYCLE_ACTION
The valid values for this variable are as follows:
AGGREGATE
- execute compile, validate, plan, and applyCOMPILE
- only compile the Snowflake configuration found in theCONFIGURATION_DIR
VALIDATE
- only validate the compiled resourcesPLAN
- only plan the validated configurationAPPLY
- only apply the planned configurationAGGREGATE-DESTROY
- execute compile, validate, plan-destroy, and destroyPLAN-DESTROY
- only plan how to tear down the Snowflake configurationDESTROY
- only execute the teardown plan
LIFECYCLE_MANAGE_OBJECT
The valid values for this variable are as follows:
ACCOUNT_LEVEL
- only manage account-level objectsDATABASE
- only manage databasesDATABASE_LEVEL
- only manage database-level objectsGRANT
- only manage grants
If no value is specified for this parameter, all object groups are managed in sequence.
DATAOPS_SOLE_DEBUG
Setting DATAOPS_SOLE_DEBUG
to a value enables SOLE'S debug mode. This generates more log-output related to SOLE'S internal processes and is useful for debugging if a job fails or behaves unexpectedly. No credentials are logged when using DATAOPS_SOLE_DEBUG
. The SOLE credentials are masked by a fixed 16-length character X
.
Example Jobs
As described in the Usage section above, the SOLE Orchestrator facilitates the implementation of the two aggregate lifecycle_actions, AGGREGATE and AGGREGATE-DESTROY. At the risk of duplicating content, providing context to the following example jobs is a good idea. Therefore, let's assume you need to create a new feature branch to develop a new report. Once the code has been written, it must be tested. To test this report, you need to run a DataOps pipeline, including the setup and tear-down Snowflake jobs.
Set Up Snowflake
The default from the DataOps Reference Project for setting up Snowflake uses the AGGREGATE
lifecycle action.
"Set Up Snowflake":
extends:
- .agent_tag
stage: "Snowflake Setup"
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE
ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
resource_group: $CI_JOB_NAME
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
Tear Down Snowflake
The default from the DataOps Reference Project for tearing down Snowflake for a feature branch uses the AGGREGATE-DESTROY
lifecycle action.
"Tear Down Snowflake":
extends:
- .agent_tag
stage: Clean Up
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE-DESTROY
ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
resource_group: $CI_JOB_NAME
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
rules:
## Block this job from master (PROD) and qa (QA) branches
- if: '$CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD || $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA'
when: never
## For all other branches, enable this job to be run manually
- when: manual
Individual Jobs
For further examples on how to use the lifecycle actions and the object hierarchies with the SOLE orchestrator refer to the lifecycle action jobs user guide.
Project Resources
None
Host Dependencies (and Resources)
None