Snowflake Object Lifecycle Orchestrator
Type | Pre-Set |
---|---|
Image | $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE |
This Snowflake Object Lifecycle Engine orchestrator is a pre-set orchestrator responsible for processing the Snowflake configuration in the /dataops/snowflake
project directory using the Snowflake Object Lifecycle Engine.
Refer to the SOLE User Guide and SOLE Reference Guide for further details.
Usage
This YAML configuration file snippet demonstrates a typical use case for the SOLE Orchestrator.
"My SOLE Job":
extends:
- .agent_tag
stage: "Snowflake Setup"
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
script:
- /dataops
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
Supported Parameters
Parameter | Required/Default | Description |
---|---|---|
CONFIGURATION_DIR | REQUIRED, defaults to $CI_PROJECT_DIR/dataops/snowflake | The project directory where the Snowflake configuration files are located. |
ARTIFACT_DIRECTORY | Optional, defaults to $CI_PROJECT_DIR/snowflake-provider | Path to where the artifacts such as the resource files, import files, and state files are uploaded. The directory must also be set to upload as an artifact in the DataOps job |
LIFECYCLE_ACTION | REQUIRED | See below for this parameter's valid values |
LIFECYCLE_MANAGE_OBJECT | Optional | See below for this parameter's valid values. If no value is specified all object groups are managed |
DATAOPS_SOLE_ACCOUNT | REQUIRED, from connection parameters | Snowflake account (If the account is not region-less or the organization name is not used, the region must be added to this variable) |
DATAOPS_SOLE_USERNAME | REQUIRED , from connection parameters | Snowflake account username |
DATAOPS_SOLE_PASSWORD | REQUIRED , from connection parameters | Snowflake account password |
DATAOPS_SOLE_ROLE | REQUIRED , from connection parameters | Snowflake role that can run queries |
DATAOPS_SOLE_WAREHOUSE | Optional , from connection parameters | Snowflake warehouse used for SOLE Grant Management |
DATAOPS_PREFIX | REQUIRED, from project settings | This prefix is added to all account-level objects and databases (except the default database) |
DATAOPS_DATABASE | REQUIRED, from project settings | The default database name. It is required if default database is specified in the configuration in the format {{env.DATAOPS_DATABASE}} |
DATAOPS_DATABASE_MASTER | Optional, from project settings | The PROD or main/master database. It is required if the production database is referenced in the configuration by the variable {{env.DATAOPS_DATABASE_MASTER}} |
DATAOPS_ENV_NAME | REQUIRED, from project settings | This suffix is added to all account-level objects and databases (except the default database) |
DATAOPS_NONDB_ENV_NAME | Optional | The suffix override value. If specified, it overrides all branch-specific suffixes with its value |
LIFECYCLE_RESOURCE_FILE | Optional, defaults to resources.tf.json | The file name that contains the resource definitions |
LIFECYCLE_IMPORT_FILE | Optional, defaults to resource_imports.json | The file name that contains the resource import statements |
LIFECYCLE_STATE_RESET | Optional | If set, the local state is reset and all defined and managed objects are re-initialized |
PROVIDER_DIR | Optional, defaults to /snowflake-provider | The provider configuration directory path. It must contain a file named provider.tf , containing the list of Snowflake providers |
DISABLE_PERSISTENT_CACHE | Optional | If set, it disables persistent cache usage by the orchestrator. Consequently, the generated files' transfer between jobs must be handled by the user |
DONT_USE_SOLE_GRANT_MANAGEMENT | Optional | If set, it disables SOLE Grant Management for managing the privilege/role grant and revokes and uses the Terraform Provider |
DATAOPS_SOLE_OVERRIDE | Optional | If set, it overrides the object delete/update prevention feature |
DATAOPS_SOLE_DEBUG | Optional | If set, it enables the debug logs for SOLE without exposing any credentials when writing to the logs |
SOLE_LOG_CLEANUP | Optional | If set, it removes all the log files from the host system's cache |
SOLE_CACHE_CLEANUP | Optional | If set, it removes the host system's cache |
DATAOPS_PRESERVE_OBJECT_NAME | Optional | If set, it enables special character support |
SET_TERRAFORM_KEYS_TO_ENV | Optional | If set, it exports SOLE credentials from the DataOps Vault to the pipeline environment |
DATAOPS_SOLE_PROD_BRANCH | Optional, defaults to master , deprecated | Specifies the branch used as the production environment |
CONFIGURATION_DIR
The parameter CONFIGURATION_DIR
specifies your project directory where the Snowflake configuration files are located. For the supported configurations refer to the SOLE Managed Object Reference.
LIFECYCLE_ACTION
The valid values for this variable are as follows:
AGGREGATE
- execute compile, validate, plan, and applyCOMPILE
- only compile the Snowflake configuration found atCONFIGURATION_DIR
VALIDATE
- only validate the compiled resourcesPLAN
- only plan the validated configurationAPPLY
- only apply the planned configurationAGGREGATE-DESTROY
- execute compile, validate, plan-destroy, and destroyPLAN-DESTROY
- only plan how to tear down the Snowflake configurationDESTROY
- only execute the teardown plan
LIFECYCLE_MANAGE_OBJECT
The valid values for this variable are as follows:
ACCOUNT_LEVEL
- only manage account-level objectsDATABASE
- only manage databasesDATABASE_LEVEL
- only manage database-level objectsGRANT
- only manage grants
If no value is specified for this parameter all object groups are managed in sequence.
DATAOPS_SOLE_DEBUG
Setting DATAOPS_SOLE_DEBUG
to a value enables debug mode for SOLE. This generates more log output related to its internal process and is useful for debugging if the job fails or behaves unexpectedly. No credentials are logged when using DATAOPS_SOLE_DEBUG
. The SOLE credentials are masked by a fixed 16-length character X
.
Example Jobs
Set up Snowflake
"Set Up Snowflake":
extends:
- .agent_tag
stage: "Snowflake Setup"
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE
ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
resource_group: $CI_JOB_NAME
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
Project Resources
None
Host Dependencies (and Resources)
None