Skip to main content

Snowflake Object Lifecycle Orchestrator

TypePre-Set
Image$DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE

This Snowflake Object Lifecycle Engine orchestrator is a pre-set orchestrator responsible for processing the Snowflake configuration in the /dataops/snowflake project directory using the Snowflake Object Lifecycle Engine.

Refer to the SOLE User Guide and SOLE Reference Guide for further details.

Usage

This YAML configuration file snippet demonstrates a typical use case for the SOLE Orchestrator.

pipelines/includes/local_includes/sole_jobs/my_sole_job.yml
"My SOLE Job":
extends:
- .agent_tag
stage: "Snowflake Setup"
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
script:
- /dataops
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

Supported Parameters

ParameterRequired/DefaultDescription
CONFIGURATION_DIRREQUIRED, defaults to $CI_PROJECT_DIR/dataops/snowflakeThe project directory where the Snowflake configuration files are located.
ARTIFACT_DIRECTORYOptional, defaults to $CI_PROJECT_DIR/snowflake-providerPath to where the artifacts such as the resource files, import files, and state files are uploaded. The directory must also be set to upload as an artifact in the DataOps job
LIFECYCLE_ACTIONREQUIREDSee below for this parameter's valid values
LIFECYCLE_MANAGE_OBJECTOptionalSee below for this parameter's valid values. If no value is specified all object groups are managed
DATAOPS_SOLE_ACCOUNTREQUIRED, from connection parametersSnowflake account (If the account is not region-less or the organization name is not used, the region must be added to this variable)
DATAOPS_SOLE_USERNAMEREQUIRED , from connection parametersSnowflake account username
DATAOPS_SOLE_PASSWORDREQUIRED , from connection parametersSnowflake account password
DATAOPS_SOLE_ROLEREQUIRED , from connection parametersSnowflake role that can run queries
DATAOPS_SOLE_WAREHOUSEOptional , from connection parametersSnowflake warehouse used for SOLE Grant Management
DATAOPS_PREFIXREQUIRED, from project settingsThis prefix is added to all account-level objects and databases (except the default database)
DATAOPS_DATABASEREQUIRED, from project settingsThe default database name. It is required if default database is specified in the configuration in the format {{env.DATAOPS_DATABASE}}
DATAOPS_DATABASE_MASTEROptional, from project settingsThe PROD or main/master database. It is required if the production database is referenced in the configuration by the variable {{env.DATAOPS_DATABASE_MASTER}}
DATAOPS_ENV_NAMEREQUIRED, from project settingsThis suffix is added to all account-level objects and databases (except the default database)
DATAOPS_NONDB_ENV_NAMEOptionalThe suffix override value. If specified, it overrides all branch-specific suffixes with its value
LIFECYCLE_RESOURCE_FILEOptional, defaults to resources.tf.jsonThe file name that contains the resource definitions
LIFECYCLE_IMPORT_FILEOptional, defaults to resource_imports.jsonThe file name that contains the resource import statements
LIFECYCLE_STATE_RESETOptionalIf set, the local state is reset and all defined and managed objects are re-initialized
PROVIDER_DIROptional, defaults to /snowflake-providerThe provider configuration directory path. It must contain a file named provider.tf, containing the list of Snowflake providers
DISABLE_PERSISTENT_CACHEOptionalIf set, it disables persistent cache usage by the orchestrator. Consequently, the generated files' transfer between jobs must be handled by the user
DONT_USE_SOLE_GRANT_MANAGEMENTOptionalIf set, it disables SOLE Grant Management for managing the privilege/role grant and revokes and uses the Terraform Provider
DATAOPS_SOLE_OVERRIDEOptionalIf set, it overrides the object delete/update prevention feature
DATAOPS_SOLE_DEBUGOptionalIf set, it enables the debug logs for SOLE without exposing any credentials when writing to the logs
SOLE_LOG_CLEANUPOptionalIf set, it removes all the log files from the host system's cache
SOLE_CACHE_CLEANUPOptionalIf set, it removes the host system's cache
DATAOPS_PRESERVE_OBJECT_NAMEOptionalIf set, it enables special character support
SET_TERRAFORM_KEYS_TO_ENVOptionalIf set, it exports SOLE credentials from the DataOps Vault to the pipeline environment
DATAOPS_SOLE_PROD_BRANCHOptional, defaults to master, deprecatedSpecifies the branch used as the production environment

CONFIGURATION_DIR

The parameter CONFIGURATION_DIR specifies your project directory where the Snowflake configuration files are located. For the supported configurations refer to the SOLE Managed Object Reference.

LIFECYCLE_ACTION

The valid values for this variable are as follows:

  • AGGREGATE - execute compile, validate, plan, and apply
  • COMPILE - only compile the Snowflake configuration found at CONFIGURATION_DIR
  • VALIDATE - only validate the compiled resources
  • PLAN - only plan the validated configuration
  • APPLY - only apply the planned configuration
  • AGGREGATE-DESTROY - execute compile, validate, plan-destroy, and destroy
  • PLAN-DESTROY - only plan how to tear down the Snowflake configuration
  • DESTROY - only execute the teardown plan

LIFECYCLE_MANAGE_OBJECT

The valid values for this variable are as follows:

If no value is specified for this parameter all object groups are managed in sequence.

DATAOPS_SOLE_DEBUG

Setting DATAOPS_SOLE_DEBUG to a value enables debug mode for SOLE. This generates more log output related to its internal process and is useful for debugging if the job fails or behaves unexpectedly. No credentials are logged when using DATAOPS_SOLE_DEBUG. The SOLE credentials are masked by a fixed 16-length character X.

Example Jobs

Set up Snowflake

pipelines/includes/local_includes/sole_jobs/set_up_snowflake.yml
"Set Up Snowflake":
extends:
- .agent_tag
stage: "Snowflake Setup"
image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE
ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
resource_group: $CI_JOB_NAME
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

Project Resources

None

Host Dependencies (and Resources)

None