Skip to main content

Lifecycle Action Jobs

A fundamental architectural principle of DataOps pipelines is that they are made up of any number of jobs, including those run by the Snowflake Object Lifecycle Engine (SOLE).

SOLE includes two types of jobs:

Aggregate Lifecycle Jobs

Aggregate jobs comprise a specific selection of individual jobs. They are the DataOps default, which is why they are described before the individual jobs in this text. They are an alternative to executing each action individually, designed to simplify the overall Snowflake object management process, reducing the level of management required by SOLE and providing a straightforward lifecycle management workflow.

The following aggregate jobs are available:

Aggregate Job

The AGGREGATE job handles all the SOLE lifecycle actions related to creating and updating Snowflake-managed objects in a specific sequence.

As highlighted in the next code snippet, the following actions are all executed in a single job as indicated by the AGGREGATE value in the LIFECYCLE_ACTION parameter:

pipelines/includes/local_includes/sole_jobs/sole_aggregate_action.yml
"Aggregate Action":
extends:
- .agent_tag
stage: Snowflake Setup
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

In summary, the AGGREGATE job compiles the user configuration into Terraform-compatible configuration code, validates this code to ensure it is executable, plans how to execute the configuration, and finally applies the plan to the Snowflake objects.

Aggregate-Destroy Job

The AGGREGATE-DESTROY job handles all actions related to the deletion (or destruction) of all Snowflake-managed objects in a specific sequence to ensure that all objects are successfully destroyed. The individual actions in this job include:

As demonstrated in the next code snippet, these actions are all executed in a single job as indicated by the AGGREGATE-DESTROY value in the LIFECYCLE_ACTION parameter. And as shown in the rules section of this YAML code snippet, this aggregate action has a list of rules that apply to when the associated actions can be executed or not.

For instance, the DESTROY action must never run under the following circumstances (as well as other scenarios not elucidated here but described in the YAML file):

pipelines/includes/local_includes/sole_jobs/sole_aggregate_destroy_action.yml
"Aggregated-Destroy Action":
extends:
- .agent_tag
stage: Clean Up
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE-DESTROY
script:
- export LIFECYCLE_OBJECT_SUFFIX=$SNOWFLAKE_SUFFIX
- /dataops
rules:
# If running in production, never allow destroy to be run
- if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD
when: never
# If running in qa, never allow destroy to be run
- if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA
when: never
# For other runs, this step is manual
- when: manual
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

In summary, the AGGREGATE-DESTROY job plans how to delete all the Snowflake objects and then applies the plan to the Snowflake environment.

Lifecycle Job Sequence

Before we look at each individual job in detail, it is imperative to structure the order of the jobs in a specific sequence, as not only is the output from one job required by its subsequent jobs, but the runtime sequence of the lifecycle actions must occur in a particular order for the SOLE jobs to execute successfully.

Why?

The best way to answer this question is to consider the following scenario:

Let's assume you have a Snowflake organization, including an account, database, schema, views, and tables, that is fully managed by SOLE. These objects are divided up into four object groups with account-level objects at the top of the hierarchy and grants at the bottom of the hierarchy as follows:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

This object group hierarchy imitates the order in which SOLE must create (and manage) all Snowflake objects belonging to an organization. For instance, it cannot create database objects without first creating the account-level objects. In the same way, it cannot create database-level objects without first creating database objects. The same rule applies to grant objects.

In the same way, SOLE must destroy (or delete) these Snowflake objects in reverse order, the grants first and the account-level objects last.

Before we dive into the details of each job, let's consider this runtime sequence for the object group hierarchy's lifecycle actions. For creating and updating the sequence is:

  • Compile SOLE configurations
  • Validate all object group hierarchies
  • Plan account-level and database objects based on the YAML configuration
  • Apply account-level and database object configurations
  • Plan database-level objects based on the YAML configuration
  • Apply database-level configurations
  • Plan grants based on the YAML configuration
  • Apply grants

For destroying the objects the sequence is:

  • Destroy-plan grants
  • Destroy grants
  • Destroy-plan database-level objects
  • Destroy database-level objects
  • Destroy-plan database and account-level objects
  • Destroy database and account-level objects

Lifecycle Job Stages

Stages for individual jobs are as crucial as the Lifecycle Action Job Sequence as stages define the order by which jobs get executed. Here is a reference guide to all the SOLE stages to help with setting up each job correctly:

stages:
- Compile Configuration
- Validate Configurations
- Plan Account-Level Objects
- Apply Account-Level Objects
- Plan Database-Level Objects
- Apply Database-Level Objects
- Plan Objects Grants
- Apply Objects Grants
- Clean Up Plan Grants
- Clean Up Grants
- Clean Up Plan Database-Level
- Clean Up Database-Level
- Clean Up Plan Account-Level
- Clean Up Account-Level

Individual Lifecycle Jobs

Individual jobs perform a single lifecycle action per job in contrast to aggregate jobs that perform multiple actions in a single job. In other words, in an individual job, there is a job definition with supported variables.

Now that we understand the two concepts of the lifecycle action job sequence and all the stages needed to implement the Snowflake object management lifecycle, let's look at each lifecycle action job in order of its execution. For the create and update jobs:

And for the destroy jobs:

Compile Job

The COMPILE job compiles user configurations to generate Terraform-supported configurations with complete namespaces, dependencies, and reference resolutions.

As shown in the next code snippet, this action is executed in a single job as indicated by the COMPILE value in the LIFECYCLE_ACTION parameter:

pipelines/includes/local_includes/sole_jobs/sole_compile_action.yml
"Compile Configuration Action":
extends:
- .agent_tag
stage: Compile Configuration Action
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: COMPILE
script:
- export LIFECYCLE_OBJECT_SUFFIX=$SNOWFLAKE_SUFFIX
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
- $CI_PROJECT_DIR/dataops/snowflake
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

Validate Job

The VALIDATE job validates the Terraform-supported configurations generated in the COMPILE job for each object group.

For example, the following code snippet runs a VALIDATE job that validates all Snowflake account-level objects as indicated by the parameter values in LIFECYCLE_ACTION being VALIDATE and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_validate_account_level_objects.yml
"Validate Account-Level Objects":
extends:
- .agent_tag
stage: Validate Account-Level Object Configurations
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: VALIDATE
LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

The value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

A VALIDATE job (or stage in the same job) must be created for each object group to validate the overarching configuration generated by the COMPILE job. The object groups must be run in this order. Otherwise, SOLE will not correctly validate the entire Snowflake-managed object configuration.

Plan Job

The PLAN job imports existing, Snowflake managed objects to a local state file and generates a plan for the APPLY job based on the Terraform-compatible configuration and what is currently in the Snowflake ecosystem. The order in which the PLAN job executes is vital and is described in the Lifecycle Action Job Sequence section.

The following code snippet is an example of a PLAN job that plans the create and update processes of all Snowflake account-level objects as indicated by the parameter values in the LIFECYCLE_ACTION being PLAN and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_plan_account_level_objects.yml
"Plan Account-Level Objects":
extends:
- .agent_tag
stage: Plan Account-Level Objects
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: PLAN
LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the VALIDATE job, the value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

A PLAN job (or stage in the same job) must be created for each object group to plan the APPLY job. The object groups must be run in the specified order. Otherwise, SOLE will not correctly plan SOLE's APPLY job.

Apply Job

The APPLY job executes the plans created by the PLAN job. The order in which the APPLY job executes the SOLE plan is vital and is described in the Lifecycle Action Job Sequence section.

The following code snippet is an example of an APPLY job that executes creates and updates all Snowflake account-level objects as indicated by the parameter values in the LIFECYCLE_ACTION being APPLY and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_apply_account_level_objects.yml
"Apply Account-Level Objects":
extends:
- .agent_tag
stage: Apply Account-Level Objects
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: APPLY
LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the PLAN job, the value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

An APPLY job (or stage in the same job) must be created for each object group to apply the plan created in the PLAN job. The object groups must be run in the specified order. Otherwise, SOLE will not correctly execute the APPLY action for the Snowflake managed objects.

Plan-Destroy Job

The PLAN-DESTROY job imports existing, Snowflake-managed objects to a local state file and generates a plan for the DESTROY job. The order in which the PLAN-DESTROY job plans the destruction of the Snowflake objects is vital and is described in the Lifecycle Action Job Sequence section.

This code snippet runs a PLAN-DESTROY job that plans the deletion all Snowflake account-level objects as indicated by the parameter values in the LIFECYCLE_ACTION being PLAN-DESTROY and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_plan_destroy_account_level_objects.yml
"Plan-Destroy Account-Level Objects":
extends:
- .agent_tag
stage: Clean Up Plan Account-Level Objects
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: PLAN-DESTROY
LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
script:
- /dataops
rules:
# If running in production branch, never allow destroy to be run
- if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD
when: never
# If running in qa, never allow the destroy to be run
- if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA
when: never
# For other runs, this step is manual
- when: manual
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the APPLY job, the value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

A PLAN-DESTROY job (or stage in the same job) must be created for each object group to plan the DESTROY job. The object groups must be run in the specified order. Otherwise, SOLE will not correctly execute the PLAN-DESTROY action for the Snowflake managed objects.

Destroy Job

The DESTROY job destroys or deletes all Snowflake managed objects specified in the plan created by the PLAN-DESTROY. The order in which the DESTROY job executes the plan is vital and is described in the Lifecycle Action Job Sequence section.

This code snippet runs a DESTROY job that plans the deletion of all Snowflake account-level objects as indicated by the values in the LIFECYCLE_ACTION (DESTROY) and LIFECYCLE_MANAGE_OBJECT(ACCOUNT_LEVEL) parameters. And as shown in the rules section of this YAML code snippet, this action has a list of rules that apply to when the DESTROY action can be executed or not.

pipelines/includes/local_includes/sole_jobs/sole_destroy_account_level_objects.yml
"Destroy Account-Level Objects":
extends:
- .agent_tag
stage: Clean Up Account-Level Objects
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: DESTROY
LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
script:
- /dataops
rules:
# If running in production branch, never allow destroy to be run
- if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD
when: never
# If running in qa, never allow the destroy to be run
- if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA
when: never
# For other runs, this step is manual
- when: manual
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
allow_failure: false
icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the APPLY job, the value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

A DESTROY job must be created for each object group to execute the plan. The object groups must be run in the specified order. Otherwise, SOLE will not correctly execute the DESTROY action for the Snowflake managed objects.