Lifecycle Action Jobs

A fundamental architectural principle of DataOps pipelines is that they are made up of any number of jobs, including those run by the Snowflake Object Lifecycle Engine (SOLE).

SOLE includes a few job types:

• Aggregate lifecycle jobs
• Plan and apply jobs
• Individual lifecycle jobs

It is essential to keep in mind that you must order jobs in a specific sequence, respecting job stages.

Aggregate lifecycle jobs

Aggregate jobs comprise a specific selection of individual jobs. They are the DataOps default, which is why they are described before the individual jobs in this text. They are an alternative to executing each action individually, designed to simplify the overall Snowflake object management process, reducing the level of management required by SOLE and providing a straightforward lifecycle management workflow.

The following aggregate jobs are available:

• Aggregate Job (AGGREGATE)
• Aggregate-Destroy Job (AGGREGATE-DESTROY)

Aggregate job

The AGGREGATE job handles all the SOLE lifecycle actions related to creating and updating Snowflake-managed objects in a specific sequence.

As highlighted in the next code snippet, the following actions are all executed in a single job as indicated by the AGGREGATE value in the LIFECYCLE_ACTION parameter:

• COMPILE
• VALIDATE
• PLAN
• APPLY

pipelines/includes/local_includes/sole_jobs/sole_aggregate_action.yml

"Aggregate Action":
  extends:
    - .agent_tag
  stage: Snowflake Setup
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

In the above example, the AGGREGATE lifecycle action in the Aggregate Action job compiles user configuration into Terraform-compatible configuration code, validates this code to ensure it is executable, plans how to execute the configuration, and finally applies the plan to the Snowflake objects.

Aggregate-destroy job

The AGGREGATE-DESTROY job handles all actions related to the deletion (or destruction) of all Snowflake-managed objects in a specific sequence to ensure that all objects are successfully destroyed. The individual actions in this job include:

• PLAN-DESTROY
• DESTROY

As demonstrated in the following code snippet, these actions are all executed in a single job as indicated by the AGGREGATE-DESTROY value in the LIFECYCLE_ACTION parameter. And as shown in the rules section of this YAML code snippet, this aggregate action has a list of rules that apply to when the associated actions can be executed or not.

For instance, the DESTROY action must never run under the following circumstances (as well as other scenarios not elucidated here but described in the YAML file):

• If the running pipeline is a Real-Time/Event Pipeline and is merging code to the main branch
• If the pipeline is running in the production (main) environment

pipelines/includes/local_includes/sole_jobs/sole_aggregate_destroy_action.yml

"Aggregated-Destroy Action":
  extends:
    - .agent_tag
  stage: Clean Up
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE-DESTROY
  script:
    - export LIFECYCLE_OBJECT_SUFFIX=$SNOWFLAKE_SUFFIX
    - /dataops
  rules:
    # If running in production, never allow destroy to be run
    - if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD
      when: never
      # If running in qa, never allow destroy to be run
    - if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA
      when: never
      # For other runs, this step is manual
    - when: manual
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

In the above example, the AGGREGATE-DESTROY action job in the Aggregated-destroy Action job plans how to delete all the Snowflake objects and then applies the plan to the Snowflake environment.

Plan and apply lifecycle jobs

The plan and apply jobs are coupled in a process that allows you to review and approve any modifications you want to make to a Snowflake database before the changes are applied.

This process gives you more control over your snowflake objects and ensures you can review and approve any changes before applying them.

The following jobs are available:

• Plan Snowflake setup job (PLAN-ALL)
• Apply plan to Snowflake job (APPLY-ALL)

warning

PLAN-ALL and APPLY-ALL are coupled and apply only to non-cloned Snowflake databases.

Prerequisites

The plan and apply jobs are available in the DataOps template project and will be automatically initiated if you set the SOLE variable DATAOPS_SOLE_ENABLE_PLAN_APPROVAL to any value or to true.

Plan Snowflake setup Job

The PLAN-ALL action job handles all the SOLE lifecycle actions related to creating and updating Snowflake-managed objects in a specific sequence.

As highlighted in the next code snippet, the following actions are all executed in a single job as indicated by the PLAN-ALL value in the LIFECYCLE_ACTION parameter:

• COMPILE
• VALIDATE
• PLAN

pipelines/includes/local_includes/sole_jobs/plan_snowflake_setup.yml

"Plan Snowflake Setup":
  extends:
    - .agent_tag
  stage: Snowflake Setup
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: PLAN-ALL
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
      - dataops/report/plan_viewer/
    name: "SOLE Plan Viewer"
    expose_as: "dataopsreport"
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

In the above example, the Plan Snowflake Setup job:

• Compiles your configuration into Terraform-compatible configuration code
• Validates this code to ensure it is executable
• Plans how to execute the configuration on the Snowflake objects
• Generates a human-readable preview of the plan

Apply plan to Snowflake job

The APPLY-ALL action job executes the plans created by the PLAN-ALL action job. The order in which the job executes the SOLE plan is vital and is described in the Lifecycle job sequence section.

The following code snippet is an example of an APPLY-ALL lifecycle action that executes, creates, and updates all Snowflake objects as indicated by the parameter values in the LIFECYCLE_ACTION being APPLY-ALL.

pipelines/includes/local_includes/sole_jobs/apply_plan_to_snowflake.yml

"Apply Plan to Snowflake":
  extends:
    - .agent_tag
  stage: Snowflake Setup
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: APPLY-ALL
    ARTIFACT_DIRECTORY: $CI_PROJECT_DIR/snowflake-artifacts
    CONFIGURATION_DIR: $CI_PROJECT_DIR/dataops/snowflake
  needs:
    - job: Initialise Reference Project
    - job: Plan Snowflake Setup
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
      - dataops/report/plan_viewer/
    name: "SOLE Plan Viewer"
    expose_as: "dataopsreport"
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  when: manual

In summary, the Apply Plan to Snowflake job applies the plan to the Snowflake environment.

Individual lifecycle jobs

Individual action jobs perform a single lifecycle action per job in contrast to aggregate jobs that perform multiple actions in a single job. In other words, an individual job has a job definition with supported variables.

Now that we understand the two concepts of the lifecycle action job sequence and all the stages needed to implement the Snowflake object management lifecycle, let's look at each lifecycle action job in order of its execution. For the create and update jobs:

• Compile Job
• Validate Job
• Plan Job
• Apply Job

And for the destroy jobs:

• Plan-Destroy Job
• Destroy Job

Compile job

The COMPILE action job compiles user configurations to generate Terraform-supported configurations with complete namespaces, dependencies, and reference resolutions.

As shown in the next code snippet, this action is executed in a single job as indicated by the COMPILE value in the LIFECYCLE_ACTION parameter:

pipelines/includes/local_includes/sole_jobs/sole_compile_action.yml

"Compile Configuration Action":
  extends:
    - .agent_tag
   stage: Compile Configuration Action
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: COMPILE
  script:
    - export LIFECYCLE_OBJECT_SUFFIX=$SNOWFLAKE_SUFFIX
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
      - $CI_PROJECT_DIR/dataops/snowflake
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

Validate job

The VALIDATE action job validates the Terraform-supported configurations generated in the COMPILE action job for each object group.

For example, the following code snippet runs a VALIDATE job that validates all Snowflake account-level objects as indicated by the parameter values in LIFECYCLE_ACTION being VALIDATE and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_validate_account_level_objects.yml

"Validate Account-Level Objects":
  extends:
    - .agent_tag
  stage: Validate Account-Level Object Configurations
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: VALIDATE
    LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

The value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

You must create a VALIDATE action job (or stage in the same job) for each object group to validate the overarching configuration generated by the COMPILE action job. The object groups must run in this order. Otherwise, SOLE will not correctly validate the entire Snowflake-managed object configuration.

Plan job

The PLAN action job imports existing Snowflake-managed objects to a local state file and generates a plan for the APPLY action job based on the Terraform-compatible configuration and what is currently in the Snowflake ecosystem. The order in which the PLAN action job executes is vital and is described in the Lifecycle Action Job Sequence section.

The following code snippet is an example of a PLAN action job that plans the create and update processes of all Snowflake account-level objects as indicated by the parameter values in the LIFECYCLE_ACTION being PLAN and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_plan_account_level_objects.yml

"Plan Account-Level Objects":
  extends:
    - .agent_tag
  stage: Plan Account-Level Objects
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: PLAN
    LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the VALIDATE action job, the value of the LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

You must create a PLAN action job (or stage in the same job) for each object group to plan the APPLY action job. The object groups must run in the specified order. Otherwise, SOLE will not correctly plan SOLE's APPLY action job.

Apply job

The APPLY action job executes the plans created by the PLAN action job. The order in which the APPLY action job executes the SOLE plan is vital and is described in the Lifecycle Action Job Sequence section.

The following code snippet is an example of an APPLY action job that executes, creates, and updates all Snowflake account-level objects as indicated by the parameter values in the LIFECYCLE_ACTION being APPLY and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_apply_account_level_objects.yml

"Apply Account-Level Objects":
  extends:
    - .agent_tag
  stage: Apply Account-Level Objects
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: APPLY
    LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
  script:
    - /dataops
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the PLAN action job, the value of the LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

You must create an APPLY action job (or stage in the same job) for each object group to apply the plan created in the PLAN action job. The object groups must run in the specified order. Otherwise, SOLE will not correctly execute the APPLY action for the Snowflake-managed objects.

Plan-destroy job

The PLAN-DESTROY action job imports existing Snowflake-managed objects to a local state file and generates a plan for the DESTROY action job. The order in which the PLAN-DESTROY action plans the destruction of the Snowflake objects is vital and is described in the Lifecycle Action Job Sequence section.

This code snippet runs a PLAN-DESTROY action job that plans the deletion of all Snowflake account-level objects as indicated by the parameter values in the LIFECYCLE_ACTION being PLAN-DESTROY and LIFECYCLE_MANAGE_OBJECT being ACCOUNT_LEVEL.

pipelines/includes/local_includes/sole_jobs/sole_plan_destroy_account_level_objects.yml

"Plan-Destroy Account-Level Objects":
  extends:
    - .agent_tag
  stage: Clean Up Plan Account-Level Objects
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: PLAN-DESTROY
    LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
  script:
    - /dataops
  rules:
    # If running in production branch, never allow destroy to be run
    - if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD
      when: never
      # If running in qa, never allow the destroy to be run
    - if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA
      when: never
      # For other runs, this step is manual
    - when: manual
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the APPLY action job, the value of LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

You must create a PLAN-DESTROY action job (or stage in the same job) for each object group to plan the DESTROY job. The object groups must run in the specified order. Otherwise, SOLE will not correctly execute the PLAN-DESTROY action for the Snowflake-managed objects.

Destroy job

The DESTROY job destroys or deletes all Snowflake-managed objects specified in the plan created by the PLAN-DESTROY. The order in which the DESTROY job executes the plan is vital and is described in the Lifecycle Action Job Sequence section.

This code snippet runs a DESTROY action job that plans the deletion of all Snowflake account-level objects as indicated by the values in the LIFECYCLE_ACTION (DESTROY) and LIFECYCLE_MANAGE_OBJECT(ACCOUNT_LEVEL) parameters. As shown in the rules section of this YAML code snippet, this action has a list of rules that apply to when the DESTROY action is executed or not.

pipelines/includes/local_includes/sole_jobs/sole_destroy_account_level_objects.yml

"Destroy Account-Level Objects":
  extends:
    - .agent_tag
  stage: Clean Up Account-Level Objects
  image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: DESTROY
    LIFECYCLE_MANAGE_OBJECT: ACCOUNT_LEVEL
  script:
    - /dataops
  rules:
      # If running in production branch, never allow destroy to be run
    - if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_PROD
      when: never
      # If running in qa, never allow the destroy to be run
    - if: $CI_COMMIT_REF_NAME == $DATAOPS_BRANCH_NAME_QA
      when: never
      # For other runs, this step is manual
    - when: manual
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  allow_failure: false
 icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}

As with the APPLY action job, the value of the LIFECYCLE_MANAGE_OBJECT parameter must be one of the following object groups:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

To execute the plan, you must create a DESTROY action job for each object group. The object groups must run in the specified order. Otherwise, SOLE will not correctly run the DESTROY action for the Snowflake-managed objects.

Lifecycle job sequence

It is imperative to order the action jobs in a specific sequence. This is because not only the output from one job is required by its subsequent jobs, but the runtime sequence of the lifecycle actions must occur in a particular order for the SOLE jobs to execute successfully.

Why?

The best way to answer this question is to consider the following scenario:

Let's assume you have a Snowflake organization, including an account, database, schema, views, and tables fully managed by SOLE. These objects are divided up into four object groups with account-level objects at the top of the hierarchy and grants at the bottom of the hierarchy as follows:

• ACCOUNT_LEVEL
• DATABASE
• DATABASE_LEVEL
• GRANT

This object group hierarchy imitates the order in which SOLE must create and manage all Snowflake objects belonging to an organization. For instance, it cannot create database objects without first creating account-level objects. In the same way, it cannot create database-level objects without first creating database objects. The same rule applies to grant objects.

In the same way, SOLE must destroy (or delete) these Snowflake objects in reverse order, the grants first and the account-level objects last.

Before diving into each job's details, consider this runtime sequence for the object group hierarchy's lifecycle actions. To create and update the objects, the sequence is:

1. Compile SOLE configurations
2. Validate all object group hierarchies
3. Plan account-level and database objects based on the YAML configuration
4. Apply account-level and database object configurations
5. Plan database-level objects based on the YAML configuration
6. Apply database-level configurations
7. Plan grants based on the YAML configuration
8. Apply grants

To destroy the objects, the sequence is:

1. Destroy-plan grants
2. Destroy grants
3. Destroy-plan database-level objects
4. Destroy database-level objects
5. Destroy-plan database and account-level objects
6. Destroy database and account-level objects
7. Destroy database and account-level objects

Lifecycle job stages

Stages for individual action jobs are as crucial as the lifecycle action job sequence, as stages define the order by which jobs get executed. Here is a reference guide to all the SOLE stages to help with setting up each job correctly:

stages:
  - Compile Configuration
  - Validate Configurations
  - Plan Account-Level Objects
  - Apply Account-Level Objects
  - Plan Database-Level Objects
  - Apply Database-Level Objects
  - Plan Objects Grants
  - Apply Objects Grants
  - Clean Up Plan Grants
  - Clean Up Grants
  - Clean Up Plan Database-Level
  - Clean Up Database-Level
  - Clean Up Plan Account-Level
  - Clean Up Account-Level