Skip to main content

Quick Start

Jobs Setup

Below are example jobs definitions to perform different types of operations with SOLE.

Individual Jobs

This set of jobs performs a single action in each job.

For each action, there is a job definition with supported variables. The order of the jobs is important as output from one is required by subsequent jobs.

Compile Job

This job compiles the configuration and generates terraform supported configuration with complete namespace, dependency and reference resolution.

Compile Configuration:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: COMPILE
stage: "Compile Configuration"
script:
- export LIFECYCLE_OBJECT_SUFFIX=$SNOWFLAKE_SUFFIX
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
- $CI_PROJECT_DIR/dataops/snowflake

Validate Job

This set of jobs validates generated configurations for each object group.

Validate <Object-Group>:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: VALIDATE
LIFECYCLE_MANAGE_OBJECT: <OBJECT_GROUP>
stage: "Validate Configurations"
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY

The value of OBJECT_GROUP must be one of the following:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

A Validate job for each object group should be set up to validate all the generated configurations.

Plan Jobs

This set of jobs imports existing, non-managed objects to the local state and generate a plan for Apply Jobs action.

See the Jobs Sequence for the order in which the Plan Jobs must be setup

Plan <Object-Group>:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: PLAN
LIFECYCLE_MANAGE_OBJECT: <OBJECT_GROUP>
stage: "Plan <Object-Group> Objects"
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY

The value of OBJECT_GROUP must be one of the following:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

Apply Jobs

This set of jobs executes the plan generated by the Plan Jobs.

See the Jobs Sequence for the order in which the Apply Jobs must be setup

Apply <Object-Group>:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: APPLY
LIFECYCLE_MANAGE_OBJECT: <OBJECT_GROUP>
stage: "Apply <Object-Group> Objects"
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY

The value of OBJECT_GROUP must be one of the following:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

Destroy-Plan Jobs

This set of jobs logs the objects that would be destroyed by SOLE in the Destroy Jobs.

See the Jobs Sequence for the order in which the Destroy-Plan Jobs must be set up.

Plan-Destroy <Object-Group>:
extends:
- .not_running_on_master_or_qa
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: PLAN-DESTROY
LIFECYCLE_MANAGE_OBJECT: <OBJECT_GROUP>
stage: "Clean Up Plan <Object-Group>"
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY

The value of OBJECT_GROUP must be one of the following:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

Destroy Jobs

This set of jobs destroy all managed objects for the specified object group as per logged output in Destroy-Plan Jobs.

See the Jobs Sequence for the order in which the Destroy Jobs must be setup

Destroy <Object-Group>:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: DESTROY
LIFECYCLE_MANAGE_OBJECT: <OBJECT_GROUP>
stage: "Clean Up <Object-Group>"
script:
- /dataops
rules:
# If merging to master, never allow the destroy to be run
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
when: never
# If running in master, never allow the destroy to be run
- if: '$CI_COMMIT_REF_NAME == "master"'
when: never
# If merging to qa, never allow the destroy to be run
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "qa"'
when: never
# If running in qa, never allow the destroy to be run
- if: '$CI_COMMIT_REF_NAME == "qa"'
when: never
# For other runs, this step is manual
- when: manual
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY
allow_failure: false

The value of OBJECT_GROUP must be one of the following:

  • ACCOUNT_LEVEL
  • DATABASE
  • DATABASE_LEVEL
  • GRANT

Jobs Sequence

The sequence of the jobs should be in the following order for a successful execution

  1. Compile
  2. Validate All Object Groups
  3. Plan Account-Level and Database
  4. Apply Account-Level and Database
  5. Plan Database-Level
  6. Apply Database-Level
  7. Plan Grants
  8. Apply Grants
  9. Destroy-Plan Database-Level
  10. Destroy Database-Level
  11. Destroy-Plan Account Level and Database
  12. Destroy Account Level and Database

Stages

The following stage setup can be referred to for quick setup

stages:
- "Compile Configuration"
- "Validate Configurations"
- "Plan Account-Level Objects"
- "Apply Account-Level Objects"
- "Plan Database-Level Objects"
- "Apply Database-Level Objects"
- "Plan Objects Grants"
- "Apply Objects Grants"
- "Clean Up Plan Database-Level"
- "Clean Up Database-Level"
- "Clean Up Plan Account-Level"
- "Clean Up Account-Level"
info

The above stage section just focuses on SOLE. All other stages required by other orchestrators/jobs are ignored

Aggregate Jobs

An alternative to executing each action individually and managing the order of execution for Object Groups, Aggregate Jobs can be used.

Aggregate Jobs combines all Setup and Tear-down actions in a single Jobs respectively.

This reduces the level of management required for SOLE and provides an easy workflow for Lifecycle Management.

Setup Aggregate Job

This Job handles all actions related to the Creation and Update of Managed objects in a sequence for successful creation

The Compilation, Import, Plan Generation and Plan Apply are all executed in a single Job.

Aggregated Action:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE
stage: "Apply Account Objects"
script:
- /dataops
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY

Destroy Aggregate Job

This Job handles all actions related to the deletion of Managed objects in a sequence for successful deletion.

Aggregated-Destroy Action:
extends:
- .agent_tag
image: $DATAOPS_SNOWFLAKELIFECYCLE_RUNNER_IMAGE
variables:
LIFECYCLE_ACTION: AGGREGATE-DESTROY
stage: "Clean Up Account-Level"
script:
- export LIFECYCLE_OBJECT_SUFFIX=$SNOWFLAKE_SUFFIX
- /dataops
rules:
# If merging to master, never allow the destroy to be run
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
when: never
# If running in master, never allow the destroy to be run
- if: '$CI_COMMIT_REF_NAME == "master"'
when: never
# If merging to qa, never allow the destroy to be run
- if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "qa"'
when: never
# If running in qa, never allow the destroy to be run
- if: '$CI_COMMIT_REF_NAME == "qa"'
when: never
# For other runs, this step is manual
- when: manual
artifacts:
when: always
paths:
- $ARTIFACT_DIRECTORY