Skip to main content

Reference Project Jobs

A core architectural principle of DataOps build pipelines involves their composition with a variety of jobs, including those in the DataOps reference project and executed by the Snowflake Object Lifecycle Engine (SOLE).

The reference project implementation incorporates extra, optional Snowflake Lifecycle jobs and execution rules that support data environments. These jobs come in a few job types:

It is essential to keep in mind that you must order jobs in a specific sequence, respecting job stages.

Snowflake prod/qa setup jobs​

The Set Up Snowflake (PROD/QA) job facilitates the setup of Snowflake environments for production (prod) and quality assurance (qa) branches.

With the AGGREGATE lifecycle action, this job ensures that the necessary resources and configurations are in place for Snowflake operations in these critical environments. The job only runs when changes are pushed to the master, main, production, prod, or qa branches. This is crucial in preparing Snowflake environments for production and quality assurance tasks.

pipelines/includes/default/snowflake_lifecycle_env.yml
Set Up Snowflake (PROD/QA):
  extends: .agent_tag
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE
    ARTIFACT_DIRECTORY: snowflake-artifacts
    CONFIGURATION_DIR: dataops/snowflake
  resource_group: $CI_JOB_NAME
  stage: Snowflake Setup
  script: /dataops
  environment:
    name: $CI_COMMIT_REF_NAME
    url: $SNOWSIGHT_URL/#/data/databases/$DATAOPS_DATABASE
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
   rules:
    - if: '$CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_REF_NAME == "main" || $CI_COMMIT_REF_NAME == "production" || $CI_COMMIT_REF_NAME == "prod" || $CI_COMMIT_REF_NAME == "qa"'
      when: on_success
    - when: never

Snowflake Dev setup jobs​

The Set Up Snowflake (DEV) job streamlines the setup of Snowflake environments for development purposes.

With the AGGREGATE lifecycle action, this job prepares the necessary environment and resources for seamless development operations. This job is automatically triggered when changes are pushed to the dev, develop, or development branches. It's vital for maintaining consistent and well-prepared development environments.

pipelines/includes/default/snowflake_lifecycle_env.yml
Set Up Snowflake (DEV):
  extends: .agent_tag
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE
    ARTIFACT_DIRECTORY: snowflake-artifacts
    CONFIGURATION_DIR: dataops/snowflake
  resource_group: $CI_JOB_NAME
  stage: Snowflake Setup
  script: /dataops
  environment:
    name: $CI_COMMIT_REF_NAME
    url: $SNOWSIGHT_URL/#/data/databases/$DATAOPS_DATABASE
    on_stop: Tear Down Snowflake
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    - if: '$CI_COMMIT_REF_NAME == "dev" || $CI_COMMIT_REF_NAME == "develop" || $CI_COMMIT_REF_NAME == "development"'
      when: on_success
    - when: never

Snowflake Feature setup jobs​

The Set Up Snowflake (Feature) job sets up Snowflake environments specifically for feature branches.

With the AGGREGATE lifecycle action, this job ensures the required resources are ready, enabling developers to work on new features within isolated Snowflake environments. The job is automatically activated for all branches except main, production, prod, qa, dev, develop, and development. It's an ideal choice for isolating Snowflake environments during feature development.

pipelines/includes/default/snowflake_lifecycle_env.yml
Set Up Snowflake (Feature):
  extends: .agent_tag
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE
    ARTIFACT_DIRECTORY: snowflake-artifacts
    CONFIGURATION_DIR: dataops/snowflake
  resource_group: $CI_JOB_NAME
  stage: Snowflake Setup
  script: /dataops
  environment:
    name: $CI_COMMIT_REF_NAME
    url: $SNOWSIGHT_URL/#/data/databases/$DATAOPS_DATABASE
    on_stop: Tear Down Snowflake
    auto_stop_in: 5 days
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    - if: '$CI_COMMIT_REF_NAME != "master" && $CI_COMMIT_REF_NAME != "main" && $CI_COMMIT_REF_NAME != "production" && $CI_COMMIT_REF_NAME != "prod" && $CI_COMMIT_REF_NAME != "qa" && $CI_COMMIT_REF_NAME != "dev" && $CI_COMMIT_REF_NAME != "develop" && $CI_COMMIT_REF_NAME != "development"'
      when: on_success
    - when: never

Snowflake teardown jobs​

With the AGGREGATE-DESTROY lifecycle action, the Tear Down Snowflake job ensures removing Snowflake environments, including all their content. This job is vital for minimizing resource consumption and maintaining environmental hygiene after operations. It never runs on main, production, prod, and qa branches and is set to be manually triggered on all other branches. The job initiates the teardown process for Snowflake environments, preventing unnecessary resource usage.

pipelines/includes/default/snowflake_lifecycle_env.yml
Tear Down Snowflake:
  extends: .agent_tag
  image: $DATAOPS_SNOWFLAKEOBJECTLIFECYCLE_RUNNER_IMAGE
  variables:
    LIFECYCLE_ACTION: AGGREGATE-DESTROY
    ARTIFACT_DIRECTORY: snowflake-artifacts
    CONFIGURATION_DIR: dataops/snowflake
    GIT_STRATEGY: none
  resource_group: $CI_JOB_NAME
  stage: Clean Up
  needs:
    - job: Initialise Pipeline
      optional: true
  script: /dataops
  environment:
    name: $CI_COMMIT_REF_NAME
    action: stop
  artifacts:
    when: always
    paths:
      - $ARTIFACT_DIRECTORY
  icon: ${SNOWFLAKEOBJECTLIFECYCLE_ICON}
  rules:
    ## Block this job from main (PROD) and qa (QA) branches
    - if: '$CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_REF_NAME == "main" || $CI_COMMIT_REF_NAME == "production" || $CI_COMMIT_REF_NAME == "prod" || $CI_COMMIT_REF_NAME == "qa"'
      when: never
    ## For all other branches, enable this job to be run manually
    - when: manual

Job stages​

The stages assigned to individual jobs are equally significant, as stages establish the precise run sequence of reference project jobs. Here is a reference to the SOLE stages to help assist in accurately configuring each job within the context of the reference project.

stages:
  - Snowflake Setup
  - Clean Up
Last updated on