Skip to content

Coalesce Orchestrator

Enterprise

Image$DATAOPS_COALESCE_RUNNER_IMAGE

Coalesce allows you to transform Snowflake data visually with its data transformation platform and powerful development capabilities.

Coalesce Build UI - project browser view

The Coalesce orchestrator triggers the refresh of a Coalesce job as part of a DataOps pipeline. This functionality makes it possible to integrate all existing Coalesce jobs into a DataOps pipeline.

Usage

The Coalesce orchestrator orchestrates Coalesce jobs via Coalesce CLI using a Coalesce token.

The orchestrator's workflow is as follows:

  • It runs the coa refresh command using Coalesce CLI Refresh, targeting the specified Coalesce instance.
  • It then polls for completion of the orchestration process.
  • Finally, it propagates the Coalesce execution status to the DataOps pipeline.
pipelines/includes/local_includes/coalesce_jobs/coalesce.yml
My Coalesce Job:
  extends:
    - .agent_tag
  stage: "Data Transformation"
  image: $DATAOPS_COALESCE_RUNNER_IMAGE
  variables:
    DATAOPS_COALESCE_TOKEN: DATAOPS_VAULT(COALESCE.TOKEN)
    DATAOPS_COALESCE_ENVIRONMENT_ID: XXXX
    DATAOPS_COALESCE_JOB_ID: XXXX
    DATAOPS_COALESCE_SNOWFLAKE_USERNAME: DATAOPS_VAULT(XXXX)
    DATAOPS_COALESCE_SNOWFLAKE_PASSWORD: DATAOPS_VAULT(XXXX)
    DATAOPS_COALESCE_SNOWFLAKE_ROLE: DATAOPS_VAULT(XXXX)
    DATAOPS_COALESCE_SNOWFLAKE_WAREHOUSE: DATAOPS_VAULT(XXXX)
  script: /dataops
  icon: ${COALESCE_ICON}

For the orchestrator to work you need to obtain the access token, the environment id, and the job id from Coalesce:

  • To generate the access token, follow these steps:
  • Click on the Deploy tab at the Coalesce UI.
  • Hover over Generate Access Token and copy the token that appears.
  • Push the value to your secrets manager under COALESCE.TOKEN so that the job parameter DATAOPS_COALESCE_TOKEN gets resolved securely during runtime.
  • To obtain the environment id, follow these steps:

  • Click on the Deploy tab at the Coalesce UI.

  • Identify your target environment, e.g. Prod
  • Copy the numerical id next to id

    Obtain the Coalesce environment id

  • Finally, update the value of DATAOPS_COALESCE_ENVIRONMENT_ID

  • To obtain, the job id to run in the given environment, follow these steps:

  • Click on the Build tab at the Coalesce UI.

  • Navigate to Jobs in the left sidebar
  • Open the job in question and copy the numerical id from the far right

    Obtain the Coalesce job id

  • Finally, update the value of DATAOPS_COALESCE_JOB_ID

Snowflake account configuration

Note that you cannot choose the account from the DataOps side. You need to configure the account at the Coalesce side. See the Coalesce environment documention for more details.

Supported parameters

Parameter Required/Default Description
COALESCE_TOKEN REQUIRED Coalesce API Token
DATAOPS_COALESCE_ENVIRONMENT_ID REQUIRED Coalesce environment id
DATAOPS_COALESCE_JOB_ID REQUIRED Coalesce job id to be refreshed
COALESCE_SNOWFLAKE_USERNAME REQUIRED Coalesce Snowflake connection username
COALESCE_SNOWFLAKE_PASSWORD REQUIRED Coalesce Snowflake connection password
COALESCE_SNOWFLAKE_ROLE REQUIRED Coalesce Snowflake connection role
COALESCE_SNOWFLAKE_WAREHOUSE REQUIRED Coalesce Snowflake connection warehouse

Example jobs

A Coalesce job retrieving all parameters from the DataOps vault.

My Coalesce Job:
  extends:
    - .agent_tag
  stage: "Data Transformation"
  image: $DATAOPS_COALESCE_RUNNER_IMAGE
  variables:
    DATAOPS_COALESCE_TOKEN: DATAOPS_VAULT(COALESCE.TOKEN)
    DATAOPS_COALESCE_ENVIRONMENT_ID: 2
    DATAOPS_COALESCE_JOB_ID: 10
    DATAOPS_COALESCE_SNOWFLAKE_USERNAME: DATAOPS_VAULT(SNOWFLAKE.SOLE.USERNAME)
    DATAOPS_COALESCE_SNOWFLAKE_PASSWORD: DATAOPS_VAULT(SNOWFLAKE.SOLE.PASSWORD)
    DATAOPS_COALESCE_SNOWFLAKE_ROLE: DATAOPS_VAULT(SNOWFLAKE.SOLE.ROLE)
    DATAOPS_COALESCE_SNOWFLAKE_WAREHOUSE: DATAOPS_VAULT(SNOWFLAKE.SOLE.WAREHOUSE)
  script: /dataops
  icon: ${COALESCE_ICON}