Skip to main content

Matillion Orchestrator

TypePre-Set
Image$DATAOPS_MATILLION_RUNNER_IMAGE

The Matillion orchestrator is a [pre-set orchestrator][preset-orchestrators] that triggers the start of a Matillion job as part of a DataOps pipeline. This functionality makes it possible to integrate all existing Matillion jobs into a DataOps pipeline.

Usage​

The Matillion orchestrator orchestrates Matillion jobs via Matillion's ETL API.

The orchestrator's workflow is as follows:

  • It posts a request to the specified Matillion instance
  • It then polls the status and fetches progress from the Matillion instance
  • Finally, it propagates the Matillion execution status to the DataOps pipeline
"pipelines/includes/local_includes/matillion_jobs/my_matillion_job.yml
My Matillion Job:
  extends:
    - .agent_tag
  stage: My Stage
  image: $DATAOPS_MATILLION_RUNNER_IMAGE
  variables:
    MATILLION_ACTION: START
    MATILLION_URL: XXXX
    MATILLION_USERNAME: DATAOPS_VAULT(XXXX)
    MATILLION_PASSWORD: DATAOPS_VAULT(XXXX)
    MATILLION_GROUP: XXXX
    MATILLION_PROJECT: XXXX
    MATILLION_JOB: XXXX
  script: /dataops
  icon: ${MATILLION_ICON}

We recommend that you configure the DataOps pipeline to continue running only if the Matillion job is successful, ensuring that the pipeline run does not transform any out-of-date data.

Supported parameters​

ParameterRequired/DefaultDescription
MATILLION_USERNAMEREQUIREDUsername to access the Matillion ETL API
MATILLION_PASSWORDREQUIREDPassword to access the Matillion ETL API
MATILLION_ACTIONREQUIREDCurrently only supports START to run a Matillion job
MATILLION_URLREQUIREDThe URL of the Matillion ETL instance
MATILLION_GROUPREQUIREDName of the Matillion group of the job to be executed
MATILLION_PROJECTREQUIREDName of the Matillion project of the job to be executed
MATILLION_JOBREQUIREDName of the Matillion job to execute
MATILLION_VERSIONdefaultVersion for Matillion ETL
MATILLION_VARIABLESNoneCorrectly-escaped block of JSON comprising key-value pairs of job variables (see example below)
MATILLION_TIMEOUT3600Matillion task timeout in seconds. If increasing the DataOps job timeout, also set this to an equivalent value
MATILLION_IGNORE_SSL_CERTFALSESet to TRUE to ignore SSL warnings, e.g. when using self-signed certificates
SET_MATILLION_KEYS_TO_ENVNoneDEPRECATED If set, the credentials from the DataOps Vault would be fetched and exposed in the environment. The preferred method is now to set sensitive parameters using the DATAOPS_VAULT() syntax (see examples)
MATILLION_USERNAME_VAULT_KEYMATILLION.DEFAULT.USERNAMEDEPRECATED If set, overrides the default vault path for username
MATILLION_PASSWORD_VAULT_KEYMATILLION.DEFAULT.PASSWORDDEPRECATED If set, overrides the default vault path for password

Example jobs​

Basic task execution​

This example demonstrates what a typical pipeline job looks like:

pipelines/includes/local_includes/matillion_jobs/start_matillion_job.yml
Start Matillion Job:
  extends:
    - .agent_tag
  stage: "Bulk Ingestion"
  image: $DATAOPS_MATILLION_RUNNER_IMAGE
  variables:
    MATILLION_ACTION: START
    MATILLION_URL: https://etl1.example.com
    MATILLION_USERNAME: DATAOPS_VAULT(matillion.etl1.username)
    MATILLION_PASSWORD: DATAOPS_VAULT(matillion.etl1.password)
    MATILLION_GROUP: My Group
    MATILLION_PROJECT: My Project
    MATILLION_JOB: my_job
    MATILLION_VERSION: default
  script: /dataops
  icon: ${MATILLION_ICON}

Passing variables to Matillion​

This job passes a payload of variables to the Matillion job:

pipelines/includes/local_includes/matillion_jobs/start_matillion_job.yml
Start Matillion Job with Variables:
  extends:
    - .agent_tag
  stage: "Bulk Ingestion"
  image: $DATAOPS_MATILLION_RUNNER_IMAGE
  variables:
    MATILLION_ACTION: START
    MATILLION_URL: https://etl1.example.com
    MATILLION_USERNAME: DATAOPS_VAULT(matillion.etl1.username)
    MATILLION_PASSWORD: DATAOPS_VAULT(matillion.etl1.password)
    MATILLION_GROUP: My Group
    MATILLION_PROJECT: My Project
    MATILLION_JOB: my_parameterized_job
    MATILLION_VERSION: default
    MATILLION_VARIABLES: >-
      {
        "my_variable_one": "VALUE 1",
        "my_variable_two": "VALUE 2"
      }
  script: /dataops
  icon: ${MATILLION_ICON}

And the following example shows how to pass sensitive variables into Matillion using the existing SNOWFLAKE. namespace from the DataOps vault:

pipelines/includes/local_includes/matillion_jobs/start_matillion_job.yml
Start Matillion Job with Sensitive Variables:
  extends:
    - .agent_tag
  stage: "Bulk Ingestion"
  image: $DATAOPS_MATILLION_RUNNER_IMAGE
  variables:
    MATILLION_ACTION: START
    MATILLION_URL: https://etl1.example.com
    MATILLION_USERNAME: DATAOPS_VAULT(matillion.etl1.username)
    MATILLION_PASSWORD: DATAOPS_VAULT(matillion.etl1.password)
    MATILLION_GROUP: My Group
    MATILLION_PROJECT: My Project
    MATILLION_JOB: my_sensitive_job
    MATILLION_VERSION: default
    MATILLION_VARIABLES: >-
      {
        "snowflake_account": "DATAOPS_VAULT(SNOWFLAKE.ACCOUNT)",
        "snowflake_username": "DATAOPS_VAULT(SNOWFLAKE.SOLE.USERNAME)",
        "snowflake_password": "DATAOPS_VAULT(SNOWFLAKE.SOLE.PASSWORD)"
      }
  script: /dataops
  icon: ${MATILLION_ICON}
Note on Matillion job variables

DataOps will, by default, pass variables into the Matillion API using the scalarVariables block. However, if you also wish to include grid variables, you can provide an entire JSON block consisting of the keys scalarVariables or gridVariables at the top level. The orchestrator will automatically detect the JSON and pass it as-is into Matillion.

Last updated on