Skip to main content

Matillion Orchestrator

TypePre-Set
Image$DATAOPS_MATILLION_RUNNER_IMAGE

The Matillion Orchestrator is a pre-set orchestrator that triggers the start of a Matillion job as part of a DataOps pipeline. This functionality makes it possible to integrate all existing Matillion jobs into a DataOps pipeline.

Usage

The Matillion Orchestrator orchestrates Matillion jobs via Matillion's ETL API.

The orchestrator's workflow is as follows:

  • It posts a request to the specified Matillion instance
  • It then polls the status and fetches progress from the Matillion instance
  • Finally, it propagates the Matillion execution status to the DataOps pipeline
"pipelines/includes/local_includes/matillion_jobs/my_matillion_job.yml
"My Matillion Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_MATILLION_RUNNER_IMAGE
variables:
MATILLION_ACTION: START
MATILLION_URL: XXXX
MATILLION_GROUP: XXXX
MATILLION_PROJECT: XXXX
MATILLION_JOB: XXXX
script:
- /dataops
icon: ${MATILLION_ICON}

We recommend that you configure the DataOps pipeline to continue running only if the Matillion job is successful, ensuring that the pipeline run does not transform any out-of-date data.

Supported Parameters

ParameterRequired/DefaultDescription
MATILLION_USERNAMEREQUIRED but by default pulled from the DataOps VaultUsername to access the Matillion ETL API
MATILLION_PASSWORDREQUIRED but by default pulled from the DataOps VaultPassword to access the Matillion ETL API
MATILLION_ACTIONREQUIREDCurrently only supports START
MATILLION_URLREQUIREDThe server IP address or domain name of the Matillion ETL instance
MATILLION_GROUPREQUIREDName of the Matillion group of the job to be executed
MATILLION_PROJECTREQUIREDName of the Matillion project of the job to be executed
MATILLION_JOBREQUIREDName of the Matillion Job to execute
MATILLION_VERSIONdefaultVersion for Matillion ETL
SET_MATILLION_KEYS_TO_ENVNoneIf set, the credentials from the DataOps Vault would be fetched and exposed in the environment
MATILLION_USERNAME_VAULT_KEYMATILLION.DEFAULT.USERNAMEIf set, overrides the default vault path for username
MATILLION_PASSWORD_VAULT_KEYMATILLION.DEFAULT.PASSWORDIf set, overrides the default vault path for password

Example Jobs

This example demonstrates what a typical pipeline job looks like:

"pipelines/includes/local_includes/matillion_jobs/start_matillion_job.yml
"Start Matillion Job":
extends:
- .agent_tag
stage: "Bulk Ingestion"
image: $DATAOPS_MATILLION_RUNNER_IMAGE
variables:
MATILLION_ACTION: START
MATILLION_URL: https://etl1.example.com/xxxx
MATILLION_GROUP: My-Group
MATILLION_PROJECT: My-Project
MATILLION_JOB: my_job
MATILLION_VERSION: default
script:
- /dataops
icon: ${MATILLION_ICON}

Project Resources

None

Host Dependencies (and Resources)

None