Matillion Orchestrator

Type	Pre-Set
Image	$DATAOPS_MATILLION_RUNNER_IMAGE

The Matillion orchestrator is a pre-set orchestrator that triggers the start of a Matillion job as part of a DataOps pipeline. This functionality makes it possible to integrate all existing Matillion jobs into a DataOps pipeline.

Usage

The Matillion orchestrator orchestrates Matillion jobs via Matillion's ETL API.

The orchestrator's workflow is as follows:

• It posts a request to the specified Matillion instance
• It then polls the status and fetches progress from the Matillion instance
• Finally, it propagates the Matillion execution status to the DataOps pipeline

"pipelines/includes/local_includes/matillion_jobs/my_matillion_job.yml

"My Matillion Job":
  extends: 
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_MATILLION_RUNNER_IMAGE
  variables:
    MATILLION_ACTION: START
    MATILLION_URL: XXXX
    MATILLION_GROUP: XXXX
    MATILLION_PROJECT: XXXX
    MATILLION_JOB: XXXX
  script: 
    - /dataops
  icon: ${MATILLION_ICON}

We recommend that you configure the DataOps pipeline to continue running only if the Matillion job is successful, ensuring that the pipeline run does not transform any out-of-date data.

Supported parameters

Parameter	Required/Default	Description
MATILLION_USERNAME	REQUIRED but by default pulled from the DataOps Vault	Username to access the Matillion ETL API
MATILLION_PASSWORD	REQUIRED but by default pulled from the DataOps Vault	Password to access the Matillion ETL API
MATILLION_ACTION	REQUIRED	Currently only supports START
MATILLION_URL	REQUIRED	The server IP address or domain name of the Matillion ETL instance
MATILLION_GROUP	REQUIRED	Name of the Matillion group of the job to be executed
MATILLION_PROJECT	REQUIRED	Name of the Matillion project of the job to be executed
MATILLION_JOB	REQUIRED	Name of the Matillion Job to execute
MATILLION_VERSION	default	Version for Matillion ETL
SET_MATILLION_KEYS_TO_ENV	None	If set, the credentials from the DataOps Vault would be fetched and exposed in the environment
MATILLION_USERNAME_VAULT_KEY	MATILLION.DEFAULT.USERNAME	If set, overrides the default vault path for username
MATILLION_PASSWORD_VAULT_KEY	MATILLION.DEFAULT.PASSWORD	If set, overrides the default vault path for password

Example jobs

This example demonstrates what a typical pipeline job looks like:

"pipelines/includes/local_includes/matillion_jobs/start_matillion_job.yml

"Start Matillion Job":
  extends: 
    - .agent_tag
  stage: "Bulk Ingestion"
  image: $DATAOPS_MATILLION_RUNNER_IMAGE
  variables:
    MATILLION_ACTION: START
    MATILLION_URL: https://etl1.example.com/xxxx
    MATILLION_GROUP: My-Group
    MATILLION_PROJECT: My-Project
    MATILLION_JOB: my_job
    MATILLION_VERSION: default
  script:
    -  /dataops
  icon: ${MATILLION_ICON}

Project resources

None

Host dependencies (and Resources)

None