Matillion Orchestrator
Type | Pre-Set |
---|---|
Image | $DATAOPS_MATILLION_RUNNER_IMAGE |
The Matillion orchestrator is a pre-set orchestrator that triggers the start of a Matillion job as part of a DataOps pipeline. This functionality makes it possible to integrate all existing Matillion jobs into a DataOps pipeline.
Usage
The Matillion orchestrator orchestrates Matillion jobs via Matillion's ETL API.
The orchestrator's workflow is as follows:
- It posts a request to the specified Matillion instance
- It then polls the status and fetches progress from the Matillion instance
- Finally, it propagates the Matillion execution status to the DataOps pipeline
"pipelines/includes/local_includes/matillion_jobs/my_matillion_job.yml
"My Matillion Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_MATILLION_RUNNER_IMAGE
variables:
MATILLION_ACTION: START
MATILLION_URL: XXXX
MATILLION_GROUP: XXXX
MATILLION_PROJECT: XXXX
MATILLION_JOB: XXXX
script:
- /dataops
icon: ${MATILLION_ICON}
We recommend that you configure the DataOps pipeline to continue running only if the Matillion job is successful, ensuring that the pipeline run does not transform any out-of-date data.
Supported parameters
Parameter | Required/Default | Description |
---|---|---|
MATILLION_USERNAME | REQUIRED but by default pulled from the DataOps Vault | Username to access the Matillion ETL API |
MATILLION_PASSWORD | REQUIRED but by default pulled from the DataOps Vault | Password to access the Matillion ETL API |
MATILLION_ACTION | REQUIRED | Currently only supports START |
MATILLION_URL | REQUIRED | The server IP address or domain name of the Matillion ETL instance |
MATILLION_GROUP | REQUIRED | Name of the Matillion group of the job to be executed |
MATILLION_PROJECT | REQUIRED | Name of the Matillion project of the job to be executed |
MATILLION_JOB | REQUIRED | Name of the Matillion Job to execute |
MATILLION_VERSION | default | Version for Matillion ETL |
SET_MATILLION_KEYS_TO_ENV | None | If set, the credentials from the DataOps Vault would be fetched and exposed in the environment |
MATILLION_USERNAME_VAULT_KEY | MATILLION.DEFAULT.USERNAME | If set, overrides the default vault path for username |
MATILLION_PASSWORD_VAULT_KEY | MATILLION.DEFAULT.PASSWORD | If set, overrides the default vault path for password |
Example jobs
This example demonstrates what a typical pipeline job looks like:
"pipelines/includes/local_includes/matillion_jobs/start_matillion_job.yml
"Start Matillion Job":
extends:
- .agent_tag
stage: "Bulk Ingestion"
image: $DATAOPS_MATILLION_RUNNER_IMAGE
variables:
MATILLION_ACTION: START
MATILLION_URL: https://etl1.example.com/xxxx
MATILLION_GROUP: My-Group
MATILLION_PROJECT: My-Project
MATILLION_JOB: my_job
MATILLION_VERSION: default
script:
- /dataops
icon: ${MATILLION_ICON}
Project resources
None
Host dependencies (and Resources)
None