Skip to main content

Talend TMC Orchestrator

Enterprise

Image$DATAOPS_TMC_RUNNER_IMAGE

The Talend TMC orchestrator runs existing ETL pipeline jobs configured in Talend Cloud's Talend Management Console (TMC) as part of a DataOps pipeline. As with the Talend TAC Orchestrator, this orchestrator integrates both these technologies (Talend TMC and DataOps.live) and allows them to co-exist.

Usage

This YAML script demonstrates how to structure a typical TMC ETL pipeline job inside a DataOps pipeline.

pipelines/includes/local_includes/tmc_jobs/my_tmc_job.yml
"My TMC Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: XXXX
TMC_ACCESS_TOKEN: DATAOPS_VAULT(XXXX)
script:
- /dataops
icon: ${TALEND_ICON}

Supported parameters

ParameterRequired/DefaultDescription
TMC_ACTIONREQUIREDMust be START
TMC_TASK_IDREQUIREDThe task identifier for the deployed task in TMC
TMC_ACCESS_TOKENREQUIREDThe user's personal access token for Talend Cloud
TMC_TIMEOUTOptional. Defaults to 300The maximum time (in seconds) to wait for the task to complete
TMC_REGIONOptional. Defaults to euThe Talend Cloud region (see the Talend docs for supported values)
TMC_TASK_PARAMETERSOptionalThe list of parameters to pass to the Talend task as JSON string

Example jobs

The following examples describe how to start a job, as specified by TMC_TASK_ID and using the user's access token (TMC_ACCESS_TOKEN).

1. Run a TMC job in the Talend EU region

pipelines/includes/local_includes/tmc_jobs/execute_tmc_job.yml
"Execute TMC Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
script:
- /dataops
icon: ${TALEND_ICON}

2. Run a TMC job in the Talend US east region

pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml
"Execute TMC Task":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
TMC_REGION: us
script:
- /dataops
icon: ${TALEND_ICON}

3. Pass parameters to a TMC job

This example passes the task parameters as JSON payload. The parameter values stem from a mix of DataOps Vault values, job variables, and constants.

pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml
"Execute TMC Task":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
TMC_TASK_PARAMETERS: |-
{
"a_user": "DATAOPS_VAULT(TALEND.PROD.a_user)",
"a_password": "DATAOPS_VAULT(TALEND.PROD.a_password)",
"which_db": "$DATAOPS_DATABASE",
"other_param": false
}
script:
- /dataops
icon: ${TALEND_ICON}