Skip to main content

Talend TMC Orchestrator

TypePre-Set
Image$DATAOPS_TMC_RUNNER_IMAGE

The Talend TMC Orchestrator, a pre-set orchestrator, executes existing ETL pipeline jobs configured in Talend Cloud's Talend Management Console (TMC) as part of a DataOps pipeline. As with the Talend TAC Orchestrator, this orchestrator integrates both these technologies (Talend TMC and DataOps) and allows them to co-exist.

Usage

This YAML script demonstrates how to structure a typical TMC ETL pipeline job run inside a DataOps pipeline.

pipelines/includes/local_includes/tmc_jobs/my_tmc_job.yml
"My TMC Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: XXXX
TMC_ACCESS_TOKEN: DATAOPS_VAULT(XXXX)
script:
- /dataops
icon: ${TALEND_ICON}

Supported Parameters

ParameterRequired/DefaultDescription
TMC_ACTIONREQUIREDMust be START
TMC_TASK_IDREQUIREDThe task identifier for the deployed task in TMC
TMC_ACCESS_TOKENREQUIREDThe user's personal access token for Talend Cloud
TMC_TIMEOUTOptional, defaults to 300The maximum time (in seconds) to wait for the task to complete
TMC_REGIONOptional, defaults to euThe Talend Cloud region (see the Talend docs for supported values)
TMC_TASK_PARAMETERSOptionalThe list of parameters to pass to the Talend task as JSON string

Example Jobs

The following examples describe how to start a job, as specified by TMC_TASK_ID and using the user's personal access token (TMC_ACCESS_TOKEN).

1. Run a TMC Job in the Talend EU Region

pipelines/includes/local_includes/tmc_jobs/execute_tmc_job.yml
"Execute TMC Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
script:
- /dataops
icon: ${TALEND_ICON}

2. Run a TMC job in the Talend US East region

pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml
"Execute TMC Task":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
TMC_REGION: us
script:
- /dataops
icon: ${TALEND_ICON}

3. Passing parameters to a TMC job

This example passes the task parameters as JSON payload. The parameter values stem from a mix of DataOps Vault values, job variables, and constants.

pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml
"Execute TMC Task":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
TMC_TASK_PARAMETERS: |-
{
"a_user": "DATAOPS_VAULT(TALEND.PROD.a_user)",
"a_password": "DATAOPS_VAULT(TALEND.PROD.a_password)",
"which_db": "$DATAOPS_DATABASE",
"other_param": false
}
script:
- /dataops
icon: ${TALEND_ICON}

Project Resources

None

Host Dependencies (and Resources)

None