Talend TMC Orchestrator
Enterprise¶
| Image | $DATAOPS_TMC_RUNNER_IMAGE |
|---|
The Talend TMC orchestrator runs existing ETL pipeline jobs configured in Talend Cloud's Talend Management Console (TMC) as part of a DataOps pipeline. As with the Talend TAC Orchestrator, this orchestrator integrates both these technologies (Talend TMC and DataOps.live) and allows them to co-exist.
Usage¶
This YAML script demonstrates how to structure a typical TMC ETL pipeline job inside a DataOps pipeline.
pipelines/includes/local_includes/tmc_jobs/my_tmc_job.yml
"My TMC Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: XXXX
TMC_ACCESS_TOKEN: DATAOPS_VAULT(XXXX)
script:
- /dataops
icon: ${TALEND_ICON}
Supported parameters¶
| Parameter | Required/Default | Description |
|---|---|---|
TMC_ACTION |
REQUIRED | Must be START |
TMC_TASK_ID |
REQUIRED | The task identifier for the deployed task in TMC |
TMC_ACCESS_TOKEN |
REQUIRED | The user's personal access token for Talend Cloud |
TMC_TIMEOUT |
Optional. Defaults to 300 |
The maximum time (in seconds) to wait for the task to complete |
TMC_REGION |
Optional. Defaults to eu |
The Talend Cloud region (see the Talend docs for supported values) |
TMC_TASK_PARAMETERS |
Optional | The list of parameters to pass to the Talend task as JSON string |
Example jobs¶
The following examples describe how to start a job, as specified by TMC_TASK_ID and using the user's access
token (TMC_ACCESS_TOKEN).
1. Run a TMC job in the Talend EU region¶
pipelines/includes/local_includes/tmc_jobs/execute_tmc_job.yml
"Execute TMC Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
script:
- /dataops
icon: ${TALEND_ICON}
2. Run a TMC job in the Talend US east region¶
pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml
"Execute TMC Task":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
TMC_REGION: us
script:
- /dataops
icon: ${TALEND_ICON}
3. Pass parameters to a TMC job¶
This example passes the task parameters as JSON payload. The parameter values stem from a mix of DataOps Vault values, job variables, and constants.
pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml
"Execute TMC Task":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_TMC_RUNNER_IMAGE
variables:
TMC_ACTION: START
TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
TMC_TASK_PARAMETERS: |-
{
"a_user": "DATAOPS_VAULT(TALEND.PROD.a_user)",
"a_password": "DATAOPS_VAULT(TALEND.PROD.a_password)",
"which_db": "$DATAOPS_DATABASE",
"other_param": false
}
script:
- /dataops
icon: ${TALEND_ICON}