Talend TMC Orchestrator

Enterprise

Image	$DATAOPS_TMC_RUNNER_IMAGE

The Talend TMC orchestrator runs existing ETL pipeline jobs configured in Talend Cloud's Talend Management Console (TMC) as part of a DataOps pipeline. As with the Talend TAC Orchestrator, this orchestrator integrates both these technologies (Talend TMC and DataOps.live) and allows them to co-exist.

Usage

This YAML script demonstrates how to structure a typical TMC ETL pipeline job inside a DataOps pipeline.

pipelines/includes/local_includes/tmc_jobs/my_tmc_job.yml

"My TMC Job":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_TMC_RUNNER_IMAGE
  variables:
    TMC_ACTION: START
    TMC_TASK_ID: XXXX
    TMC_ACCESS_TOKEN: DATAOPS_VAULT(XXXX)
  script:
    - /dataops
  icon: ${TALEND_ICON}

Supported parameters

Parameter	Required/Default	Description
TMC_ACTION	REQUIRED	Must be START
TMC_TASK_ID	REQUIRED	The task identifier for the deployed task in TMC
TMC_ACCESS_TOKEN	REQUIRED	The user's personal access token for Talend Cloud
TMC_TIMEOUT	Optional. Defaults to 300	The maximum time (in seconds) to wait for the task to complete
TMC_REGION	Optional. Defaults to eu	The Talend Cloud region (see the Talend docs for supported values)
TMC_TASK_PARAMETERS	Optional	The list of parameters to pass to the Talend task as JSON string

Example jobs

The following examples describe how to start a job, as specified by TMC_TASK_ID and using the user's access token (TMC_ACCESS_TOKEN).

1. Run a TMC job in the Talend EU region

pipelines/includes/local_includes/tmc_jobs/execute_tmc_job.yml

"Execute TMC Job":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_TMC_RUNNER_IMAGE
  variables:
    TMC_ACTION: START
    TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
    TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
  script:
    - /dataops
  icon: ${TALEND_ICON}

2. Run a TMC job in the Talend US east region

pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml

"Execute TMC Task":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_TMC_RUNNER_IMAGE
  variables:
    TMC_ACTION: START
    TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
    TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
    TMC_REGION: us
  script:
    - /dataops
  icon: ${TALEND_ICON}

3. Pass parameters to a TMC job

This example passes the task parameters as JSON payload. The parameter values stem from a mix of DataOps Vault values, job variables, and constants.

pipelines/includes/local_includes/tmc_jobs/execute_tmc_task.yml

"Execute TMC Task":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_TMC_RUNNER_IMAGE
  variables:
    TMC_ACTION: START
    TMC_TASK_ID: 5bd04205776eb50ea0e2eb66
    TMC_ACCESS_TOKEN: DATAOPS_VAULT(TALEND.TMC.ACCESS_TOKEN)
    TMC_TASK_PARAMETERS: |-
      {
        "a_user":         "DATAOPS_VAULT(TALEND.PROD.a_user)",
        "a_password":     "DATAOPS_VAULT(TALEND.PROD.a_password)",
        "which_db":       "$DATAOPS_DATABASE",
        "other_param":    false
      }
  script:
    - /dataops
  icon: ${TALEND_ICON}