Talend TAC Orchestrator

Enterprise

Image	$DATAOPS_TAC_RUNNER_IMAGE

The Talend TAC orchestrator runs existing Talend ETL jobs configured in the Talend Administration Center as part of a DataOps pipeline. This orchestrator allows the two technologies (Talend and DataOps.live) to co-exist.

note

Before this orchestrator runs as part of a DataOps pipeline, you must register the ETL jobs in Talend's Job Conductor in your Talend Administration Center instance.

Usage

This YAML script demonstrates how to structure a typical TAC ETL pipeline job inside a DataOps pipeline.

pipelines/includes/local_includes/tac_jobs/my_tac_job.yml

"My TAC Job":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_TAC_RUNNER_IMAGE
  variables:
    TAC_ACTION: START
    TAC_URL: XXXX
    TAC_USERNAME: XXXX
    TAC_PASSWORD: DATAOPS_VAULT(XXX)
    TAC_TASK_ID: XXXX
  script:
    - /dataops
  icon: ${TALEND_ICON}

Supported parameters

Parameter	Required/Default	Description
TAC_ACTION	REQUIRED	Must be START
TAC_URL	REQUIRED	The URL of the TAC instance to use
TAC_USERNAME	REQUIRED	The TAC user with access to start jobs in the Job Conductor
TAC_PASSWORD	REQUIRED	The TAC user's password
TAC_TASK_ID	REQUIRED	The job ID for the deployed task in the TAC Job Conductor
TAC_INSECURE	Optional	If set, it fetches the results of the deployed task on TAC in an insecure mode. It allows skipping the verification step to make sure the connection is secure before transferring the task results. For more information, see Curl Documentation.

Example jobs

This YAML script demonstrates a typical TAC use case, that is, starting an ETL pipeline to ingest data into a Snowflake database:

pipelines/includes/local_includes/tac_jobs/run_tac_job.yml

"Run TAC Job":
  extends:
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_TAC_RUNNER_IMAGE
  variables:
    TAC_ACTION: START
    TAC_URL: https://tac1.example.com/org.talend.administration
    TAC_USERNAME: DATAOPS_VAULT(TALEND.TAC1.USERNAME)
    TAC_PASSWORD: DATAOPS_VAULT(TALEND.TAC1.PASSWORD)
    TAC_TASK_ID: 123456
  script:
    - /dataops
  icon: ${TALEND_ICON}