Informatica Cloud Taskflow Orchestrator

Enterprise

Image	$DATAOPS_INFORMATICACLOUD_RUNNER_IMAGE

The Informatica Cloud orchestrator triggers the start of an Informatica taskflow and monitors its execution as part of a DataOps pipeline. Using it, can you leverage all existing Informatica cloud taskflows into a DataOps pipeline.

An Informaticta taskflow controls the execution sequence of a data transfer task, dynamic mapping task, mapping task, PowerCenter task, or synchronization task based on the output of the previous task. A taskflow defines the control flow logic within Informatica cloud. It covers various data integration and transformation needs, including ETL and ELT for Snowflake, for Informatica.

Usage

The Informatica Cloud orchestrator orchestrates taskflows via the Informatica Cloud REST API.

The orchestrator's workflow is as follows:

1. Triggers the specified taskflow using your Informatica cloud organization URL.
2. Polls the status and fetches progress for the taskflow.
3. Propagates the Informatica Cloud taskflow execution status to the DataOps pipeline.

"My Informatica taskflow job":
  extends:
    - .agent_tag
  stage: "Data Transformation"
  image: $DATAOPS_INFORMATICACLOUD_RUNNER_IMAGE
  variables:
    INFORMATICA_TASKFLOW_URL: https://<your_org>.informaticacloud.com/active-bpel/rt/TaskflowID
    INFORMATICA_TASK_TYPE: TASKFLOW
    INFORMATICA_USERNAME: <username>
    INFORMATICA_PASSWORD: <password>
  script:
    - /dataops
  icon: ${INFORMATICA_ICON}

We recommend configuring the DataOps pipeline to continue to run, if the Informatica Cloud taskflwo job succeeds, ensuring that the pipeline run does not transform any out-of-date data.

Supported parameters

Parameter	Required/Default	Description
INFORMATICA_USERNAME	REQUIRED	Username to access the Informatica Cloud API
INFORMATICA_PASSWORD	REQUIRED	Password to access the Informatica Cloud API
INFORMATICA_TASK_TYPE	REQUIRED	Currently only supports TASKFLOW
INFORMATICA_TASKFLOW_URL	REQUIRED	The REST API URL for the taskflow. Normally starting with dm-em.informaticacloud.com or dm-us.informaticacloud.com.
INFORMATICA_TASKFLOW_ARGS	Optional	Arguments passed to Informatica Cloud taskflow
INFORMATICA_TIMEOUT	Optional. Defaults to 3600	Informatica Cloud taskflow timeout in seconds. If you increased the DataOps job timeout, set this to an equivalent value, and vice-versa.

Example jobs

This example demonstrates what a typical pipeline job looks like:

Taskflow job

pipelines/includes/local_includes/informatica_jobs/my_informatica_job.yml

"Informatica data ingestion":
  extends:
    - .agent_tag
  stage: Data Ingestion
  image: $DATAOPS_INFORMATICACLOUD_RUNNER_IMAGE
  variables:
    INFORMATICA_TASKFLOW_URL: https://<your_org>.informaticacloud.com/active-bpel/rt/TaskflowID
    INFORMATICA_TASK_TYPE: TASKFLOW
    INFORMATICA_USERNAME: DATAOPS_VAULT(INFORMATICA.USERNAME)
    INFORMATICA_PASSWORD: DATAOPS_VAULT(INFORMATICA.PASSWORD)
  script:
    - /dataops
  icon: ${INFORMATICA_ICON}

Taskflow job with arguments

pipelines/includes/local_includes/informatica_jobs/my_informatica_job.yml

"Informatica data ingestion":
  extends:
    - .agent_tag
  stage: Data Ingestion
  image: $DATAOPS_INFORMATICACLOUD_RUNNER_IMAGE
  variables:
    INFORMATICA_TASKFLOW_URL: https://<your_org>.informaticacloud.com/active-bpel/rt/TaskflowID
    INFORMATICA_TASK_TYPE: TASKFLOW
    INFORMATICA_USERNAME: DATAOPS_VAULT(INFORMATICA.USERNAME)
    INFORMATICA_PASSWORD: DATAOPS_VAULT(INFORMATICA.PASSWORD)
    INFORMATICA_TASKFLOW_ARGS: |
      {
        "arg_1": "value"
      }
  script:
    - /dataops
  icon: ${INFORMATICA_ICON}