API Orchestrator

Type	Flexible
Image	$DATAOPS_API_RUNNER_IMAGE

The API orchestrator is a flexible orchestrator that enables pipelines to interact with RESTful API services across various applications. The included curl and jq applications enable querying remote services using scripts as well as parsing/displaying the JSON output. It also interacts with the DataOps vault to allow vault variables to be used in these scripts.

Usage

Example use cases of this orchestrator include:

• Interacting with a third-party data ingestion tool to trigger ingestion at the beginning of a pipeline
• Triggering a downstream reporting system to refresh the data at the end of a pipeline.

For instance, you can interface with a job's external endpoints as follows:

pipelines/includes/local_includes/api_service_jobs/my_api_job.yml

"My API Job":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_API_RUNNER_IMAGE
  variables:
  script:
    - curl https://some.api.service/api/v1/widgets
  icon: ${API_ICON}

Supported parameters

None

Example jobs

This example is of the API orchestrator interfacing with an ingestion job's endpoint:

pipelines/includes/local_includes/api_service_jobs/my_api_ingestion_job.yml

"My API ingestion job":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_API_RUNNER_IMAGE
  variables:
  script:
    - curl https://myendpoint.com/dosomething
  icon: ${API_ICON}

This second example uses the API orchestrator to call a third-party API:

pipelines/includes/local_includes/api_service_jobs/call_external_api.yml

"Call external API":
  extends:
    - .agent_tag
  stage: "Call API"
  image: $DATAOPS_API_RUNNER_IMAGE
  variables:
    MY_API_URL: https://some.api.service/api/v1/widgets
    MY_WIDGETS_FILE: $CI_PROJECT_DIR/widgets.json
  script:
    - curl $MY_API_URL > $MY_WIDGETS_FILE
    - jq $MY_WIDGETS_FILE
  icon: ${API_ICON}