API Orchestrator

Professional

Enterprise

Image	$DATAOPS_API_RUNNER_IMAGE

The API orchestrator enables pipelines to interact with RESTful API services across various applications. The included curl and jq applications enable querying remote services using scripts as well as parsing/displaying the JSON output. It also interacts with the DataOps vault to allow vault variables to be used in these scripts.

Usage

Example use cases of this orchestrator include:

• Interacting with a third-party data ingestion tool to trigger ingestion at the beginning of a pipeline
• Triggering a downstream reporting system to refresh the data at the end of a pipeline.

For instance, you can interface with a job's external endpoints as follows:

pipelines/includes/local_includes/api_service_jobs/my_api_job.yml

"My API Job":
  extends:
    - .agent_tag
  stage: "My Stage"
  image: $DATAOPS_API_RUNNER_IMAGE
  variables:
  script:
    - curl https://some.api.service/api/v1/widgets
  icon: ${API_ICON}

Supported parameters

None

Example jobs

This example is of the API orchestrator interfacing with an ingestion job's endpoint:

pipelines/includes/local_includes/api_service_jobs/my_api_ingestion_job.yml

"My API ingestion job":
  extends:
    - .should_run_ingestion
    - .agent_tag
  stage: "Batch Ingestion"
  image: $DATAOPS_API_RUNNER_IMAGE
  variables:
  script:
    - curl https://myendpoint.com/dosomething
  icon: ${API_ICON}

This second example uses the API orchestrator to call a third-party API:

pipelines/includes/local_includes/api_service_jobs/call_external_api.yml

"Call external API":
  extends:
    - .agent_tag
  stage: "Call API"
  image: $DATAOPS_API_RUNNER_IMAGE
  variables:
    MY_API_URL: https://some.api.service/api/v1/widgets
    MY_WIDGETS_FILE: $CI_PROJECT_DIR/widgets.json
  script:
    - curl $MY_API_URL > $MY_WIDGETS_FILE
    - jq $MY_WIDGETS_FILE
  icon: ${API_ICON}