Skip to main content

API Orchestrator

Professional
Enterprise

Image$DATAOPS_API_RUNNER_IMAGE

The API orchestrator enables pipelines to interact with RESTful API services across various applications. The included curl and jq applications enable querying remote services using scripts as well as parsing/displaying the JSON output. It also interacts with the DataOps vault to allow vault variables to be used in these scripts.

Usage

Example use cases of this orchestrator include:

  • Interacting with a third-party data ingestion tool to trigger ingestion at the beginning of a pipeline
  • Triggering a downstream reporting system to refresh the data at the end of a pipeline.

For instance, you can interface with a job's external endpoints as follows:

pipelines/includes/local_includes/api_service_jobs/my_api_job.yml
"My API Job":
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_API_RUNNER_IMAGE
variables:
script:
- curl https://some.api.service/api/v1/widgets
icon: ${API_ICON}

Supported parameters

None

Example jobs

This example is of the API orchestrator interfacing with an ingestion job's endpoint:

pipelines/includes/local_includes/api_service_jobs/my_api_ingestion_job.yml
"My API ingestion job":
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_API_RUNNER_IMAGE
variables:
script:
- curl https://myendpoint.com/dosomething
icon: ${API_ICON}

This second example uses the API orchestrator to call a third-party API:

pipelines/includes/local_includes/api_service_jobs/call_external_api.yml
"Call external API":
extends:
- .agent_tag
stage: "Call API"
image: $DATAOPS_API_RUNNER_IMAGE
variables:
MY_API_URL: https://some.api.service/api/v1/widgets
MY_WIDGETS_FILE: $CI_PROJECT_DIR/widgets.json
script:
- curl $MY_API_URL > $MY_WIDGETS_FILE
- jq $MY_WIDGETS_FILE
icon: ${API_ICON}