Skip to main content

API Orchestrator

TypeFlexible
Image$DATAOPS_API_RUNNER_IMAGE

The DataOps API Orchestrator is a flexible orchestrator that enables pipelines to interact with RESTful API services across a wide range of applications. The included curl and jq applications enable querying remote services using scripts as well as parsing/displaying the JSON output. It also interacts with the DataOps vault to allow vault variables to be used in these scripts.

Usage

Example uses of this orchestrator include:

  • Interacting with a third-party data ingestion tool to trigger ingestion at the beginning of a pipeline
  • Triggering a downstream reporting system to refresh the data at the end of a pipeline.

For instance, you can interface with a job's external endpoints as follows:

pipelines/includes/local_includes/api_service_jobs/my_api_job.yml
"My API Job": 
extends:
- .agent_tag
stage: "My Stage"
image: $DATAOPS_API_RUNNER_IMAGE
variables:
script:
- curl https://some.api.service/api/v1/widgets
icon: ${API_ICON}

Supported Parameters

None

Example Jobs

This example is of the DataOps API orchestrator interfacing with an ingestion job's endpoint:

pipelines/includes/local_includes/api_service_jobs/my_api_ingestion_job.yml
"My API ingestion job":
extends:
- .should_run_ingestion
- .agent_tag
stage: "Batch Ingestion"
image: $DATAOPS_API_RUNNER_IMAGE
variables:
script:
- curl https://myendpoint.com/dosomething
icon: ${API_ICON}

This second example uses the API orchestrator to call a third-party API:

pipelines/includes/local_includes/api_service_jobs/call_external_api.yml
"Call external API":
extends:
- .agent_tag
stage: "Call API"
image: $DATAOPS_API_RUNNER_IMAGE
variables:
MY_API_URL: https://some.api.service/api/v1/widgets
MY_WIDGETS_FILE: $CI_PROJECT_DIR/widgets.json
script:
- curl $MY_API_URL > $MY_WIDGETS_FILE
- jq $MY_WIDGETS_FILE
icon: ${API_ICON}

Project Resources

None

Host Dependencies (and Resources)

None