Skip to main content

Using the DataOps API

As reiterated in the API documentation introduction, as a design principle, anything you can do from the DataOps User Interface (UI) can be done via a clean and powerful HTTP REST API. There are many use cases for this, but most importantly, Federated Orchestration.

People have tried to use generic automation tools, such as Airflow, Azure DevOps, CircleCI, and GitLab. However, these technologies are all built for the software world (Airflow is a little more generic) but have no expertise in other areas.

No system can be an expert in everything in a modern, complex world. This has given rise to domain-specific orchestrators, i.e., orchestrators built to understand a specific domain's technologies and requirements, such as security, network, AAA, and virtualization.

note

The DataOps platform is the Domain-Specific Orchestrator for Data.

This image shows how an Enterprise Orchestrator is used to kick off DataOps pipelines from outside the DataOps platform. In other words, instead of DataOps calling a domain-specific orchestrator, the client system uses this API to schedule or kickoff DataOps pipelines.


external-integration __shadow__


Now that we understand the DataOps Platform API, let's look at the following demo project showing how we can orchestrate the DataOps platform from an external platform API:

Variable Setup

The first step is to set up the following variables in a single script:

  • Access token
  • Pipeline
  • App URL
  • Environment branch you want to run the pipeline for
  • Project name (this needs to be url-encoded)
# Personal access token obtained from https://app.dataops.live/-/profile/personal_access_tokens
export AUTH_TOKEN= # your personal authentication token

# PIPELINE file you want to run
export PIPELINE_FILE=custom-ci.yml

# App URL
export URL=https://app.dataops.live/api

# branch
export BRANCH=master

# project name
export PROJECT="dataops-demo-project%2Fdemonstrator-projects%2Fexternal-integration-demo"

Check Everything is Working

Once the variables have been set up as highlighted above, the next step is to check that everything is working. This curl statement uses all these variables to get the details of the PIPELINE_FILE you are about to run.

curl -s --request GET --header "PRIVATE-TOKEN: $AUTH_TOKEN"   --header "Content-Type: application/json"  "$URL/v4/projects/$PROJECT/repository/files/$PIPELINE_FILE?ref=$BRANCH"  | jq

The output should look as follows:

{
"file_name": "custom-ci.yml",
"file_path": "custom-ci.yml",
"size": 196,
"encoding": "base64",
"content_sha256": "969bfc1d317f7b2b5408dbbe8e73e74f4274a7be2b0a617781fda43d7bad482e",
"ref": "master",
"blob_id": "d6ab7f4d18788f7dd90eb33a9d7cbb3dee04fb41",
"commit_id": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"last_commit_id": "9595e9535bb8710d31a7f5f9ae20844ad026840e",
"content": "aW5jbHVkZTogCiAgIyBJbmNsdWRlcyBhbGwgdGhlIHJlcXVpcmVkIGNvbmZpZyBhbmQgZGVmYXVsdCBmaWxlcwogIC0gL3BpcGVsaW5lcy9pbmNsdWRlcy9ib290c3RyYXAueW1sCgogICMgQW55IG51bWJlciBvZiBhZGRpdGlvbmFsIGZpbGVzIGNhbiBiZSBpbmNsdWRlZCBhZnRlciB0aGlzIHBvaW50CiAgLSAvc2NyaXB0cy9lbnYueW1sIAogCg=="
}

Kick Off a Pipeline

Use the following statement to kick off a pipeline:

curl -s --request POST --header "PRIVATE-TOKEN: $AUTH_TOKEN"   \
--header "Content-Type: application/json" \
--data "{ \"variables\": [ {\"key\": \"_PIPELINE_FILE_NAME\", \"variable_type\": \"env_var\", \"secret_value\": \"$PIPELINE_FILE\"}, {\"key\": \"PIPELINE_VAR_1\", \"value\": \"hello\"}, {\"key\": \"PIPELINE_VAR_2\", \"value\": \"world\"} ] }" \
"$URL/v4/projects/$PROJECT/pipeline?ref=$BRANCH" | jq

The output should be as follows:

{
"id": 326852,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "pending",
"created_at": "2022-03-28T19:59:26.385+01:00",
"updated_at": "2022-03-28T19:59:26.481+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/326852",
"before_sha": "0000000000000000000000000000000000000000",
"tag": false,
"yaml_errors": null,
"user": {
"id": 227,
"name": "DataOpsExternalIntegrationDemo",
"username": "DataOpsExternalIntegrationDemo",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/04242d06be8b91777cf51b4a1a3a8d54?s=80&d=identicon",
"web_url": "https://app.dataops.live/DataOpsExternalIntegrationDemo"
},
"started_at": null,
"finished_at": null,
"committed_at": null,
"duration": null,
"coverage": null,
"detailed_status": {
"icon": "status_pending",
"text": "pending",
"label": "pending",
"group": "pending",
"tooltip": "pending",
"has_details": false,
"details_path": "/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/326852",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_pending-5bdf338420e5221ca24353b6bff1c9367189588750632e9a871b7af09ff6a2ae.png"
}
}

All Recent Pipelines

Use the following statement to view all recent pipelines in the project:

curl -s --request GET --header "PRIVATE-TOKEN: $AUTH_TOKEN" \
"$URL/v4/projects/$PROJECT/pipelines" | jq "."

The output from this command should look as follows:

[
{
"id": 326852,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "success",
"created_at": "2022-03-28T19:59:26.385+01:00",
"updated_at": "2022-03-28T19:59:40.246+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/326852"
},
{
"id": 89296,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "success",
"created_at": "2021-07-06T16:30:36.735+01:00",
"updated_at": "2021-07-06T16:30:49.779+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/89296"
},
{
"id": 57596,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "failed",
"created_at": "2021-05-04T11:02:10.871+01:00",
"updated_at": "2021-05-04T11:02:13.556+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/57596"
}
]

Pipeline Status

Use the following statements to check on the status of a specific pipeline, in this case the last one with id 57596:

export PIPELINE_ID=57596

curl -s --request GET --header "PRIVATE-TOKEN: $AUTH_TOKEN" \
"$URL/v4/projects/$PROJECT/pipelines/$PIPELINE_ID" | jq "."

The output should be as follows:

{
"id": 57596,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "failed",
"created_at": "2021-05-04T11:02:10.871+01:00",
"updated_at": "2021-05-04T11:02:13.556+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/57596",
"before_sha": "0000000000000000000000000000000000000000",
"tag": false,
"yaml_errors": null,
"user": {
"id": 3,
"name": "Guy Adams",
"username": "guy",
"state": "active",
"avatar_url": "https://app.dataops.live/uploads/-/system/user/avatar/3/avatar.png",
"web_url": "https://app.dataops.live/guy"
},
"started_at": "2021-05-04T11:02:12.029+01:00",
"finished_at": "2021-05-04T11:02:13.549+01:00",
"committed_at": null,
"duration": 1,
"coverage": null,
"detailed_status": {
"icon": "status_failed",
"text": "failed",
"label": "failed",
"group": "failed",
"tooltip": "failed",
"has_details": false,
"details_path": "/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/57596",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_failed-41304d7f7e3828808b0c26771f0309e55296819a9beea3ea9fbf6689d9857c12.png"
}
}

All Pipeline Jobs

Use the following code to see all the jobs in a specific pipeline:

export PIPELINE_ID=57596

curl -s --request GET --header "PRIVATE-TOKEN: $AUTH_TOKEN" \
"$URL/v4/projects/$PROJECT/pipelines/$PIPELINE_ID/jobs" | jq "."

The output should be similar to the following image:

[
{
"id": 752022,
"status": "failed",
"stage": "Demo Jobs",
"name": "Show Variables",
"ref": "master",
"tag": false,
"coverage": null,
"allow_failure": false,
"created_at": "2021-05-04T11:02:10.885+01:00",
"started_at": "2021-05-04T11:02:11.919+01:00",
"finished_at": "2021-05-04T11:02:13.491+01:00",
"duration": 1.571394,
"user": {
"id": 3,
"name": "Guy Adams",
"username": "guy",
"state": "active",
"avatar_url": "https://app.dataops.live/uploads/-/system/user/avatar/3/avatar.png",
"web_url": "https://app.dataops.live/guy",
"created_at": "2020-01-16T13:15:52.186+00:00",
"bio": "",
"bio_html": "",
"location": "",
"public_email": "",
"skype": "",
"linkedin": "",
"twitter": "",
"website_url": "",
"organization": "",
"job_title": "",
"work_information": null
},
"commit": {
"id": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"short_id": "d137b2a5",
"created_at": "2021-03-18T10:14:49.000+00:00",
"parent_ids": [
"72ffc60dd31948f5a42b3e9ec5e73d4ab91c8565"
],
"title": "[skip ci] Deleted External Integration.png1, External Integration-Diagram.pptx1 files",
"message": "[skip ci] Deleted External Integration.png1, External Integration-Diagram.pptx1 files",
"author_name": "Guy Adams",
"author_email": "guy.adams@datalytyx.com",
"authored_date": "2021-03-18T10:14:49.000+00:00",
"committer_name": "Guy Adams",
"committer_email": "guy.adams@datalytyx.com",
"committed_date": "2021-03-18T10:14:49.000+00:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/commit/d137b2a5784ef91dbbe2ed32543e334123db198b"
},
"pipeline": {
"id": 57596,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "failed",
"created_at": "2021-05-04T11:02:10.871+01:00",
"updated_at": "2021-05-04T11:02:13.556+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/57596"
},
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/jobs/752022",
"artifacts": [
{
"file_type": "trace",
"size": 1222,
"filename": "job.log",
"file_format": null
}
],
"runner": {
"id": 222,
"description": "demonstrator-projects",
"ip_address": "18.134.131.125",
"active": true,
"is_shared": false,
"name": "gitlab-runner",
"online": true,
"status": "online"
},
"artifacts_expire_at": null
}
]

Specific Job Details

Use the following code to view the details for a specific job:

export JOB_ID=752022

curl -s --request GET --header "PRIVATE-TOKEN: $AUTH_TOKEN" \
"$URL/v4/projects/$PROJECT/jobs/$JOB_ID" | jq "."

The output should be as follows:

{
"id": 752022,
"status": "failed",
"stage": "Demo Jobs",
"name": "Show Variables",
"ref": "master",
"tag": false,
"coverage": null,
"allow_failure": false,
"created_at": "2021-05-04T11:02:10.885+01:00",
"started_at": "2021-05-04T11:02:11.919+01:00",
"finished_at": "2021-05-04T11:02:13.491+01:00",
"duration": 1.571394,
"user": {
"id": 3,
"name": "Guy Adams",
"username": "guy",
"state": "active",
"avatar_url": "https://app.dataops.live/uploads/-/system/user/avatar/3/avatar.png",
"web_url": "https://app.dataops.live/guy",
"created_at": "2020-01-16T13:15:52.186+00:00",
"bio": "",
"bio_html": "",
"location": "",
"public_email": "",
"skype": "",
"linkedin": "",
"twitter": "",
"website_url": "",
"organization": "",
"job_title": "",
"work_information": null
},
"commit": {
"id": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"short_id": "d137b2a5",
"created_at": "2021-03-18T10:14:49.000+00:00",
"parent_ids": [
"72ffc60dd31948f5a42b3e9ec5e73d4ab91c8565"
],
"title": "[skip ci] Deleted External Integration.png1, External Integration-Diagram.pptx1 files",
"message": "[skip ci] Deleted External Integration.png1, External Integration-Diagram.pptx1 files",
"author_name": "Guy Adams",
"author_email": "guy.adams@datalytyx.com",
"authored_date": "2021-03-18T10:14:49.000+00:00",
"committer_name": "Guy Adams",
"committer_email": "guy.adams@datalytyx.com",
"committed_date": "2021-03-18T10:14:49.000+00:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/commit/d137b2a5784ef91dbbe2ed32543e334123db198b"
},
"pipeline": {
"id": 57596,
"sha": "d137b2a5784ef91dbbe2ed32543e334123db198b",
"ref": "master",
"status": "failed",
"created_at": "2021-05-04T11:02:10.871+01:00",
"updated_at": "2021-05-04T11:02:13.556+01:00",
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/pipelines/57596"
},
"web_url": "https://app.dataops.live/dataops-demo-project/demonstrator-projects/external-integration-demo/-/jobs/752022",
"artifacts": [
{
"file_type": "trace",
"size": 1222,
"filename": "job.log",
"file_format": null
}
],
"runner": {
"id": 222,
"description": "demonstrator-projects",
"ip_address": "18.134.131.125",
"active": true,
"is_shared": false,
"name": "gitlab-runner",
"online": true,
"status": "online"
},
"artifacts_expire_at": null
}

View Job Logs

Use the following code to see the logs for a specific job:

export JOB_ID=752022

curl -s --request GET --header "PRIVATE-TOKEN: $AUTH_TOKEN" \
"$URL/v4/projects/$PROJECT/jobs/$JOB_ID/trace"

The output should be as follows:

Running with gitlab-runner 12.10.2 (c5874a4b)
on demonstrator-projects -yrQ2aqz
Preparing the "docker" executor
Using Docker executor with image datalytyx/dataops-python3-runner:stable ...
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Pulling docker image datalytyx/dataops-python3-runner:stable ...
Using docker image sha256:0e842a3d4c858a922abd875cef063f53fec8997e1e5be77ca401cd92528a60f1 for datalytyx/dataops-python3-runner:stable ...
Preparing environment
Authenticating with credentials from $DOCKER_AUTH_CONFIG
Uploading artifacts for failed job
Authenticating with credentials from $DOCKER_AUTH_CONFIG
ERROR: Job failed (system failure): Error response from daemon: error while creating mount source path '/app': mkdir /app: read-only file system (docker.go:882:0s)