Skip to main content

Using the REST API

How to use the API​

You can access the DataOps.live application programming interface (API) using HTTPS GET and POST requests. There are many tools that enable you to make these requests. For this documentation, we will use curl (a tool to transfer data from or to a server) in a Unix-like environment.

Base URL​

All DataOps.live API requests use the same base URL:

https://app.dataops.live/api/v4

This URL will prefix all requests.

Valid API request​

The following is a basic example of a request to DataOps.live:

curl "https://app.dataops.live/api/v4/projects"

Example response:

[]
tip

Add the --include option to show server response headers and give useful debugging output.

Authentication​

All API requests require authentication to return data. There are several ways you can authenticate with the DataOps.live API:

  • Personal access token (PAT)
  • Project access token
  • Group access token
  • Pipeline trigger token (limited endpoints only)

If the authentication information is not valid or is missing, DataOps.live returns an error message with a status code of 401:

{
  "message": "401 Unauthorized"
}

Access tokens​

Personal, project, or group access tokens are used similarly, i.e., via a private-token request header.

You can generate a PAT from your DataOps.live profile.

warning

Tokens are secret. Handle them securely like you would a sign-in password.

Project and group tokens are generated from the Settings -> Access Tokens menu.

Example of using personal, project, or group access token in a request header:

curl --header "private-token: your_access_token" "https://app.dataops.live/api/v4/projects"

Example response:

[
  {
    "id": 21816,
    "description": "Example project description",
    "name": "Example project name",
    ...
  }
]
tip

When working on the command line, consider using a JSON formatter like jq to beautify the JSON responses.

Project access token​

When a pipeline job is about to run, a unique token is generated and injected as a CI_JOB_TOKEN predefined variable. Use the CI/CD job token to authenticate with specific API endpoints during the job run.

You can create an allowlist of DataOps projects or DataOps custom reference projects to access your project using their CI_JOB_TOKEN. For example, project A can add project B to the allowlist. This allows CI/CD jobs in project B (the "allowed project") to authenticate API calls and access resources in project A.

By default, a project's allowlist includes only itself. If project A is public or internal, project B can access it without being added to the allowlist.

Disabling the allowlist poses a security risk, so project maintainers or owners should ensure it remains enabled. Add projects to the allowlist only when cross-project access is required.

If you turn off the allowlist, all projects are considered in the allowlist, and the job token is limited only by the user's access permissions.

You should turn this setting off for reference projects. For all other projects, leave it turned on.

Pipeline trigger tokens​

A pipeline trigger token is scoped only for triggering a pipeline.

For detailed steps on getting a pipeline trigger token, refer to our Running pipelines - Pipeline trigger.

API rate limits​

API requests are rate-limited per user. User in this context applies to the user that the access token belongs to. This includes "bot" users in the case of project and group access tokens.

The rate limit for the DataOps.live API is 600 requests per minute.

When a user exceeds the rate limit, the following requests are blocked with the HTTPS response code 429. In this scenario, the response will contain headers allowing the requester to retry after a specific period of time. See the table below for the headers and their meaning.

Response headers​

HeaderExampleDescription
RateLimit-Limit600The request quota for the client each minute.
RateLimit-Observed601Number of requests associated with the client in the time window.
RateLimit-Remaining0Remaining quota in the time window. The result of RateLimit-Limit - RateLimit-Observed.
RateLimit-Reset1699977600Unix time-formatted time when the request quota is reset.
RateLimit-ResetTimeTue, 14 Nov 2023 16:00:00 GMTRFC2616-formatted date and time when the request quota is reset.
Retry-After25Remaining duration in seconds until the quota is reset. This is a standard HTTP header.

Pagination​

Offset-based pagination​

Sometimes, the response spans many pages. When listing resources, you can pass the following query parameters:

ParameterDescription
pagePage number (default: 1)
per_pageNumber of items to list per page (default: 20, max: 100)

Example of using the page and per_page query parameters:

curl --header "private-token: your_access_token" "https://app.dataops.live/api/v4/namespaces?per_page=30&page=1"

Example response:

[
  {
    "id": 320941,
    "name": "Example group name",
    "path": "example-group-name",
    "kind": "group",
    ...
  }
]

Learn more about query parameters in a URL query string via Wikipedia.

Request payload​

JSON​

DataOps.live supports a JSON request payload when making PUT and POST requests.

Example of sending a request with a data payload:

curl --request POST --header "Content-Type: application/json" \
     --data '{"name":"<example-name>", "description":"<example-description>"}' "https://app.dataops.live/api/v4/projects"

Form fields​

DataOps.live supports using the form submission type when making PUT and POST requests.

Example of sending a request with a form type:

curl --request POST \
     --form "name=<example-name>" \
     --form "description=<example-description>" \
     --form "permissions[project_access]=null"
     "https://app.dataops.live/api/v4/projects"a

Content type​

The DataOps.live API supports the application/JSON content type by default.

Last updated on