Skip to main content

Monitoring Pipelines

Finding pipelines by name on the platform​

Starting February 24, 2024, you can filter pipelines not only by Trigger author, Branch name, Tag name, Status and Source, but also by Pipeline name.

Filtering by name lets you observe, analyze, and act on pipeline runs for a specific *-ci.yml file.

no commit or merge request pipelines

You can only filter for scheduled pipelines, pipelines running directly from CI/CD -> Pipelines, and pipelines triggered via the API.

You can't filter pipelines created from commits or merge requests.

The filtering uses the workflow name of your data pipelines, which is defaulted to the pipeline filename. Using the workflow name means the Pipelines overview page will show the pipeline filename *-ci.yml, rather than the last commit message on the branch. If you want to customize your search, read how you can set a custom pipeline name instead.

To filter pipelines by their names:

  1. Navigate to your project on the data product platform and open the pipelines page at CI/CD → Pipelines.

  2. Click in the filter field and select Pipeline name.

    Pipeline filter options !!shadow!!

  3. Click the pipeline name you wish to filter for, the default full_ci.yml in this example.

    Pipeline CI options !!shadow!!

    This lists the pipelines by the name you selected.

    Pipelines filtered !!shadow!!

Keep in mind that filtering only works on pipelines with the pipeline name configured. Therefore, you cannot filter past pipelines where the name was unset.

Finding pipelines by name with the API​

You can also filter pipelines by name within the API. The same constraints as for the platform apply, i.e. you can only filter scheduled pipelines, pipelines running directly from CI/CD -> Pipelines, and pipelines triggered via the API .

Using the API endpoint below, you can query for pipelines that ran using the specified pipeline name.

https://app.dataops.live/api/v4/projects/<PROJECT_ID>>/pipelines?name=<PIPELINE_NAME>

Example response:

API response
[
   {
   "id": 1,
   "iid": 1,
   "project_id": 1,
   "sha": "392e75fcfd4*******",
   "ref": "main",
   "status": "manual",
   "source": "web",
   "created_at": "2024-02-20T12:07:21.200+00:00",
   "updated_at": "2024-02-20T12:11:06.882+00:00",
   "web_url": "https://app.dataops.live/example-group/example-project/-/pipelines/1",
   "name": "full-ci.yml"
   },
   {
   "id": 2,
   "iid": 2,
   "project_id": 1,
   "sha": "995cf21666d*******",
   "ref": "main",
   "status": "canceled",
   "source": "web",
   "created_at": "2024-02-20T12:00:14.326+00:00",
   "updated_at": "2024-02-20T12:07:21.942+00:00",
   "web_url": "https://app.dataops.live/example-group/example-project/-/pipelines/2",
   "name": "full-ci.yml"
   },
...
]

Setting a custom pipeline name​

The pipeline name is set by default in the DataOps Reference Project using the workflow: section with a name: keyword. That means you do not have to make any changes to your project. If you prefer a pipeline name different from the filename, add and modify the following configuration to your -ci.yml files.

workflow:
  name: $_PIPELINE_FILE_NAME
Last updated on