Skip to main content

How to Auto-Cancel Duplicate Pipelines

If an instance of a pipeline starts before the previous instance has been completed, use the outlined approach to abort the new instance so that multiple pipelines do not run concurrently.

Note

It is, of course, possible to use resource groups to "pause" the new pipeline and wait for the previous jobs to complete. But this approach can be safer if we run hourly-ci.yml on a schedule and each run takes longer than 60 minutes to complete.

Sample Code​

The following job can be included in a pipeline to check the currently-running pipelines and abort if there's already one running the same pipeline file on the same branch. The newer pipeline will fail with an error so that notifications will be sent (depending on the project's configuration), and action can be taken as appropriate.

pipelines/includes/local_includes/auto_abort.yml
Check for Running Pipeline:
  extends: .agent_tag
  image: $DATAOPS_UTILS_RUNNER_IMAGE
  stage: Pipeline Initialisation
  variables:
    API_AUTH_HEADER: "PRIVATE-TOKEN: $LOCAL_PROJECT_TOKEN"
    API_URL_PROJECT: https://app.dataops.live/api/v4/projects/$CI_PROJECT_ID
    PIPELINE_URL_BASE: https://app.dataops.live/dataops-internal/sam/am-i-running/-/pipelines/
  script: |
    echo "This is project $CI_PROJECT_ID (ref $CI_COMMIT_REF_NAME) running pipeline $CI_PIPELINE_ID ($_PIPELINE_FILE_NAME)"
    pipeline_ids=$(curl -s -H "$API_AUTH_HEADER" "$API_URL_PROJECT/pipelines?ref=$CI_COMMIT_REF_NAME&status=running" | jq '.[] | select(.id != ($ENV.CI_PIPELINE_ID|tonumber)) | .id')
    for id in $pipeline_ids; do
      pipeline_name=$(curl -s -H "$API_AUTH_HEADER" "$API_URL_PROJECT/pipelines/$id/variables" | jq -r '.[] | select(.key == "_PIPELINE_FILE_NAME") | .value')
      if [[ "$pipeline_name" == "$_PIPELINE_FILE_NAME" ]]; then
        echo -e "\e[31m[ERROR] Another pipeline is also running $_PIPELINE_FILE_NAME in $CI_COMMIT_REF_NAME:\e[0m $PIPELINE_URL_BASE$id"
        exit 1
      fi
    done
    echo -e "\e[92m[OK] No other pipelines are running $_PIPELINE_FILE_NAME in $CI_COMMIT_REF_NAME\e[0m"

API Authentication​

Since the scope of the built-in CI_JOB_TOKEN variable does not cover these parts of the DataOps.live API, it is necessary to provide alternative credentials. One approach that we have used here is to create a project access token against this project, using the api scope, and store this in a project variable (we have used LOCAL_PROJECT_TOKEN in this code).

Explanation of the Code​

The following variables have been defined in the job for clarity:

  • API_AUTH_HEADER Header passed to the curl command for authentication against the DataOps.live API.
  • API_URL_PROJECT Base URL fragment for the DataOps.live API endpoint that relates to the current project.
  • PIPELINE_URL_BASE Base URL fragment for constructing data product platform URLs to pipelines (for convenience in error messages).
echo "This is project $CI_PROJECT_ID (ref $CI_COMMIT_REF_NAME) running pipeline $CI_PIPELINE_ID ($_PIPELINE_FILE_NAME)"
pipeline_ids=$(curl -s -H "$API_AUTH_HEADER" "$API_URL_PROJECT/pipelines?ref=$CI_COMMIT_REF_NAME&status=running" | jq '.[] | select(.id != ($ENV.CI_PIPELINE_ID|tonumber)) | .id')
for id in $pipeline_ids; do
  pipeline_name=$(curl -s -H "$API_AUTH_HEADER" "$API_URL_PROJECT/pipelines/$id/variables" | jq -r '.[] | select(.key == "_PIPELINE_FILE_NAME") | .value')
  if [[ "$pipeline_name" == "$_PIPELINE_FILE_NAME" ]]; then
    echo -e "\e[31m[ERROR] Another pipeline is also running $_PIPELINE_FILE_NAME in $CI_COMMIT_REF_NAME:\e[0m $PIPELINE_URL_BASE$id"
    exit 1
  fi
done
echo -e "\e[92m[OK] No other pipelines are running $_PIPELINE_FILE_NAME in $CI_COMMIT_REF_NAME\e[0m"
  • Line 2: Using the pipelines API, obtain the IDs of any running pipelines on the current branch/tag, excluding the current pipeline.
  • Line 4: Using the pipeline variables API, obtain the file name of each pipeline's configuration.
  • Lines 5-8: Print an error and exit (non-zero, causing job/pipeline to fail) if the other pipeline has the same pipeline file name as this one.
Last updated on