Skip to main content

Running Pipelines

Multiple pipeline configurations​

Your project can have multiple pipeline configuration files that define the jobs needed to fulfill the needs of your data pipeline.

You must add the pipeline configuration files to the root of the project. The platform will consider any file ending with -ci.yml as a pipeline configuration file, e.g., full-ci.yml, my-data-pipe-ci.yml.

Methods to run pipelines​

You can run pipelines using a variety of methods. Each method can define which pipeline configuration file to use.

The following is a list of the methods you can use to start a pipeline:

Run pipeline form​

Use this method when you want to run an ad-hoc pipeline via the web user interface and when the pipelines don't have a regular processing time or testing pipeline changes.

Steps to use this method:

  1. Navigate to your project.

  2. Navigate to CI/CD → Pipelines.

    Main nav CICD menu item highlighted __shadow__

  3. Click Run Pipeline. You will be redirected to the Run Pipeline form view.

    Pipelines view highlighting Run pipeline button __shadow__

  4. Fill in the Run Pipeline form.

    Run pipeline form with fields highlighted __shadow__

    1. Select the branch or tag where your pipeline configuration file exists.
    2. Select the pipeline type as your chosen pipeline configuration file.

  5. Click Run Pipeline to start the pipeline. You will be redirected to the pipeline in progress view, where your pipeline jobs are created and ready to start.

New schedule form​

Use this method when you want to run a pipeline configuration file at regular intervals which make running pipelines more consistent.

Steps to use this method:

  1. Navigate to your project.

  2. Navigate to CI/CD → Schedules.

    Main nav CICD Schedules menu item highlighted __shadow__

  3. Click New schedule. You will be redirected to the Schedule a new pipeline form view.

    Schedules view with New schedule btn highlighted __shadow__

  4. Fill in the Schedule a new pipeline form.

    New schedule form with fields highlighted __shadow__

    1. Enter a description that appears on the pipeline schedules view.
    2. Select an interval pattern or create your own.
    3. Select the cron timezone.
    4. Select the branch or tag where your pipeline configuration file exists.
    5. Select the pipeline type as your chosen pipeline configuration file.

  5. Click Save pipeline schedule. You will be redirected to the Schedules view where you can see your new schedule.

Commit message​

Use this method when you author a change to your data pipeline that you would like to run as soon as you commit and push the change. This makes the development loop quicker by running a specific pipeline file per development commit.

The platform looks for your pipeline configuration filename anywhere in the commit message in the following form:

[<file-name> ci]

The following are examples of commit messages where the pipeline file is called new-ci.yml:

  • Some commit message [new-ci.yml ci]
  • [new-ci.yml ci] another message
  • Longer commit message [new-ci.yml ci] and more

When you push any of these messages, it will run the new-ci.ym pipeline configuration file.

info

A message that includes [skip ci] guarantees none of the pipelines will run.

Steps to use this method in the Web IDE:

  1. Navigate to your project.

  2. Click Web IDE. You will be redirected to the Web IDE view showing your repo files and an editor.

    Project view with Web IDE btn highlighted __shadow__

  3. Make a change to the repo.

  4. Click Commit. This shows additional form options and switches the editor to a differential view of the changes.

  5. Fill in the commit form.

    Web IDE commit changes form __shadow__

  6. Select the pipeline type as your chosen pipeline configuration file. This updates your commit message in the multiline text field.

  7. Click Commit. The Web IDE will refresh and allow edits again.

  8. Click the pipelines 🚀 button (top right of the Web IDE). This shows the pipeline's side panel view and provides a live progress feed of the running pipeline.

Steps to use this method on the command line. Note that these steps assume you have cloned your project locally and made a change:

  1. git add --all
  2. git commit -m "Some commit message [new-ci.yml]"
  3. git push

CLI commit changes with message __shadow__

To see the running pipeline:

  1. Navigate to your project.
  2. Navigate to CI/CD → Pipelines, to see the new-ci.yml pipeline running.

API (REST)​

Use this method to integrate DataOps.live data pipelines with scripts and software to use an external scheduling system or event-driven data pipeline architecture.

DataOps.live has a POST REST endpoint /api/v4/projects/YOUR_PROJECT_ID/trigger/pipeline that you can use to trigger a project's data pipeline. You will need your project ID, which you can find on the project overview page.

The endpoint is expecting a minimum of 4 pieces of HTTP form data:

  • Pipeline trigger token for authorization token=YOUR_PIPELINE_TRIGGER_TOKEN
  • Project ID for the project the pipeline belongs to
  • Git ref to use ref=YOUR_REF
  • The pipeline configuration file to use 'variables[_PIPELINE_FILE_NAME]=YOUR_PIPELINE_FILE_NAME'
note

The _PIPELINE_FILE_NAME is a reserved environment variable for DataOps.live and references the pipeline configuration file.

You will need to be authorized to trigger the pipeline using a pipeline trigger token.

Steps to get a pipeline trigger token:

  1. Navigate to your project.
  2. Navigate to Settings → CI/CD.
  3. Click Expand in the Pipeline triggers section. The page shows settings for pipeline trigger tokens.
  4. Fill in the Manage your project's triggers form.
  5. Click Add trigger. The page refreshes, and your trigger appears in the list.

You can copy the displaying token to call the API and run your pipeline.

To use this method on the command line, you will need to add your trigger token and project ID.

Run a curl command with form options:

curl -X POST --fail -F token=YOUR_PIPELINE_TRIGGER_TOKEN -F ref=YOUR_REF -F 'variables[_PIPELINE_FILE_NAME]=YOUR_PIPELINE_FILE_NAME' https://app.dataops.live/api/v4/projects/YOUR_PROJECT_ID/trigger/pipeline

Which pipeline configuration takes precedence?​

There are situations where you can define pipeline configuration files in a request, a "commit message," and have a project default simultaneously. The platform has an order of precedence to decide which pipeline file to run.

An ordered list of questions to decide which pipeline file to run (the first question to answer "yes" will be the chosen pipeline file):

  1. Is this a scheduled run? Then the pipeline file defined in the schedule is used.
  2. Is this run from the Web UI, API, or a parent pipeline? Then the pipeline file defined in the request is used.
  3. Is this run from a Git push or merge request, and does this run include a pipeline file commit message? Then the file defined in the commit is used.
  4. Is there a project default? Then the project default is used.
  5. Nothing defined? Then dataops-ci.yml is used.
info

You don't need a dataops-ci.yml. If the file is missing, then nothing happens.

FAQ​

What happens when my commit message has a pipeline file name, and I start a pipeline using the Web UI?​

The only time we use the commit message as the pipeline file name is when that commit is pushed or merged. In this scenario, the pipeline file name set by the Web UI has precedence over the commit message.

What happens when my commit message has a pipeline file name that does not exist?​

The precedence checks will see if there is a project default configuration file. If there is no project default, it will check for a dataops-ci.yml. If there is no dataops-ci.yml then nothing happens.

Last updated on