Skip to main content

Running DataOps Pipelines

You can create multiple, different pipeline configuration files on the DataOps platform.

tip

Pipeline configurations are added to the root of the project. The system will consider any file ending with -ci.yml as a pipeline configuration file.

For instance:

config-files __shadow__

Ways to Run Pipelines

There are essentially two ways to run a pipeline:

Run a Pipeline Manually

Here is a simple workflow describing how to run a pipeline manually:

  • Navigate to the CI/CD > Pipelines sub-menu
  • Click on the Run Pipeline button and fill in the information on the following screen:

run-pipeline-manually __shadow__

  • You can select any pipeline configuration file from the highlighted pipeline-type dropdown.

pipeline-type __shadow__

  • After filling in the required information, click the Run Pipeline button at the bottom to run the pipeline.

run-pipeline-button __shadow__

Run a Pipeline Automatically

danger

A default pipeline must be defined in the project's CI/CD settings; otherwise, an automated pipeline won't start.

Pipelines are automatically triggered whenever you push code to a source code repository. The correct pipeline configuration file is selected based on the following rules:

Select the Configuration File Based on Commit Message

You can tell DataOps to run a pipeline using a specific configuration file via a commit message. To do so, just add the file name anywhere in the commit message in the format .<file-name>-ci.yml.

The following example commit message shows that system will look for the file named .abc-ci.yml and run the pipeline if this file exists and is valid.

This is a commit message [.abc-ci.yml ci]
danger

The pipeline won't automatically run if the file mentioned in the commit message is invalid (the file is missing).

Select the Default Configuration File Defined in CI/CD Settings

If a pipeline configuration file is not specified in a commit message, DataOps will fetch the default file from the default CI/CD settings and run the corresponding pipeline.