Skip to main content

Scheduling DataOps Pipelines

Scheduling one or more pipeline(s) forms an integral part of any DataOps project.

Why?

In summary, there are several reasons for this. However, the most significant reason is that it saves time by scheduling recurring tasks. For instance, let's assume you have transactional tables that must be updated every 15 minutes to ensure that the linked data products remain up to date.

The way to schedule a pipeline is from within a project by navigating to CI / CD -> Schedules and clicking on New schedule:

scheduling-pipelines __shadow__

Pipeline Scheduling

When creating a new schedule, be sure to pick the correct pipeline (<name>-ci.yml or full-ci.yml) file.