Skip to main content

Pipeline Parameters

In a standard DataOps configuration, the <pipeline-name>-ci.yml (or full-ci.yml) pipeline file describes several well-defined behaviors, implemented by setting related pipeline parameters inside a variables: key-value pair, inside the CI/CD settings, or as parameters for a one-time or scheduled pipeline run.

tip

An ingestion job is any job that inherits from the .should_run_ingestion execution rule.

Parameter (Key)Possible ValuesBehavior/Description
FORCE_INGESTIONA single job nameForces the specified ingestion job to run
FORCE_ALL_INGESTIONTRUESimilar to FORCE_INGESTION but will force all ingestion jobs to run
DATAOPS_SOLE_DEBUGTRUEEnables the debugging of SOLE without writing sensitive information to the logs
DATAOPS_DEBUGTRUECan be specified at any level for a whole pipeline or an individual job
CACHE_CLEANUPOptional, defaults to FALSEIf set to TRUE, or any value, deletes the entire local cache for the current pipeline run

FORCE_INGESTION

tip

The job name must match the value in the JOB_NAME parameter.

Even if an ingestion job within a DataOps pipeline is not meant to run because the pipeline is running on the dev or feature branch, this parameter forces the job to run anyway.

The need to force a single ingestion job to run is a widespread use case when developing new ingestion jobs. For instance, let's assume you have five existing ingestion jobs and create a sixth ingestion job in your feature branch. You won't necessarily want the original five jobs to run (which they won't) if you use the force ingestion parameters to force the sixth ingestion job to run.

For instance, a typical ingestion job is as follows:

pipelines/includes/local_includes/my-ingestion.yml
My Ingestion:
extends:
- .agent_tag
- .should_run_ingestion
stage: "Data Ingestion"
variables:
JOB_NAME: my_sixth_ingestion_job # Not used inside the job, but used to match FORCE_INGESTION
script:
- /dataops

The <pipeline-name>-ci.yml pipeline file then includes the parameter FORCE_INGESTION with the job name, my_sixth_ingestion_job, which will force this job to run during the pipeline run.

For another use case, see Force Ingestion in Development Branches.

FORCE_ALL_INGESTION

When set to TRUE, the FORCE_ALL_INGESTION parameter tells the pipeline to run all the ingestion jobs, irrespective of whether the context is the production environment or not.

DATAOPS_SOLE_DEBUG

Setting this parameter causes DataOps jobs to output additional debug information for the Snowflake Object Lifecycle Engine (SOLE) without disclosing sensitive information to the logs.

tip

This parameter can be specified at any level for a whole pipeline or a particular job.

caution

Use this with care - while all credentials are masked and using this parameter is secure, you should not permanently turn this on. Enabling this option creates excessive numbers of log files for single runs.

DATAOPS_DEBUG

Setting this parameter to true causes DataOps jobs to output additional debug information.

caution

Use this with care, as it can often expose vault secrets to the job output. Once used, delete the pipeline instance runs so that these secrets are not disclosed.

CACHE_CLEANUP

If this parameter is true, the pipeline run deletes the pipeline cache. For a detailed discussion on how and when to use this parameter, see the DataOps Pipeline Cache.