Skip to main content

DataOps Pipeline Parameters

In a standard DataOps configuration, the <pipeline-name>-ci.yml (or full-ci.yml) pipeline file describes several well-defined behaviors, implemented by setting related pipeline parameters inside a variables: key-value pair, inside the CI/CD settings, or as parameters for a one-time or scheduled pipeline run.


An ingestion job is any job that inherits from the .should_run_ingestion execution rule.

Parameter (Key)Possible ValuesBehavior/Description
FORCE_INGESTIONA single job nameThis parameter forces the specified ingestion job to run.
FORCE_ALL_INGESTIONTRUESimilar to FORCE_INGESTION but will force all ingestion jobs to run
DATAOPS_SOLE_DEBUGTRUEThis parameter enables the debugging of SOLE without writing sensitive information to the logs.
DATAOPS_DEBUGTRUEThis parameter can be specified at any level, for a whole pipeline or an individual job.
CACHE_CLEANUPOptional, defaults to FALSESet to TRUE, or any value, deletes the entire local cache for the current pipeline run.



The job name must match the value in the JOB_NAME parameter.

Even if an ingestion job within a DataOps pipeline is not meant to run because the pipeline is running on the dev or feature branch, this parameter forces the job to run anyway.

The need to force a single ingestion job to run is a widespread use case when developing new ingestion jobs. For instance, let's assume you have five existing ingestion jobs and create a sixth ingestion job in your feature branch. You won't necessarily want the original five jobs to run (which they won't) if you use the force ingestion parameters to force the sixth ingestion job to run.

For instance, a typical ingestion job is as follows:

My Ingestion:
- .agent_tag
- .should_run_ingestion
stage: "Data Ingestion"
JOB_NAME: my_sixth_ingestion_job # Not used inside the job, but used to match FORCE_INGESTION
- /dataops

The <pipeline-name>-ci.yml pipeline file then includes the parameter FORCE_INGESTION with the job name, my_sixth_ingestion_job, which will force this job to run during the pipeline run.


When set to TRUE, the FORCE_ALL_INGESTION parameter tells the pipeline to run all the ingestion jobs, irrespective of whether the context is production environment or not.


Setting this parameter causes DataOps jobs to output additional debug information for the Snowflake Object Lifecycle Engine (SOLE) without disclosing sensitive information to the logs.


This parameter can be specified at any level for a whole pipeline or a particular job.


Use this with care - while all credentials are masked and using this parameter is secure you should not permanently turn this on. Enabling this option creates excessive numbers of logs files for single runs.


Setting this parameter to true causes DataOps jobs to output additional debug information.


Use this with care as it can often expose vault secrets to the job output. Once used, delete the pipeline instance runs so that these secrets are not disclosed.


If this parameter is set to true, the pipeline run deletes the pipeline cache. For a detailed discussion on how and when to use this parameter, see the DataOps Pipeline Cache.