Skip to main content

DataOps Pipeline Parameters

In a standard DataOps configuration, the <pipeline-name>-ci.yml (or full-ci.yml) pipeline file describes several well-defined behaviors, implemented by setting related pipeline parameters inside a variables: key-value pair, inside the CI/CD settings, or as parameters for a one-time or scheduled pipeline run.

tip

An ingestion job is any job that inherits from the .should_run_ingestion execution rule.

Parameter (Key)Possible ValuesBehavior/Description
FORCE_INGESTIONA single job nameThis parameter forces the specified ingestion job to run.
FORCE_ALL_INGESTIONTRUESimilar to FORCE_INGESTION but will force all ingestion jobs to run
DATAOPS_SOLE_DEBUGTRUEThis parameter enables the debugging of SOLE without writing sensitive information to the logs.
DATAOPS_DEBUGTRUEThis parameter can be specified at any level, for a whole pipeline or an individual job.
CACHE_CLEANUPOptional, defaults to FALSESet to TRUE, or any value, deletes the entire local cache for the current pipeline run.

FORCE_INGESTION

tip

The job name must match the value in the JOB_NAME parameter.

Even if an ingestion job within a DataOps pipeline is not meant to run because the pipeline is running on the dev or feature branch, this parameter forces the job to run anyway.

The need to force a single ingestion job to run is a widespread use case when developing new ingestion jobs. For instance, let's assume you have five existing ingestion jobs and create a sixth ingestion job in your feature branch. You won't necessarily want the original five jobs to run (which they won't) if you use the force ingestion parameters to force the sixth ingestion job to run.

For instance, a typical ingestion job is as follows:

pipelines/includes/local_includes/my-ingestion.yml
My Ingestion:
extends:
- .agent_tag
- .should_run_ingestion
stage: "Data Ingestion"
variables:
JOB_NAME: my_sixth_ingestion_job # Not used inside the job, but used to match FORCE_INGESTION
script:
- /dataops

The <pipeline-name>-ci.yml pipeline file then includes the parameter FORCE_INGESTION with the job name, my_sixth_ingestion_job, which will force this job to run during the pipeline run.

FORCE_ALL_INGESTION

When set to TRUE, the FORCE_ALL_INGESTION parameter tells the pipeline to run all the ingestion jobs, irrespective of whether the context is production environment or not.

DATAOPS_SOLE_DEBUG

Setting this parameter causes DataOps jobs to output additional debug information for the Snowflake Object Lifecycle Engine (SOLE) without disclosing sensitive information to the logs.

tip

This parameter can be specified at any level for a whole pipeline or a particular job.

caution

Use this with care - while all credentials are masked and using this parameter is secure you should not permanently turn this on. Enabling this option creates excessive numbers of logs files for single runs.

DATAOPS_DEBUG

Setting this parameter to true causes DataOps jobs to output additional debug information.

caution

Use this with care as it can often expose vault secrets to the job output. Once used, delete the pipeline instance runs so that these secrets are not disclosed.

CACHE_CLEANUP

If this parameter is set to true, the pipeline run deletes the pipeline cache. For a detailed discussion on how and when to use this parameter, see the DataOps Pipeline Cache.