Skip to main content

Pipeline Parameters

note

For an in-depth explanation of the DataOps pipeline types and structure, see the Pipeline Overview documentation.

In a standard DataOps configuration, the <pipeline-name>-ci.yml (or full-ci.yml) pipeline file describes several well-defined behaviors, implemented by setting related pipeline parameters inside a variables: key-value pair, inside the CI/CD settings, or as parameters for a one-time or scheduled pipeline run.

tip

An ingestion job is any job that inherits from the .should_run_ingestion execution rule.

Parameter (Key)Possible ValuesBehavior/Description
FORCE_INGESTIONA single job nameForces the specified ingestion job to run
FORCE_ALL_INGESTIONTRUESimilar to FORCE_INGESTION but will force all ingestion jobs to run
DATAOPS_SOLE_DEBUG1Enables the debugging of SOLE without writing sensitive information to the logs
DATAOPS_DEBUG1Can be specified at any level for a whole pipeline or an individual job — any confidential, secret values in the logs will be masked
CACHE_CLEANUPOptional, defaults to FALSEIf set to TRUE, or any value, it deletes the entire local cache for the current pipeline run

FORCE_INGESTION​

tip

The job name must match the value in the JOB_NAME parameter.

Even if an ingestion job within a DataOps pipeline is not meant to run because the pipeline is running on the dev or feature branch, this parameter forces the job to run anyway.

The need to force a single ingestion job to run is a widespread use case when developing new ingestion jobs. For instance, let's assume you have five existing ingestion jobs and create a sixth ingestion job in your feature branch. You won't necessarily want the original five jobs to run (which they won't) if you use the force ingestion parameters to force the sixth ingestion job to run.

For instance, a typical ingestion job is as follows:

pipelines/includes/local_includes/my-ingestion.yml
My Ingestion:
  extends:
    - .agent_tag
    - .should_run_ingestion
  stage: "Data Ingestion"
  variables:
    JOB_NAME: my_sixth_ingestion_job # Not used inside the job, but used to match FORCE_INGESTION
  script:
    - /dataops

The <pipeline-name>-ci.yml pipeline file then includes the parameter FORCE_INGESTION with the job name, my_sixth_ingestion_job, which will force this job to run during the pipeline run.

For another use case, see Force Ingestion in Development Branches.

FORCE_ALL_INGESTION​

When set to TRUE, the FORCE_ALL_INGESTION parameter tells the pipeline to run all the ingestion jobs, irrespective of whether the context is the production environment or not.

DATAOPS_SOLE_DEBUG​

Setting this parameter to 1 causes DataOps jobs to output additional debug information for the Snowflake Object Lifecycle Engine (SOLE) without disclosing sensitive information to the logs.

tip

You can specify this parameter at any level, e.g., for the whole pipeline or a particular job.

warning

Use this with care - while all credentials are masked and using this parameter is secure, you should not permanently turn this on. Enabling this option creates excessive numbers of log files for single runs.

DATAOPS_DEBUG​

Setting this parameter to 1 causes DataOps jobs to output additional debug information.

info

If you set the parameter to 1, all secret values displayed in the logs will be masked. See Vault Concepts for more details on secret masking.

CACHE_CLEANUP​

If this parameter is set to TRUE, or any value, the pipeline run deletes the pipeline cache. For a detailed discussion on how and when to use this parameter, see the DataOps Pipeline Cache.

Last updated on