In a standard DataOps configuration, the
full-ci.yml) pipeline file describes several well-defined behaviors, implemented by setting related pipeline parameters inside a
variables: key-value pair, inside the CI/CD settings, or as parameters for a one-time or scheduled pipeline run.
An ingestion job is any job that inherits from the
.should_run_ingestion execution rule.
|Parameter (Key)||Possible Values||Behavior/Description|
|A single job name||This parameter forces the specified ingestion job to run.|
|Similar to |
|This parameter enables the debugging of SOLE without writing sensitive information to the logs.|
|This parameter can be specified at any level, for a whole pipeline or an individual job.|
|Optional, defaults to ||Set to |
The job name must match the value in the
Even if an ingestion job within a DataOps pipeline is not meant to run because the pipeline is running on the dev or feature branch, this parameter forces the job to run anyway.
The need to force a single ingestion job to run is a widespread use case when developing new ingestion jobs. For instance, let's assume you have five existing ingestion jobs and create a sixth ingestion job in your feature branch. You won't necessarily want the original five jobs to run (which they won't) if you use the force ingestion parameters to force the sixth ingestion job to run.
For instance, a typical ingestion job is as follows:
stage: "Data Ingestion"
JOB_NAME: my_sixth_ingestion_job # Not used inside the job, but used to match FORCE_INGESTION
<pipeline-name>-ci.yml pipeline file then includes the parameter
FORCE_INGESTION with the job name,
my_sixth_ingestion_job, which will force this job to run during the pipeline run.
When set to
FORCE_ALL_INGESTION parameter tells the pipeline to run all the ingestion jobs, irrespective of whether the context is production environment or not.
Setting this parameter causes DataOps jobs to output additional debug information for the Snowflake Object Lifecycle Engine (SOLE) without disclosing sensitive information to the logs.
This parameter can be specified at any level for a whole pipeline or a particular job.
Use this with care - while all credentials are masked and using this parameter is secure you should not permanently turn this on. Enabling this option creates excessive numbers of logs files for single runs.
Setting this parameter to true causes DataOps jobs to output additional debug information.
Use this with care as it can often expose vault secrets to the job output. Once used, delete the pipeline instance runs so that these secrets are not disclosed.
If this parameter is set to true, the pipeline run deletes the pipeline cache. For a detailed discussion on how and when to use this parameter, see the DataOps Pipeline Cache.