Pipeline Cache
For an in-depth explanation of the DataOps pipeline types and structure, see the Pipeline Overview documentation.
The pipeline cache is saved on the DataOps.live runner host machine in the /agent_cache
folder or the cache folder specified during the Runner Installation.
A folder with a name set to pipeline id/number is created inside /agent_cache
for each pipeline.
This pipeline cache is designed to reduce pipeline build- and runtime by allowing components or dependencies from a particular pipeline run to be used in subsequent runs without being created or downloaded at the start of every pipeline run.
This cache also saves and retrieves the encrypted secrets stored in the DataOps.live vault. This vault contains the credentials specified in YAML files in the host machine's /secrets
folder.
See the Vault document for more information.
By default, the pipeline cache is not removed from the host machine. However, if required, it can be destroyed by setting the environment variable CACHE_CLEANUP
. The following code snippet shows how to set CACHE_CLEANUP
to ensure the cache is removed from the host machine at the end of the pipeline run.
clean-cache:
extends:
- .agent_tag
image: $DATAOPS_BASE_RUNNER_IMAGE
stage: Clean Up
variables:
CACHE_CLEANUP: 1
This cleans up the whole cache for the current pipeline if /dataops
is called with CACHE_CLEANUP
.
If any other job runs after the cache is deleted, it may lead to failure or an incorrect result. Therefore, it is best to avoid setting the CACHE_CLEANUP
environment variable.