Skip to main content

DataOps Runner Disk Space Maintenance

Clearing DataOps Runner docker images

The DataOps team does not have access to clients' runners for security reasons. However, we do provide useful tips on how to preserve the health of your runner's environment. One particularly useful one is performing a regular docker system prune command with a CRON job. This command removes:

  • all stopped containers
  • all networks not used by at least one container
  • all dangling images
  • all dangling build cache

With the usage of the DataOps Runner, the above can over time inevitably accumulate into a massive amount of unnecessarily used disk space. The solution for this is simple:

Setting a CRON job to perform the docker system prune

As a first step create a file in the /etc/cron.weekly/ folder:

cd /etc/cron.weekly
sudo nano docker_system_prune.sh

Afterward, populate the file:

docker_system_prune.sh
#!/bin/bash
docker system prune

All done!

Clearing the DataOps Runner persistent cache

DataOps introduces the persistent_cache as a means to store information between different pipeline runs. This cache cannot be cleared from the platform and it's unadvisable to clear it automatically. However, over time it can also accumulate a large amount of data, most of which will not be used.

You will find the persistent_cache as a folder under the agent_cache on your runner instance. It holds the following example structure:

/agent_cache/persistent_cache/PROJECT_NAME/BRANCH_NAME

You can safely delete the cache for branches that no longer exist or will not be used anymore manually.