Skip to main content

DataOps Kubernetes Runner Installation

Step 1 - Secrets and volumes

You will need Kubernetes secrets and persistent volumes claims created before installing the runner Helm chart.

Secrets

Docker registry

You will need an image pull secret to allow the pods to pull from the DataOps repository.

Create the secret in your cluster:

kubectl create secret docker-registry docker-creds \
--docker-server=docker.io \
--docker-username=dataopsreadserviceuser \
--docker-password=qf2h9372fg3ioug384 \
--docker-email=support@dataops.live
note

These examples install the runner in the default Kubernetes namespace.

DataOps runner registration token

You will need a secret to hold the runner registration token.

The registration token is generated automatically in DataOps.live and is used to link together the runner you are about to create with your specific DataOps Project or Group.

note

These registration tokens are scoped to either the project or group you get them from.

Follow these steps to obtain your Registration Token:

  1. Connect to the DataOps Platform UI
  2. Open the group (preferred) or project you want to create the runner for
    1. choosing the group makes the runner available to all projects in that group
  3. Go to Settings → CI/CD
  4. Find the Runners section and click Expand
  5. Inside the Specific runners section under Set up a group runner manually, you will find the registration token
  6. Copy it

Group Runner Token

Create the secret in your cluster and replace REGISTRY_TOKEN with the copied token:

kubectl create secret generic reg-token \
--from-literal=runner-registration-token=REGISTRY_TOKEN \
--from-literal=runner-token=""
note

runner-token="" must remain an empty string. This is automatically populated during registration.

Volumes

We recommend using storage classes to dynamically create your persistent volumes, in this configuration, you will need two persistent volume claims (PVC) that are then used to mount two paths for the runner.

Storage class for Azure File Storage:

afs-storageclass.yml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: afs-sc
provisioner: file.csi.azure.com # replace with "kubernetes.io/azure-file" if aks version is less than 1.21
allowVolumeExpansion: true
mountOptions:
- dir_mode=0777
- file_mode=0777
- uid=0
- gid=0
- mfsymlinks
- cache=strict
- actimeo=30
parameters:
skuName: Premium_LRS

For example, the two Persistent Volume Claim using the previous Azure StorageClass:

azure-file-storage-pvc-cache.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pipeline-cache.dataops.live
spec:
accessModes:
- ReadWriteOnce
storageClassName: afs-sc
resources:
requests:
storage: 5Gi
azure-file-storage-pvc-config.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: local-config.dataops.live
spec:
accessModes:
- ReadWriteOnce
storageClassName: afs-sc
resources:
requests:
storage: 5Gi

Learn more about Azure Dynamic Storage related to Kubernetes in the Microsoft docs.

Step 2 - Adding DataOps Helm chart repository

To install the DataOps Runner you will need access to the Helm chart. Helm charts package templated Kubernetes manifests so they can be configured to fit customer environments.

Install the repo:

helm repo add dataops http://charts.dataops.live

Test the runner chart is available for installation:

helm search repo dataops

Update your local repository index with the latest versions of the chart:

helm repo update

Step 3 - Configure and install the runner

Create a new file dataops-values.yml that will configure the DataOps runner Helm chart for your environment:

dataops-values.yml
imagePullSecrets:
# Name of Docker registry credential created earlier.
- name: "docker-creds"

# Main runner configuration
runners:
# Equivalent to runner config.toml file contents.
# https://docs.gitlab.com/runner/configuration/advanced-configuration.html
config: |
[[runners]]
[runners.kubernetes]
namespace = "default"
[[runners.kubernetes.volumes.pvc]]
name = "pipeline-cache.dataops.live"
mount_path = "/agent_cache"
[[runners.kubernetes.volumes.pvc]]
name = "local-config.dataops.live"
mount_path = "/local_config"
# Name of runner, used to identify in the cluster.
name: AGENT_NAME
# Tag used in jobs to specify this runner.
tags: AGENT_TAG
# Registration token secret name created earlier.
secret: reg-token

config section highlights

  • namespace - Namespace to run Kubernetes jobs in.
  • [[runners.kubernetes.volumes.pvc]] - PVC configuration for orchestrators. Note that two PVCs are required, pipeline-cache.dataops.live and local-config.dataops.live.
info

Discover the specific configuration syntax for specifying your chosen volume storage from the Kubernetes executor docs.

Install

Install the runner using your chart values:

helm upgrade --install runner dataops/dataops-runner -f dataops-values.yml
tip

DataOps runner chart installed! 🎉