DataOps Docker Runner Upgrade

Overview

The following is an overview of the upgrade process:

1. Stage 1 Install the new runner with new agent_tag
2. Stage 2 Create a feature branch and change your configuration to use this new agent tag and run pipeline tests
3. Stage 3 Merge this feature branch to Dev and run pipeline tests. Continue until you are happy with all testing and then merge to main
4. Stage 4 Once you are fully happy, docker rm the old agent and you are done

Note on branch names

We are using main as our production branch, yours may be named master

Note on permissions

You will need permission to run Docker commands and modify root-owned files using sudo.

Installing the new runner with new agent_tag

Step 1 - Export settings for upgrade

These steps are similar to installing a runner with some upgrade specific tweaks.

To make running commands consistent, copy and modify the following environment variables and export them to your command line environment.

Copy and run in your terminal:

export IMAGE_TAG='latest'  # DockerHub image tag and the version of the runner
export IMAGE=dataopslive/dataops-runner:$IMAGE_TAG  # DockerHub repo path and tag
export AGENT_TAG=<YOUR_RUNNER_AGENT_TAG>  # DataOps runner tag
export AGENT_NAME=$AGENT_TAG-$IMAGE_TAG  # Runner full name
export DATAOPS_URL=https://app.dataops.live/  # DataOps.live SaaS platform
export REGISTRATION_TOKEN=<YOUR_REGISTRATION_TOKEN>  # Token from Settings → CI/CD → Runners

You can use the IMAGE_TAG value latest-next to enable additional features not yet in latest. See Platfom release notes for more info.

More information about AGENT_TAG

To upgrade without impacting other branches/environments, use a different AGENT_TAG to the existing .agent_tag's.

Agent tags can be seen in Settings → CI/CD → Runners and look like blue pills.

Step 2 - Register new runner

Register the new runner using the new runner image:

docker run --rm -v /srv/dataops-runner-$AGENT_NAME/config:/etc/gitlab-runner $IMAGE register --non-interactive --executor "docker" --docker-image dataopslive/dataops-utils-orchestrator:5-stable --url "$DATAOPS_URL" --registration-token "$REGISTRATION_TOKEN" --description "$AGENT_NAME" --tag-list "$AGENT_TAG" --run-untagged="false" --locked="true"

This new runner is now visible in Settings → CI/CD → Runners.

Step 3 - Update new runner configuration

Update the new runner configuration file generated as a result of the registration step:

sudo sed -i 's/concurrent = .*/concurrent = 8/' /srv/dataops-runner-$AGENT_NAME/config/config.toml
sudo sed -i 's/check_interval = .*/check_interval = 10/' /srv/dataops-runner-$AGENT_NAME/config/config.toml
sudo sed -i  's/    volumes =.*$/    volumes = ["\/app:\/local_config:rw","\/agent_cache:\/agent_cache:rw", "\/secrets:\/secrets:ro"]/' /srv/dataops-runner-$AGENT_NAME/config/config.toml

caution

If you added any custom config for the old runner, you will have to manually update the new runner config to include it. If you are unsure, please check /srv/dataops-runner-<old_agent_name>/config/config.toml, then update /srv/dataops-runner-$AGENT_NAME/config/config.toml as necessary.

Step 4 - Start new runner

Start the new runner alongside the old runner:

docker run -d --name $AGENT_NAME --restart always -v /srv/dataops-runner-$AGENT_NAME/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock $IMAGE

Jobs will be picked up

Success! New runner deployed and able to pick up jobs with AGENT_TAG tag.

Note that this is in parallel with the existing runner

Updating the agent_tag on feature branch

Step 1 - Create a new feature branch

From the Project → Repository view, click the plus (+) dropdown button and select New branch.

In the New Branch view, enter a new branch name and ensure you are branching from your Dev branch.

note

There are several ways to create a new branch

Step 2 - Update agent tag

Edit the .yml file where you manage agent tags. In a standard DataOps project this is usually located in pipelines/includes/config/agent_tag.yml

caution

Ensure you are on your new branch

Update the tag to the AGENT_TAG used during the new runner install.

Commit your change to this feature branch.

Step 3 - Monitor testing

When you commit the agent tag change, you will need to trigger your pipeline and all jobs will use the new runner.

Go to your Project → CI/CD → Pipelines and use the Run pipeline button.

1. Select your feature branch to run
2. Select the pipeline type that will run tests
3. Use the Run pipeline button

This will start a pipeline and you can monitor the progress of the new runner.

To confirm that the new runner is being used:

1. Select any job from the pipeline just triggered
2. View its detail and logs
3. At the head of the logs, you will find your AGENT_TAG.

When you are happy that jobs are performing well e.g. tests are passing and transformations are happening successfully, then continue to the next step.

Merging and testing

Step 1 - Merge the feature branch into Dev branch

Once your checks are complete on the feature branch, it is time to merge the agent tag change into the Dev branch.

Create a merge request from Project → Merge requests. Use the New merge request button.

note

Recent changes to a branch may also create a notification and helper Create merge request button, which you can also use.

1. Add a description and any required reviewer.
2. Use the Create merge request button.
3. Review and approve the merge request.
4. Merge the request into Dev.

Now trigger a Dev pipeline run to test the runner in Dev. Go to your Project → CI/CD → Pipelines and use the Run pipeline button. Full instructions are on Step 3 - Monitor testing instructions to trigger the pipeline.

When you are happy that jobs are performing well e.g. tests are passing and transformations are happening successfully, then continue to the next step.

Step 2 - Merge Dev branch into main

Once your checks are complete on the Dev branch, it is time to merge the agent tag change into the main branch.

Follow the instructions in Step 1 - Merge feature branch into Dev branch, changing the source branch to Dev and target branch to main.

info

At this point the old runner is no longer running jobs, however, don't remove it just yet because it's your fail back if anything goes wrong in main, revert the agent tag change and it should continue on the old version

Cleaning up

Step 1 - Stop and remove the old runner

Export the helper variables:

export OLD_AGENT_NAME=<YOUR_OLD_RUNNER_AGENT_NAME>

Stop and remove old runner:

docker stop $OLD_AGENT_NAME
docker rm $OLD_AGENT_NAME

Step 2 - Remove image

Clean up old images using docker rmi IMAGE:TAG, where IMAGE:TAG is the old runner image stopped and removed in Step 3.

Example

docker rmi dataopslive/dataops-runner:v13.8.0

note

Installing a runner without an IMAGE_TAG will result in Docker using the latest tag instead. In that case, you will need to take care in identifying the correct image to remove.