Skip to main content

DataOps Docker runner installation

Step 1 - Fetch registration tokens from DataOps

The Registration Token is generated automatically in DataOps.live and is used to link together the runner you are about to create with your specific DataOps Project or Group.

note

These registration tokens are scoped

Follow these steps to obtain your Registration Token:

  1. Connect to the DataOps Platform UI
  2. Open the group (preferred) or project you want to create the runner for
    1. choosing the group makes the runner available to all projects in that group
  3. Go to Settings → CI/CD
  4. Find the Runners section and click Expand
  5. Inside the Specific runners section under Set up a group runner manually, you will find the registration token
  6. Copy it

Group Runner Token

Step 2 - Connect to Docker Hub

On your runner host command line run:

docker login --username  dataopsreadserviceuser --password qf2h9372fg3ioug384
caution

Try prefixing the command with sudo if it doesn't work. However, the usermod -aG docker hasn't been done correctly and may cause future issues.

note

The dataopsreadserviceuser is a read-only service account to allow you to pull the dataopslive/dataops-runner image.

Step 3 - Register the runner

To make running commands consistent, copy and modify the following environment variables and export them to your command line environment.

export IMAGE_TAG='latest'  # DockerHub image tag and the version of the runner
export IMAGE=dataopslive/dataops-runner:$IMAGE_TAG # DockerHub repo path and tag
export AGENT_TAG=<YOUR_RUNNER_NAME> # DataOps runner tag
export AGENT_NAME=$AGENT_TAG-$IMAGE_TAG # Runner full name
export DATAOPS_URL=https://app.dataops.live/ # DataOps.live SaaS platform
export REGISTRATION_TOKEN=<YOUR_REGISTRATION_TOKEN> # Token from the UI in step 1

make sure you adjust the IMAGE_TAG value to the value provided as part of the release notes if you need to choose a specific version other than the latest.

then:

docker run --rm -v /srv/dataops-runner-$AGENT_NAME/config:/etc/gitlab-runner $IMAGE register --non-interactive --executor "docker" --docker-image dataopslive/dataops-utils-orchestrator:5-stable --url "$DATAOPS_URL" --registration-token "$REGISTRATION_TOKEN" --description "$AGENT_NAME" --tag-list "$AGENT_TAG" --run-untagged="false" --locked="true" --access-level="not_protected"

You should now go back to the UI and, in the same location, see the new one you created, for example:

My New Runner

The runner has been given a random identifier that cannot be changed. The 'Last Contact' 'Never' in the example above indicates that the new runner has registered but is not yet running.

Step 4 - Update runner configuration

Several key configurations are required that are not set using the standard register command. Set these by running the following commands on your server:

# allow the agent to run up to 8 concurrent jobs
sudo sed -i 's/concurrent = .*/concurrent = 8/' /srv/dataops-runner-$AGENT_NAME/config/config.toml
# have agent poll server every 1 second
sudo sed -i 's/check_interval = .*/check_interval = 10/' /srv/dataops-runner-$AGENT_NAME/config/config.toml
# mounts the /app into /local_config inside every runner that is started by this agent.
sudo sed -i 's/ volumes =.*$/ volumes = ["\/app:\/local_config:rw","\/agent_cache:\/agent_cache:rw", "\/secrets:\/secrets:ro"]/' /srv/dataops-runner-$AGENT_NAME/config/config.toml

Step 5 - Start the runner

docker run -d --name $AGENT_NAME --restart always -v /srv/dataops-runner-$AGENT_NAME/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock $IMAGE

or, for extra debugging:

docker run -e DEBUG=true -d --name $AGENT_NAME --restart always -v /srv/dataops-runner-$AGENT_NAME/config:/etc/gitlab-runner -v /var/run/docker.sock:/var/run/docker.sock $IMAGE

You should now see:

My New Runner

Test this out!

At this point, you should be able to run a pipeline (e.g., the full-ci.yml created from the template project). If the first job in the pipeline changes to a blue pie, the job is running on your Runner, everything is connected, and you can move on.

If not, check your setup and, if needed, contact support@dataops.live.

initial job running

Start and Stop the DataOps Runner

Once you have completed Steps 1 to 5 of the initial runner setup, you don't need to repeat the steps every time to start or stop the runner.

To start:

export AGENT_TAG=my-documentation-runner   # Change this to your desired name xxx-runner
docker start $AGENT_TAG

To stop:

export AGENT_TAG=my-documentation-runner   # Change this to your desired name xxx-runner
docker stop $AGENT_TAG

Credentials and Secrets

The DataOps Platform/Runner model's basic security model is that the platform and repository contain all the information about what should be done. But they have none of the credentials actually to do it. These credentials are stored on your DataOps Runner so that no one else has access to them.

tip

This process is described in more detail in our vault concepts section, and it is advisable to read this page before proceeding any further.

DataOps Vault Setup

DataOps requires a directory from the host called /secrets with a /secrets/vault.yml and /secrets/vault.salt

To create the minimum base vault configuration simply run:

sudo mkdir -p /secrets
echo {} | sudo tee /secrets/vault.yml > /dev/null
echo $RANDOM | md5sum | head -c 20 | sudo tee /secrets/vault.salt > /dev/null

Full details on when and how to add to these are in our vault concepts section.