How to Create a Custom Before Script

The before_script runs ahead of each job's main script block. The default lives in the DataOps Reference Project. It sets various dynamic variables, such as DATAOPS_DATABASE and variables relating to branch/environment names, which are then available to the apps and scripts running in the job's main part.

It is possible to create an additional before_script, either in a single project or in a custom reference project, that adds to or overrides selected DataOps variables to enable custom behavior.

Creating the script

Create a file in your project (or reference project) that contains a shell script (Bash) to set variables and perform other logic. For variables to be available to the subsequent script, your script must echo these out to stdout.

Please note

Don't echo anything else to stdout, as this breaks the variable export mechanism. If you need to print any text to the user, do this via stderr, e.g.:

>&2 echo "This is a note for the logs..."

Here is an example script that overrides the environment suffix for account-level database objects in production:

#!/usr/bin/env bash

export DATAOPS_DATABASE_MASTER=$DATAOPS_PREFIX
if [[ $DATAOPS_ENV_NAME == $DATAOPS_ENV_NAME_PROD ]]; then
  >&2 echo "Running in main/PROD: setting DATAOPS_DATABASE to $DATAOPS_PREFIX"
  export DATAOPS_DATABASE=$DATAOPS_DATABASE_MASTER
  export DATABASE=$DATAOPS_DATABASE
fi

echo "DATAOPS_DATABASE=$DATAOPS_DATABASE"
echo "DATABASE=$DATABASE"
echo "DATAOPS_DATABASE_MASTER=$DATAOPS_DATABASE_MASTER"

Your additional before_script always executes after the DataOps before_script, so you can redefine the pre-computed values of built-in variables as above.

Remember...

Your new before_script is a standard shell script, so will need to have the execute bit set. Otherwise, you will see permission-denied errors when running pipelines. You can use the chmod tool to set this in our browser-based development environment (DevReady) or your local development environment.

Enabling script for projects

In most cases, you require only a single variable setting to enable your custom before_script.

In a single project

Edit your project's variables.yml file and add the following variable definition using the path to your script defined above.

pipelines/includes/config/variables.yml

variables:
  ...
  DATAOPS_EXTRA_BEFORE_SCRIPTS: $CI_PROJECT_DIR/scripts/before_script.sh
  ...

In a reference project

If you're using a custom reference project, it makes sense to add your custom before script here, such that it takes effect in all the projects that use your reference project.

Based on the fact that the reference projects clone into a specific location in the pipeline's runtime workspace, the configuration path is a little different from the single project configuration:

pipelines/includes/config/variables.yml

variables:
  ...
  DATAOPS_EXTRA_BEFORE_SCRIPTS: $CI_PROJECT_DIR/reference-projects/my-reference-project/scripts/before_script.sh
  ...

Replace my-reference-project and the path following it with the correct name and path for your reference project.