Skip to main content

How to Create a Custom Before Script

The before script runs ahead of each job's main script block. The default lives in the DataOps Reference Project. It sets various dynamic variables, such as DATAOPS_DATABASE and variables relating to branch/environment names, which are then available to the apps and scripts that run in the main part of the job.

It is possible to create an additional before script, either in a single project or in a custom reference project, that add to or override selected DataOps variables to enable custom behavior.

Create the Script

Create a file in your project (or reference project) that contains a shell script (Bash) to set variables and perform other logic. For variables to be available to the subsequent script, your script must echo these out to stdout.

Please note

Don't echo anything else out to stdout, as this breaks the variable export mechanism. If you need to print any text to the user, do this via stderr, e.g.:

>&2 echo "This is a note for the logs..."

Here is an example script that overrides the environment suffix for account-level database objects in production:

scripts/before_script.sh
#!/usr/bin/env bash

export DATAOPS_DATABASE_MASTER=$DATAOPS_PREFIX
if [[ $DATAOPS_ENV_NAME == $DATAOPS_ENV_NAME_PROD ]]; then
>&2 echo "Running in master/PROD: setting DATAOPS_DATABASE to $DATAOPS_PREFIX"
export DATAOPS_DATABASE=$DATAOPS_DATABASE_MASTER
export DATABASE=$DATAOPS_DATABASE
fi

echo "DATAOPS_DATABASE=$DATAOPS_DATABASE"
echo "DATABASE=$DATABASE"
echo "DATAOPS_DATABASE_MASTER=$DATAOPS_DATABASE_MASTER"

Your additional before script always executes after the DataOps before script, so you can redefine the pre-computed values of built-in variables as above.

Enable for Projects

In most cases, you require just a single variable setting to enable your custom before script.

In a Single Project

Edit your project's variables.yml file and add the following variable definition, using the path to your script defined above.

pipelines/includes/config/variables.yml
variables:
...
DATAOPS_EXTRA_BEFORE_SCRIPTS: $CI_PROJECT_DIR/scripts/before_script.sh
...

In a Reference Project

If you're using a custom reference project, it makes sense to add your custom before script here, such that it takes effect in all the projects that use your reference project.

Based on the fact that the reference projects clone into a specific location in the pipeline's runtime workspace, the configuration path is a little different from the single project configuration:

pipelines/includes/config/variables.yml
variables:
...
DATAOPS_EXTRA_BEFORE_SCRIPTS: $CI_PROJECT_DIR/reference-projects/my-reference-project/scripts/before_script.sh
...

Replace my-reference-project and the path following it with the correct name and path for your reference project.