Skip to main content

Secrets Manager Orchestrator

TypePre-Set
Image$DATAOPS_SECRETSMANAGER_RUNNER_IMAGE

The pre-set orchestrator DataOps Secrets Manager Orchestrator allows a DataOps pipeline to retrieve passwords, keys, and other sensitive information from a remote Secrets Manager service and seamlessly adds them to the pipeline's vault.

This orchestrator currently supports:

These secrets are stored in and managed by a third-party remote secrets manager rather than on the orchestrator host. The Secrets Manager Orchestrator is configured with the secrets' location to fetch them and insert them into the DataOps Vault at runtime, allowing access to all the jobs in the pipeline.

Usage

There are several ways to use this orchestrator, as demonstrated in this section:

1. Basic Usage

pipelines/includes/local_includes/secrets_management_jobs/secrets_manager_load.yml
"Load Secrets":
extends:
- .agent_tag
stage: "Vault Initialisation"
image: $DATAOPS_SECRETSMANAGER_RUNNER_IMAGE
variables:
script:
- /dataops
icon: ${SECRETSMANAGER_ICON}

2. Using JSON

It is possible to store JSON objects in a secret (an entity within the secrets manager) using any supported technologies rather than individual values. For instance: use the parameter SECRETS_EXPAND_JSON to enable the orchestrator to expand and merge data from a JSON object into the DataOps Vault, as follows:

Set SECRETS_EXPAND_JSON to 1 (or True). Then, add an example secret named SNOWFLAKE and its values:

{
"MASTER": {
"USERNAME":"DATAOPS_MASTER",
"PASSWORD":"abcde12345"
}
}

The result in the Vault will look like this:

SNOWFLAKE:
MASTER:
USERNAME: DATAOPS_MASTER
PASSWORD: abcde12345

3. Supported Secrets Managers

As highlighted above, we currently support the following secrets managers:

AWS Secrets Manager

To correctly merge secrets into the vault, the key names must be a fully-namespaced vault path in the usual dotted notation. For example, the keys in the AWS secret must be SNOWFLAKE.MASTER.USERNAME and SNOWFLAKE.MASTER.PASSWORD, as displayed in figure 1 below, to end up with a vault that looks like this:

SNOWFLAKE:
MASTER:
USERNAME: DATAOPS_MASTER
PASSWORD: abcde12345

Figure 1: AWS Secret Keys

secrets_mgr_example.png __shadow__

To retrieve a single secret from the AWS Secrets Manager, set the value of SECRETS_SELECTION to the secret's name (as displayed in the AWS console).

AWS SSM Parameter Store

To correctly merge secrets into the vault from the AWS SSM Parameter Store, the parameter names must match a fully-namespaced vault path (with optional prefix) that will automatically have the slashes replaced with dots.

For example, a parameter named /SNOWFLAKE/MASTER/USERNAME will be stored in the vault under key SNOWFLAKE.MASTER.USERNAME.

If the parameter name has a prefix like /dataops/SNOWFLAKE/MASTER/USERNAME, you can remove this by using the variable SECRETS_STRIP_PREFIX (use the value /dataops/ for the above example key).

To select a subset of parameters using a path prefix, set the SECRETS_SELECTION value to the path. For example:

SECRETS_SELECTION: /dataops/
SECRETS_STRIP_PREFIX: /dataops/

Azure Key Vault

In order to correctly merge secrets into the vault, the parameter names must match a fully-namespaced vault path. As secret names in Key Vault cannot contain dots, dashes must be used as separators.

For example, a secret named SNOWFLAKE-MASTER-USERNAME will be stored in the vault under key SNOWFLAKE.MASTER.USERNAME.

The default behavior of the secrets manager orchestrator is to retrieve all secrets in the vault specified. However, it is possible to retrieve only a single secret by setting the variable SECRETS_SELECTION to the secret's name.

Supported Parameters

This section is subdivided into the following sections:

General Parameters

ParameterRequired/DefaultDescription
SECRETS_MANAGERREQUIRED, defaults to AWS_SECRETS_MANAGERSSM Parameter Store: AWS_PARAMETER_STORE, Azure Key Vault: AZURE_KEY_VAULT
SECRETS_SELECTIONOptionalSpecify a name to retrieve a single secret (AWS Secrets Manager) or the name prefix to retrieve (AWS Parameter Store), otherwise all available secrets are retrieved
SECRETS_SELECTION_FILTEROptionalSpecify a prefix or substring to match against secret names to retrieve a subset of secrets (AWS Secrets Manager only)
SECRETS_STRIP_PREFIXOptionalRemove a prefix from key names (SSM only)
SECRETS_EXPAND_JSONREQUIRED, defaults to FalseHandle compound secrets stored as JSON by merging the whole structure into the DataOps Vault

AWS-Specific Parameters

ParameterRequired/DefaultDescription
SECRETS_AWS_REGIONREQUIRED, defaults to eu-west-2Use this AWS region
SECRETS_AWS_USE_ROLEREQUIRED, defaults to FalseSet to True to use implicit authentication from this orchestrator's EC2 instance role
SECRETS_AWS_ACCESS_KEY_LOCATIONREQUIRED, defaults to AWS.DEFAULT.S3_KEYUse keys from this vault location when authenticating with AWS
SECRETS_AWS_SECRET_KEY_LOCATIONREQUIRED, defaults to AWS.DEFAULT.S3_SECRETUse keys from this vault location when authenticating with AWS

Azure-Specific Parameters

ParameterRequired/DefaultDescription
SECRETS_AZURE_USE_MANAGED_IDENTITYREQUIRED, defaults to 1Use the managed identity associated with the orchestrator's VM to authenticate with the Key Vault. Set to 0 (zero) to use client secret instead.
SECRETS_AZURE_TENANT_IDOptionalIf not using a managed identity, this is the Azure tenant ID
SECRETS_AZURE_CLIENT_IDOptionalIf not using a managed identity, this is the Azure client ID
SECRETS_AZURE_KEY_VAULT_URLOptionalURL to the Key Vault instance to access
SECRETS_AZURE_CLIENT_SECRET_LOCATIONREQUIRED, defaults to AZURE.DEFAULT.CLIENT_SECRETIf not using a managed identity, this is the DataOps Vault location of the client secret for authentication

Example Jobs

This job is actually the standard Load Secrets job from all pipelines, as defined in the DataOps Reference Project.

pipelines/includes/local_includes/secrets_manager_jobs/load_secrets.yml
"Load Secrets":
extends:
- .agent_tag
stage: "Vault Initialisation"
image: $DATAOPS_SECRETSMANAGER_RUNNER_IMAGE
variables:
script:
- export DATAOPS_TEMPLATES_DIR=$(dirname $DATAOPS_VAULT_CONTENT)
- cp $DATAOPS_REFERENCE_PROJECT_DIR/$DATAOPS_REFERENCE_PROJECT_NAME/runner-scripts/22-vault-template /runner-scripts/
- /dataops
icon: ${SECRETSMANAGER_ICON}

Project Resources

None

Host Dependencies (and Resources)

None