Skip to main content

Secrets Manager Orchestrator

TypePre-Set
Image$DATAOPS_SECRETSMANAGER_RUNNER_IMAGE

The pre-set orchestrator DataOps Secrets Manager Orchestrator allows a DataOps pipeline to retrieve passwords, keys, and other sensitive information from a remote Secrets Manager service and seamlessly adds them to the pipeline's vault.

This orchestrator currently supports:

Secrets are stored in and managed by a third-party remote Secrets Manager rather than on the orchestrator host. The Secrets Manager Orchestrator is configured with the secrets' location to fetch them and insert them into the DataOps Vault at runtime, allowing access to all the jobs in the pipeline.

Usage

You can use the orchestrator in different ways:

1. Basic Usage

pipelines/includes/local_includes/secrets_management_jobs/secrets_manager_load.yml
"Load Secrets":
extends:
- .agent_tag
stage: "Vault Initialisation"
image: $DATAOPS_SECRETSMANAGER_RUNNER_IMAGE
variables:
script:
- /dataops
icon: ${SECRETSMANAGER_ICON}

2. Using JSON

Store JSON objects in a secret (an entity within the secrets manager) using any supported technologies rather than individual values. For example, use the parameter SECRETS_EXPAND_JSON to enable the orchestrator to expand and merge data from a JSON object into the DataOps Vault, as follows:

  1. Set SECRETS_EXPAND_JSON to 1 (or True)
  2. Add an example secret named SNOWFLAKE and its values
SNOWFLAKE
{
"TRANSFORM": {
"USERNAME": "DATAOPS_ADMIN",
"PASSWORD": "abcde12345"
}
}

The result in the Vault will look like this:

SNOWFLAKE:
TRANSFORM:
USERNAME: DATAOPS_ADMIN
PASSWORD: abcde12345

Supported Secrets Managers

We currently support the following Secrets Managers:

AWS Secrets Manager

To correctly merge secrets into the vault, the key names must be a fully-namespaced vault path in the usual dotted notation. For example, the keys in the AWS secret must be SNOWFLAKE.TRANSFORM.USERNAME and SNOWFLAKE.TRANSFORM.PASSWORD, as displayed in figure 1 below, to end up with a vault that looks like this:

SNOWFLAKE:
TRANSFORM:
USERNAME: DATAOPS_ADMIN
PASSWORD: abcde12345

Figure 1: AWS Secret Keys

secrets_mgr_example.png __shadow__

To retrieve a single secret from the AWS Secrets Manager, set the value of SECRETS_SELECTION to the secret's name (as displayed in the AWS console).

AWS SSM Parameter Store

To correctly merge secrets into the vault from the AWS SSM Parameter Store, the parameter names must match a fully-namespaced vault path (with optional prefix) that will automatically have the slashes replaced with dots.

For example, a parameter named /SNOWFLAKE/TRANSFORM/USERNAME is stored in the vault under the key SNOWFLAKE.TRANSFORM.USERNAME.

If the parameter name has a prefix like /dataops/SNOWFLAKE/TRANSFORM/USERNAME, you can remove this by using the variable SECRETS_STRIP_PREFIX (use the value /dataops/ for the above example key).

To select a subset of parameters using a path prefix, set the SECRETS_SELECTION value to the path. For example:

SECRETS_SELECTION: /dataops/
SECRETS_STRIP_PREFIX: /dataops/

Azure Key Vault

To correctly merge secrets into the vault, the parameter names must match a fully-namespaced vault path. As secret names in Key Vault cannot contain dots, dashes must be used as separators.

For example, a secret named SNOWFLAKE-TRANSFORM-USERNAME will be stored in the vault under the key SNOWFLAKE.TRANSFORM.USERNAME.

The default behavior of the secrets manager orchestrator is to retrieve all secrets in the vault specified. However, it is possible to retrieve only a single secret by setting the variable SECRETS_SELECTION to the secret's name.

HashiCorp Vault

As with the AWS services, secret names can use slash separators which will be converted to dots when the secrets are loaded into DataOps. Currently, the Vault integration supports KV version 1 and KV version 2 secrets. When using KV version 2, DataOps by default pulls the latest version of any specific secret.

Use the parameter SECRETS_SELECTION to return a subset of the secrets in the selected mount point.

Authentication Sequence

When you add multiple authentication parameters, they will be tried in the following order:

  1. JWT
  2. Token
  3. Username/password

Prerequisites to Using JWT

If you use the JWT provided by DataOps in the variable CI_JOB_JWT, make sure to have the following configuration in HashiCorp Vault:

  • JWT authentication enabled and configured to use jwks_url="https://app.dataops.live/-/jwks" and bound_issuer="app.dataops.live"
  • A policy that allows list and read capabilities to the secrets that DataOps will be using
  • A role that uses the above policy and adds bound claims to the project/namespace IDs, branch/environment as required
Example JWT Policy
{
"role_type": "jwt",
"policies": ["dataops_read"],
"token_explicit_max_ttl": 60,
"user_claim": "user_email",
"bound_claims_type": "glob",
"bound_claims": {
"project_id": "12345"
}
}

You can view the token by using this example job:

Example JWT Job
Example JWT Job:
extends: .agent_tag
image: $DATAOPS_UTILS_RUNNER_IMAGE
stage: Pipeline Initialisation
script:
- echo "$CI_JOB_JWT" > CI_JOB_JWT.txt
artifacts:
paths:
- CI_JOB_JWT.txt

Supported Parameters

The following sections list the details of the below parameters:

General Parameters

ParameterRequired/DefaultDescription
SECRETS_EXPAND_JSONREQUIRED, defaults to FalseHandle compound secrets stored as JSON by merging the whole structure into the DataOps Vault
SECRETS_MANAGERREQUIRED, defaults to AWS_SECRETS_MANAGERSpecify one of the values: SSM Parameter Store AWS_PARAMETER_STORE, Azure Key Vault AZURE_KEY_VAULT, HashiCorp Vault HASHICORP_VAULT
SECRETS_SELECTIONOptionalSpecify a name to retrieve a single secret (AWS Secrets Manager) or the name prefix to retrieve (AWS Parameter Store), otherwise all available secrets are retrieved
SECRETS_SELECTION_FILTEROptionalSpecify a prefix or substring to match against secret names and retrieve a subset of secrets
SECRETS_STRIP_PREFIXOptionalRemove a prefix from key names (SSM only)

AWS-Specific Parameters

ParameterRequired/DefaultDescription
SECRETS_AWS_REGIONREQUIRED, defaults to eu-west-2Use this AWS region
SECRETS_AWS_USE_ROLEREQUIRED, defaults to FalseSet to True to use implicit authentication from this orchestrator's EC2 instance role
SECRETS_AWS_ACCESS_KEY_LOCATIONREQUIRED, defaults to AWS.DEFAULT.S3_KEYUse keys from this vault location when authenticating with AWS
SECRETS_AWS_SECRET_KEY_LOCATIONREQUIRED, defaults to AWS.DEFAULT.S3_SECRETUse keys from this vault location when authenticating with AWS

Azure-Specific Parameters

ParameterRequired/DefaultDescription
SECRETS_AZURE_CLIENT_SECRET_LOCATIONREQUIRED, defaults to AZURE.DEFAULT.CLIENT_SECRETIf not using a managed identity, this is the DataOps Vault location of the client secret for authentication
SECRETS_AZURE_USE_MANAGED_IDENTITYREQUIRED, defaults to 1Use the managed identity associated with the orchestrator's VM to authenticate with the Key Vault. Set to 0 (zero) to use client secret instead.
SECRETS_AZURE_TENANT_IDOptionalIf not using a managed identity, this is the Azure tenant ID
SECRETS_AZURE_CLIENT_IDOptionalIf not using a managed identity, this is the Azure client ID
SECRETS_AZURE_KEY_VAULT_URLOptionalURL to the Key Vault instance to access

HashiCorp Vault-Specific Parameters

ParameterRequired/DefaultDescription
SECRETS_HASHICORP_VAULT_MOUNT_POINTREQUIREDVault mount point to use
SECRETS_HASHICORP_VAULT_URLREQUIREDURL of the Vault instance that will be used
SECRETS_HASHICORP_VAULT_NAMESPACEOptionalVault namespace to connect to
SECRETS_HASHICORP_VAULT_JWT_ROLEOptionalIf specified, JWT authentication will be attempted using the CI_JOB_JWT and the role with this name
SECRETS_HASHICORP_VAULT_TOKENOptionalIf provided, token authentication will be attempted
SECRETS_HASHICORP_VAULT_USERNAMEOptionalIf provided (along with SECRETS_HASHICORP_VAULT_PASSWORD), userpass authentication will be attempted
SECRETS_HASHICORP_VAULT_PASSWORDOptionalRequired (along with SECRETS_HASHICORP_VAULT_USERNAME) for userpass authentication
SECRETS_HASHICORP_VAULT_KV_VERSIONOptionalDefault value is 2, set it to 1 only if you use a version 1 vault mount point

Example Jobs

This job is the standard Load Secrets job from all pipelines, as defined in the DataOps Reference Project.

pipelines/includes/local_includes/secrets_manager_jobs/load_secrets.yml
"Load Secrets":
extends:
- .agent_tag
stage: "Vault Initialisation"
image: $DATAOPS_SECRETSMANAGER_RUNNER_IMAGE
variables:
script:
- export DATAOPS_TEMPLATES_DIR=$(dirname $DATAOPS_VAULT_CONTENT)
- cp $DATAOPS_REFERENCE_PROJECT_DIR/$DATAOPS_REFERENCE_PROJECT_NAME/runner-scripts/22-vault-template /runner-scripts/
- /dataops
icon: ${SECRETSMANAGER_ICON}