Skip to main content

Alternative Authentication Methods

Feature release status badge: PriPrev
PriPrev

Overview

It is critical for any development environment, whether local or cloud-based, to access target environments. In the case of DataOps development, this primarily means access to Snowflake.

Storing credentials in the DataOps CDE

The default and most straightforward method, as detailed in the CDE Setup and Configuration, is to store the five key variables in the DataOps CDE. However, keeping some or all of these in the DataOps CDE may not be desirable for some organizations. Therefore some other methods are described on this page. These will be extended over time.

Manual input

Some users/organizations will want to keep all their credentials in some local/other stores and input these each time a DataOps CDE workspace is instantiated. The /dataops-cde/scripts/dataops_cde_init.sh script will look for a file called /ephemeral_creds/creds.env. If found, it will load the contents of this file as environment variables into each command/terminal.

Safe storage

The /ephemeral_creds/ is a RAM disk - it looks like a file system store, but it exists in volatile memory only. It is never written to a physical disk, never stored, and never backed up anywhere. Not persisting it is for security reasons. When a workspace times out (after 30 minutes of no use), the file system is stored as an inactive workspace for you to come back to with all your actual file changes still present. However, the RAM drops along with your credentials when a workspace times out. Therefore you will need to re-add these following the same procedure when you restart a stopped workspace.

The easiest way to populate /ephemeral_creds/creds.env is like this:

export DBT_ENV_SECRET_ACCOUNT="<account name>"
export DBT_ENV_SECRET_PASSWORD="<password>"
export DBT_ENV_SECRET_USER="<username>"
export DBT_ENV_ROLE="DATAOPS_WRITER" # In a default project, this will be DATAOPS_WRITER
export DBT_ENV_WAREHOUSE="DATAOPS_TRANSFORMATION" # In a default project, this will be DATAOPS_TRANSFORMATION
printenv | grep DBT_ENV_ | awk '{print "export "$1}' > /ephemeral_creds/creds.env

When this is populated, you will see output like this:

cde alternative method __shadow__

It's possible to mix this method with the default. For example, you could store DBT_ENV_SECRET_ACCOUNT, DBT_ENV_SECRET_USER, DBT_ENV_ROLE, and DBT_ENV_WAREHOUSE using the default method and only add the password manually, e.g.:

export DBT_ENV_SECRET_PASSWORD="<password>"
printenv | grep DBT_ENV_ | awk '{print "export "$1}' > /ephemeral_creds/creds.env