Alternative Authentication Methods

PriPrev

Overview

It is critical for any development environment, whether local or cloud-based, to access target environments. In the case of DataOps development, this primarily means access to Snowflake.

Storing credentials in the DDE Cloud

The default and most straightforward method, as detailed in the setup instructions, is to store the five key variables in the DDE Cloud. However, keeping some or all of these in the DDE Cloud may not be desirable for some organizations. Therefore some other methods are described on this page. These will be extended over time.

Manual input

Some users/organizations will want to keep all their credentials in some local/other stores and input these each time a DDE Cloud workspace is instantiated. The /dataops-cde/scripts/dataops_cde_init.sh script looks for a file called /ephemeral_creds/creds.env. If found, it loads the contents of this file as environment variables into each command/terminal.

Safe storage

The /ephemeral_creds/ is a RAM disk — it looks like a file system store, but it exists in volatile memory only. It is never written to a physical disk, never stored, and never backed up anywhere. Not persisting it is for security reasons. When a workspace times out (after 30 minutes of no use), the file system is stored as an inactive workspace for you to come back to with all your actual file changes still present. However, the RAM and your credentials drop when a workspace times out. Therefore, you must re-add these following the same procedure when you restart a stopped workspace.

The easiest way to populate /ephemeral_creds/creds.env is like this:

export DBT_ENV_SECRET_ACCOUNT="<account name>"
export DBT_ENV_SECRET_PASSWORD="<password>"
export DBT_ENV_SECRET_USER="<username>"
export DBT_ENV_ROLE="DATAOPS_WRITER" # In a default project, this will be DATAOPS_WRITER
export DBT_ENV_WAREHOUSE="DATAOPS_TRANSFORMATION" # In a default project, this will be DATAOPS_TRANSFORMATION
printenv | grep DBT_ENV_ | awk '{print "export "$1}' > /ephemeral_creds/creds.env

When this is populated, you will see output like this:

It's possible to mix this method with the default. For example, you could store DBT_ENV_SECRET_ACCOUNT, DBT_ENV_SECRET_USER, DBT_ENV_ROLE, and DBT_ENV_WAREHOUSE using the default method and only add the password manually, e.g.:

export DBT_ENV_SECRET_PASSWORD="<password>"
printenv | grep DBT_ENV_ | awk '{print "export "$1}' > /ephemeral_creds/creds.env