DDE DevReady

PriPrev

danger

The DDE is in private preview. Therefore, the service may be unavailable for periods of time. There also may be occasions where service elements are rebuilt, and things like your stored credentials are removed, and you will need to re-add. This poses no risk to the rest of DataOps and is completely isolated.

As good development practice dictates, whether developing on local machines or in the cloud, never rely on these for safe storage of configuration/code. Always create a new branch before any changes, and regularly commit your changes to this branch.

What is DDE DevReady?

The DDE DevReady is a browser-based tool that automates dev environments for each of your tasks in seconds. It gives you highly optimized data development experience for key DataOps use cases without installation and helps you speed up the development process when writing software.

DevReady setup

Initial setup within the DataOps SaaS platform

To let the DDE DevReady automatically work with your DataOps project so that you can start coding immediately, you must first do the below setup steps.

1. Under the top-level Projects menu, click your chosen project to open its details.

2. From the project menu, click DataOps DDE.

   

   You will see a confirmation message if this is your first time doing this.

3. Click Enable DataOps Cloud Development Environment.

   

   This brings you back to your project, but now the DDE DevReady is enabled.

4. Create a .gitpod.yml file in the root directory of your repository.

   tasks:
     - name: Setup
     - before: |
         /dataops-dde/scripts/dataops-dde-setup.sh
   image: dataopslive/dataops-development-workspace:5-stable

   tasks defines how the DDE DevReady prepares and builds your project and how it starts the project's development server, and image represents the docker image used for workspaces.

   note

   The DDE DevReady works per branch, so this file must exist in each one. This isn't an issue once this file is in your main branch. Yet when starting, if you open the DDE DevReady and it doesn't look or behave as you expect, it's very likely that you opened a branch without this file configured correctly.

5. Ensure that the .gitignore file in the repository root includes the following:

   .gitignore

   dataops/modelling/dbt_packages
   snowflake.log
   note

   In the same way, the .gitignore file with this code should be present for each of your branches. Otherwise, the platform will not look or behave as you expect.

6. Register the file associations that are using template rendering with the Better Jinja extension:

   .vscode/settings.json

   {
     "files.associations": {
       "*.yml": "jinja-yaml",
       "*.yaml": "jinja-yaml",
       "*.html": "jinja-html",
       "*.sql": "jinja-sql"
     }
   }

7. Close and delete the workspace via DDE DevReady workspace management, and start a new workspace.

   Whenever you make a change in .gitpod.yml, you must close and delete the current workspace and start a new one to retrieve the changes.

8. Verify if all works well by validating the SOLE configuration. To do this:

   1. Export the environment variables like below — followed by a restart of the workspace.

      terminal

         gp env DATAOPS_SOLE_ACCOUNT=<your Snowflake account>
         gp env DATAOPS_SOLE_USERNAME=<your Snowflake username>
         gp env DATAOPS_SOLE_PASSWORD=<your Snowflake password>
         gp env DATAOPS_SOLE_ROLE=<your Snowflake role>
         gp env DATAOPS_SOLE_WAREHOUSE=<your Snowflake warehouse>
      note

      The old variable names e.g. starting with DBT_ are still compatible and functional. However, we recommend transitioning to the above new names e.g. starting with DATAOPS_MATE_ in your setup to stay current with the latest updates.

   2. Once you have these in place, run the command below.

      terminal

      sole-validate

Any SOLE compilation error will show on the VS Code console. For more information, check out SOLE compilation and validation.

Giving DevReady access to your DataOps account

1. Click DataOps DDE again to open up a workspace in the DataOps Development Environment. If you haven't done this recently, you may be asked to reauthenticate:

   

2. In the new window, click Continue with app.data....

   The first time you authenticate, you may see the below message (although this requirement is being removed in newer DDE DevReady Releases):

   

   Click Authorized to allow the DDE DevReady environment to authenticate back to the DataOps SaaS platform.

3. In the dialog box that opens, only select the default VS Code Browser.

   

   Other editors may not work and certainly will not have the optimized DataOps Developer Experience. You should now see the DDE DevReady that looks like this:

   

Giving DevReady access to Snowflake

The DataOps.live extension provides a step-by-step guide to help you set up the Snowflake connection for DevReady, simplifying your experience in the development environment.

The setup walkthrough offers detailed instructions on how to give access to your Snowflake account. It also offers a more secure way of storing your credentials.

When you open the development environment for the first time without setting any credentials beforehand, a notification message prompts you to open the setup walkthrough.

A two-step getting started displays on the VS Code. Following the on-screen guidance lets you:

• Choose the dbt version for your workspace
• Set your Snowflake credentials

The GIF below provides a visual walkthrough, followed by a step-by-step procedure.

1. Click Choose dbt Version → Set dbt Version.

   

2. On top of the VS Code, select from the dropdown the dbt version for the workspace.

   Available versions correspond to those in the MATE orchestrator used in pipelines. You can also enter a custom dbt version bundled with MATE.

3. Click Setup Snowflake Connection → Run Setup Wizard.

4. On top of the VS Code, choose a storage method for your Snowflake credentials.

   Two options are available:

   • DataOps.live account: Offers secure storage — no need to reenter when reopening or starting a new workspace for the same project.
   • Visual Studio Code secret storage: Stores credentials within the workspace but only lasts for the lifetime of that workspace. You'll need to reenter them for a new workspace.

   For a more permanent solution without using your DataOps.live account, consider using DDE DevPod through your local VS Code client. You can set this up in your account preferences by selecting VS Code as your preferred editor.

5. Enter your Snowflake credentials following the on-screen instructions.

   These credentials are for tools like dbt in DDE to connect with your Snowflake. Once you enter them, the DDE setup will configure your environment. You only need to do this once for each workspace.

Workspaces automatically stop after 30 minutes of inactivity and get deleted if left inactive for 14 days. However, you can prevent deletion by pinning a workspace.

Optional MATE configuration

The DDE DevReady builds a profiles.yml file assuming the profile name is snowflake_operations. If this is not the case for you, you can set DBT_PROFILE_NAME. For example:

gp env DBT_PROFILE_NAME=my_snowflake_profile_name

Testing this

Congratulations, you should now be fully set up to use the DDE DevReady. You can get into a clean, ready-to-develop workspace by clicking DDE DevReady within seconds.

Workspaces and network access

The DDE DevReady manages the developer environments as workspaces. A workspace is operated as a self-contained docker image shipping with a prepackaged set of tools.

Every user's workspace is a dedicated docker container on an EKS cluster in the same VPC as the DataOps SaaS platform. The user accesses the workspace environment over the public internet. The Git operations like push or checkout use a secure connection between the DataOps SaaS and the workspace. Access to Snowflake is performed from the workspace via a direct, secure connection.

Alternative authentication methods

Overview

It is critical for any development environment, whether local or browser-based, to access target environments. In the case of DataOps development, this primarily means access to Snowflake.

Storing credentials in DevReady

The default and most straightforward method, as detailed in the DDE DevReady setup instructions, is to store the five key variables in the DDE DevReady. However, keeping some or all of these in the DDE DevReady may not be desirable for some organizations. Therefore some other methods are described on this page. These will be extended over time.

Manual input

Some users/organizations will want to keep all their credentials in some local/other stores and input these each time a DDE DevReady workspace is instantiated. The /dataops-cde/scripts/dataops_cde_init.sh script looks for a file called /ephemeral_creds/creds.env. If found, it loads the contents of this file as environment variables into each command/terminal.

Safe storage

The /ephemeral_creds/ is a RAM disk — it looks like a file system store, but it exists in volatile memory only. It is never written to a physical disk, never stored, and never backed up anywhere. Not persisting it is for security reasons.

When a workspace times out (after 30 minutes of no use), the file system is stored as an inactive workspace for you to come back to with all your actual file changes still present. However, the RAM and your credentials drop when a workspace times out. Therefore, you must re-add these following the same procedure when you restart a stopped workspace.

The easiest way to populate /ephemeral_creds/creds.env is like this:

terminal

export DATAOPS_MATE_ACCOUNT=<your Snowflake account>
export DATAOPS_MATE_PASSWORD=<your Snowflake password>
export DATAOPS_MATE_USER=<your Snowflake username>
export DATAOPS_MATE_ROLE=DATAOPS_WRITER # In a default project, this will be DATAOPS_WRITER
export DATAOPS_MATE_WAREHOUSE=DATAOPS_TRANSFORMATION # In a default project, this will be DATAOPS_TRANSFORMATION
printenv | grep DATAOPS_MATE_| awk '{print "export "$1}' > /ephemeral_creds/creds.env

note

The old variable names e.g. starting with DBT_ are still compatible and functional. However, we recommend transitioning to the above new names e.g. starting with DATAOPS_MATE_ in your setup to stay current with the latest updates.

When this is populated, you will see output like this:

It's possible to mix this method with the default. For example, you could store DATAOPS_MATE_ACCOUNT, DATAOPS_MATE_USER, DATAOPS_MATE_ROLE, and DATAOPS_MATE_WAREHOUSE using the default method and only add the password manually, e.g.:

terminal

export DATAOPS_MATE_PASSWORD="<password>"
printenv | grep DATAOPS_MATE_ | awk '{print "export "$1}' > /ephemeral_creds/creds.env