Setup

PriPrev

danger

The DDE Cloud is in private preview. Therefore, the service may be unavailable for periods of time. There also may be occasions where service elements are rebuilt, and things like your stored credentials are removed, and you will need to re-add. This poses no risk to the rest of DataOps and is completely isolated.

As good development practice dictates, whether developing on local machines or in the cloud, never rely on these for safe storage of configuration/code. Always create a new branch before any changes, and regularly commit your changes to this branch.

Initial setup within the DataOps SaaS application

To let DDE Cloud automatically work with your DataOps project so that you can start coding right away, you must first do the below setup steps

1. Under the top-level Projects menu, click on your chosen project to open its details.

2. From the project menu, click DataOps CDE.

   

3. If this is the first time you have done this, you will see the confirmation message that displays. Click Enable DataOps Cloud Development Environment.

   

   This brings you back to your project, but now the DataOps CDE is enabled for you.

4. Create a .gitpod.yml file in the root directory of your repository.

   tasks:
     - name: Setup
     - before: |
         /dataops-cde/scripts/dataops_cde_setup.sh
   image: dataopslive/dataops-gitpod-workspace:5-stable

   tasks defines how the DDE Cloud prepares and builds your project and how it starts the project's development server, and image represents the docker image used for workspaces.

   note

   The DDE Cloud works per branch, so this file must exist in each branch. Once this file works its way into your main branches, this isn't an issue. Yet when starting, if you open the DDE Cloud and it doesn't look or behave as you expect, it's very likely that you opened a branch without this file configured correctly.

5. Ensure that the .gitignore file in the repository root includes the following:

   dataops/modelling/dbt_packages
   .vscode/settings.json
   snowflake.log
   note

   In the same way, the .gitignore file with this code should be present for each of your branches. Otherwise, the cloud platform will not look or behave as you expect.

Initial setup with DDE Cloud

1. Click DataOps CDE again to open up a workspace in the DataOps CDE environment. If you haven't done this recently, you may be asked to reauthenticate:

   

2. In the new window, click Continue with app.data....

   The first time you authenticate, you may see (although this requirement is being removed in newer DDE Cloud Releases):

   

   You must click Authorized to allow the DDE Cloud environment to authenticate back to the DataOps SaaS platform.

3. In the dialog box that opens, only select the default VS Code Browser.

   

   Other editors may not work and certainly will not have the optimized DataOps Developer Experience. You should now see the DDE Cloud that looks like this:

   

Initial setup of credentials

Credentials for working with Snowflake are captured as variables.

1. Set the key variables for the environment.

   info

   These credentials will be used to communicate to Snowflake for the MATE model test and execution. The credentials set in the variables should not represent the same user as the service account you are using for DataOps pipelines.

   We recommended that you use the same role as the service account. If you were to test and run models from the DDE Cloud using a different role, it might not have the correct permissions when you commit this code, and the DataOps pipeline job tries to access them.

   You can set the variables from the DDE Cloud Admin UI https://code.dataops.live/variables, or https://code.dev.dataops.live/variables for the preview environment.

   You can also get here from within the DDE Cloud workspace by clicking Gitpod in the bottom left and then Gitpod: Open Settings from the dropdown list:

   

   And selecting Variables from the left panel of the new window.

   

   Read more about Variables and Scopes.

   Alternatively, you can set the variables from a terminal within DDE Cloud by using the gp env command, e.g.:

    gp env DBT_ENV_SECRET_ACCOUNT="<account name>"
    gp env DBT_ENV_SECRET_PASSWORD="<password>"
    gp env DBT_ENV_SECRET_USER="<username>"
    gp env DBT_ENV_ROLE="DATAOPS_WRITER" # In a default project this will be DATAOPS_WRITER
    gp env DBT_ENV_WAREHOUSE="DATAOPS_TRANSFORMATION" # In a default project this will be DATAOPS_TRANSFORMATION

   which you do directly in the DDE Cloud terminal:

   

   However you enter your variables, you should end up with them looking similar to this:

   

2. Once you set these variables, you must start a new workspace to pick up these changes. The easiest way to do this is to delete the workspace and create a new one.

Optional MATE configuration

The DDE Cloud builds a profiles.yml file assuming the profile name is snowflake_operations. If this is not the case for you, you can set DBT_PROFILE_NAME. For example:

gp env DBT_PROFILE_NAME=my_snowflake_profile_name

Testing this

Congratulations, you should now be fully set up to use the DDE Cloud. You can get into a clean, ready-to-develop workspace by clicking DDE Cloud within seconds.