The DDE Cloud is in private preview. Therefore, the service may be unavailable for periods of time. There also may be occasions where service elements are rebuilt, and things like your stored credentials are removed, and you will need to re-add. This poses no risk to the rest of DataOps and is completely isolated.
As good development practice dictates, whether developing on local machines or in the cloud, never rely on these for safe storage of configuration/code. Always create a new branch before any changes, and regularly commit your changes to this branch.
Initial setup within the DataOps SaaS application
To let DDE Cloud automatically work with your DataOps project so that you can start coding right away, you must first do the below setup steps
Under the top-level Projects menu, click on your chosen project to open its details.
From the project menu, click DataOps CDE.
If this is the first time you have done this, you will see the confirmation message that displays. Click Enable DataOps Cloud Development Environment.
This brings you back to your project, but now the DataOps CDE is enabled for you.
.gitpod.ymlfile in the root directory of your repository.
- name: Setup
- before: |
tasksdefines how the DDE Cloud prepares and builds your project and how it starts the project's development server, and
imagerepresents the docker image used for workspaces.note
The DDE Cloud works per branch, so this file must exist in each branch. Once this file works its way into your main branches, this isn't an issue. Yet when starting, if you open the DDE Cloud and it doesn't look or behave as you expect, it's very likely that you opened a branch without this file configured correctly.
Ensure that the
.gitignorefile in the repository root includes the following:
In the same way, the
.gitignorefile with this code should be present for each of your branches. Otherwise, the cloud platform will not look or behave as you expect.
Initial setup with DDE Cloud
Click DataOps CDE again to open up a workspace in the DataOps CDE environment. If you haven't done this recently, you may be asked to reauthenticate:
In the new window, click Continue with app.data....
The first time you authenticate, you may see (although this requirement is being removed in newer DDE Cloud Releases):
You must click Authorized to allow the DDE Cloud environment to authenticate back to the DataOps SaaS platform.
In the dialog box that opens, only select the default VS Code Browser.
Other editors may not work and certainly will not have the optimized DataOps Developer Experience. You should now see the DDE Cloud that looks like this:
Initial setup of credentials
Credentials for working with Snowflake are captured as variables.
Set the key variables for the environment.info
These credentials will be used to communicate to Snowflake for the MATE model test and execution. The credentials set in the variables should not represent the same user as the service account you are using for DataOps pipelines.
We recommended that you use the same role as the service account. If you were to test and run models from the DDE Cloud using a different role, it might not have the correct permissions when you commit this code, and the DataOps pipeline job tries to access them.
You can set the variables from the DDE Cloud Admin UI https://code.dataops.live/variables, or https://code.dev.dataops.live/variables for the preview environment.
You can also get here from within the DDE Cloud workspace by clicking Gitpod in the bottom left and then Gitpod: Open Settings from the dropdown list:
And selecting Variables from the left panel of the new window.
Read more about Variables and Scopes.
Alternatively, you can set the variables from a terminal within DDE Cloud by using the
gp envcommand, e.g.:
gp env DBT_ENV_SECRET_ACCOUNT="<account name>"
gp env DBT_ENV_SECRET_PASSWORD="<password>"
gp env DBT_ENV_SECRET_USER="<username>"
gp env DBT_ENV_ROLE="DATAOPS_WRITER" # In a default project this will be DATAOPS_WRITER
gp env DBT_ENV_WAREHOUSE="DATAOPS_TRANSFORMATION" # In a default project this will be DATAOPS_TRANSFORMATION
which you do directly in the DDE Cloud terminal:
However you enter your variables, you should end up with them looking similar to this:
Once you set these variables, you must start a new workspace to pick up these changes. The easiest way to do this is to delete the workspace and create a new one.
Optional MATE configuration
The DDE Cloud builds a
profiles.yml file assuming the profile name is
snowflake_operations. If this is not the case for you, you can set
DBT_PROFILE_NAME. For example:
gp env DBT_PROFILE_NAME=my_snowflake_profile_name
Congratulations, you should now be fully set up to use the DDE Cloud. You can get into a clean, ready-to-develop workspace by clicking DDE Cloud within seconds.