Skip to main content

DevReady Setup and Configuration

Feature release status badge: PubPrev
PubPrev

What is DevReady — DataOps development environment?

DevReady is a browser-based tool that automates development environments for each of your tasks in seconds. You can use DevReady with Visual Studio (VS) Code that supports full-featured Git integration.

DevReady gives you highly optimized data development experience for key DataOps use cases without installation and helps you speed up the development process when writing software.

DataOps development environment in your browser: deployment diagram !!shadow!!

Setup overview

Pre-requisites

  • Some tools that you might run in DevReady for fast feedback create local files that should not be committed to your project later. To avoid mistakes make sure the following is in your .gitignore file:

    .gitignore
    dataops/modelling/dbt_packages
    snowflake.log
  • If you are using SSO with your Snowflake account, you will need to first setup a Snowflake OAuth Security Integration.

DevReady setup

Initial setup within the data product platform

To let the DevReady automatically work with your DataOps project so that you can start coding immediately, you must first do the below setup steps.

  1. Under the top-level Projects menu, click your chosen project to open its details.

  2. Click DataOps.live | Develop from the project menu.

    DevReady menu on the platform !!shadow!!

    You will see a confirmation message if this is your first time doing this.

  3. Click Enable DataOps.live | Develop.

    enable DevReady dialog box !!shadow!!

    This brings you back to your project, but now the DevReady is enabled.

Giving DevReady access to your DataOps account

  1. Click DataOps.live | Develop from the menu of your open project to open up a workspace in the DataOps development environment. If you haven't done this recently, you may be asked to reauthenticate:

    open a workspace in the DevReady environment dialog box !!shadow!!

  2. In the new window, click Continue with app.data....

    The first time you authenticate, you may see the below message (although this requirement is being removed in newer DDE DevReady Releases):

    authentication window to the data product platform !!shadow!!

    Click Authorized to allow the DevReady environment to authenticate back to the data product platform.

  3. In the dialog box that opens, only select the default VS Code Browser.

    dialog box that lists the available editors !!shadow!!

    Other editors may not work and certainly will not have the optimized DataOps developer experience.

    You should now see the DevReady that looks like this:

    DevReady overview !!shadow!!

    If you return to the DevReady, you can open it by clicking Open Walkthrough on the popup

    DevReady popup !!shadow!!

    Or by clicking the DataOps icon on the sidebar then selecting DataOps.live | Develop Setup.

    DevReady sidebar !!shadow!!

Giving DevReady access to Snowflake

The DataOps.live extension provides a step-by-step guide to help you set up the Snowflake connection for DevReady and store your credentials securely, simplifying your experience in the development environment.

When you open the development environment for the first time without setting any credentials beforehand, a notification message prompts you to open the setup walkthrough. Once open, a two-step getting started displays on the VS Code. Following the on-screen guidance lets you:

  • Set your Snowflake credentials
  • Choose the dbt version for your workspace

Snowflake connection

  1. Click Setup Snowflake ConnectionRun Setup Wizard.

  2. Choose from Username & Password or Single Sign On.

    DevReady incorporates Single Sign-On capabilities to streamline and secure user access across Snowflake accounts. For more information, see the Secure user access to Snowflake documentation.

Username & Password
  1. On top of the VS Code, choose a storage method for your Snowflake credentials.

    Two options are available:

    • DataOps.live account: Offers secure storage — no need to reenter when reopening or starting a new workspace for the same project.
    • Visual Studio Code secret storage: Stores credentials within the workspace but only lasts for the lifetime of that workspace. You'll need to reenter them for a new workspace.

    For a more permanent solution without using your DataOps.live account, consider using DevPod through your local VS Code client. You can set this up in your account preferences by selecting VS Code as your preferred editor.

  2. Enter your Snowflake credentials following the on-screen instructions.

    These credentials are for tools like dbt in the DataOps development environment to connect with your Snowflake. Once you enter them, the DataOps development environment setup will configure your environment. You only need to do this once for each workspace.

Single Sign On
  1. If your DevReady SSO setup has been configured to allow multiple warehouses, you will be asked to choose a warehouse from the dropdown menu at the top of the window.

  2. If your DevReady SSO Setup has been configured to allow multiple roles, you will be asked to choose a role from the dropdown menu at the top of the window.

  3. Next you will be asked to confirm you wish to sign in with SSO.

  4. The built-in VS Code trusted URL mechanism will prompt you to open a browser window to complete the SSO process.

    DevReady OAuth Trusted Domain !!shadow!!

  5. You will be redirected to your Snowflake account where you can choose to log in with your SSO provider.

  6. Snowflake will ask you to confirm the account and role you are granting DevReady to use.

    DevReady OAuth Allow !!shadow!!

  7. You will instructed to close the browser window and return to VS Code.

  8. You should be back on your DevReady window now and will see DevReady setting up your workspace and there should be a popup in the bottom right corner of VS Code confirming you are now connected to Snowflake.

    DevReady OAuth Success !!shadow!!

dbt version

  1. Click Choose dbt VersionSet dbt Version.

  2. On top of the VS Code, select from the dropdown the dbt version for the workspace.

    Available versions correspond to those in the MATE orchestrator used in pipelines. You can also enter a custom dbt version bundled with MATE. For more information, see MATE Orchestrator.

  3. Click Setup Snowflake ConnectionRun Setup Wizard.

Workspaces automatically stop after 30 minutes of inactivity and get deleted if left inactive for 14 days. However, you can prevent deletion by pinning a workspace.

Checking your DevReady setup

MATE

To test that MATE is working correctly:

  1. Navigate to a .sql file in the dataops/modelling/models/ directory.

  2. Choose Execute dbt SQL in the top right of the screen.

Run Model !!shadow!!

SOLE

Check if everything works well by verifying the SOLE configuration using one of the following methods.

  1. Run the command:

    terminal
    sole-validate
  2. On the left vertical bar, click the icon DataOps.liveSOLE validate.

Any SOLE compilation error will show on the VS Code console. For more information, check out SOLE compilation and validation.

Congratulations, you should now be fully set up to use the DevReady. You can get into a clean, ready-to-develop workspace by clicking DevReady within seconds.

Workspaces and network access

info

DevReady workspaces run in a DataOps.live VPC. If you have Snowflake IP allowlists implemented, you must add the 13.41.139.252 IP address to your allowlist.

From Tuesday, 20th February 2024, the IP addresses you must add to your allowlist are 18.135.200.239, 18.169.227.255 and 52.56.96.175

The DevReady manages the developer environments as workspaces. A workspace is operated as a self-contained docker image shipping with a prepackaged set of tools.

Every user's workspace is a dedicated docker container on an EKS cluster in the same VPC as the data product platform. The user accesses the workspace environment over the public internet. The Git operations like push or checkout use a secure connection between the data product platform and the workspace. Access to Snowflake is performed from the workspace via a direct, secure connection.

DevReady high level architecture !!shadow!!

Custom startup jobs

You can register your own custom startup jobs in the DevReady. These jobs will run after the user has completed the setup. These jobs are configured in dataops-develop.yml.

Example

dataops-develop.yml
startup_script:
terminal: one
script:
- echo "Hello from startup_script"

Example with dependencies

dataops-develop.yml
startup_script:
terminal: custom_startup
script:
- echo "Hello from startup_script"

startup_script_part_2:
terminal: custom_startup
needs:
- startup_script
script:
- echo "Hello from startup_script_part_2"

Note the same terminal name is used for both jobs.

Example with dependencies and multiple terminals

dataops-develop.yml
startup_script_a:
terminal: custom_startup_a
script:
- echo "Hello from startup_script_a"

startup_script_b:
terminal: custom_startup_b
script:
- echo "Hello from startup_script_b"

Note different terminal names are used for both jobs.

Override the default startup

You can even override the default setup terminal by adding a setup job of your own, but be sure to include the default setup script items as well.

base setup job
setup:
terminal: setup
script:
- /dataops-dde/scripts/setup/30-pre-setup.sh
- /dataops-dde/scripts/setup/50-setup.sh
- /dataops-dde/scripts/setup/70-post-setup.sh
- /dataops-dde/scripts/setup/99-setup-complete.sh