Skip to main content

Setup

Feature release status badge: PriPrev
PriPrev

danger

The DataOps Development Environment (DDE) is in private preview. Therefore, the service may be unavailable for periods of time. There also may be occasions where service elements are rebuilt, and things like your stored credentials are removed, and you will need to re-add. This poses no risk to the rest of DataOps and is completely isolated.

As good development practice dictates, whether developing on local machines, remotely, or in the cloud, never rely on these for safe storage of configuration/code. Always create a new branch before any changes, and regularly commit your changes to this branch.

If you do not want to install anything locally besides VS Code, you can set up Docker on a remote computer and connect it to development.

The simplest way to do this is to use the Remote SSH extension in VS Code to connect to the remote computer and clone your repo that contains your devcontainer.json file or create a new one.

Prerequisites for your remote machine​

You must have the following on your remote machine before you can start:

  • Docker installed

  • Be authenticated to Docker Hub for DataOps images:

    On your command line, run the following command:

      docker login --username  dataopsreadserviceuser --password qf2h9372fg3ioug384

  • Confirm that you can successfully run:

      docker run -it dataopslive/dataops-development-workspace:5-latest bash -c "date"

    And see a result like:

      ubuntu@ip-172-31-7-62:~$ docker run -it dataopslive/dataops-development-workspace:5-latest bash -c "date"
      Mon 20 Feb 2023 12:26:48 PM UTC

Prerequisites for your local machine​

SSH automatically from your local machine to your remote machine​

  1. Configure your SSH client. Find more help on using the SSH config file.

      Host <dev_container_remote>
        HostName <remote_ip_address>
        User <user>
        IdentityFile <path_to_key_file>

  2. Confirm that you can connect to the remote machine from your local machine through VS Code.

    SSH Window

Setting up the DDE Remote​

  1. Open or clone your desired repository.

  2. Create a folder called .devcontainer.

    Note that this change is inside the repository, so if another user uses the DDE, you may already see this folder.

  3. Create a file called devcontainer.json inside the .devcontainer folder.

    Note that this change is inside the repository, so if another user uses the DDE, you may already see this file. Here is a minimal example:

    {
      "name": "DataOpsDDE",
      "image": "dataopslive/dataops-development-workspace:5-latest"
    }

  4. Reopen the folder inside a container by running the Dev Containers: Reopen in container command from either the command palette or the quick actions status bar item.

    DevCon Prompt

Read more about how to develop on a remote Docker instance.

Configuring the DDE Remote​

Configuring development credentials​

To configure credentials for this environment, you must to set the following environment variables.

Here are the steps to configure the DDE:

  1. Create a file inside the .devcontainer folder called devcontainer.env — if you are using a remote host, you need to create this file on the host.

  2. Add the below to this file, replacing the example values with your own credentials.

    DBT_ENV_SECRET_ACCOUNT=ACCOUNT_NAME
    DBT_ENV_SECRET_PASSWORD=PASSWORD123
    DBT_ENV_SECRET_USER=USERNAME
    DBT_ENV_ROLE=DATAOPS_WRITER # In a default project, this will be DATAOPS_WRITER
    DBT_ENV_WAREHOUSE=DATAOPS_TRANSFORMATION # In a default project, this will be DATAOPS_TRANSFORMATION

  3. Add a run argument to your devcontainer.json file to load the devcontainer.env file.

    {
      "runArgs": ["--env-file", ".devcontainer/devcontainer.env"]
    }

There are various ways to configure VS Code and Devcontainers to store and inject environment variables, some of which are documented at Accessing credentials.

Configuring Git credentials​

When trying to do your first commit, VS Code might prompt you to set your Git username and email. The commands it suggests need to be modified to remove the --global flag, as the change will not persist if the container is re-created.

Instead, your commands should look like this:

git config user.email "you@example.com"
git config user.name "Your Name"

Now, your Git config is set locally on the files mounted from the host computer and will persist.

Advanced configuration​

You can extend your container environment to your own needs, but you will always benefit from building on top of the built-in tooling provided by the DDE. Here is a reference for what you can configure through the devcontainer.json file.

Last updated on