Skip to main content

MATE Use Cases

Feature release status badge: PubPrev
PubPrev

There are many use cases that the DataOps development environment optimizes, including transformation and automated testing using the DataOps Modeling and Transformation Engine (MATE). The development environment and MATE streamline your development process. This allows for swift modifications and instant execution of specific models, enabling you to test outcomes without any delay in data loading.

info

Examples on this page are using DevReady, the DataOps development environment in your browser.

Switching dbt version in the DataOps development environment​

The DataOps development environment uses a default dbt version, but you can change this version through the DataOps.live VS Code extension.

warning

By the end of October, the default dbt version will be dbt 1.5, rather than 1.0.

By the end of November, the DataOps development environment will no longer support dbt 1.0. We strongly advise upgrading to a later supported dbt version, like 1.4 or 1.5, and benefit from improved performance and access to important updates.

If you have already set up the DataOps development environment, you can change the dbt version by clicking the DataOps.live icon in the left vertical bar and then clicking the Setup Walkthrough button in the top left.

You can choose from a list of dbt versions that come with the MATE packages or choose a custom version of dbt in which case you only get the packages defined in your packages.yml file.

Setting dbt version for a single user or for all project users​

You can set the version of dbt per user or project by manually setting it at the user or workspace scopes provided by VS Code.

Setting dbt version for a user through your VS Code settings:

Setting dbt version for a project through your VS Code settings:

The only difference is the scope at which you set dbt version, user scope only sets it for the user but workspace scope will put it in a file in the project at the path .vscode/settings.json

dbt version settings !!shadow!!

Above, you can see the settings.json file with the version of dbt for the current project/workspace, this one takes precedence over any others. Versions of dbt are also set at the user and remote settings scopes but only the workspace level setting will be used.

MATE packages in the DataOps development environment​

The DataOps development environment will include MATE packages but only for supported versions of dbt. You can still use MATE packages with an unsupported dbt version, but you must install them manually. For more information about switching dbt version in the MATE orchestrator, see Switching dbt Version.

Modifying an existing MATE model​

This example shows how to do simple modifications to an existing DataOps model and preview the results automatically after entering each new SQL line. Let's assume you need to edit a product table to add new columns with specific names and populate them with content.

  1. Under your top-level Projects menu, click the project you want to open.

  2. Click DataOps.live | develop from the project menu buttons to open the development environment.

  3. Navigate to the project modeling SQL file and duplicate the statement about the column name.

    The column NAME is duplicated automatically.

  4. Edit the statements and name the first column productcategory and the second productsubcategory.

  5. Add the coalesce function to the column statements to fill the columns with Misc.

    The columns are automatically populated with the new text.

  6. Enter another SQL statement to add the new ListPrice column to the table.

Adding a test to an existing MATE model​

This example shows how to add a test to a layer in an existing DataOps model and automatically preview the results of running the tests. Let's assume you want to test selecting the data for a specific customer.

  1. Under your top-level Projects menu, click the project you want to open.

  2. Click DataOps.live | Develop from the project menu buttons to open the development environment.

  3. Navigate to the project YAML file and add tests to check for null values in specific columns in the customer table.

  4. Select dbt Power User on the left vertical bar and click the run button on the panel's top right corner.

    The terminal console shows the tests' results defined in the .yml file.

  5. Enter another test condition in the YAML file to test relationships to a field in another table and rerun the test.

Testing MATE model changes​

This example shows how to check if the changes added to the DataOps model haven't broken anything downstream.

  1. Open the project modeling SQL file and add a test to retrieve the ID of all business entities.
  2. Run the test via the dbt Power User icon.

The terminal console shows the results pointing out any invalid identifier.

  1. Remove the statement from the SQL file and run the children's models via the dbt Power User icon.

  2. Keep switching between running the test to retrieve the ID of all business entities and running the children's models till you reach the expected results.

Delta runs​

A powerful feature of dbt is the ability to work out which model definitions have changed since a specific baseline. The /dataops-cde/scripts/dataops_cde_setup.sh script, which runs every time a workspace is started creates this baseline and stores it in /workspace/state/. If you want to run a command (build/test will run the same way) for all the models that you have modified in some way since this baseline, you can run:

cd dataops/modelling/
dbt run --select state:modified+ --state /workspace/state/
Last updated on