Skip to main content

MATE Use Cases

There are many use cases that the DataOps development environment optimizes, including transformation and automated testing using the DataOps Modeling and Transformation Engine (MATE). The development environment and MATE streamline your development process. This allows for swift modifications and instant execution of specific models, enabling you to test outcomes without any delay in data loading.

info

Examples on this page are using DevReady, the DataOps development environment in your browser.

Modifying an existing MATE model​

This example shows how to do simple modifications to an existing DataOps model and preview the results automatically after entering each new SQL line. Let's assume you need to edit a product table to add new columns with specific names and populate them with content.

  1. Under your top-level Projects menu, click the project you want to open.

  2. Click DataOps.live | develop from the project menu buttons to open the development environment.

  3. Navigate to the project modeling SQL file and duplicate the statement about the column name.

    The column NAME is duplicated automatically.

  4. Edit the statements and name the first column productcategory and the second productsubcategory.

  5. Add the coalesce function to the column statements to fill the columns with Misc.

    The columns are automatically populated with the new text.

  6. Enter another SQL statement to add the new ListPrice column to the table.

Adding a test to an existing MATE model​

This example shows how to add a test to a layer in an existing DataOps model and automatically preview the results of running the tests. Let's assume you want to test selecting the data for a specific customer.

  1. Under your top-level Projects menu, click the project you want to open.

  2. Click DataOps.live | Develop from the project menu buttons to open the development environment.

  3. Navigate to the project YAML file and add tests to check for null values in specific columns in the customer table.

  4. Select dbt Power User on the left vertical bar and click the run button on the panel's top right corner.

    The terminal console shows the tests' results defined in the .yml file.

  5. Enter another test condition in the YAML file to test relationships to a field in another table and rerun the test.

Testing MATE model changes​

This example shows how to check if the changes added to the DataOps model haven't broken anything downstream.

  1. Open the project modeling SQL file and add a test to retrieve the ID of all business entities.
  2. Run the test via the dbt Power User icon.

The terminal console shows the results pointing out any invalid identifier.

  1. Remove the statement from the SQL file and run the children's models via the dbt Power User icon.

  2. Keep switching between running the test to retrieve the ID of all business entities and running the children's models till you reach the expected results.

Delta runs​

A powerful feature of dbt is the ability to work out which model definitions have changed since a specific baseline. The /dataops-cde/scripts/dataops_cde_setup.sh script, which runs every time a workspace is started creates this baseline and stores it in /workspace/state/. If you want to run a command (build/test will run the same way) for all the models that you have modified in some way since this baseline, you can run:

cd dataops/modelling/
dbt run --select state:modified+ --state /workspace/state/
Last updated on