MATE Packages
Bundled packages
The DataOps Template Rendering topic notes that packages are integral to the MATE ecosystem. They are vital for importing macros, models, and other logic into a DataOps project. An excellent example of a package is dbt_utils
, a library of open-source macros you can use and reuse across DataOps MATE projects. Another vital package used in MATE transformations is the dataops
package. MATE ships with the packages dbt_utils
and dataops
by default.
The full set of bundled MATE packages includes:
- dbt-1.7
- dbt-1.5
- dbt-1.4
- Datavault-UK/automate_dv (version: 0.10.1)
- tnightengale/dbt_meta_testing (version: 0.3.6)
- dbt-labs/dbt_utils (version: 1.1.1)
- data-mie/dbt_profiler (version: 0.8.1)
The package Datavault-uk/dbtvault
has been renamed to Datavault-UK/automate_dv
in dbt 1.7.
To prevent potential failures, ensure that you update the usage of the package by replacing dbtvault
with automate_dv
. For example:
- Previous Configuration
- New Configuration
{%- set source_model = "v_stg_orders" -%}
{%- set src_pk = "ORDER_PK" -%}
{%- set src_nk = "ORDERKEY" -%}
{%- set src_ldts = "LOAD_DATE" -%}
{%- set src_source = "RECORD_SOURCE" -%}
{{ dbtvault.hub(src_pk=src_pk, src_nk=src_nk, src_ldts=src_ldts,
src_source=src_source, source_model=source_model) }}
{%- set source_model = "v_stg_orders" -%}
{%- set src_pk = "ORDER_PK" -%}
{%- set src_nk = "ORDERKEY" -%}
{%- set src_ldts = "LOAD_DATE" -%}
{%- set src_source = "RECORD_SOURCE" -%}
{{ automate_dv.hub(src_pk=src_pk, src_nk=src_nk, src_ldts=src_ldts,
src_source=src_source, source_model=source_model) }}
- Datavault-uk/dbtvault (version: 0.9.5)
- tnightengale/dbt_meta_testing (version: 0.3.6)
- dbt-labs/dbt_utils (version: 1.1.1)
- data-mie/dbt_profiler (version: 0.8.1)
When upgrading a dbt version, potential breaking changes may affect specific existing setups:
- If moving from 1.4 to 1.5, MATE jobs could fail if their configuration contains an empty
tests:
keyword in your YAML file. You can resolve this by removing the emptytests:
keyword.
Check dbt migration for detailed guidance to help you resolve any potential issues.
- Datavault-uk/dbtvault (version: 0.9.4)
- tnightengale/dbt_meta_testing (version: 0.3.6)
- dbt-labs/dbt_utils (version: 1.0.0)
- data-mie/dbt_profiler (version: 0.6.2)
We strongly advise upgrading to the latest dbt version. However, when upgrading from dbt 1.4 to 1.5, MATE jobs could fail if their configuration contains an empty tests:
keyword in your YAML file. You can resolve this by removing the empty tests:
keyword.
Check dbt migration for detailed guidance to help you resolve any potential issues.
Many packages are available on hub.getdbt.com.
Installing additional packages
If the bundled MATE packages are insufficient, it is possible to install additional ones. To do this, you must modify the packages.yml
file in the project's dataops/modelling
directory.
Specify a package using one of the following methods depending on where your package is stored.