Databricks MATE Orchestrator
| Image | $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE |
|---|
The Databricks Modelling and Transformation (MATE) orchestrator runs the dbt project in your DataOps project against a Databricks SQL warehouse, using dbt Core 1.12 and the dbt-databricks adapter. It takes the same TRANSFORM_* variables as the MATE orchestrator for Snowflake, so a MATE job file ports with few changes.
With the default dbt profile, each model is built in the catalog named by DATAOPS_DATABASE, so every environment, including each feature branch, writes to its own catalog. To set up the workspace, the service principal and the vault, see the Databricks Setup Checklist, which also lists the differences from Snowflake.
Usage¶
A Databricks MATE job extends the .databricks_modelling_and_transformation_base job from the DataOps Reference Project:
"Build all Models":
extends:
- .databricks_modelling_and_transformation_base
- .agent_tag
stage: "Data Transformation"
image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
variables:
TRANSFORM_ACTION: RUN
script:
- /dataops
icon: ${TRANSFORM_ICON}
The base job sets the image and saves the dbt project's target/ directory as an artifact. A job carries no Databricks credentials: the profile and the token come from the vault.
Porting a Snowflake MATE job
Change .modelling_and_transformation_base to .databricks_modelling_and_transformation_base. If the job sets image: $DATAOPS_TRANSFORM_RUNNER_IMAGE, change it too, or remove it: a job's image: overrides the base job's.
dbt runs only inside /dataops, so a dbt command in the job's script: after /dataops fails.
Supported parameters¶
| Parameter | Required/Default | Description |
|---|---|---|
TRANSFORM_ACTION |
REQUIRED | One of RUN, TEST, SEED or COMPILE. See Transform actions |
TRANSFORM_PROJECT_PATH |
Optional. Defaults to $CI_PROJECT_DIR/dataops/modelling |
The directory of the dbt project |
DATABASE_PROFILE |
Optional. Defaults to databricks_operations |
The dbt profile to use from the profile template |
DATABASE_TARGET |
Optional. Defaults to other |
The dbt target to use from the profile template |
TRANSFORM_MODEL_SELECTOR |
Optional | Passed to dbt as --select, for example tag:finance or models/staging |
TRANSFORM_YAML_SELECTOR |
Optional | Passed to dbt as --selector, naming a YAML selector |
TRANSFORM_VARS |
Optional | Passed to dbt as --vars, for example "{ var_one: Value one, var_two: 2 }" |
FULL_REFRESH |
Optional | If set, passes --full-refresh to RUN and SEED. With TEST or COMPILE, it's ignored with a warning |
TREAT_TEST_ERRORS_AS_FAILED |
Optional | If set, the JUnit report reports an erroring result as a failure |
TREAT_TEST_WARNS_AS_FAILED |
Optional | If set, the JUnit report reports a warning as a failure instead of skipped |
REPORT_NAME |
Optional. Defaults to report.xml |
The JUnit report's file name |
REPORT_DIR |
Optional. Defaults to $CI_PROJECT_DIR |
The directory the JUnit report is written to. It must already exist. A relative path is relative to TRANSFORM_PROJECT_PATH |
DATAOPS_DEBUG |
Optional | If set, passes --debug to dbt |
A variable set to an empty string counts as unset.
Transform actions¶
TRANSFORM_ACTION |
Equivalent dbt command | Notes |
|---|---|---|
RUN |
dbt run | Writes a JUnit report |
TEST |
dbt test | Writes a JUnit report |
SEED |
dbt seed | Seed files are in the dbt project's seeds/ directory by default |
COMPILE |
dbt compile | Compiles the models without building them |
The value is matched exactly, so run instead of RUN fails the job. With TRANSFORM_ACTION unset, no dbt command runs and the job succeeds.
Differences from the Snowflake MATE orchestrator¶
These fail the job before any dbt command runs:
TRANSFORM_ACTIONset toBUILD,SNAPSHOT,DOCS,OPERATION,SOURCEorRENDER. The reference project'sGenerate Docsjob setsDOCS, so don't include it in a pipeline that runs on Databricks.TRANSFORM_EXTRA_PARAMS_BEFOREorTRANSFORM_EXTRA_PARAMS_AFTERset to any value.
These have no effect:
-
DATAOPS_FEATURE_MATE_DBT_VERSION. The Snowflake MATE orchestrator uses it to choose the dbt version, 1.7 by default. This orchestrator always runs dbt 1.12, where a generic test's arguments go underarguments:. The dbt 1.7 form logs a deprecation warning, and fails the job when dbt treats warnings as errors, so move the arguments in a project written for dbt 1.7: -
TRANSFORM_FORCE_DEPS. The orchestrator doesn't rundbt deps, so the packages inpackages.ymlaren't installed, and it doesn't include the packages bundled with MATE, such asdataops. A project's macros and tests must be in its ownmacros/andtests/folders, not in a package. TRANSFORM_MODEL_INDIRECT_SELECTION_MODE,TRANSFORM_DEBUG_TIMING,TRANSFORM_PARTIAL_PARSEandJSON_DIR. Each logs a warning that it has no effect.
dbt sources aren't generated from SOLE configuration or from data products.
Pipelines that also run Snowflake MATE jobs
A pipeline-wide TRANSFORM_PROJECT_PATH, DATABASE_PROFILE or DATABASE_TARGET reaches both orchestrators' jobs. In a pipeline that runs both, set these on each job, or on a base job of your own, rather than in variables.yml or as CI/CD variables.
The dbt profile¶
The orchestrator renders a dbt profile from the vault before dbt runs. It reads DATABRICKS.HOST and DATABRICKS.TRANSFORM.*; see Populate the vault.
When your project has no dataops/profiles/profiles.template.yml, the orchestrator uses this default. Its qa and prod targets are copies of other:
databricks_operations:
target: other
outputs:
other:
type: databricks
host: "{{ DATABRICKS.HOST }}"
http_path: "{{ DATABRICKS.TRANSFORM.HTTP_PATH }}"
token: "{% raw %}{{ env_var('DATAOPS_DATABRICKS_DBT_TOKEN') }}{% endraw %}"
catalog: "{{ env.DATAOPS_DATABASE }}"
schema: BASESCHEMA
threads: {{ DATABRICKS.TRANSFORM.THREADS }}
schema: BASESCHEMA applies only to models with no +schema: configuration.
If your project has a dataops/profiles/profiles.template.yml, the orchestrator renders that file instead. Projects created from the DataOps template project have one that holds only a Snowflake profile:
- In a project that uses only Databricks, delete the file to use the default above, or replace its contents with it.
- In a project that uses both, add
databricks_operationsto it. The whole file is rendered, so the vault needs both theSNOWFLAKEand theDATABRICKSsections, unless you wrap each profile in{% if SNOWFLAKE is defined %}or{% if DATABRICKS is defined %}.
A custom profile must take its token from DATAOPS_DATABRICKS_DBT_TOKEN with the token: line above, {% raw %} tags included.
Authentication¶
Before each dbt command, the orchestrator obtains an OAuth token with only the sql and unity-catalog scopes. It lasts an hour and isn't renewed, so a single dbt command must finish within an hour; split a longer run into several jobs, for example by TRANSFORM_MODEL_SELECTOR.
Test reporting¶
After RUN and TEST, the orchestrator writes a JUnit report of dbt's results to $REPORT_DIR/$REPORT_NAME, whether dbt passed or failed. To show the results in the pipeline's Tests tab, add the report as an artifact:
"Test all Models":
extends:
- .databricks_modelling_and_transformation_base
- .agent_tag
stage: "Transformation Testing"
image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
variables:
TRANSFORM_ACTION: TEST
script:
- /dataops
artifacts:
when: always
reports:
junit: $CI_PROJECT_DIR/report.xml
icon: ${TESTING_ICON}
Warnings are reported as skipped unless TREAT_TEST_WARNS_AS_FAILED is set, and errors as errors unless TREAT_TEST_ERRORS_AS_FAILED is set. The report never changes the job's result, which is always dbt's.
Example jobs¶
Load seed files¶
Drops and recreates each seed table, so a seed file whose columns have changed still loads:
"Load Seeds":
extends:
- .databricks_modelling_and_transformation_base
- .agent_tag
stage: "Data Ingestion"
image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
variables:
TRANSFORM_ACTION: SEED
FULL_REFRESH: 1
script:
- /dataops
icon: ${TRANSFORM_ICON}
Build tagged models¶
Builds all the models tagged finance:
"Build Finance Models":
extends:
- .databricks_modelling_and_transformation_base
- .agent_tag
stage: "Data Transformation"
image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
variables:
TRANSFORM_ACTION: RUN
TRANSFORM_MODEL_SELECTOR: tag:finance
script:
- /dataops
artifacts:
when: always
reports:
junit: $CI_PROJECT_DIR/report.xml
icon: ${TRANSFORM_ICON}
Test sources¶
Runs the tests on every source, and on the tests tagged source_test, before any model is built:
"Test all Sources":
extends:
- .databricks_modelling_and_transformation_base
- .agent_tag
stage: "Source Testing"
image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
variables:
TRANSFORM_ACTION: TEST
TRANSFORM_MODEL_SELECTOR: source:* tag:source_test
script:
- /dataops
artifacts:
when: always
reports:
junit: $CI_PROJECT_DIR/report.xml
icon: ${TESTING_ICON}