Skip to content

Databricks MATE Orchestrator

Image$DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE

The Databricks Modelling and Transformation (MATE) orchestrator runs the dbt project in your DataOps project against a Databricks SQL warehouse, using dbt Core 1.12 and the dbt-databricks adapter. It takes the same TRANSFORM_* variables as the MATE orchestrator for Snowflake, so a MATE job file ports with few changes.

With the default dbt profile, each model is built in the catalog named by DATAOPS_DATABASE, so every environment, including each feature branch, writes to its own catalog. To set up the workspace, the service principal and the vault, see the Databricks Setup Checklist, which also lists the differences from Snowflake.

Usage

A Databricks MATE job extends the .databricks_modelling_and_transformation_base job from the DataOps Reference Project:

pipelines/includes/local_includes/mate_jobs/build_all_models.yml
"Build all Models":
  extends:
    - .databricks_modelling_and_transformation_base
    - .agent_tag
  stage: "Data Transformation"
  image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
  variables:
    TRANSFORM_ACTION: RUN
  script:
    - /dataops
  icon: ${TRANSFORM_ICON}

The base job sets the image and saves the dbt project's target/ directory as an artifact. A job carries no Databricks credentials: the profile and the token come from the vault.

Porting a Snowflake MATE job

Change .modelling_and_transformation_base to .databricks_modelling_and_transformation_base. If the job sets image: $DATAOPS_TRANSFORM_RUNNER_IMAGE, change it too, or remove it: a job's image: overrides the base job's.

dbt runs only inside /dataops, so a dbt command in the job's script: after /dataops fails.

Supported parameters

Parameter Required/Default Description
TRANSFORM_ACTION REQUIRED One of RUN, TEST, SEED or COMPILE. See Transform actions
TRANSFORM_PROJECT_PATH Optional. Defaults to $CI_PROJECT_DIR/dataops/modelling The directory of the dbt project
DATABASE_PROFILE Optional. Defaults to databricks_operations The dbt profile to use from the profile template
DATABASE_TARGET Optional. Defaults to other The dbt target to use from the profile template
TRANSFORM_MODEL_SELECTOR Optional Passed to dbt as --select, for example tag:finance or models/staging
TRANSFORM_YAML_SELECTOR Optional Passed to dbt as --selector, naming a YAML selector
TRANSFORM_VARS Optional Passed to dbt as --vars, for example "{ var_one: Value one, var_two: 2 }"
FULL_REFRESH Optional If set, passes --full-refresh to RUN and SEED. With TEST or COMPILE, it's ignored with a warning
TREAT_TEST_ERRORS_AS_FAILED Optional If set, the JUnit report reports an erroring result as a failure
TREAT_TEST_WARNS_AS_FAILED Optional If set, the JUnit report reports a warning as a failure instead of skipped
REPORT_NAME Optional. Defaults to report.xml The JUnit report's file name
REPORT_DIR Optional. Defaults to $CI_PROJECT_DIR The directory the JUnit report is written to. It must already exist. A relative path is relative to TRANSFORM_PROJECT_PATH
DATAOPS_DEBUG Optional If set, passes --debug to dbt

A variable set to an empty string counts as unset.

Transform actions

TRANSFORM_ACTION Equivalent dbt command Notes
RUN dbt run Writes a JUnit report
TEST dbt test Writes a JUnit report
SEED dbt seed Seed files are in the dbt project's seeds/ directory by default
COMPILE dbt compile Compiles the models without building them

The value is matched exactly, so run instead of RUN fails the job. With TRANSFORM_ACTION unset, no dbt command runs and the job succeeds.

Differences from the Snowflake MATE orchestrator

These fail the job before any dbt command runs:

  • TRANSFORM_ACTION set to BUILD, SNAPSHOT, DOCS, OPERATION, SOURCE or RENDER. The reference project's Generate Docs job sets DOCS, so don't include it in a pipeline that runs on Databricks.
  • TRANSFORM_EXTRA_PARAMS_BEFORE or TRANSFORM_EXTRA_PARAMS_AFTER set to any value.

These have no effect:

  • DATAOPS_FEATURE_MATE_DBT_VERSION. The Snowflake MATE orchestrator uses it to choose the dbt version, 1.7 by default. This orchestrator always runs dbt 1.12, where a generic test's arguments go under arguments:. The dbt 1.7 form logs a deprecation warning, and fails the job when dbt treats warnings as errors, so move the arguments in a project written for dbt 1.7:

    # dbt 1.7
    - accepted_values:
        values: ["open", "closed"]
    # dbt 1.12
    - accepted_values:
        arguments:
          values: ["open", "closed"]
    
  • TRANSFORM_FORCE_DEPS. The orchestrator doesn't run dbt deps, so the packages in packages.yml aren't installed, and it doesn't include the packages bundled with MATE, such as dataops. A project's macros and tests must be in its own macros/ and tests/ folders, not in a package.

  • TRANSFORM_MODEL_INDIRECT_SELECTION_MODE, TRANSFORM_DEBUG_TIMING, TRANSFORM_PARTIAL_PARSE and JSON_DIR. Each logs a warning that it has no effect.

dbt sources aren't generated from SOLE configuration or from data products.

Pipelines that also run Snowflake MATE jobs

A pipeline-wide TRANSFORM_PROJECT_PATH, DATABASE_PROFILE or DATABASE_TARGET reaches both orchestrators' jobs. In a pipeline that runs both, set these on each job, or on a base job of your own, rather than in variables.yml or as CI/CD variables.

The dbt profile

The orchestrator renders a dbt profile from the vault before dbt runs. It reads DATABRICKS.HOST and DATABRICKS.TRANSFORM.*; see Populate the vault.

When your project has no dataops/profiles/profiles.template.yml, the orchestrator uses this default. Its qa and prod targets are copies of other:

dataops/profiles/profiles.template.yml
databricks_operations:
  target: other
  outputs:
    other:
      type: databricks
      host: "{{ DATABRICKS.HOST }}"
      http_path: "{{ DATABRICKS.TRANSFORM.HTTP_PATH }}"
      token: "{% raw %}{{ env_var('DATAOPS_DATABRICKS_DBT_TOKEN') }}{% endraw %}"
      catalog: "{{ env.DATAOPS_DATABASE }}"
      schema: BASESCHEMA
      threads: {{ DATABRICKS.TRANSFORM.THREADS }}

schema: BASESCHEMA applies only to models with no +schema: configuration.

If your project has a dataops/profiles/profiles.template.yml, the orchestrator renders that file instead. Projects created from the DataOps template project have one that holds only a Snowflake profile:

  • In a project that uses only Databricks, delete the file to use the default above, or replace its contents with it.
  • In a project that uses both, add databricks_operations to it. The whole file is rendered, so the vault needs both the SNOWFLAKE and the DATABRICKS sections, unless you wrap each profile in {% if SNOWFLAKE is defined %} or {% if DATABRICKS is defined %}.

A custom profile must take its token from DATAOPS_DATABRICKS_DBT_TOKEN with the token: line above, {% raw %} tags included.

Authentication

Before each dbt command, the orchestrator obtains an OAuth token with only the sql and unity-catalog scopes. It lasts an hour and isn't renewed, so a single dbt command must finish within an hour; split a longer run into several jobs, for example by TRANSFORM_MODEL_SELECTOR.

Test reporting

After RUN and TEST, the orchestrator writes a JUnit report of dbt's results to $REPORT_DIR/$REPORT_NAME, whether dbt passed or failed. To show the results in the pipeline's Tests tab, add the report as an artifact:

pipelines/includes/local_includes/mate_jobs/test_all_models.yml
"Test all Models":
  extends:
    - .databricks_modelling_and_transformation_base
    - .agent_tag
  stage: "Transformation Testing"
  image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
  variables:
    TRANSFORM_ACTION: TEST
  script:
    - /dataops
  artifacts:
    when: always
    reports:
      junit: $CI_PROJECT_DIR/report.xml
  icon: ${TESTING_ICON}

Warnings are reported as skipped unless TREAT_TEST_WARNS_AS_FAILED is set, and errors as errors unless TREAT_TEST_ERRORS_AS_FAILED is set. The report never changes the job's result, which is always dbt's.

Example jobs

Load seed files

Drops and recreates each seed table, so a seed file whose columns have changed still loads:

pipelines/includes/local_includes/mate_jobs/load_seeds.yml
"Load Seeds":
  extends:
    - .databricks_modelling_and_transformation_base
    - .agent_tag
  stage: "Data Ingestion"
  image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
  variables:
    TRANSFORM_ACTION: SEED
    FULL_REFRESH: 1
  script:
    - /dataops
  icon: ${TRANSFORM_ICON}

Build tagged models

Builds all the models tagged finance:

pipelines/includes/local_includes/mate_jobs/build_finance_models.yml
"Build Finance Models":
  extends:
    - .databricks_modelling_and_transformation_base
    - .agent_tag
  stage: "Data Transformation"
  image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
  variables:
    TRANSFORM_ACTION: RUN
    TRANSFORM_MODEL_SELECTOR: tag:finance
  script:
    - /dataops
  artifacts:
    when: always
    reports:
      junit: $CI_PROJECT_DIR/report.xml
  icon: ${TRANSFORM_ICON}

Test sources

Runs the tests on every source, and on the tests tagged source_test, before any model is built:

pipelines/includes/local_includes/mate_jobs/test_all_sources.yml
"Test all Sources":
  extends:
    - .databricks_modelling_and_transformation_base
    - .agent_tag
  stage: "Source Testing"
  image: $DATAOPS_DATABRICKS_MATE_ORCHESTRATOR_IMAGE
  variables:
    TRANSFORM_ACTION: TEST
    TRANSFORM_MODEL_SELECTOR: source:* tag:source_test
  script:
    - /dataops
  artifacts:
    when: always
    reports:
      junit: $CI_PROJECT_DIR/report.xml
  icon: ${TESTING_ICON}