Generating Sources for Tables Managed Outside SOLE

We have seen in Auto-Generating Sources from SOLE Ingestion Tables how to avoid manually adding new dbt sources every time you create a table by sharing table definitions between SOLE and MATE.

However, what if the tables are externally managed, outside the scope of SOLE? DataOps.live provides a convenient script — source-builder — that can save you time when creating MATE dbt source definitions for such tables.

Be careful!

The source-builder script is provided as-is and without warranty. Although it is not a part of the data product platform, it has been well tested, and our Support team will be happy to help if they can.

Prerequisites

To run the source-builder script, you must:

• Have a working Python 3.6+ environment — it has only been fully tested on Ubuntu Linux so far (Windows should work, but your mileage may vary)
• Install dbt locally, or at least create a profiles.yml as the script uses a local dbt profiles.yml file as the source of connection and authentication details

You can download and install the sources from the Model Builder project on DataOps.live — check the README for instructions.

To best use this script, you may need to have your DataOps project cloned locally (using Git over HTTPS).

Usage

The source-builder script quickly retrieves some or all of your table structures from Snowflake and converts them into dbt source definition files in YAML format.

usage: source-builder [-h] [--version] [--profiles-dir PROFILES_DIR] [--profile PROFILE] [--database DATABASE]
                      [--schema SCHEMA] [--warehouse WAREHOUSE] [--project-dir PROJECT_DIR]
                      [--sources-dir SOURCES_DIR] [--force]

Build dbt source definitions from a Snowflake database.

optional arguments:
  -h, --help            show this help message and exit
  --version, -v         show program's version number and exit
  --profiles-dir PROFILES_DIR
                        Which directory to look in for the profiles.yml file (default: /home/sam/.dbt)
  --profile PROFILE, -p PROFILE
                        Which profile to load (default: default)
  --database DATABASE, -d DATABASE
                        Database to interrogate (default None)
  --schema SCHEMA, -s SCHEMA
                        Schema to interrogate (default: None)
  --warehouse WAREHOUSE, -w WAREHOUSE
                        Warehouse to use (default: None)
  --project-dir PROJECT_DIR, -o PROJECT_DIR
                        Target directory for source files (default: .)
  --sources-dir SOURCES_DIR, -i SOURCES_DIR
                        Sources directory within the project directory (default: sources)
  --force, -f           Force overwriting output files/directories