We have seen in Auto-Generating Sources from SOLE Ingestion Tables how to avoid manually adding new dbt sources every time you create a table by sharing table definitions between SOLE and MATE.
However, what if the tables are externally managed, outside the scope of SOLE?
DataOps.live provides a convenient script —
source-builder — that can save you time when creating MATE dbt source definitions for such tables.
source-builder script is provided as-is and without warranty. Although it is not a part of the data product platform,
it has been well tested, and our Support team will be happy to help if they can.
To run the
source-builder script, you must:
- Have a working Python 3.6+ environment — it has only been fully tested on Ubuntu Linux so far (Windows should work, but your mileage may vary)
- Install dbt locally, or at least create a
profiles.ymlas the script uses a local dbt
profiles.ymlfile as the source of connection and authentication details
To best use this script, you may need to have your DataOps project cloned locally (using Git over HTTPS).
source-builder script quickly retrieves some or all of your table structures from Snowflake and converts them into dbt source definition files in YAML format.
usage: source-builder [-h] [--version] [--profiles-dir PROFILES_DIR] [--profile PROFILE] [--database DATABASE]
[--schema SCHEMA] [--warehouse WAREHOUSE] [--project-dir PROJECT_DIR]
[--sources-dir SOURCES_DIR] [--force]
Build dbt source definitions from a Snowflake database.
-h, --help show this help message and exit
--version, -v show program's version number and exit
Which directory to look in for the profiles.yml file (default: /home/sam/.dbt)
--profile PROFILE, -p PROFILE
Which profile to load (default: default)
--database DATABASE, -d DATABASE
Database to interrogate (default None)
--schema SCHEMA, -s SCHEMA
Schema to interrogate (default: None)
--warehouse WAREHOUSE, -w WAREHOUSE
Warehouse to use (default: None)
--project-dir PROJECT_DIR, -o PROJECT_DIR
Target directory for source files (default: .)
--sources-dir SOURCES_DIR, -i SOURCES_DIR
Sources directory within the project directory (default: sources)
--force, -f Force overwriting output files/directories