Handy Tools
Not content to just provide a great Modelling and Transformation Engine, DataOps also makes available a couple of very handy scripts that can save a lot of time generating source and model definitions.
You will need a working Python 3.6+ environment to run these on, and they have only been fully tested on Ubuntu Linux so far (Windows should work but your mileage may vary!).
Also, these scripts use a local dbt profiles.yml
file as the source of connection and authentication details - so you'll either need to install dbt locally, or at least create a profiles.yml
.
The sources can be downloaded and installed from the Model Builder project on DataOps - check the README for instructions.
You may need to have your DataOps project cloned locally (using git over HTTPS) to best use these tools.
These tools are provided as-is and without warranty. Although they are not a part of the DataOps platform, they have been well tested, and our support team will be happy to help if they can.
Source builder
The source-builder
script will slurp some or all of your table structures from Snowflake and convert
them into dbt source definition files in YAML format.
Usage
usage: source-builder [-h] [--version] [--profiles-dir PROFILES_DIR] [--profile PROFILE] [--database DATABASE]
[--schema SCHEMA] [--warehouse WAREHOUSE] [--project-dir PROJECT_DIR]
[--sources-dir SOURCES_DIR] [--force]
Build dbt source definitions from a Snowflake database.
optional arguments:
-h, --help show this help message and exit
--version, -v show program's version number and exit
--profiles-dir PROFILES_DIR
Which directory to look in for the profiles.yml file (default: /home/sam/.dbt)
--profile PROFILE, -p PROFILE
Which profile to load (default: default)
--database DATABASE, -d DATABASE
Database to interrogate (default None)
--schema SCHEMA, -s SCHEMA
Schema to interrogate (default: None)
--warehouse WAREHOUSE, -w WAREHOUSE
Warehouse to use (default: None)
--project-dir PROJECT_DIR, -o PROJECT_DIR
Target directory for source files (default: .)
--sources-dir SOURCES_DIR, -i SOURCES_DIR
Sources directory within the project directory (default: sources)
--force, -f Force overwriting output files/directories
Model builder
Also in the same package, the model-builder
script does not need to connect to Snowflake, but will
instead scan a local set of dbt source definitions and scaffold a set of basic SQL models from them.
usage: model-builder [-h] [-v] {from-sources,from-json} ...
Build dbt models from dbt source definitions.
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
subcommands:
{from-sources,from-json}
from-sources Build models from existing dbt source definitions
from-json Build model from JSON document