Handy Tools

Not content to just provide a great Modelling and Transformation Engine, DataOps also makes available a couple of very handy scripts that can save a lot of time generating source and model definitions.

You will need a working Python 3.6+ environment to run these on, and they have only been fully tested on Ubuntu Linux so far (Windows should work but your mileage may vary!).

Also, these scripts use a local dbt profiles.yml file as the source of connection and authentication details - so you'll either need to install dbt locally, or at least create a profiles.yml.

The sources can be downloaded and installed from the Model Builder project on DataOps - check the README for instructions.

You may need to have your DataOps project cloned locally (using git over HTTPS) to best use these tools.

Be careful!

These tools are provided as-is and without warranty. Although they are not a part of the DataOps platform, they have been well tested, and our support team will be happy to help if they can.

Source builder

The source-builder script will slurp some or all of your table structures from Snowflake and convert them into dbt source definition files in YAML format.

Usage

usage: source-builder [-h] [--version] [--profiles-dir PROFILES_DIR] [--profile PROFILE] [--database DATABASE]
                      [--schema SCHEMA] [--warehouse WAREHOUSE] [--project-dir PROJECT_DIR]
                      [--sources-dir SOURCES_DIR] [--force]

Build dbt source definitions from a Snowflake database.

optional arguments:
  -h, --help            show this help message and exit
  --version, -v         show program's version number and exit
  --profiles-dir PROFILES_DIR
                        Which directory to look in for the profiles.yml file (default: /home/sam/.dbt)
  --profile PROFILE, -p PROFILE
                        Which profile to load (default: default)
  --database DATABASE, -d DATABASE
                        Database to interrogate (default None)
  --schema SCHEMA, -s SCHEMA
                        Schema to interrogate (default: None)
  --warehouse WAREHOUSE, -w WAREHOUSE
                        Warehouse to use (default: None)
  --project-dir PROJECT_DIR, -o PROJECT_DIR
                        Target directory for source files (default: .)
  --sources-dir SOURCES_DIR, -i SOURCES_DIR
                        Sources directory within the project directory (default: sources)
  --force, -f           Force overwriting output files/directories

Model builder

Also in the same package, the model-builder script does not need to connect to Snowflake, but will instead scan a local set of dbt source definitions and scaffold a set of basic SQL models from them.

usage: model-builder [-h] [-v] {from-sources,from-json} ...

Build dbt models from dbt source definitions.

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit

subcommands:
  {from-sources,from-json}
    from-sources        Build models from existing dbt source definitions
    from-json           Build model from JSON document