Skip to main content

Additional Installed Tools

Feature release status badge: PriPrev
PriPrev

Here is a list of the Command line utilities that ship by default with the DataOps Development Environment (DDE).

danger

The tools installed in the DDE are subject to change without warning.

dataops CLI

Configuration/Authentication

None required.

Usage example

dataops solegen "create schema SALES_RECORD.SALES comment='This is a test DataOps.live schema.';"

Produces:

dataops INFO Reading config from create schema SALES_RECORD.SALES comment='This is a test DataOps.live schema.';
dataops INFO Extracting data from a raw DDL query.
dataops INFO Parsing schema object: SALES.
databases:
SALES_RECORD:
schemas:
SALES:
comment: This is a test DataOps.live schema.
is_managed: false
is_transient: false
manage_mode: all
dataops INFO Written output to stdout

Additional information

Read more about the dataops CLI.

dbt CLI

Configuration/Authentication

~/.dbt/profiles.yml is built/rebuilt every time dbt is run using the following variables:

  • DBT_ENV_SECRET_ACCOUNT
  • DBT_ENV_SECRET_PASSWORD
  • DBT_ENV_SECRET_USER
  • DBT_ENV_ROLE
  • DBT_ENV_WAREHOUSE

Usage example

cd dataops/modelling && dbt parse

produces:

15:11:22  Running with dbt=1.2.1
15:11:22 Start parsing.
15:11:22 Dependencies loaded
15:11:22 ManifestLoader created
15:11:22 Manifest loaded
15:11:22 [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 3 unused configuration paths:
- models.MyTemplate.dataops_meta
- models.MyTemplate.dataops_meta.pipelines
- snapshots

15:11:22 Manifest checked
15:11:23 Flat graph built
15:11:23 Manifest loaded
15:11:23 Performance info: target/perf_info.json
15:11:23 Done.

Additional information

Read more about the dbt CLI.

glab CLI

Configuration/Authentication

If the environment variable DATAOPS_ACCESS_TOKEN is set, then glab is automatically authenticated.

Usage example

glab ci list

produces:

Showing 30 pipelines on dataops-demo-project/truedataops-22 (Page 1)

(success) • #851831 kaggledev (about 1 day ago)
(success) • #851817 kaggledev (about 1 day ago)
(success) • #851748 kaggledev (about 1 day ago)
(success) • #851730 kaggledev (about 1 day ago)
(success) • #851483 kaggledev (about 1 day ago)
(canceled) • #851482 kaggledev (about 1 day ago)
(success) • #851422 kaggledev (about 1 day ago)
(success) • #851415 kaggledev (about 1 day ago)
(canceled) • #851413 kaggledev (about 1 day ago)
(success) • #851284 kaggledev (about 1 day ago)
(canceled) • #851282 kaggledev (about 1 day ago)
(canceled) • #851281 kaggledev (about 1 day ago)
(canceled) • #851279 kaggledev (about 1 day ago)
(canceled) • #851276 kaggledev (about 1 day ago)
(skipped) • #851275 kaggledev (about 1 day ago)
(manual) • #851264 kaggledev (about 1 day ago)
(manual) • #851075 kaggledev (about 1 day ago)
(failed) • #851074 kaggle_dev (about 1 day ago)
(manual) • #851038 dp_versioning (about 1 day ago)
(manual) • #850819 dp_versioning_sean (about 1 day ago)
(manual) • #850766 dpversioning (about 1 day ago)
(manual) • #849322 dp_versioning (about 1 day ago)
(manual) • #849268 dp_versioning (about 1 day ago)
(manual) • #849172 dp_versioning (about 1 day ago)
(manual) • #848585 dp_versioning (about 1 day ago)
(manual) • #848390 dp_versioning (about 2 days ago)
(manual) • #848388 dp_versioning (about 2 days ago)
(manual) • #848380 dp_versioning (about 2 days ago)
(manual) • #848313 dp_versioning (about 2 days ago)
(manual) • #847459 dp_versioning (about 2 days ago)

Additional information

Read more about the GitLab CLI.

streamlit

Configuration/Authentication

Done within the application.

Usage example

cd dataops/streamlit/orders/ && streamlit run app.py

produces:

Collecting usage statistics. To deactivate, set browser.gatherUsageStats to False.


You can now view your Streamlit app in your browser.

Network URL: http://10.0.5.2:8501
External URL: http://18.202.147.41:8501

Additional information

Read more about Streamlit.

dataops-render

Resides here: /dataops-cde/scripts/

This is the same render engine that runs in jobs within a pipeline to render the .template.xxx files.

You can learn more about it in the DataOps template rendering section.

Configuration/Authentication

None.

Usage example

/home/gitpod/.pyenv/versions/3.8.13/bin/python /dataops-cde/scripts/dataops-render -o --file $STREAMLIT_FOLDER/orders/app.template.py render-template

produces:

/workspace/truedataops-22/dataops/streamlit/orders/app.py [exists, overwritten]

Additional information

Read more about the DataOps template rendering.

before_script.sh

Resides here: /dataops-cde/scripts/

This is a cut-down version of the full before_script used in a DataOps pipeline, mainly to populate key environment-specific variables.

It is designed to be used by other parts of the DDE to ensure that the correct target account, database, etc., are being targeted. However, it can be useful to run it directly to ensure the correct variables are being populated.

DATAOPS_DATABASE is the most critical of these.

Configuration/Authentication

None.

Usage example

/dataops-cde/scripts/before_script.sh

produces:

DATAOPS_PREFIX=DATAOPS_TDO_22
DATAOPS_ENV_NAME=FB_DP_VERSIONING
DATAOPS_ENV_NAME_PROD=PROD
DATAOPS_ENV_NAME_QA=QA
DATAOPS_ENV_NAME_DEV=DEV
DATAOPS_BRANCH_NAME_PROD=main
DATAOPS_BRANCH_NAME_QA=qa
DATAOPS_BRANCH_NAME_DEV=dev
DATAOPS_DATABASE=DATAOPS_TDO_22_FB_DP_VERSIONING
DATABASE=DATAOPS_TDO_22_FB_DP_VERSIONING
DATAOPS_DATABASE_MASTER=DATAOPS_TDO_22_PROD
DATAOPS_NONDB_ENV_NAME=FB_DP_VERSIONING
DATAOPS_BEFORE_SCRIPT_HAS_RUN=YES

Additional information

Read more about a custom before script.

If you have created a custom before_script.sh in your DataOps project and want to use that within your DDE, don't hesitate to get in touch with our Support team.

sqlfmt

A SQL formatter as documented here.

Configuration/Authentication

None required.

Usage example

cd dataops/modelling/models
sqlfmt .
note

Running the formatter changes all the models from the current directory downwards. It could change hundreds or even thousands of models. Remember, you only need to stage and commit the modified models you want. Also, it would be best if you reran any changed models before you commit, validating they are still working as expected.

sqlfluff

A SQL linter (and fixer) as documented here.

Configuration/Authentication

None required.

Usage example

To run linting and report on the results:

cd dataops/modelling/models
sqlfluff lint --dialect snowflake

To run linting, excluding certain rules and report on the results:

cd dataops/modelling/models
sqlfluff lint --dialect snowflake --exclude-rules L036,L014

SQLFluff can also be used to fix certain issues:

cd dataops/modelling/models
sqlfluff fix --dialect snowflake --exclude-rules L036,L014
note

When you run the fix command, it may change many models from the current directory downwards. It could change hundreds or even thousands of models. Remember, you only need to stage and commit the modified models you want. Also, it would be best if you reran any changed models before you commit, validating they are still working as expected.