Additional Installed Tools
Here is a list of the Command line utilities that ship by default with the DataOps Development Environment (DDE).
The tools installed in the DDE are subject to change without warning.
dataops CLI
Configuration/Authentication
None required.
Usage example
dataops solegen "create schema SALES_RECORD.SALES comment='This is a test DataOps.live schema.';"
Produces:
dataops INFO Reading config from create schema SALES_RECORD.SALES comment='This is a test DataOps.live schema.';
dataops INFO Extracting data from a raw DDL query.
dataops INFO Parsing schema object: SALES.
databases:
SALES_RECORD:
schemas:
SALES:
comment: This is a test DataOps.live schema.
is_managed: false
is_transient: false
manage_mode: all
dataops INFO Written output to stdout
Additional information
Read more about the dataops CLI.
dbt CLI
Configuration/Authentication
~/.dbt/profiles.yml is built/rebuilt every time dbt is run using the following variables:
- DBT_ENV_SECRET_ACCOUNT
- DBT_ENV_SECRET_PASSWORD
- DBT_ENV_SECRET_USER
- DBT_ENV_ROLE
- DBT_ENV_WAREHOUSE
Usage example
cd dataops/modelling && dbt parse
produces:
15:11:22 Running with dbt=1.2.1
15:11:22 Start parsing.
15:11:22 Dependencies loaded
15:11:22 ManifestLoader created
15:11:22 Manifest loaded
15:11:22 [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 3 unused configuration paths:
- models.MyTemplate.dataops_meta
- models.MyTemplate.dataops_meta.pipelines
- snapshots
15:11:22 Manifest checked
15:11:23 Flat graph built
15:11:23 Manifest loaded
15:11:23 Performance info: target/perf_info.json
15:11:23 Done.
Additional information
Read more about the dbt CLI.
glab CLI
Configuration/Authentication
If the environment variable DATAOPS_ACCESS_TOKEN is set, then glab is automatically authenticated.
Usage example
glab ci list
produces:
Showing 30 pipelines on dataops-demo-project/truedataops-22 (Page 1)
(success) • #851831 kaggledev (about 1 day ago)
(success) • #851817 kaggledev (about 1 day ago)
(success) • #851748 kaggledev (about 1 day ago)
(success) • #851730 kaggledev (about 1 day ago)
(success) • #851483 kaggledev (about 1 day ago)
(canceled) • #851482 kaggledev (about 1 day ago)
(success) • #851422 kaggledev (about 1 day ago)
(success) • #851415 kaggledev (about 1 day ago)
(canceled) • #851413 kaggledev (about 1 day ago)
(success) • #851284 kaggledev (about 1 day ago)
(canceled) • #851282 kaggledev (about 1 day ago)
(canceled) • #851281 kaggledev (about 1 day ago)
(canceled) • #851279 kaggledev (about 1 day ago)
(canceled) • #851276 kaggledev (about 1 day ago)
(skipped) • #851275 kaggledev (about 1 day ago)
(manual) • #851264 kaggledev (about 1 day ago)
(manual) • #851075 kaggledev (about 1 day ago)
(failed) • #851074 kaggle_dev (about 1 day ago)
(manual) • #851038 dp_versioning (about 1 day ago)
(manual) • #850819 dp_versioning_sean (about 1 day ago)
(manual) • #850766 dpversioning (about 1 day ago)
(manual) • #849322 dp_versioning (about 1 day ago)
(manual) • #849268 dp_versioning (about 1 day ago)
(manual) • #849172 dp_versioning (about 1 day ago)
(manual) • #848585 dp_versioning (about 1 day ago)
(manual) • #848390 dp_versioning (about 2 days ago)
(manual) • #848388 dp_versioning (about 2 days ago)
(manual) • #848380 dp_versioning (about 2 days ago)
(manual) • #848313 dp_versioning (about 2 days ago)
(manual) • #847459 dp_versioning (about 2 days ago)
Additional information
Read more about the GitLab CLI.
streamlit
Configuration/Authentication
Done within the application.
Usage example
cd dataops/streamlit/orders/ && streamlit run app.py
produces:
Collecting usage statistics. To deactivate, set browser.gatherUsageStats to False.
You can now view your Streamlit app in your browser.
Network URL: http://10.0.5.2:8501
External URL: http://18.202.147.41:8501
Additional information
Read more about Streamlit.
dataops-render
Resides here: /dataops-cde/scripts/
This is the same render engine that runs in jobs within a pipeline to render the .template.xxx files.
You can learn more about it in the DataOps template rendering section.
Configuration/Authentication
None.
Usage example
/home/gitpod/.pyenv/versions/3.8.13/bin/python /dataops-cde/scripts/dataops-render -o --file $STREAMLIT_FOLDER/orders/app.template.py render-template
produces:
/workspace/truedataops-22/dataops/streamlit/orders/app.py [exists, overwritten]
Additional information
Read more about the DataOps template rendering.
before_script.sh
Resides here: /dataops-cde/scripts/
This is a cut-down version of the full before_script used in a DataOps pipeline, mainly to populate key environment-specific variables.
It is designed to be used by other parts of the DDE to ensure that the correct target account, database, etc., are being targeted. However, it can be useful to run it directly to ensure the correct variables are being populated.
DATAOPS_DATABASE is the most critical of these.
Configuration/Authentication
None.
Usage example
/dataops-cde/scripts/before_script.sh
produces:
DATAOPS_PREFIX=DATAOPS_TDO_22
DATAOPS_ENV_NAME=FB_DP_VERSIONING
DATAOPS_ENV_NAME_PROD=PROD
DATAOPS_ENV_NAME_QA=QA
DATAOPS_ENV_NAME_DEV=DEV
DATAOPS_BRANCH_NAME_PROD=main
DATAOPS_BRANCH_NAME_QA=qa
DATAOPS_BRANCH_NAME_DEV=dev
DATAOPS_DATABASE=DATAOPS_TDO_22_FB_DP_VERSIONING
DATABASE=DATAOPS_TDO_22_FB_DP_VERSIONING
DATAOPS_DATABASE_MASTER=DATAOPS_TDO_22_PROD
DATAOPS_NONDB_ENV_NAME=FB_DP_VERSIONING
DATAOPS_BEFORE_SCRIPT_HAS_RUN=YES
Additional information
Read more about a custom before script.
If you have created a custom before_script.sh in your DataOps project and want to use that within your DDE, don't hesitate to get in touch with our Support team.
sqlfmt
A SQL formatter as documented here.
Configuration/Authentication
None required.
Usage example
cd dataops/modelling/models
sqlfmt .
Running the formatter changes all the models from the current directory downwards. It could change hundreds or even thousands of models. Remember, you only need to stage and commit the modified models you want. Also, it would be best if you reran any changed models before you commit, validating they are still working as expected.
sqlfluff
A SQL linter (and fixer) as documented here.
Configuration/Authentication
None required.
Usage example
To run linting and report on the results:
cd dataops/modelling/models
sqlfluff lint --dialect snowflake
To run linting, excluding certain rules and report on the results:
cd dataops/modelling/models
sqlfluff lint --dialect snowflake --exclude-rules L036,L014
SQLFluff can also be used to fix certain issues:
cd dataops/modelling/models
sqlfluff fix --dialect snowflake --exclude-rules L036,L014
When you run the fix command, it may change many models from the current directory downwards. It could change hundreds or even thousands of models. Remember, you only need to stage and commit the modified models you want. Also, it would be best if you reran any changed models before you commit, validating they are still working as expected.