DataOps CLI

The DataOps CLI (Command-Line Interface) makes it easy for users to interact and manage DataOps-related functionalities through command-line operations on our data product platform.

The DataOps CLI supports a few tools that help you:

• Generate a YAML configuration for SOLE objects from existing DDL statements. Check the SOLE Generator documentation for more information.
• Generate help for data product modules and available commands for SOLE and MATE when you work within the DataOps development environment. Check the DataOps development environment CLI commands topic for more information.

Prerequisites

Before installing the DataOps CLI, make sure you have the following prerequisites installed on your machine:

Linux / MacOS

• Python 3.8+ from python.org
• Pip

Windows Powershell

• Python from the Windows Store or Python 3.8+ from python.org
• Pip

Installing DataOps CLI

Once you have the prerequisites, you can then install the DataOps CLI tool similarly to any other Python package.

Step 1 - Create a personal access token

Create a personal access token (PAT) that you will use to gain access to the CLI package.

1. Open https://app.dataops.live/-/profile/personal_access_tokens

   

   1. Token name is a friendly name to remind you of its purpose, for example dataops-pypi.
   2. Expiration date is optional.
   3. Select scopes are the permissions you are allowing for the new token.

2. Select the checkbox of the following required scope: read_api.

3. Create the token by clicking Create personal access token.

   The page refreshes, and a new token shows at the top.

   

4. Make a copy of the token in a secure place.

   warning

   This is the only time you see this token. When you navigate away from this page, the token will not reappear.

5. Get your username from your account menu in the top right of DataOps.live. Your username doesn't include the @ symbol.

   

Step 2 - Set up your Python environment

Linux / MacOS

1. Add the two credentials into your terminal environment so that they are available during configuration:

   export DATAOPS_USERNAME=<username-from-step-1> DATAOPS_PAT=<personal-access-token-from-step-1>

2. (Optionally) If you do not want to install dataops-cli globally on your machine, create a Python virtual environment before activating it:

   python3 -m venv venv
   source venv/bin/activate

3. Add the DataOps.live package repository by updating your "pip" configuration:

   pip config set global.index-url https://$DATAOPS_USERNAME:$DATAOPS_PAT@app.dataops.live/api/v4/projects/5844/packages/pypi/simple

   The pip config set command adds the configuration you describe to your pip configuration file (usually located in your home directory).

4. Install the package using pip:

   pip install dataops-cli
   Running into errors?

   For 401 errors, check your DATAOPS_USERNAME and DATAOPS_PAT credentials are correct. For other issues, consider reaching out to our Support team with a copy of the error.

5. To confirm that the installation was successful, try showing the --help or --version:

   dataops --help

   This should give you a brief description of the available options for the DataOps CLI, as shown in the image below:

   

   Use the help menu to discover the tools available.

Windows Powershell

1. (Optionally) If you do not want to install dataops-cli globally on your machine, create a Python virtual environment before activating it:

   python3 -m venv venv
   .\venv\Scripts\activate

2. Add the DataOps.live package repository by updating your "pip" configuration. You must substitute the two credentials from "Step 1" in the command below:

   python3 -m pip config set global.index-url https://DATAOPS_USERNAME:DATAOPS_PAT@app.dataops.live/api/v4/projects/5844/packages/pypi/simple

   The pip config set command adds the configuration you describe to your pip configuration file, usually located in your home directory.

3. Install the package using pip:

   python3 -m pip install dataops-cli
   Running into errors?

   For 401 errors, check your DATAOPS_USERNAME and DATAOPS_PAT credentials are correct. For other issues, consider reaching out to our support team with a copy of the error.

4. To confirm that the installation was successful, try showing the --help or --version:

   dataops --help

   This should give you a brief description of the available options for the DataOps CLI, as shown in the image below:

   

   Use the help menu to discover the tools available.

Once you have the DataOps CLI tool installed, you can check the separate sub-tools for more details on how to use it.

Sub-tools:

• dataops sole gen --help
• dataops mate --help
• dataops data-product --help

Step 3 - Install command-line autocomplete (optional)

After you have installed the dataops-cli, you can optionally install the command-line autocomplete helper. Only currently works for bash/zsh/fish shells.

Autocomplete for the dataops-cli lets you use the TAB key to complete commands and options. For example, you could type: dataops - and press TAB to see additional options:

Continuing to press TAB will cycle through the options, highlighting the next, pre-filling your command line:

1. Install the command-line autocomplete by using the dataops-autocomplete command, automatically installed when installing the dataops-cli. For example, you can install autocomplete for zsh using the following:

   dataops-autocomplete zsh

   This adds a line to your shell file to load into your shell environment.

   important

   Follow the onscreen instructions or start a new terminal to enable autocomplete.

Uninstalling DataOps CLI

To uninstall the dataops-cli tool, use pip commands:

pip uninstall dataops-cli

Did you install command-line autocomplete?

To uninstall autocomplete, remove the added DATAOPS-AUTOCOMPLETE line in your shell file:

1. For bash shell users, look in your ~/.bashrc
2. For zsh shell users, look in your ~/.zshrc

fish shell users must remove .config/fish/completions/dataops.fish.

Creating a Python virtual environment

To install the necessary package for a Python virtual environment on an Ubuntu system:

1. Run the script:

   sudo apt-get install python3.10-venv

   For more information about installing Python packages, see Creation of virtual environments.

2. Give a name to the directory where to install the virtual environment, for example, dataops-test.

   python3 -m venv ~/venvs/dataops-test

3. Activate the virtual environment by running:

   source ~/venvs/dataops-test/bin/activate

DataOps development environment CLI commands

The commands below are accessible only within the DataOps development environment.

Data product

This command prints the help for the data products module, e.g., check, builds and checks for backward compatibility, and viewer which opens a new window displaying manifest visualization enriched with dbt run data.

dataops data-product --help

For more information about data products, check the Building data product documentation.

SOLE

This command prints the help for the available commands for SOLE, e.g., gen and validate.

dataops sole --help

For more information about the sole validation, check the SOLE compilation and validation documentation.

MATE

This command prints the help for the available MATE commands, e.g., model-builder and source-builder.

dataops mate --help

For more information regarding source and model builders, check the Generating sources for tables managed outside SOLE documentation.