Get the DataOps Runner from Snowflake Marketplace
The DataOps.live app in the Snowflake Marketplace allows the management of DataOps.live runners entirely within your Snowflake environment using Snowpark Container Services.
To use this app, you must already have an active DataOps.live subscription.
Getting started
-
Install the DataOps.live app from the Snowflake Marketplace.
-
Grant the requested account-level permissions and allow connectivity to external endpoints.
-
Activate the app and launch the UI using the buttons provided.
-
Wait for the Streamlit-based user interface to load.
Dependent objects and permissions
This app will need to be granted some account-level privileges in order to function, and it will also need access to an External Access Integration to communicate with DataOps.live services.
Account-level permissions required
- Bind Service Endpoint will allow runner services to connect to the DataOps.live platform.
- Create Compute Pool will allow this app to create a default compute pool as part of the installation.
- Create Warehouse will allow this app to create a default warehouse as part of the installation.
External connectivity
Allow the creation of External Access Integration (EAI) for connectivity.
To allow DataOps runners and pipeline jobs to connect to the DataOps.live platform, the app installation includes creating an External Access Integration and Network Rules. The app pre-configures these and executes them in Snowflake, so additional user action is unnecessary.
Additional connectivity
Additional network rules can be attached to this EAI to allow pipeline jobs to access other resources (e.g., other cloud databases, APIs, etc.). Simply create network rules with the necessary access, then attach them to this EAI. Please ensure that the default network rule (that allows access to app.dataops.live) is not removed when attaching new rules.
To obtain the name of the EAI that is created by the installation, click the Review button and then expand the Advanced Options section of the pop-up to reveal the External Access Integration name.
Compute resources
DataOps.live runners will execute as long-running Snowflake SPCS Services, using a configured compute pool. The app installation will create a compute pool of size small that can be used as the default pool for runners.
Additionally, processes running in pipeline jobs often need a Snowflake Warehouse to execute statements. The installation process creates a default warehouse that can be used as the default for all statements unless otherwise configured.
Managing runners
User interface showing the list of runners.
Add a runner
Complete the requested information by clicking the + button in the title bar.
Basic information
- Runner name will be used to identify the runner in this app and will also form part of the SPCS service identifier.
- Agent tags are the tag names that will be registered with the DataOps.live platform (use commas to separate multiple values).
- Registration token is retrieved from the DataOps.live platform for the purposes of adding new runners.
Snowflake configuration
This section captures the Snowflake authentication details for use when DataOps.live jobs connect to Snowflake. Please refer to the DataOps.live setup documentation for more information.
Advanced configuration
- Compute pool is set to the default pool which is created as part of the app installation, but any other compute pool can be selected if this app has been granted usage privileges to it.
- Warehouse is set to the default warehouse which is created as part of the app installation, but any other warehouse can be selected if this app has been granted usage privileges to it.
Remove a runner
Click the Show button next to the runner, then click the Drop button at the bottom of the pop-up.
View runner logs
Click the Logs button next to the runner. Log information can be updated as needed by clicking the Refresh button.