How-To Guides
This section offers several valuable and useful "How-To" guides:
📄️ Add Tables/Views to an Existing Share
How-to guide for adding tables or views to an existing share in Snowflake using the MATE (Modelling and Transformation) orchestrator
📄️ Auto-Cancel Duplicate Pipelines
How-to guide for configuring a check job to abort a pipeline if another is already running in the same ref
📄️ Auto-Cancel Running Queries
How-to guide for automatically canceling any running Snowflake queries in the event of pipeline failure
📄️ Build Only Changed MATE Models
How-to guide for adding a runner script that builds or tests only the models that have been changed since the previous pipeline commit
📄️ Change Column Datatype in a Table with data
How-to guide for changing the data type of a column of a table that is already populated with data
📄️ Clone Schemas in Feature Branches
How-to guide for cloning in feature branches only the schemas specified in a configurable list
📄️ Configure a Production-Only Runner
How-to guide for creating a DataOps runner that only runs jobs in the production environment on the main branch
📄️ Configure Select Statement in a Snowflake PIPE
How-to guide for configuring the select_statement parameter of the Snowflake PIPE object using the Snowflake Lifecycle Engine.
📄️ Create Incremental Models in MATE
How-to guide for incrementally loading data from a staging table to a history table in the curation layer in a DataOps pipeline with the MATE orchestrator
📄️ Create a Custom Before Script
How-to guide for creating a custom before_script that sets dynamic variables to be used by the app and script that run in the job
📄️ Create a Custom Orchestrator Image
How-to guide for creating a custom DataOps.live orchestrator packing your prebuilt business logic and extending existing orchestrators or using generic docker images
📄️ Create a Custom Reference Project
How-to guide for creating a custom reference project that helps to avoid the needless repetition of content
📄️ Detect if a File has Changed
How-to guide for triggering a specific action in a pipeline only if the state of a file has changed within a job
📄️ Create Fallback Jobs
How-to guide for creating fallback jobs. When a job fails, you can specify another image to perform the task.
📄️ Git Clone over HTTPS
How-to guide for performing a Git clone over HTTPS
📄️ Git Clone when SSO is Enabled
How-to guide for performing a Git clone when SSO is enabled where you need to create a personal access token for users that do not have a password set
📄️ Host a dbt Package
How-to guide for hosting a dbt package in the DataOps.live data product platform to easily manage common macros, models, and other modeling and transformation resources
📄️ Configure the Runner Health Check Script
How-to guide for configuring the health check script to monitor your DataOps runner.
📄️ Manage Log Output
How-to guide for configuring logging in DataOps.live when running pipelines and log size recommendations
📄️ Manage Multiple Databases
How-to guide for creating and working with more than just the default DataOps database in SOLE and MATE.
📄️ Pass Variables from a Pipeline to MATE
How-to guide for setting up a pipeline job to pass a variable from the DataOps pipeline to a MATE model
📄️ YAML CI/CD Config File Wildcards
How-to guide for supporting wildcards to break down your YAML configuration files into multiple smaller files and improve reusability
📄️ Prevent DataOps Pipelines from Running Concurrently
How-to guide for ensuring that a new DataOps pipeline does not start before the current pipeline run is completed
📄️ Automatic Retry of Failed Jobs
How-to guide for automatically retrying a failed job and self-healing pipelines. Showing a configuration for setting the number of retry attempts
📄️ Runner Disk Space Maintenance
How-to guide for keeping the disk space usage of your DataOps Runner within a quota
📄️ Git Error - Server Certificate Verification Failed
How-to guide for solving the error server certificate verification failed. CAfile: none CRLfile: none
📄️ Serve MATE Pipeline Documentation Offline
How-to guide for retrieving, storing, and restoring MATE documentation for future offline browsing
📄️ How to Set up Git Pre-Commit Hooks for a DataOps Project
How-to guide for setting up git pre-commit hooks automating quality checks of code and data before committing. Save development time and reduce errors.
📄️ Set up Multiple Pull Policies on the DataOps Runner
How-to guide for adding multiple pull policies. They provide mitigation strategies for container registry downtimes.
📄️ Use a Third-Party Git Repository
How-to guide for using an external Git repository like GitHub, Bitbucket, or Azure DevOps for central code management
📄️ Update Tags on Existing Runners
How-to guide for updating the tags of existing runners
📄️ Use Datetime and Time Modules in Jinja
How-to guide using Python's datetime and time modules in your Jinja templates and dbt models.
📄️ Use Parent-Child Pipelines
How-to guide for triggering one pipeline from another using variables between parent and child pipelines
📄️ Use Snowflake Tags
How-to guide for creating and applying Snowflake tags in SOLE and MATE
📄️ Use SSH with Git
How-to guide for using SSH in DataOps.live to communicate with Git securely