How-To Guides
This section offers several valuable and useful "How-To" guides:
📄️ Build only Changed MATE Models
When actively developing MATE models in a feature branch, especially with larger projects,
📄️ Clone Schemas in Feature Branches
When the DataOps database becomes sufficiently large that cloning it into a dev or feature branch
📄️ Configure a Production-Only Runner
How-to guide for creating a DataOps runner that will only run jobs in the production environment on the main branch.
📄️ Create incremental models in MATE
How-to guide for incrementally loading data from a staging table to a history table in the curation layer in a DataOps pipeline with the MATE orchestrator.
📄️ Create a Custom Before Script
The before script runs ahead of each job's main script block. The default lives in the
📄️ Create a Custom Orchestrator Image
How to create a customer DataOps.live orchestrator packing your prebuilt business logic extending existing orchestrators or using generic docker images.
📄️ Create a Custom Reference Project
Once your DataOps account has more than one project, you might start thinking about how to avoid
📄️ Detect if a File has Changed
Sometimes, it may be necessary to consider triggering a certain action in a pipeline only
📄️ Git Clone when SSO is Enabled
How to perform a git clone when SSO is enabled. Step-by-step guide on creating a personal access token for users that do not have a password set.
📄️ Host a dbt Package
The ability to manage common macros, models and other modeling and transformation resources using dbt Packages is very important to the principles of DataOps, reducing code duplication and centralizing management. DataOps provides the capability to host dbt packages right inside the platform with simple authentication using tokens.
📄️ Managing Log Output
Configure logging in DataOps.live when running pipelines and log size recommendations.
📄️ Passing variables from a Pipeline to MATE
How-to guide on passing a variable from a DataOps pipeline to a MATE model. The example includes how to set up your MATE pipeline job, a sample model, and the result.
📄️ YAML CI/CD Config File Wildcards
For experienced DataOps engineers who require more flexibility in creating DataOps pipelines, we support using wildcards in the -ci.yml file's include:, resulting in the ability to break your -ci.yml files into multiple smaller files to improve reusability and readability. In other words, it is now simpler to reuse the same configuration files in multiple different places and make pipeline config files less verbose, more dynamic, simpler, and easier to read.
📄️ Preventing DataOps Pipelines from Running Concurrently
By default, DataOps runs pipelines in parallel. Not only do pipelines run in parallel, but jobs of the same pipeline stage also run in parallel. Thus, you can effortlessly sequence jobs within a pipeline by adding them to different stages.
📄️ Automatic retry of failed jobs
How-to guide for automatically retrying a failed job and self-healing pipelines. Showing a configuration for setting the number of retry attempts.
📄️ DataOps Runner Disk Space Maintenance
How-to guide for keeping the disk space usage of your DataOps Runner within a quota.
📄️ Git error - server certificate verification failed
How-to guide for solving the error server certificate verification failed. CAfile: none CRLfile: none
📄️ Serving the MATE Pipeline Documentation Offline
How-to guide for retrieving, storing and restoring MATE documentation for future offline browsing
📄️ Use a third-party Git Repository
It is possible to manage a DataOps project using an external Git repository like GitHub, Bitbucket, or Azure DevOps for central code management. In this case, synchronization of the external repository can be set up such that all changes
📄️ Use SSH with Git
DataOps uses the SSH protocol to securely communicate with Git.
📄️ Using Visual Studio Code with DataOps
How to use Visual Studio Code with the DataOps.live SaaS application. Recommended extensions, development workflow tips, and setup instructions.