Advanced Topics
Data product project structure
The data product creator easily integrates with a standard template project structure predefined on the data product platform. See DataOps Create Template.
When starting a new project using the data product creator, it's best to begin with a fresh project as the app functions optimally with a project structure similar to the template.
The data product project based on the template looks like below, customer_orders is the new data product in this example:
The template project structure includes default configurations under the dataops folder, pre-scripts in the scripts subfolder, and repository configurations in the config folder.
Additionally, the gitpod.yml file specifies the development environment (DevReady) version used in the data product creator.
Snowflake credentials are automatically passed from the data product creator to DataOps.live platform as project variables, used during pipeline runs.
Upon finalizing the workflow in the data product creator, it generates a new branch in the repository with the data product name, along with essential files including:
- data product definition
- job configurations
- pipeline file
- SOLE config
- MATE sources and models.
The data product definition, job configuration, and SOLE configuration are organized within a separate folder named after the data product, customer_orders in this example.
The pipeline file resides in the main folder and it has the data product name.
MATE sources and models are located under the main MATE project folder, organized in distinct sub-folders under sources and models.
AI Assist chat
The AI Assist chat is an AI-powered copilot that improves your user experience on the data product platform by getting instant responses to your queries while creating data products and troubleshooting challenges. It integrates into the data product creation workflow, offering helpful suggestions and guidance as you build models through natural language prompts.
Once you've generated the data pipeline that creates the data product through the UI and switched to the development environment, the assist chat steps in. It takes the business problem you describe when defining your data product and suggests transformational models to solve it. While it's not perfect, it gets you pretty close — about 90% complete — by suggesting code based on your natural language prompts.
You can keep asking for suggestions until you're satisfied, then accept and tweak the code as needed. Plus, the chat provides valuable context you might not have considered on your own. It's like having a knowledgeable teammate at your fingertips.
Spendview and the data product creator
Check out our Spendview for Snowflake to monitor and maintain the underlying infrastructure behind the data products created via the creator. With Spendview for Snowflake, you can track spending across various projects and warehouses within the same Snowflake account used to retrieve the metadata of the data product dataset. Gain valuable insights into usage patterns and user activity, enabling you to optimize resources and make informed decisions.
Plus, our graphical metadata representations across the registered Snowflake accounts make it easy to identify cost-driver issues and take immediate action. Stay in control of your DataOps projects and products and drive better financial outcomes with Spendview for Snowflake.