Stage
You can provide configuration to Snowflake Object Lifecycle Engine for the following operations with stage:
- Manage the lifecycle of new and existing stages
- Manage the grants of a stage
Usage
We have introduced SOLE for Data Products as a new framework for SOLE to help you easily build an ecosystem of data products. Learn more about SOLE for Data Products which is currently available as a private preview.
- Stage in Current Configuration
- Stage in SOLE for Data Products
databases:
<database-name>:
schemas:
<schema-name>:
stages:
<stage-name>:
<configuration-key>: <value>
- stage:
name: <stage-name>
<configuration-key>: <value>
database: rel(database.<database-name>)
schema: rel(schema.<schema-name>)
Supported parameters
The engine supports the parameters listed below.
Configuration Key | Required/Optional | Data Types and Values | Description |
---|---|---|---|
comment | Optional | String | Specifies a comment for the stage |
copy_options | Optional | String: See here for a definition of copy_options | Specifies one (or more) copy options for the stage. See the Snowflake doc section for copy options. |
credentials | Optional | String | Specifies the credentials for the stage, required only if the storage location is private/protected - not required for public buckets/containers |
deleted | Optional | Boolean: True enables deletion prevention, False does nothing | Specifies what objects are allowed to be deleted |
directory | Optional | String: See here for a definition of directory | Specifies the directory settings for the stage. See the Snowflake doc section for more info. |
encryption | Optional | String | Required only for loading from/unloading into encrypted files and not required if storage location and files are decrypted |
file_format | Optional | String/Map: See here for a definition of file_format | Specifies the file format for the stage. Note: A change will always be detected for the file_format parameter in subsequent pipeline runs even when there isn't any in the case of:- Object names having lowercase characters or containing special characters in the parameter. - Specifying any format option(s) with a value set to its default value. |
grants | Optional | Map: See Supported Stage Grants to Roles | List of Privileges and Roles to which privileges are granted on the current stage |
manage_mode | Optional | String: all (default), none , grants | Configures what properties to manage for the stage. See Changing Manage Mode before changing the value. |
storage_integration | Optional | String | Specifies the name of the storage integration used to delegate authentication responsibility for external cloud storage to a Snowflake identity and access management (IAM) entity. See Storage Integration Usage for more info. |
URL | Optional | String | Specifies the URL for the stage. See Storage Integration Usage for more info. |
During subsequent pipeline runs, a force-replacement behavior might be observed in PLAN and APPLY phases for a few parameters like directory
.
Storage integration usage
You can specify storage integration in an external stage if Access is managed via Storage Integration.
In the configuration of Storage Integration, paths to allow external storage can be specified as well. If such paths have been specified, then the stage using the Storage Integration must have the value of parameter URL
as one of the allowed paths.
In the event, that both Storage Integration and Stage are being updated, an error could occur due to an invalid URL
parameter.
To resolve such an error, apply the following procedure:
- Disable the stage (comment out the definition in the configuration) and its dependencies and run a pipeline. This would delete the stage from the environment.
- Enable the stage (uncomment the definition) with updated configuration and run the pipeline again to initialize the stage from scratch.
Supported stage grants to roles
Following are the privileges you can grant to roles in the stage definition:
- ALL PRIVILEGES
- OWNERSHIP
- USAGE
- Only for External Stage
- READ
- Only for Internal Stage
- WRITE
- Only for Internal Stage
Implicit READ for internal stage
If Only WRITE is granted to a Role, READ privilege is implicitly added as well.
directory
parameters
The Directory parameter supports the following parameters.
Configuration Key | Required/Optional | Data Types and Values |
---|---|---|
auto_refresh | Optional | Boolean |
enable | Optional | Boolean |
notification_integration | Optional | String |
refresh_on_create | Optional | Boolean |
The below parameters are supported in the directory
parameter based on the type of stage and storage provider:
- For Internal stages:
enable
- For External Stage
- For Amazon S3:
enable
,auto_refresh
andrefresh_on_create
- For Google Cloud Storage and Microsoft Azure:
enable
,auto_refresh
,refresh_on_create
andnotification_integration
:::
- For Amazon S3:
Examples
directory
parameter
for internal stages
- Stage in Current Configuration
- Stage in SOLE for Data Products
databases:
SALES_RECORD:
comment: "product sales record"
schemas:
SALES_SCHEMA:
comment: "sales records for year 2021"
stages:
INTERNAL_STAGE:
comment: "files stage"
directory:
enable: true
- stage:
name: INTERNAL_STAGE
database: rel(database.SALES_RECORD)
schema: rel(schema.SALES_SCHEMA)
comment: "files stage"
directory:
enable: true
for Amazon S3
- Stage in Current Configuration
- Stage in SOLE for Data Products
databases:
SALES_RECORD:
comment: "product sales record"
schemas:
SALES_SCHEMA:
comment: "sales records for year 2021"
stages:
S3_STAGE:
url: "s3://{{AWS.DATAOPS.STAGING.S3_BUCKET}}"
credentials: "AWS_KEY_ID='{{AWS.DATAOPS.STAGING.ACCESS_KEY_ID}}' AWS_SECRET_KEY='{{AWS.DATAOPS.STAGING.SECRET_ACCESS_KEY}}'"
directory:
enable: true
auto_refresh: true
refresh_on_create: true
- stage:
name: S3_STAGE
database: rel(database.SALES_RECORD)
schema: rel(schema.SALES_SCHEMA)
url: "s3://{{AWS.DATAOPS.STAGING.S3_BUCKET}}"
credentials: "AWS_KEY_ID='{{AWS.DATAOPS.STAGING.ACCESS_KEY_ID}}' AWS_SECRET_KEY='{{AWS.DATAOPS.STAGING.SECRET_ACCESS_KEY}}'"
directory:
enable: true
auto_refresh: true
refresh_on_create: true
for Google cloud storage
- Stage in Current Configuration
- Stage in SOLE for Data Products
databases:
SALES_RECORD:
comment: "product sales record"
schemas:
SALES_SCHEMA:
comment: "sales records for year 2021"
stages:
GCS_STAGE:
url: "gcs://load/files/"
storage_integration: "my_storage_int"
directory:
enable: true
auto_refresh: true
refresh_on_create: true
notification_integration: "MY_NOTIFICATION_INT"
- stage:
name: GCS_STAGE
database: rel(database.SALES_RECORD)
schema: rel(schema.SALES_SCHEMA)
url: "gcs://load/files/"
storage_integration: "my_storage_int"
directory:
enable: true
auto_refresh: true
refresh_on_create: true
notification_integration: "MY_NOTIFICATION_INT"
for Microsoft Azure
- Stage in Current Configuration
- Stage in SOLE for Data Products
databases:
SALES_RECORD:
comment: "product sales record"
schemas:
SALES_SCHEMA:
comment: "sales records for year 2021"
stages:
AZURE_STAGE:
url: "azure://account.blob.core.windows.net/load/files"
storage_integration: "my_storage_int"
directory:
enable: true
auto_refresh: true
refresh_on_create: true
notification_integration: "MY_NOTIFICATION_INT"
- stage:
name: AZURE_STAGE
database: rel(database.SALES_RECORD)
schema: rel(schema.SALES_SCHEMA)
url: "azure://account.blob.core.windows.net/load/files"
storage_integration: "my_storage_int"
directory:
enable: true
auto_refresh: true
refresh_on_create: true
notification_integration: "MY_NOTIFICATION_INT"