Stage

You can provide configuration to Snowflake Object Lifecycle Engine for the following operations with stage:

• Manage the lifecycle of new and existing stages
• Manage the grants of a stage

Usage

note

We have introduced SOLE for Data Products as a new framework for SOLE to help you easily build an ecosystem of data products. The major difference is in how you define Snowflake objects in the configuration file. Rather than having a grouped collection of objects, SOLE for Data Products goes for modular, self-describing, and explicit object definition.
Learn more about SOLE for Data Products, currently available as a public preview.

We have also introduced Data products as an extra layer on top of the data product platform capabilities making managing data products easier than ever. Learn more about Data Products, currently available as a private preview.

Default Configuration

databases:
  <database-name>:
    schemas:
      <schema-name>:
        stages:
          <stage-name>:
            <configuration-key>: <value>

Data Products Configuration

- stage:
    name: <stage-name>
    <configuration-key>: <value>
    database: rel(database.<database-name>)
    schema: rel(schema.<schema-name>)

Supported parameters

The engine supports the parameters listed below.

Configuration Key	Required/Optional	Data Types and Values	Description
comment	Optional	String	Specifies a comment for the stage
copy_options	Optional	String: See here for a definition of copy_options	Specifies one (or more) copy options for the stage. See the Snowflake doc section for copy options.
credentials	Optional	String	Specifies the credentials for the stage, required only if the storage location is private/protected - not required for public buckets/containers
deleted	Optional	Boolean: True enables deletion prevention, False does nothing	Specifies what objects are allowed to be deleted
directory	Optional	String: See here for a definition of directory	Specifies the directory settings for the stage. See the Snowflake doc section for more info.
encryption	Optional	String	Required only for loading from/unloading into encrypted files and not required if storage location and files are decrypted
file_format	Optional	String/Map: See here for a definition of file_format	Specifies the file format for the stage. Note: A change will always be detected for the file_format parameter in subsequent pipeline runs even when there isn't any in the case of: - Object names having lowercase characters or containing special characters in the parameter. - Specifying any format option(s) with a value set to its default value.
grants	Optional	Map: See Supported Stage Grants to Roles	List of Privileges and Roles to which privileges are granted on the current stage
manage_mode	Optional	String: all (default), none, grants	Configures what properties to manage for the stage. See Changing Manage Mode before changing the value.
storage_integration	Optional	String	Specifies the name of the storage integration used to delegate authentication responsibility for external cloud storage to a Snowflake identity and access management (IAM) entity. See Storage Integration Usage for more info.
URL	Optional	String	Specifies the URL for the stage. See Storage Integration Usage for more info.

warning

During subsequent pipeline runs, a force-replacement behavior might be observed in PLAN and APPLY phases for a few parameters like directory.

Storage integration usage

You can specify storage integration in an external stage if Access is managed via Storage Integration.

In the configuration of Storage Integration, you can also specify paths to allow external storage. If you set such paths, the stage using the Storage Integration must have the parameter URL value as one of the allowed paths.

In the event, that both Storage Integration and Stage are being updated, an error could occur due to an invalid URL parameter.
To resolve such an error, apply the following procedure:

• Disable the stage (comment out the definition in the configuration) and its dependencies and run a pipeline. This would delete the stage from the environment.
• Enable the stage (uncomment the definition) with updated configuration and run the pipeline again to initialize the stage from scratch.

Supported stage grants to roles

Following are the privileges you can grant to roles in the stage definition:

• ALL PRIVILEGES
• OWNERSHIP
• USAGE - only for External Stage
• READ - only for Internal Stage
• WRITE - only for Internal Stage

ALL PRIVILEGES handling

When you define ALL PRIVILEGES in the SOLE configuration file, you grant all the privileges listed above to roles on this object except OWNERSHIP. However, the management of ALL PRIVILEGES in SOLE differs from its handling in Snowflake. See Handling ALL PRIVILEGES in SOLE for more information.

warning

WRITE operations on replicated stages will only succeed if you set the parameter BLOCK_NON_READLIST_OPERATIONS_ON_STAGES_IN_SECONDARY to FALSE. This change is needed because of recent updates in Snowflake releases, where TRUE becomes the default value.

Implicit READ for internal stage

If Only WRITE is granted to a Role, READ privilege is implicitly added as well.

directory parameters

The Directory parameter supports the following parameters.

Configuration Key	Required/Optional	Data Types and Values
auto_refresh	Optional	Boolean
enable	Optional	Boolean
notification_integration	Optional	String
refresh_on_create	Optional	Boolean

note

The below parameters are supported in the directory parameter based on the type of stage and storage provider:

• For Internal stages: enable
• For External Stage:
  • For Amazon S3: enable, auto_refresh and refresh_on_create
  • For Google Cloud Storage and Microsoft Azure: enable, auto_refresh, refresh_on_create and notification_integration :::

Examples

directory parameter

For internal stages

Default Configuration

databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for year 2021"
        stages:
          INTERNAL_STAGE:
            comment: "files stage"
            directory:
              enable: true

Data Products Configuration

- stage:
    name: INTERNAL_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    comment: "files stage"
    directory:
      enable: true

For Amazon S3

Default Configuration

databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for year 2021"
        stages:
          S3_STAGE:
            url: "s3://{{AWS.DATAOPS.STAGING.S3_BUCKET}}"
            credentials: "AWS_KEY_ID='{{AWS.DATAOPS.STAGING.ACCESS_KEY_ID}}' AWS_SECRET_KEY='{{AWS.DATAOPS.STAGING.SECRET_ACCESS_KEY}}'"
            directory:
              enable: true
              auto_refresh: true
              refresh_on_create: true

Data Products Configuration

- stage:
    name: S3_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "s3://{{AWS.DATAOPS.STAGING.S3_BUCKET}}"
    credentials: "AWS_KEY_ID='{{AWS.DATAOPS.STAGING.ACCESS_KEY_ID}}' AWS_SECRET_KEY='{{AWS.DATAOPS.STAGING.SECRET_ACCESS_KEY}}'"
    directory:
      enable: true
      auto_refresh: true
      refresh_on_create: true

For Google cloud storage

Default Configuration

databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for year 2021"
        stages:
          GCS_STAGE:
            url: "gcs://load/files/"
            storage_integration: "my_storage_int"
            directory:
              enable: true
              auto_refresh: true
              refresh_on_create: true
              notification_integration: "MY_NOTIFICATION_INT"

Data Products Configuration

- stage:
    name: GCS_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "gcs://load/files/"
    storage_integration: "my_storage_int"
    directory:
      enable: true
      auto_refresh: true
      refresh_on_create: true
      notification_integration: "MY_NOTIFICATION_INT"

For Microsoft Azure

Default Configuration

databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for year 2021"
        stages:
          AZURE_STAGE:
            url: "azure://account.blob.core.windows.net/load/files"
            storage_integration: "my_storage_int"
            directory:
              enable: true
              auto_refresh: true
              refresh_on_create: true
              notification_integration: "MY_NOTIFICATION_INT"

Data Products Configuration

- stage:
    name: AZURE_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "azure://account.blob.core.windows.net/load/files"
    storage_integration: "my_storage_int"
    directory:
      enable: true
      auto_refresh: true
      refresh_on_create: true
      notification_integration: "MY_NOTIFICATION_INT"