Skip to main content

Stage

Configuration can be provided to Snowflake Object Lifecycle Engine for the following operation with stage:

  • Manage Lifecycle of new and existing Stage
  • Manage Grants of Stage

Supported Parameters

The engine supports the parameters listed below.

  • URL: Specifies the URL for the stage.
  • CREDENTIALS: Specifies the credentials for the stage, required only if the storage location is private/protected; not required for public buckets/containers.
    • Configuration key: credentials
    • Data Type: String
  • COPY_OPTIONS: Specifies one (or more) copy options for the stage.
    • Configuration key: copy_options
    • Data Type: String . See here for copy_options parameters.
      info

      See Snowflake Docs section for copy options.

  • ENCRYPTION: Required only for loading from/unloading into encrypted files; not required if storage location and files are decrypted.
    • Configuration key: encryption
    • Data Type: String
  • FILE_FORMAT: Specifies the file format for the stage.
    • Configuration key: file_format
    • Data Type: String/Map. See here for file_format parameters
      info

      See Snowflake Docs section for file format.

      note

      A change will always be detected for the FILE_FORMAT parameter in subsequent pipeline runs even when there isn't any in the case of:
      i) For object names having lowercase characters or containing special characters in the parameter.
      ii) Specifying any format option(s) with a value set to its default value.

  • STORAGE_INTEGRATION: Specifies the name of the storage integration used to delegate authentication responsibility for external cloud storage to a Snowflake identity and access management (IAM) entity.
  • COMMENT: Specifies a comment for the stage.
    • Configuration key: comment
    • Data Type: String
  • DIRECTORY: Specifies the directory settings for the stage.
    • Configuration key: directory
    • Data Type: String See here for directory parameters.
      info

      See Snowflake Docs section for more info

  • MANAGE_MODE: Configures what properties to manage for the Stage.
    • Configuration key: manage_mode
    • Data Type: String
    • Possible Values:
      • none
      • grants
      • all(Default)
  • GRANTS: List of Privileges and Roles to which privileges are granted to on the current Stage.
    • Configuration key: grants
    • Data Type: Map
caution

During subsequent pipeline runs a force-replacement behavior might be observed in PLAN and APPLY phase for few parameters like directory.

Storage Integration Usage

A Storage Integration can be specified in an external stage if Access is managed via Storage Integration.

In the configuration of Storage Integration, paths to allow external storage can be specified as well. If such paths have been specified, then the stage using the Storage Integration must have the value of parameter URL as one of the allowed paths.

In the event, that both Storage Integration and Stage are being updated, an error could occur due to an invalid URL parameter.
To resolve such an error, apply the following procedure:

  • Disable the stage (comment out the definition in the configuration) and its dependencies and run a pipeline. This would delete the stage from the environment.
  • Enable the stage (uncomment the definition) with updated configuration and run the pipeline again to initialize the stage from scratch.

Supported Stage Grants to Roles

Following is the list of Privileges Grant to Roles that can be specified in the stage definition

  • ALL PRIVILEGES
  • USAGE
    • Only for External Stage only
  • READ
    • Only for Internal Stage only
  • WRITE
    • Only for Internal Stage only

Implicit READ for Internal Stage

If Only WRITE is granted to a Role, READ privilege is implicitly added as well.

Directory parameters

The Directory parameter supports the following parameters.

  • ENABLE:
    • Configuration Key: enable
    • Data Type: Boolean
  • AUTO_REFRESH:
    • Configuration Key: auto_refresh
    • Data Type: Boolean
  • REFRESH_ON_CREATE:
    • Configuration Key: refresh_on_create
    • Data Type: Boolean
  • NOTIFICATION_INTEGRATION:
    • Configuration Key: notification_integration
    • Data Type: String
note

Below parameters are supported in the DIRECTORY parameter based on the type of stage and storage provider:

  • For Internal stages: ENABLE
  • For External Stage
    • For Amazon S3: ENABLE, AUTO_REFRESH and REFRESH_ON_CREATE
    • For Google Cloud Storage and Microsoft Azure: ENABLE, AUTO_REFRESH, REFRESH_ON_CREATE and NOTIFICATION_INTEGRATION

Basic syntax

stages:
<stage-name>:
<configuration-key>: <value>

Examples

Directory parameter

databases:
SALES_RECORD:
comment: "product sales record"
schemas:
SALES_SCHEMA:
comment: "sales records for year 2021"
stages:
S3_STAGE:
comment: "files stage"
directory:
enable: true