Skip to content

Stage

You can provide configuration to Snowflake Object Lifecycle Engine for the following operations with stage:

  • Manage the lifecycle of new and existing stages
  • Manage the grants of a stage

Usage

- stage:
    name: <stage-name>
    <configuration-key>: <value>
    database: rel(database.<database-name>)
    schema: rel(schema.<schema-name>)
databases:
  <database-name>:
    schemas:
      <schema-name>:
        stages:
          <stage-name>:
            <configuration-key>: <value>

Supported parameters

The engine supports the parameters listed below.

Configuration Key Required/Optional Data Types and Values Description
comment Optional String Specifies a comment for the stage
copy_options Optional String: See here for a definition of copy_options Specifies one (or more) copy options for the stage. See the Snowflake doc section for copy options.
credentials Optional String Specifies the credentials for the stage. Only required if the storage location is private/protected. Not required for public buckets/containers.
deleted Optional Boolean: True enables deletion prevention, False does nothing Specifies what objects are allowed to be deleted
depends_on Optional List of String: See Object Dependencies for a definition of depends_on List of SOLE-managed objects that this stage depends on. Objects defined in the list will be added to the generated HCL dependencies list to ensure proper creation order.
directory Optional String: See here for a definition of directory Specifies the directory settings for the stage. See the Snowflake doc section for more info.
endpoint Optional String Specifies the endpoint for the stage for S3-compatible storage. The fully-qualified domain that points to the S3-compatible API endpoint.
encryption Optional String Required only for loading from/unloading into encrypted files and not required if storage location and files are decrypted
file_format Optional String/Map: See here for a definition of file_format Specifies the file format for the stage.
Note: A change will always be detected for the file_format parameter in subsequent pipeline runs even when there isn't any in the case of:
- Object names having lowercase characters or containing special characters in the parameter.
- Specifying any format option(s) with a value set to its default value.
grants Optional Map: See Supported Stage Grants to Roles List of Privileges and Roles to which privileges are granted on the current stage
manage_mode Optional String: all (default), none, grants Configures what properties to manage for the stage.
See Changing Manage Mode before changing the value.
storage_integration Optional String Specifies the name of the storage integration used to delegate authentication responsibility for external cloud storage to a Snowflake identity and access management (IAM) entity. See Storage Integration Usage for more info.
URL Optional String Specifies the URL for the stage. See Storage Integration Usage for more info.

Warning

During subsequent pipeline runs, a force-replacement behavior might be observed in PLAN and APPLY phases for a few parameters like directory.

Storage integration usage

You can specify storage integration in an external stage if Access is managed via Storage Integration.

You can also specify paths to allow external storage in the Storage Integration configuration. If you set such paths, the stage using the Storage Integration must have the parameter URL value as one of the allowed paths.

In the event that both Storage Integration and Stage are being updated, an error could occur due to an invalid URL parameter.
To resolve such an error, apply the following procedure:

  • Disable the stage (comment out the definition in the configuration) and its dependencies and run a pipeline. This would delete the stage from the environment.
  • Enable the stage (uncomment the definition) with updated configuration and run the pipeline again to initialize the stage from scratch.

Supported stage grants to roles

Following are the privileges you can grant to roles in the stage definition:

  • ALL PRIVILEGES
  • OWNERSHIP
  • USAGE - only for External Stage
  • READ - only for Internal Stage
  • WRITE - only for Internal Stage

ALL PRIVILEGES handling

When you define ALL PRIVILEGES in the SOLE configuration file, you grant all the privileges listed above to roles on this object except OWNERSHIP. However, the management of ALL PRIVILEGES in SOLE differs from its handling in Snowflake. See Handling ALL PRIVILEGES in SOLE for more information.

Warning

WRITE operations on replicated stages will only succeed if you set the parameter BLOCK_NON_READLIST_OPERATIONS_ON_STAGES_IN_SECONDARY to FALSE. This change is needed because of recent updates in Snowflake releases, where TRUE becomes the default value.

Implicit READ for internal stage

If only WRITE is granted to a Role, the READ privilege is implicitly added as well.

directory parameters

The Directory parameter supports the following parameters.

Configuration Key Required/Optional Data Types and Values
auto_refresh Optional Boolean
enable Optional Boolean
notification_integration Optional String
refresh_on_create Optional Boolean

Note

The below parameters are supported in the directory parameter based on the type of stage and storage provider:

  • For Internal stages: enable
  • For External Stage:
  • For Amazon S3: enable, auto_refresh and refresh_on_create
  • For Google Cloud Storage and Microsoft Azure: enable, auto_refresh, refresh_on_create and notification_integration

Examples

directory parameter

For internal stages

- stage:
    name: INTERNAL_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    comment: "files stage"
    directory:
      enable: true
databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for year 2021"
        stages:
          INTERNAL_STAGE:
            comment: "files stage"
            directory:
              enable: true

For Amazon S3

- stage:
    name: S3_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "s3://{{AWS.DATAOPS.STAGING.S3_BUCKET}}"
    credentials: "AWS_KEY_ID='{{AWS.DATAOPS.STAGING.ACCESS_KEY_ID}}' AWS_SECRET_KEY='{{AWS.DATAOPS.STAGING.SECRET_ACCESS_KEY}}'"
    directory:
      enable: true
      auto_refresh: true
      refresh_on_create: true
databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for the year 2024"
        stages:
          S3_STAGE:
            url: "s3://{{AWS.DATAOPS.STAGING.S3_BUCKET}}"
            credentials: "AWS_KEY_ID='{{AWS.DATAOPS.STAGING.ACCESS_KEY_ID}}' AWS_SECRET_KEY='{{AWS.DATAOPS.STAGING.SECRET_ACCESS_KEY}}'"
            directory:
              enable: true
              auto_refresh: true
              refresh_on_create: true

For Google cloud storage

- stage:
    name: GCS_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "gcs://load/files/"
    storage_integration: "my_storage_int"
    directory:
      enable: true
      auto_refresh: true
      refresh_on_create: true
      notification_integration: "MY_NOTIFICATION_INT"
databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for the year 2024"
        stages:
          GCS_STAGE:
            url: "gcs://load/files/"
            storage_integration: "my_storage_int"
            directory:
              enable: true
              auto_refresh: true
              refresh_on_create: true
              notification_integration: "MY_NOTIFICATION_INT"

For Microsoft Azure

- stage:
    name: AZURE_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "azure://account.blob.core.windows.net/load/files"
    storage_integration: "my_storage_int"
    directory:
      enable: true
      auto_refresh: true
      refresh_on_create: true
      notification_integration: "MY_NOTIFICATION_INT"
databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for the year 2024"
        stages:
          AZURE_STAGE:
            url: "azure://account.blob.core.windows.net/load/files"
            storage_integration: "my_storage_int"
            directory:
              enable: true
              auto_refresh: true
              refresh_on_create: true
              notification_integration: "MY_NOTIFICATION_INT"

For Wasabi Storage (or any S3-compatible storage)

- stage:
    name: TEST_STAGE
    database: rel(database.SALES_RECORD)
    schema: rel(schema.SALES_SCHEMA)
    url: "s3compat://dataops"
    endpoint: "s3.us-west-1.wasabisys.com"
    credentials: "AWS_KEY_ID='<ACCESS_KEY>' AWS_SECRET_KEY='<SECRET_ACCESS_KEY>'"
databases:
  SALES_RECORD:
    comment: "product sales record"
    schemas:
      SALES_SCHEMA:
        comment: "sales records for the year 2024"
        stages:
          TEST_STAGE:
            url: "s3compat://dataops"
            endpoint: "s3.us-west-1.wasabisys.com"
            credentials: "AWS_KEY_ID='<ACCESS_KEY>' AWS_SECRET_KEY='<SECRET_ACCESS_KEY>'"