Skip to main content

Explicit Object Deletion

This mechanism helps you protect your data and avoid accidental deletions. It ensures that Snowflake objects aren't deleted unexpectedly. Common scenarios involving unintentional deletes include:

  • Remove the code related to an object, database, or user
  • Comment out the code related to an object, database, or user
  • Resolve Git merge conflicts incorrectly
  • Change the name of the object or Database

SOLE prevents the accidental deletion of your data by enforcing stricter rules on deleting objects in Snowflake. This topic discusses the following:

Enabling explicit deletion​

To enable this feature, set the variable DATAOPS_PREVENT_OBJECT_DELETION to 1 in your project and commit to the variables.yml file. See Project Variables for more information.

caution

Running a pipeline with LIFECYCLE_STATE_RESET ignores the restriction on deleting objects set by DATAOPS_PREVENT_OBJECT_DELETION.

Preventing accidental deletion of objects​

When you enable this feature, you can delete one or more objects explicitly by setting the deleted attribute on them. This lets SOLE know that the deletion is intentional.

note

Deleting a parent object will also delete any child objects even if they are not marked as deleted.

dataops/snowflake/database.yml
databases:
  DATABASE_1:
    comment: "new"
    deleted: true
caution

Running a pipeline with LIFECYCLE_STATE_RESET ignores the restriction on deleting objects set by DATAOPS_PREVENT_OBJECT_DELETION.

In the following example, even though only the database is marked as deleted, the schema will also be deleted because it belongs to the database.

dataops/snowflake/database.yml
databases:
  DATABASE_1:
    comment: "new"
    deleted: true
    schemas:
      SCHEMA_1:
        # deleted: true
        comment: schema1
        grants:
          CREATE TABLE:
            - DATAOPS_SOLE_ADMIN_ROLE

In the same way, having this feature enabled lets SOLE know if the deletion is unintentional. If part of your configuration is removed, this will be considered a missing object that hasn't been explicitly marked to be deleted and will cause the SOLE job to fail and thus protects you against data loss.

For example, if you manually remove the configuration above for DATABASE_1, the SOLE job will fail, and you should see some logging like the following:

[ DATABASE ] -  [WARNING] - [16:25:59.0654] - Found missing object: DATABASE_1
[ DATABASE ] -  [ ERROR] - [16:25:59.0654] - Found missing objects that would be unintentionally deleted. See above for details.

Moving objects between projects​

Sometimes, you may want to stop managing an object in SOLE so that you can handle it manually or move it to a different project. To do this:

  1. Set the object manage_mode property to none.

    dataops/snowflake/database.yml
    databases:
      DATABASE_3:
        comment: DATABASE 3
        manage_mode: none

    SOLE stops managing the object but it will still exist in Snowflake. Then if you were to remove this configuration from your project, SOLE would not complain as it's no longer managing the object and thus cannot be unintentionally deleted.

  2. Add the object to SOLE in another project, remove manage_mode, and run the pipeline.

    The object is imported into the new project and you can start managing it there.

Last updated on