Skip to main content

Explicit Object Deletion

This mechanism helps you protect your data and avoid accidental deletions. It ensures that Snowflake objects aren't deleted unexpectedly. Common scenarios involving unintentional deletes include:

  • Remove the code related to an object, database, or user
  • Comment out the code related to an object, database, or user
  • Resolve Git merge conflicts incorrectly
  • Change the name of the object or Database

SOLE prevents the accidental deletion of your data by enforcing stricter rules on deleting objects in Snowflake. This topic discusses the following:

Managing explicit deletion​

Deletion prevention is turned on by default. If you need to turn it off, set the variable DATAOPS_PREVENT_OBJECT_DELETION to 0 in your project and commit the change to the variables.yml file. For more information, see our documentation on project variables.

caution

Running a pipeline with LIFECYCLE_STATE_RESET ignores the restriction on deleting objects set by DATAOPS_PREVENT_OBJECT_DELETION.

Preventing accidental deletion of objects​

You can explicitly delete one or more objects by setting the deleted attribute on them. Doing so lets SOLE know that the deletion is intentional.

note

Deleting a parent object will also delete any child objects, even if they are not marked as deleted.

dataops/snowflake/database.yml
- database:
    name: DATABASE_1
    comment: "new"
    deleted: true
caution

Running a pipeline with LIFECYCLE_STATE_RESET ignores the restriction on deleting objects set by DATAOPS_PREVENT_OBJECT_DELETION.

In the following example, even though only the database is marked as deleted, the schema will also be deleted because it belongs to the database.

dataops/snowflake/database.yml
- database:
    name: DATABASE_1
    comment: "new"
    deleted: true

- schema:
    name: SCHEMA_1
    # deleted: true
    database: rel(database.DATABASE_1)
    comment: schema1

In the same way, having this feature enabled lets SOLE know if the deletion is unintentional. If part of your configuration is removed, this will be considered a missing object that has not been explicitly marked to be deleted, which will cause the SOLE job to fail and thus protect you against data loss.

For example, if you manually remove the configuration above for DATABASE_1, the SOLE job will fail, and you should see some logging like the following:

[ DATABASE ] -  [WARNING] - [16:25:59.0654] - Found missing object: DATABASE_1
[ DATABASE ] -  [ ERROR] - [16:25:59.0654] - Found missing objects that would be unintentionally deleted. See above for details.

Moving objects between projects​

Sometimes, you may want to stop managing an object in SOLE so that you can handle it manually or move it to a different project. To do this:

  1. Set the object manage_mode property to none.

    dataops/snowflake/database.yml
    - database:
        name: DATABASE_3
        comment: "DATABASE 3"
        manage_mode: none

    SOLE stops managing the object, but it will still exist in Snowflake. Then, if you were to remove this configuration from your project, SOLE would not complain as it is no longer managing the object and thus cannot be unintentionally deleted.

  2. Add the object to SOLE in another project, remove manage_mode, and run the pipeline.

    The object is imported into the new project, and you can start managing it there.

Last updated on