Skip to main content

State Management

Before we look at how SOLE manages a Snowflake object's state, it is crucial to understand what state is in the context of DataOps and SOLE.

State (or local state) is the "condition" of all the Snowflake objects at a given point in time.

In other words, SOLE asks the following questions when inspecting the organization's Snowflake infrastructure:

  • What objects exist?
  • What do they look like, or how are they structured?

In practice, SOLE maps its configuration to the real world or the physical Snowflake ecosystem, allowing SOLE to use the existing state of the objects during subsequent DataOps pipeline runs.

This information is saved to a local state file and used during SOLE's PLAN and PLAN-DESTROY lifecycle actions and is saved in the persistent cache of the DataOps Runner's host system.

SOLE state management​

The concept of state management builds on this definition of local state within the DataOps ecosystem. In summary, SOLE requires the local state information to function. And based on this requirement, state management is an integral part of the Snowflake Object Lifecycle Engine's processes.

Each environment and branch maintains its own set of states. Therefore, SOLE writes a local state file for each object group. Consequently, it is reasonable to conclude that, depending on the size of the organization and the number of Snowflake objects SOLE manages, there will be a significant number of state files to maintain and manage.

These state files are stored on the host system at the following path:

/<cache_directory>/persistent_cache/<dataops_project_name>/<branch_name>/snowflakelifecycle-runner/<resource_group>/<resource_group>.tfstate

The values in angle brackets (<>) translate into the following variables:

  • cache_directory: The host system cache directory

    • The default is /agent_cache
    • To retrieve the value for your DataOps Runner, see the volume mounts in the /srv/<agent_name>/config/config.toml file
    • Refer to the DataOps Runner initial setup info

  • dataops_project_name: The project name in lower-case

  • branch_name: The branch name in lower-case

  • resource_group: The object group name

note

State is maintained for all object groups, except for grants.

State reset​

If the state file is corrupted or does not match the actual Snowflake infrastructure, pipeline failures can occur.

In such scenarios, you can reset the local state file to perform a fresh inspection of Snowflake. The reset can be triggered by the variable LIFECYCLE_STATE_RESET set to 1.

info

Resetting the local state file deletes the existing local state for the specified resource group and re-imports all managed Snowflake Objects.

You can configure the state file reset at the project level in pipelines/includes/config/variables.yml to reset the state of all object groups. Or it can be configured at an individual object group level as a parameter to the PLAN or PLAN-DESTROY jobs.

note

The individual object group state reset function is not supported in AGGREGATE jobs.

Last updated on