Managing OpenSearch Index Lifecycle with ISM Policies

Index State Management (ISM) in OpenSearch is a powerful feature that helps automate the lifecycle of your indices. One common use case is automatically deleting old indices to save storage space and maintain cluster performance. Let’s walk through creating a policy that handles this task.

What is Index State Management?

ISM allows you to define policies that automatically transition indices through different states based on conditions like age, size, or document count. Instead of manually managing indices, you can set up rules that OpenSearch follows automatically.

Creating a Delete Policy

Here’s an example policy that deletes indices after 90 days:

{
  "policy": {
    "policy_id": "delete-old-indices",
    "description": "Policy that deletes indices after 90 days",
    "default_state": "initial",
    "states": [
      {
        "name": "initial",
        "actions": [],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "90d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "5m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": [
          "events-archive-*"
        ],
        "priority": 1
      }
    ]
  }
}

How This Policy Works

The policy defines two states:

1. Initial state: This is where new indices start. The policy waits here until the index reaches 90 days old, then transitions to the delete state.

2. Delete state: This state performs the actual deletion with retry logic in case the operation fails temporarily.

The ism_template section automatically applies this policy to any index matching the pattern events-archive-*. You can adjust this pattern to match your specific index naming convention.

Managing ISM Policies via API

Creating a New Policy

To create a new ISM policy, use the PUT request:

PUT _plugins/_ism/policies/delete-old-indices
{
  "policy": {
    "policy_id": "delete-old-indices",
    "description": "Policy that deletes indices after 90 days",
    // ... rest of policy definition
  }
}

Updating an Existing Policy

Updating policies requires a bit more work due to OpenSearch’s optimistic concurrency control. You’ll need to get the current policy first to retrieve the sequence number and primary term:

GET _plugins/_ism/policies/delete-old-indices

This returns metadata including _seq_no and _primary_term. Use these values when updating:

PUT _plugins/_ism/policies/delete-old-indices?if_seq_no=5&if_primary_term=1
{
  "policy": {
    // updated policy definition
  }
}

Best Practices

• Test first: Always test your policies on non-production data to ensure they work as expected.
• Set appropriate retry logic: The example includes retry configuration to handle temporary failures.
• Monitor policy execution: Check the ISM plugin’s status regularly to ensure policies are running correctly.
• Be careful with patterns: Make sure your index patterns are specific enough to avoid accidentally affecting unintended indices.

Wrapping Up

ISM policies provide a robust way to automate index management in OpenSearch. The delete policy shown here is just one example – you can create more complex policies that include steps like moving to cold storage or changing replica counts before deletion.

Remember to always test your policies thoroughly before applying them to production indices, as deletion operations cannot be undone.