X
Popular Searches

How To Rotate and Delete Old Elasticsearch Records After a Month

Elasticsearch indices can quickly fill up with gigabytes of data, especially if you’re logging from multiple servers many times a second. To manage data, Elasticsearch

Deleting Using The “Delete By Query” API

Elasticsearch offers a “Delete By Query” API, that will remove all documents matching a query. You can use this to match timestamps greater or less than a certain date, albeit a bit crudely:

POST indexname/_delete_by_query
{
  "query": {
    "range" : {
      "@timestamp" : {
        "gte" : "09/02/2020",
        "lte" : "11/02/2020",
        "format": "dd/MM/yyyy||yyyy"
      }
    }
  }
}

However, this query is really slow. It scales linearly with document size. If you have enough documents that you need to be rotating them to prevent your Elasticsearch instance from bursting into flames, you probably can’t delete records this way, and will need to use time-based indices instead.

A Better Method: Time Based Indices

In Elasticsearch, you don’t usually use indexes directly. Your dashboards use index patterns, which can match multiple indexes at once. The reason for this is that the indexes themselves can act as groups of data, such as grouping by day or month.

It’s much easier to manage and rotate entire indices, so if you had each ingester configured to add the current date to the index name,

index: "indexname-%{+yyyy.MM.dd}"
Advertisement

Of course, this requires you to configure the ingest pipeline to write to the daily index. You’ll need to set up your loggers to ingest data in this format.

Once that’s done though, you can create a new Index Lifecycle Policy to handle the automatic rollover of data. This option is available under “Stack Management” in the Kibana dashboard.

You can configure multiple phases of index rollover, but for this purpose it’s easier to just disable rollover and enable the delete phase, configuring it to remove indices older than X number of days.

Then, to actually apply it to an index template, you’ll need to select “Add Policy To Index Template” under “Actions” in the lifecycle policy list.

Select the index pattern you wish to add, and the policy should take effect immediately, and your old indices in the pattern will be deleted.

Anthony Heddings Anthony Heddings
Anthony Heddings is the resident cloud engineer for LifeSavvy Media, a technical writer, programmer, and an expert at Amazon's AWS platform. He's written hundreds of articles for How-To Geek and CloudSavvy IT that have been read millions of times. Read Full Bio »

The above article may contain affiliate links, which help support CloudSavvy IT.