How to create snapshot and restore snapshot with elasticsearch

In this post we will show you how to create snapshots and restore those snapshots with ElasticSearch easily. This is a very easy method if you want to migrate your current elasticsearch cluster to a new version, which cannot be performed on major upgrades, and you don't want to loose any data.

First you will need to add the repo.path location to your elasticsearch.yml. This will be a path on your local filesystem or you can use some plugins which would take snapshots to azure, amazon, or google cloud (available only on v5.x currently alpha).

We will create a folder under /etc/elasticsearch named backup.

#mkdir -p /etc/elasticsearch/backup

This will be the path where the snapshots repositories will be created. Now we will add this path to the elasticsearch.yml by editing the /etc/elasticsearch/elasticsearch.yml config file and adding the following at the end of the file:

cat >> /etc/elasticsearch/elasticsearch.yml << EOF
path.repo: ["/etc/elasticsearch/backup"]
EOF

Now we restart the elasticsearch service on the node and make sure that the backup path is writable by the elasticsearch user.

#systemctl restart elasticsearch
#chown -R elasticsearch. /etc/elasticsearch/backup

First elasticsearch needs to know the backup path by registering a backup repository:

#curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d {
  "type": "fs",
  "settings": {
     "location": "/etc/elasticsearch/backup",
     "compress": true
  }
}'

Once elasticsearch knows about the backup repository we can create a backup of the entire cluster or specific indices. We will do a full backup using a single command:

#curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_1?wait_for_completion=true"

To restore a snapshot it cannot be easier than it is:

#curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_1/_restore?wait_for_completion=true"

You can list your snapshots by running the following curl command:

#curl -XGET 'localhost:9200/_snapshot/my_backup/_all?pretty'
{
  "snapshots" : [ {
    "snapshot" : "latest",
    "version_id" : 5000003,
    "version" : "5.0.0-alpha3",
    "indices" : [ "logstash-2016.06.04", "metricbeat-2016.06.10", "metricbeat-2016.06.15", "logstash-2016.06.06", "metricbeat-2016.06.04", "packetbeat-2016.06.07", "packetbeat-2016.06.15", "logstash-2016.06.20", "logstash-2016.06.03", "metricbeat-2016.06.09", "logstash-2016.06.24", "packetbeat-2016.06.08", "metricbeat-2016.06.24", ".monitoring-es-2-2016.06.20", "metricbeat-2016.06.20", "packetbeat-2016.06.12", "logstash-2016.06.26", ".monitoring-es-2-2016.06.27", "logstash-2016.06.12", "packetbeat-2016.06.25", "logstash-2016.06.14", "logstash-2016.06.15", "packetbeat-2016.06.17", "metricbeat-2016.06.25", "metricbeat-2016.06.16", "logstash-2016.06.08", ".kibana", "packetbeat-2016.06.24", "packetbeat-2016.06.04", "metricbeat-2016.06.08", "metricbeat-2016.06.21", "metricbeat-2016.06.14", ".monitoring-es-2-2016.06.26", "logstash-2016.06.16", ".monitoring-data-2", "packetbeat-2016.06.09", "packetbeat-2016.06.26", "metricbeat-2016.06.13", "metricbeat-2016.06.17", ".monitoring-es-2-2016.06.25", "logstash-2016.06.05", "packetbeat-2016.06.10", "metricbeat-2016.06.27", "metricbeat-2016.06.26", "packetbeat-2016.06.02", "metricbeat-2016.06.03", "packetbeat-2016.06.13", "packetbeat-2016.06.14", "packetbeat-2016.06.21", "logstash-2016.06.07", "packetbeat-2016.06.03", "logstash-2016.06.02", "packetbeat-2016.06.16", "logstash-2016.06.17", "metricbeat-2016.06.07", "metricbeat-2016.06.06", "logstash-2016.06.09", "logstash-2016.06.25", "packetbeat-2016.06.06", "logstash-2016.06.10", "logstash-2016.06.27", "logstash-2016.06.11", "metricbeat-2016.06.11", "packetbeat-2016.06.11", "logstash-2016.06.13", "packetbeat-2016.06.20", "metricbeat-2016.06.12", "metricbeat-2016.06.02", "packetbeat-2016.06.27" ],
    "state" : "SUCCESS",
    "start_time" : "2016-06-27T19:07:31.299Z",
    "start_time_in_millis" : 1467054451299,
    "end_time" : "2016-06-27T19:10:57.853Z",
    "end_time_in_millis" : 1467054657853,
    "duration_in_millis" : 206554,
    "failures" : [ ],
    "shards" : {
      "total" : 321,
      "failed" : 0,
      "successful" : 321
    }
  }, {
    "snapshot" : "snapshot_1",
    "version_id" : 5000003,
    "version" : "5.0.0-alpha3",
...
    "state" : "SUCCESS",
    "start_time" : "2016-06-27T20:32:47.651Z",
    "start_time_in_millis" : 1467059567651,
    "end_time" : "2016-06-27T20:33:00.297Z",
    "end_time_in_millis" : 1467059580297,
    "duration_in_millis" : 12646,
    "failures" : [ ],
    "shards" : {
      "total" : 321,
      "failed" : 0,
      "successful" : 321
    }
  }, {
    "snapshot" : "snapshot_2",
    "version_id" : 5000003,
    "version" : "5.0.0-alpha3",
....
    "state" : "SUCCESS",
    "start_time" : "2016-06-27T20:51:50.555Z",
    "start_time_in_millis" : 1467060710555,
    "end_time" : "2016-06-27T20:51:59.761Z",
    "end_time_in_millis" : 1467060719761,
    "duration_in_millis" : 9206,
    "failures" : [ ],
    "shards" : {
      "total" : 321,
      "failed" : 0,
      "successful" : 321
    }
  } ]
}

For deleting a snapshot you can do the following curl -XDELETE 'localhost:9200/_snapshot/my_backup/snapshotname' I will delete the snapshot called latest.

#curl -XDELETE 'localhost:9200/_snapshot/my_backup/latest'

Then to check the list of snapshots we run again the following command:

#curl -XGET 'localhost:9200/_snapshot/my_backup/_all?pretty'
{
  "snapshots" : [ {
    "snapshot" : "snapshot_1",
    "version_id" : 5000003,
    "version" : "5.0.0-alpha3",
    ....
    "state" : "SUCCESS",
    "start_time" : "2016-06-27T20:32:47.651Z",
    "start_time_in_millis" : 1467059567651,
    "end_time" : "2016-06-27T20:33:00.297Z",
    "end_time_in_millis" : 1467059580297,
    "duration_in_millis" : 12646,
    "failures" : [ ],
    "shards" : {
      "total" : 321,
      "failed" : 0,
      "successful" : 321
    }
  }, {
    "snapshot" : "snapshot_2",
    "version_id" : 5000003,
    "version" : "5.0.0-alpha3",
    ....
    "state" : "SUCCESS",
    "start_time" : "2016-06-27T20:51:50.555Z",
    "start_time_in_millis" : 1467060710555,
    "end_time" : "2016-06-27T20:51:59.761Z",
    "end_time_in_millis" : 1467060719761,
    "duration_in_millis" : 9206,
    "failures" : [ ],
    "shards" : {
      "total" : 321,
      "failed" : 0,
      "successful" : 321
    }
  } ]
}

If you don't put the wait_for_completion=true when creating a snapshot the process will run in the background. You can take the status of the snapshot by running the following command:

#curl -XGET -u zozo http://localhost:9200/_snapshot/my_backup/snapshot_3/_status?pretty
{
  "snapshots" : [ {
    "snapshot" : "snapshot_3",
    "repository" : "my_backup",
    "state" : "STARTED",
    "shards_stats" : {
      "initializing" : 26,
      "started" : 0,
      "finalizing" : 0,
      "done" : 295,
      "failed" : 0,
      "total" : 321
},

As you can see this will list the status of the snapshot showing at time this command was executed has finished 296 shards and initialised 26 shards.