How to create your own in house elasticsearch alerts.

Since I am using elasticsearch for getting server side healthchecks, statistics and check on services which are important for the blog to function, I was wondering if I can use elasticsearch to write my own alerts in case something goes down. Therefore I have installed heartbeat on the elasticsearch server which is very easy to install and configure like all the beats elastic provide. I have configured it to check on the website uptime every 5 seconds.
Now I have used kibana to come up with the queries. First we will do the timestamp limit and since our alerting check will be running every 5 minutes I create a filter on heartbeat-* index between now and now-5m

Then click on Edit Query DSL and take the query format from there:

Now it's time to build the query to filter for monitor.status: "down":

Now that we have that information it's time to build the query:

curl -s http://localhost:9200/heartbeat-*/_search?pretty -H 'Content-Type: application/json' -d '
{
  "query":{
    "bool":{
      "must":[
        {
          "range" : {
            "@timestamp" : {
              "gte" : "now-5m",
              "lt" : "now"
            }
          }
        },
        {
          "term": {
            "monitor.status": "down"
          }
        }
      ]
    }
  }
}'

Now since we have the query built we can filter anything we would need out it is for getting all the informations for putting it in the e-mail.

I am using linux provided tools to get the informations I need. My very simple bash script looks like this:

#!/bin/bash
STATUS_CHECK='{ "query":{ "bool":{ "must":[ { "range" : { "@timestamp" : { "gte" : "now-5m", "lt" : "now" } } }, { "term": { "monitor.status": "down" } } ] } } }'
URL_REQ_STATUS="curl -s http://localhost:9200/heartbeat-*/_search?pretty"
while true
do
  STATUS_DOWN_CURL=$($URL_REQ_STATUS -H 'Content-Type: application/json' -d "$STATUS_CHECK")
  STATUS_DOWN=$($URL_REQ_STATUS -H 'Content-Type: application/json' -d "$STATUS_CHECK" | grep status |  awk '{print $3}' | grep "[a-z]" | wc -l)
  STATUS_DOWN_HOST=$($URL_REQ_STATUS -H 'Content-Type: application/json' -d "$STATUS_CHECK" | grep '"id"' | awk '{print $3}' | tail -1)
  STATUS_DOWN_ENV=$($URL_REQ_STATUS -H 'Content-Type: application/json' -d "$STATUS_CHECK" | grep env | awk '{print $3}' | tail -1)
  if [[ ${STATUS_DOWN} > 0 ]]; then
    echo "The servce $STATUS_DOWN_HOST is unresponsive on $STATUS_DOWN_ENV." | mail -s "Alert: HTTP service down" -a "From: alerts@example.com" email@example.com
  fi
  sleep 300
done

I went one step further and created a systemd unit file which would run the script in the background. I did that by creating a file called elasticalert.service at /usr/lib/systemd/system/.

[Unit]
Description=Elasticalert
After=elasticsearch.service

[Service]
Type=simple
WorkingDirectory=/root
ExecStart=/bin/bash elasticalert.sh
Restart=always

[Install]
WantedBy=multi-user.target

All you can do now is to make sure it starts on reboot and start it up by running the following commands:

#systemctl enable elasticalert.service
#systemctl start elasticalert.service

Note: The script is very simple and it will send an e-mail every 5 minutes as long as the query will send a match for the filters. That might be even a couple of minutes after the service is back online.