Configure elasticsearch logstash filebeats with shield to monitor nginx access.log

In this post I will show how to install and configure elasticsearch for authentication with shield and configure logstash to get the nginx logs via filebeat and send it to elasticsearch.
Why we do need filebeat when we have packetbeat? It is a good question. The short answer it is that packetbeat does not support https traffic therefore if your site it is under https the packetbeat will not show you exactly the traffic which your website will get.
I will assume that you have elasticsearch 2.x, logstash 2.x and filebeat installed and configured on your system ready to take traffic.

To install shield you first need to install license plugin. To do so you will need to navigate to /usr/share/elasticsearch and then run the following commands:

/usr/share/elasticsearch#bin/plugin install license
/usr/share/elasticsearch#bin/plugin install shield

Once that is done you need to create a series of users with the required roles to it. In this scenario we will create an admin user called admin, an user named beat which would be used on logstash to connect to the elasticsearch

/usr/share/elasticsearch:bin/shield/esusers useradd admin -r admin -p shieldadmin

the -r option will define the role you add the users to
the -p will set the password for the user you add.

The roles for shield are defined at /etc/elasticsearch/shield/roles.yml

In order to make different beats packages to connect on elasticsearch I have added all the basic indexes for packetbeats, topbeats and filebeats into the roles.yml under the logstash role which looks like this:

logstash:
  cluster: indices:admin/template/get, indices:admin/template/put
  indices:
    'logstash-*':
      privileges: indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
    'packetbeat-*':
      privileges: indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
    'filebeat-*':
      privileges: indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
    'topbeat-*':
      privileges: indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index

Now that's configured we are ready to create a beats user for the shield:

/usr/share/elasticsearch:bin/shield/esusers useradd beats -r logstash -p beatspassword

Now that is done we will need to restart the elasticsearch:

#systemctl restart elasticsearch

Now let's configure logstash for accepting logs from filebeats. The configuration is comparable with the one from this post with few changes. The logstash config file is located at /etc/logstash/conf.d/logstash.conf.

input {
    beats {
        host => "0.0.0.0"
        port => "5400"
    }
}
filter {
    if [type] == "nginx-access" {
        grok {
            match => { 'message' => '%{IPORHOST:clientip} %{USER:ident} %{USER:agent} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request}(?: HTTP/%{NUMBER:httpversion})?|)\" %{NUMBER:answer} (?:%{NUMBER:byte}|-) (?:\"(?:%{URI:referrer}|-))\" (?:%{QS:referree}) %{QS:agent}' }
         }
         geoip {
             source => "clientip"
             target => "geoip"
             database => "/etc/logstash/GeoLiteCity.dat"
             add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
             add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
        }
        mutate {
             convert => [ "[geoip][coordinates]", "float" ]
       }
   }
}
output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        hosts => ["127.0.0.1:9200"]
        user => beats
        password => beatspassword
    }
}

Note: if you add other options to the elasticsearch output make sure that the user and password options are always the last ones otherwise the logstash will not be able to authenticate into the elasticsearch.
Next we need to download the geoIP database to /etc/logstash:

#cd /etc/logstash
#wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
#gunzip GeoLiteCity.dat.gz

Now that we have the logstash configured let's test the configuration file before we start the service.

#/opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/logstash.conf
Configuration OK

We got the confirmation of a correct configuration file so we can start the logstash without any issues and configure it so that it would start at boot time..

#systemctl start logstash
#systemctl enable logstash

Now the last part of this post is to configure the filebeat on the webserver where you have the nginx running

As before I will assume that filebeat it is installed on the system so we can move to configure the filebeat.yml located at /etc/filebeat.

filebeat:
  prospectors:
    -
      paths:
        - /var/log/nginx/*/access.log
      input_type: log
      exclude_files: [".gz$"]
      document_type: nginx-access
      scan_frequency: 10s
      harvester_buffer_size: 16384
      max_bytes: 10485760
      max_backoff: 10s
      backoff_factor: 2
      spool_size: 2048
      idle_timeout: 5s
      registry_file: /var/lib/filebeat/registry
output:
  logstash:
    hosts: ["<logstash_ip>:5400"]
    worker: 1
    compression_level: 3
    index: filebeat
shipper:
  tags: ["service-X", "web"]
  refresh_topology_freq: 10
  topology_expire: 15
  queue_size: 1000
logging:
  to_syslog: true
  level: error

As you see I am using as a path /var/log/nginx/*/access.log since I am creating separate logs for each website I am hosting. If you want to monitor only one site then you should change the path to match the access.log for your website.

Now that I have filebeat configured it is time to install it and set it up for boot time startup.

#systemctl start filebeat
#systemctl enable filebeat