Monitor your servers with Elasticsearch 2.x and Beats and display it in Kibana
In this post we will cover the installation and configuration of elasticsearch 2.x, and configure packetbeat and topbeat for monitoring web services databases and server load. After all these are set up we will be using Kibana to show the data in a graphical interface.
Elasticsearch##
First we will follow the steps and installation of elasticsearch on various distributions.
For Debian/Ubuntu you need to run the following:
#wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
#echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
#sudo apt-get update && sudo apt-get install elasticsearch
For Centos/Fedora you need to run the following:
#rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
#cat > /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1
EOF
#yum install elasticsearch
Now it's time to configure the elasticsearch. To do this you must edit /etc/elasticsearch/elasticsearch.yml with your favorite editor. For simplicity I am making elasticsearch to listen on all interfaces since I am running it into NAT-ed environment. My elasticsearc.yml is the most basic configuration since I just need one node to run for starters.
cluster.name: z0z0.me
bootstrap.mlockall: true
network.host: 0.0.0.0
http.port: 9200
Once this is done it's time to configure your elasticsearch to start at startup and start it. On distributions running systemd you will be running the following commands:
#systemctl enable elasticsearch
#systemctl start elasticsearch
On the distributons running sysV init using debian/ubuntu you will be running the following commands:
#update-rc.d elasticsearch defaults 95 10
#service elasticsearch start
Running centos/fedora you will be running the following commands:
#chkconfig elasticsearch on
#service elasticsearch start
You can check if elasticsearch is running using netstat:
#netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::9200 :::* LISTEN -
tcp6 0 0 :::9300 :::* LISTEN -
Now that elasticsearch is installed and started up it's time to install the Beats on the webserver/db server and configure them. Beats is containing various packets each of them doing something different. Filebeat is reading files and sending it to logstash for further manipulations before it is sent to elasticsearch. Packetbeat is readinf the network packets which are using different configured ports for different services and is directly comunicating with Elasticseach. Topbeat is reading server usage and just like Packetbeat is comunicating directly with elasticsearch.
Now that all those are clear it's time to install packetbeat and topbeat and configure them for various distributions:
Packetbeat##
On Debian/Ubuntu:
#curl https://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
#echo "deb https://packages.elastic.co/beats/apt stable main" | sudo tee -a /etc/apt/sources.list.d/beats.list
#apt-get update && apt-get install packetbeat
For sysV init run the following:
#sudo update-rc.d packetbeat defaults 95 10
On Centos/Fedora:
#rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
#cat > /etc/yum.repos.d/beats.repo << EOF
[beats]
name=Elastic Beats Repository
baseurl=https://packages.elastic.co/beats/yum/el/$basearch
enabled=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
gpgcheck=1
EOF
#yum install packetbeat
On distributions using systemD run the following:
#systemctl enable packetbeat
Now let's configure packetbeat. To do so you need to edit the /etc/packetbeat/packetbeat.yml. We are running a webserver a db server and a redis server. For monitoring them will need to have the following configurations:
interfaces:
device: any
protocols:
http:
ports: [80, 8080, 8000, 5000, 8002]
mysql:
ports: [3306]
redis:
ports: [6379]
output:
elasticsearch:
hosts: ["http://<elasticsearch IP>:9200"]
logging:
to_files: true
files:
path: /var/log/mybeat
name: mybeat
rotateeverybytes: 10485760
keepfiles: 7
selectors: ["*"]
level: error
Before we start the packetbeat service it is important to import the elasticsearch template into the packetbeat. The template can be found at /etc/packetbeat/packetbeat.template.json. Copy the content of the file and run the following command:
curl -XPUT '<elasticsearch IP>:9200/_template/packetbeat' -d '
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"client_location": {
"type": "geo_point"
},
"params": {
"index": "analyzed",
"norms": {
"enabled": false
},
"type": "string"
},
"query": {
"doc_values": true,
"index": "not_analyzed",
"type": "string"
},
"request": {
"index": "analyzed",
"norms": {
"enabled": false
},
"type": "string"
},
"response": {
"index": "analyzed",
"norms": {
"enabled": false
},
"type": "string"
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "packetbeat-*"
}'
Now it's time to start the packetbeat service. To do so you will need to run the following commands:
For SysV init:
#service packetbeat start
For systemD:
#systemctl start packetbeat
Topbeat##
For topbeat follow the same steps as for packetbeat only replace packetbeat with topbeat.
Topbeat config looks almost the same as the packetbeat one but the input section changes. So I will put here the changed input section of the topbeat:
input:
period: 10
procs: [".*"]
stats:
system: true
proc: true
filesystem: true
The output and logging sections you can copy from packetbeat configuration file.
Topbeat does not need to have the template imported if you already imported packetbeat templates.
To start topbeat you need to run the following commands:
For SysV init:
#service topbeat start
For systemD:
#systemctl start topbeat
Once we have all these up and running we need to install kibana to configure and configure it to have the data displayed.
KIBANA##
Since Kibana is a web service running in nodejs you only need to download it and unarchive it in a safe place on your servers. We will be running it from /opt
#cd /opt
opt# wget https://download.elastic.co/kibana/kibana/kibana-4.3.0-linux-x64.tar.gz
/opt# tar -xzvf kibana-4.3.0-linux-x64.tar.gz
/opt# mv kibana-4.3.0-linux-x64 kibana4
/opt# cd kibana4
Now we will need to edit the configuration file located in /opt/kibana4/config/kibana.yml:
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://127.0.0.1:9200"
These are the minimal necesary configurations needed for kibana to start.
To start kibana you need to run the kibana script from the bin folder. Since Kibana does not have a daemon to start it in background I would be running it using nohup
#nohup /opt/kibana4/bin/kibana &
Now let's load kibana beat dashboard. To do so please follow the steps bellow:
#curl -L -O http://download.elastic.co/beats/dashboards/beats-dashboards-1.0.0.tar.gz
#tar xzvf beats-dashboards-1.0.0.tar.gz
#cd beats-dashboards-1.0.0/
#./load.sh
If your elasticsearch is running on different host then kibana then you will need to mention the elasticsearch server in the load.sh script:
#./load.sh http://<elasticsearch IP>:9200
After loading the dashboard and you are browsing the kibana on port 5601 by browsing http://{IP}:5601 Kibana will raise an error of no default index pattern error. You can fix that by refreshing the page and then setting one of the indexes as the default one.
Kibana settings
To open the loaded dashboard you click on the Dashboard menu item and click on the load dashboard icon and select Packetbeat Dashboard from the list. As soon your elasticsearch will start receiving data your dashboard will start looking like this:
Kibana packetbeat dashboard.
Your elasticsearch is now ready to be used to monitor your systems and gather data from the configured servers.