DEPLOY A LOGGING INFRA : ELK STACK v5

This setup will run elk

• elasticsearch v5 database
• Logstash v2: receiving and parsing logs <-- tried v5 but it is so slow! unusable...
• Kibana v5: Web interface
• Ngnix Proxy : for SSL + password access
• Docker container is the main source of logs, but we could send anything to syslog 5000/udp
• Filebeat collect files.log and send them to syslog 5001/tcp

Prerequisite:

• Linux like OS
• Docker, docker-compose

More info: you can find an overview of that setup on my blog: https://greg.satoshi.tech/

1. Get all files from github

git clone https://github.com/gregbkr/elk-dashboard-docker elk && cd elk

2. Fix

Fix an issue with hungry es v5

sudo sysctl -w vm.max_map_count=262144

make it persistent:

nano /etc/sysctl.conf
vm.max_map_count=262144

2. Run all containers for version 5:

docker-compose up -d

(For the old version 2, use: docker-compose -f docker-compose-v2.yml up -d)

and send few logs with nc or socat:

nc -w0 -u localhost 5000 <<< "TEST1"
echo "`date +\%Y-\%m-\%dT\%H:\%M:\%S` vm:`hostname` service:.com.health msg:TEST2" | socat -t 0 - UDP:localhost:5000

3. Log on kibana to see the result

http://localhost:5601 (direct without proxy)

https://localhost:5600 (!HTTPS ONLY! enter the credentials admin/Kibana05)

Initialize the index: pressing the green "create" button when log starting to come.

#---------------------- Config -------------------------------

5. Logstash

Validate your config

You can run logstash to easily collect input string. Each input you paste in the invite will be process and output on screen by logstash. To setup, please run this container:

docker run -it --rm --name logstash -p 5001:5000 -p 5001:5000/udp -v $PWD/logstash-conf:/opt/logstash/conf.d logstash:2  -f /opt/logstash/conf.d/logstash-test.conf

And paste log messages like this and check if the output is correct. Make modification to logstash-test.conf and restart logstash container to refresh.

<14>2015-08-31T15:20:00 vagrant-ubuntu-trusty-64 docker/proxyelk[870]: 2015/07/20 17:19:13 routing all to syslog://dev.local:5000

If you got the field tags = ParseFailure, means your parsing is wrong somewhere... :-(

Modify log parsing

You can use https://grokdebug.herokuapp.com/ in order to check a log parsing.

To refresh logstash after a modification in the logstash.conf file:

docker restart elk_logstash_1

Sending logs via docker driver

When your tests seems fine and you want containers to automatically send their log output:

nano docker-compose-yml    <-- uncomment the logging sections
docker-compose restart

You should see in kibana that elk is collecting your containers logs.

#------ Backup and Restore and optimize ------

Easy backup and restore via elasticdump

Backup index data to a file :

docker run --rm -ti -v /root/backup:/data sherzberg/elasticdump --all=true --input=http://ip:9200/ --output=/data/elkexport.json

restore

docker run --rm -ti -v /root/backup:/data sherzberg/elasticdump --bulk=true --input=/data/elkexport.json --output=http://ip:9200/

Index management (backup, restore, rotate)

More info in file: nano INDEX-MGMT.md

#------ Issues ------

Logstash v5

Try to use image: logstash:5 And run a

docker logs -f elk_logstash_1

You will see that logstashv5 start very slowly (5min sometimes)...