Solutions tested on:
Zabbix 4.4+
Kafka 2+
Burrow
CMAK
Grafana
yum install -y zabbix-java-gataway
mcedit /etc/zabbix/zabbix_java_gateway.conf
Uncoment and set START_POLLERS=10
mcedit /etc/zabbix/zabbix_server.conf
Uncoment and set to StartJavaPollers=5 Change IP for JavaGateway=IP_address_java_gateway
/etc/init.d/zabbix-java-gataway restart
chkconfig --level 345 zabbix-java-gataway on
/etc/init.d/zabbix-java-gataway start
cd /opt/kafka/bin
mcedit kafka-run-class.sh
change from
# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "
fi
to
# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 - Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
fi
Add to /etc/supervisord.conf that lines
[program:kafka]
command=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
directory=/opt/kafka/
autostart=true
autorestart=true
stopasgroup=true
startsecs=10
startretries=999
log_stdout=true
log_stderr=true
logfile=/var/log/kafka/supervisord-kafka.out
logfile_maxbytes=20MB
logfile_backups=10
/etc/init.d/supervisord restart
For systemd is possible to use unit like example below.
For this example kafka is installed in /usr/local/kafka/kafka_2.12-2.5.0
[Unit]
Description=Apache Kafka
Wants=network.target
After=network.target
[Service]
LimitNOFILE=32768
User=kafka
Environment=JAVA=/usr/bin/java
Environment="KAFKA_USER=kafka"
Environment="KAFKA_HOME=/usr/local/kafka/kafka_2.12-2.5.0"
Environment="SCALA_VERSION=2.12"
Environment="KAFKA_CONFIG=/usr/local/kafka/config"
Environment="KAFKA_BIN=/usr/local/kafka/bin"
Environment="KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/usr/local/kafka/config/log4j.properties"
Environment="KAFKA_OPTS="
Environment="KAFKA_HEAP_OPTS=-Xmx512M -Xms256M"
Environment="KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"
Environment="KAFKA_LOG_DIR=/var/log/kafka"
Environment="KAFKA_JMX_OPTS=-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=<REPLACE_WITH_YOUR_JMX_PORT> -Dcom.sun.management.jmxremote.rmi.port=<REPLACE_WITH_YOUR_JMX_PORT> -Djava.rmi.server.hostname=<REPLACE_WITH_YOUR_HOSTNAME>"
ExecStart=/usr/local/kafka/bin/kafka-server-start.sh ${KAFKA_CONFIG}/server.properties
SuccessExitStatus=0 143
Restart=on-failure
RestartSec=15
[Install]
WantedBy=multi-user.target
Please pay additional attention on options:
-Dcom.sun.management.jmxremote.port=<REPLACE_WITH_YOUR_JMX_PORT>
-Dcom.sun.management.jmxremote.rmi.port=<REPLACE_WITH_YOUR_JMX_PORT>
To avoid using dynamic TCP port and firewall/NAT issues it is better to set static port setting.
systemctl daemon-reload
systemctl restart kafka-service
##Import template Log in to your zabbix web
Click Configuration->Templates->Import
Download template zbx_kafka_templates.xml and upload to zabbix Then add this template to Kafka and configure JMX interfaces on zabbix
Enter Kafka IP address and JMX port If you see jmx icon, you configured JMX monitoring good!
if you have problems you can check JMX using this script #!/usr/bin/env bash
ZBXGET="/usr/bin/zabbix_get"
if [ $# != 5 ]
then
echo "Usage: $0 <JAVA_GATEWAY_HOST> <JAVA_GATEWAY_PORT> <JMX_SERVER> <JMX_PORT> <KEY>"
exit;
fi
QUERY="{\"request\": \"java gateway jmx\",\"conn\": \"$3\",\"port\": $4,\"keys\": [\"$5\"]}"
$ZBXGET -s $1 -p $2 -k "$QUERY"
eg.: ./zabb_get_java zabbix-java-gateway-ip 10052 server-test-ip 12345 'jmx[java.lang:type=Threading,PeakThreadCount]'
For monitoring kafka consumers you should install Burrow daemon and jq tools on kafka host
ssh clone https://github.com/helli0n/kafka-monitoring.git
cd kafka/kafkaconsumers
cp -r burrow /opt/
cp burrow/ /etc/init.d/burrow_script
chkconfig --level 345 burrow_script on
You should change config file in /opt/burrow/burrow.cfg
cd /usr/bin
wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
mv jq-linux64 jq
chmod +x jq
# jq
# jq
jq - commandline JSON processor [version 1.5]
Usage: jq [options] <jq filter> [file...]
jq is a tool for processing JSON inputs, applying the
given filter to its JSON text inputs and producing the
filter's results as JSON on standard output.
The simplest filter is ., which is the identity filter,
copying jq's input to its output unmodified (except for
formatting).
For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq
Some of the options include:
-c compact instead of pretty-printed output;
-n use `null` as the single input value;
-e set the exit status code based on the output;
-s read (slurp) all inputs into an array; apply filter to it;
-r output raw strings, not JSON texts;
-R read raw strings, not JSON texts;
-C colorize JSON;
-M monochrome (don't colorize JSON);
-S sort keys of objects on output;
--tab use tabs for indentation;
--arg a v set variable $a to value <v>;
--argjson a v set variable $a to JSON value <v>;
--slurpfile a f set variable $a to an array of JSON texts read from <f>;
See the manpage for more options.
cp kafka_consumers.sh /etc/zabbix/
cp userparameter_kafkaconsumer.conf /etc/zabbix/zabbix_agentd.d
/etc/init.d/burrow_script start
/etc/init.d/zabbix-agent restart
cp kafka_consumers.py /etc/zabbix/
cp userparameter_kafkaconsumer_py.conf /etc/zabbix/zabbix_agentd.d
pip install requests
/etc/init.d/burrow_script start
/etc/init.d/zabbix-agent restart
Upload template zbx_templates_kafkaconsumers.xml and mapping value zbx_valuemaps_kafkaconsumers.xml to zabbix server using UI and link template to Kafka host
If it doesn't work you can check it use /etc/zabbix/kafka_consumers.sh e.g.: # /etc/zabbix/kafka_consumers.sh discovery {"data":[{ "{#CONSUMER}": "CONSUMER0", "{#PARTITION}": 0, "{#TOPIC}": "TOPIC0" },{ "{#CONSUMER}": "CONSUMER1", "{#PARTITION}": 0, "{#TOPIC}": "TOPIC1" }]}
Need to deploy grafana-xxl docker container and add zabbix server as a source. Upload Grafana dashboard template
https://github.com/helli0n/kafka-monitoring/wiki/Kafka-monitoring
https://engineering.linkedin.com/apache-kafka/burrow-kafka-consumer-monitoring-reinvented
https://github.com/linkedin/Burrow/wiki
https://community.hortonworks.com/articles/28103/monitoring-kafka-with-burrow.html
https://github.com/RiotGamesMinions/zabbix_jmxdiscovery
https://zabbix.org/wiki/Docs/howto/zabbix_get_jmx
https://www.zabbix.org/wiki/ConfigureJMX
https://www.zabbix.com/documentation/4.4/manual/discovery/low_level_discovery/jmx