Monitor Your Coherence Clusters using Grafana and Prometheus (Coherence CE)

Introduction

NOTE: This repository is no longer being maintained. Please see the updated projects in https://github.com/tmiddlet2666/coherence-playground.

This document is a step by step example of how to enable metrics on Coherence cluster members, capture those metrics via Prometheus and display then in Grafana using the dashboards from the Coherence Operator project.

Note: This release of the coherence-grafana-metrics repository has instructions specifically for the Community Edition (CE) of Coherence. Please change to the v1.0.0 branch for Commercial Coherence instructions.

This is an example only and you can use this as a guide to adding Grafana monitoring to your cluster.

See the following for more information:

• Coherence Community Edition on GitHub
• Coherence Community Home Page
• Coherence documentation on Metrics
• Coherence Operator on GitHub
• Coherence Operator Metrics Documentation
• Coherence Operator Grafana Dashboards in Detail
• Grafana

Note: These instructions will work for Coherence CE versions 14.1.1-0-1 and above.

If you notice any errors in this documentation, please raise a P/R or issue.

Sample Dashboards

Below are a couple of the sample dashboards. For details of all the dashboards, please see the Coherence Operator Documentation.

Main Dashboards

Members Summary

Prerequisites

You must have the following:

• Docker Desktop for Mac or the equivalent Docker environment for you O/S.
• Maven 3.5.4+
• JDK 11+
• Cloned this repository via git clone https://github.com/tmiddlet2666/coherence-grafana-metrics.git

Note: This document has been written for Mac/Linux. Where appropriate, alternative Windows commands have been shown.

1. Generate the required dependencies

cd coherence-grafana-metrics
mvn dependency:build-classpath

Save the output of [INFO] Dependencies classpath: above to the METRICS_CP variable in start-server.sh or start-server.cmd.

Eg. replace <INSERT FULL CLASSPATH HERE> with the classpath contents.

export METRICS_CP="<INSERT FULL CLASSPATH HERE>"

Also ensure COHERENCE_HOME is set correctly to the coherence directory as below:

2. Start Coherence cache servers

Note: For Windows, replace start-server.sh with start-server.cmd.

Startup a cache server with metrics enabled on port 9612 with a role of CoherenceServer

./start-server.sh 9612 member1 CoherenceServer

Startup a second cache server with metrics enabled on port 9613

./start-server.sh 9613 member2 CoherenceServer

Optionally startup a third cache server with metrics enabled on port 9613

./start-server.sh 9614 member3 CoherenceServer

You should see the following indicating the metrics service is started in each of the cache server logs.

  ProxyService{Name=MetricsHttpProxy, State=(SERVICE_STARTED), Id=5, OldestMemberId=1}

3. Start the Console to add data

export COH_JAR=~/.m2/repository/com/oracle/coherence/ce/coherence/22.06/coherence-22.06.jar
java -Dcoherence.distributed.localstorage=false -cp $COH_JAR com.tangosol.net.CacheFactory

Enter the following at the prompt to create a cache and add 100,000 random objects:

cache test1
bulkput 100000 100 0 100
cache test2
bulkput 40000 100 0 100

Type bye to exit the Console.

4. Create the Prometheus Docker image

Edit prometheus.yml and ensure the static configs are set as below:

    static_configs:
    - targets: ['host.docker.internal:9612', 'host.docker.internal:9613', 'host.docker.internal:9614', 'host.docker.internal:9615']

Note: replace host.docker.internal with the actual IP and host if you are running a Coherence cluster a separate machine.

Build the docker image using:

docker build -t prometheus_coherence .

This will create the image prometheus_coherence:latest with the above prometheus.yaml.

5. Run the Docker images

export HOST=127.0.0.1
docker run -d -p $HOST:9090:9090 prometheus_coherence:latest

docker run -d -p $HOST:3000:3000 grafana/grafana:8.5.6

Note: Change HOST to a value that is suitable for your setup.

6. Clone the Coherence Operator repository

Issue the following to clone the Coherence Operator repository.

git clone https://github.com/oracle/coherence-operator.git

7. Access Prometheus

Login to Prometheus and confirm that the targets have been discovered by going to the following URL: http://127.0.0.1:9090/targets

You should see the targets you started in an UP state.

8. Access Grafana and create a datasource

Login to Grafana using the following URL: http://127.0.0.1:3000/ - default user admin/admin

Add a default Prometheus data source called prometheus with an endpoint of http://host.docker.internal:9090.

Ensure that you make this datasource the default datasource if it is not already.

Note: Change the host.docker.internal to an actual host name if you are running Prometheus outside of docker.

9. Import the Grafana dashboards

Login to Grafana and click on the + then Import and Upload JSON File. Select each of the dashboards in the coherence-operator/dashboards/grafana directory you cloned above and import them into Grafana.

Note: The Federation and Elastic Data dashboards will not display anything as this functionality is not available in Coherence CE.

10. Access the Main Grafana dashboards

Access Grafana using the following URL: http://127.0.0.1:3000/d/coh-main/coherence-dashboard-main.

11. Cleanup

Ensure you kill your docker images you started.