Skip to main content

Prometheus Monitoring (Cluster)

Introduction​

In this guide we will cover how to monitor your Node and Farming Cluster with Prometheus and Grafana

Prerequisites​

Prior to using this guide you should have Prometheus set up as a service:

You should also have Grafana running via one of the following methods

Expose Metrics​

If following my guides, the Node and Farmer are already set to expose metrics. If not following my guides metrics can be enabled by adding the --prometheus-listen-on flag.

Node

--prometheus-listen-on 127.0.0.1:9080

Controller

--prometheus-listen-on 127.0.0.1:9081

Cache

--prometheus-listen-on 127.0.0.1:9082

Plotter

--prometheus-listen-on 127.0.0.1:9083

Farmer

--prometheus-listen-on 127.0.0.1:9084

Restart your Node & Cluster components after making the above changes.

Update Config​

The Prometheus config file must be updated to include the endpoints for the Farmer and Node metrics. Open up the config

sudo nano /etc/prometheus/prometheus.yml

Go to the bottom and add the following job. Note that localhost is being used here because the Cluster and Prometheus are on the same host. If your Cluster is on a different host, then use the internal IP of that host.

  - job_name: 'autonomys'
    static_configs:
      - targets: ['localhost:9080']
        labels:
          instance: 'HOSTNAME'
          type: 'node'
      - targets: ['localhost:9081']
        labels:
          instance: 'HOSTNAME'
          type: 'controller'
      - targets: ['localhost:9082']
        labels:
          instance: 'HOSTNAME'
          type: 'cache'
      - targets: ['localhost:9083']
        labels:
          instance: 'HOSTNAME'
          type: 'plotter'
      - targets: ['localhost:9084']
        labels:
          instance: 'HOSTNAME'
          type: 'farmer'

Then restart Prometheus

sudo systemctl restart prometheus

Metrics are now being ingested by Prometheus and can be exposed in Grafana.

Grafana​

There is no dashboard for monitoring a cluster, however the non-cluster dashboard can be used as a starting point. Open up Grafana, select the "Dashboard" menu item. Then "New", and then "Import". For the "dashboard URL or ID" enter "20443". Then click load.

auto-dash-id

The dashboard is a bit old, but it is currently the best option. Update the name to something like "Autonomys Dashboard". Then add something to the uid to make it unique. Finally, select the Prometheus data source previously set up and click "Import".

import-settings

Some panels will likely not be working and need to be updated to use the cluster metrics. If a cluster dashboard is not created by someone in the community, I will create one eventually.