Dashboard Component Monitoring

It’s possible to extract performance status and counters from all main pipeline components board components.

Overview 

Two monitoring component types are defined:

monitor: a sender of requests to monitored processes
monitored process: a process that sends responses to requests sent by a monitor

This specification uses RabbitMQ-specific terminology.

monitor component requirements 

A monitor process is responsible for creating a queue to receive responses to requests.

All requests must:

be sent to the RabbitMQ exchange named mon.requests
contain a property called reply_to whose value is the response queue name

monitored process component requirements 

Monitored processes must:

create a unique name that will be used for routing unicast monitor requests (this name can be the same as the queue created below)
create a queue and bind it to the global mon.requests exchange with the following routing keys:
- broadcast
- the unique name created above
send a proper response message to the queue named in the reply_to property of the request message

protocol specification 

Requests and responses are json strings and are sent as the body of queue messages.

All requests are validated against the following schema before processing:

MESSAGE_SCHEMA

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "enum": [
        "consensus",
        "PING"
      ]
    },
    "hostname": {
      "type": "string"
    },
    "apptype": {
      "type": "string"
    },
    "class": {
      "type": "string",
      "enum": [
        "HEARTBEAT",
        "REQUEST_VOTE",
        "VOTE_FOR_LEADER"
      ]
    },
    "sender": {
      "type": "string"
    },
    "term": {
      "type": "string"
    }
  },
  "required": [
    "type"
  ],
  "additionalProperties": false
}

PING 

request 

This message is sent by a monitor, with type = “ping”

response 

This message is sent from a monitored process in response to a ping request.

The response will be formatted according to the following schema:

PONG_SCHEMA

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "key": {
      "type": "string"
    },
    "type": {
      "type": "string"
    },
    "hostname": {
      "type": "string"
    },
    "pid": {
      "type": "integer"
    },
    "uptime": {
      "type": "number"
    },
    "counters": {
      "type": "object"
    },
    "status": {
      "type": "object"
    },
    "module version": {
      "type": "string"
    }
  },
  "required": [
    "key",
    "type",
    "hostname",
    "pid",
    "uptime",
    "module version"
  ],
  "additionalProperties": false
}

For most process types, the status element in the response will be formatted according to the following schema:

DEFAULT_STATUS_SCHEMA

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "watchdog": {
      "type": "boolean"
    },
    "error": {
      "type": "boolean"
    },
    "message": {
      "type": "string"
    },
    "restarted": {
      "type": "boolean"
    },
    "error_count": {
      "type": "integer"
    },
    "status_counters": {
      "type": "object",
      "properties": {
        "monitoring_error_count": {
          "type": "integer"
        }
      },
      "required": [
        "monitoring_error_count"
      ],
      "additionalProperties": false
    }
  },
  "required": [
    "error",
    "message",
    "restarted",
    "status_counters"
  ],
  "additionalProperties": false
}

CONSENSUS 

All messages related to Raft consensus management are sent with type = “consensus”. See Dashboard Consensus Protocol. for more information.