Special Event Generators

NREN Isolation

NREN Isolation

Any number of nren isolation checker processes can run simultaneously. They handshake with each other and for a Raft cluster, and elect one process as a leader.

All processes listen for bgp session traps

Usage: nren-isolation-checker [OPTIONS]

    Options:
        --hostname TEXT                 RabbitMQ hostname ['test-dashboard-
                                                                        storage01.geant.org', 'test-dashboard-
                                                                        storage02.geant.org', 'test-dashboard-
                                                                        storage03.geant.org']
        --monitoring TEXT               monitoring requests exchange name
                                                                        [mon.requests]
        --correlator TEXT               alarm state broadcast exchange name
                                                                        [dashboard.alarms.broadcast]
        --isolation TEXT                isolation listener worker queue name
                                                                        [dashboard.isolation]
        --broadcast TEXT                isolation state broadcast exchange name
                                                                        [dashboard.isolation.broadcast]
        --username TEXT                 RabbitMQ username [dashboard]
        --password TEXT                 RabbitMQ user password [password]
        --vhost TEXT                    RabbitMQ vhost [/dashboard]
        --watchdog INTEGER              watchdog check frequency in seconds (no
                                                                        watchdog if unset)
        --timeout INTEGER               number of seconds without correlator state
                                                                        messages to indicate error [240]
        --timeout_reconnect/--no-timeout_reconnect
                                                                        reconnect to rmq if no traps received in
                                                                        'timeout' seconds [True]
        --email_host TEXT               Email server hostname [prod-mail.geant.net]
        --email_port INTEGER            Email server port [25]
        --email_from TEXT               from address for messages to the TT system
                                                                        [alarm@geant.org]
        --email_to TEXT                 Email recipient address(es) for TT messages
                                                                        [required]
        --email_username TEXT           Email authentication username [None]
        --email_password TEXT           Email authentication password [None]
        --isogroup TEXT                 Additional groups to include in the
                                                                        isolation decision
        --nren TEXT                     Additional/updated nren groups to include
                                                                        (format: INT:STRING)
        --inventory TEXT                Inventory Provider base uris
                                                                        ['https://test-inprov01.geant.org/',
                                                                        'https://test-inprov02.geant.org/']
        --cache_hostname TEXT           Cache db hostname [test-dashboard-
                                                                        storage03.geant.org]
        --cache_dbport INTEGER          Cache db hostname [3306]
        --cache_username TEXT           Cache db username [dbcache]
        --cache_password TEXT           Cache db username [cache-secret]
        --cache_dbname TEXT             Cache db name [services_cache]
        --classifier TEXT               classifier exchange name [dashboard.deduplicated]
        --inventory_token TEXT          API Token for accessing the Inventory Provider service [default is None]
        --help                          Show this message and exit.

The default list of groups that are used to identify isolation mappings is below. This list can be extended with the –isogroup option.

DEFAULT_ACCESS_GROUP_NAMES
[
  "eGEANT"
]

State Checking

Active Correlator Endpoint State Checker

Any number of active endpoint state checker processes can run simultaneously. They all listen for correlator alarm state broadcast messages, and then distribute the active endpoints across all processes and perform the active state checks. If any worker finds a particular endpoint to be up it notifies the correlator.

start note right correlator state snapshot end note fork :Alarms Consumer-1; fork again :Alarms Consumer-2; fork again :Alarms Consumer-2; end fork

start note right correlator state snapshot end note :broker; note right dashboard.alarms.broadcast exchange end note fork :Alarms Consumer X; note right instance that received the state message end note end fork :broker; note right dashboard.endpoints.broadcast exchange end note fork :Endpoint Checker; fork again :Endpoint Checker; fork again :Endpoint Checker; fork again :Endpoint Checker; fork again :Endpoint Checker; note right several workers started on each node end note end fork :broker; note right dashboard.classified exchange end note :Correlator; note right process state update messages end note stop

Data Flow

../_images/data-flow1.svg

Next Data Flow Design …

../_images/data-flow2.svg

Router Isolation

Router Isolation Detector

To be implemented, but here’s the original design schematic:

../_images/data-flow3.svg

Heartbeat

Heartbeat Messages

Heartbeat messages (not to be confused with heartbeats from any other part of dashboard) are sent at regular intervals into the exchange preceding classifier. Heartbeats do nothing except update the ‘last-received-ts’ counter which keeps watchdog from being triggered when there aren’t many traps coming in. These heartbeats mostly follow the same route that trap data takes, into classifier, then passed to correlator, then to the notification exchange. Finally, the Elasticsearch archiver archives heartbeats alongside other notifications. All other notifiers consume but do nothing with heartbeats. As of writing these currently are:

  • Argus notifier

  • OTRS notifier

  • Slack notifier

  • TTS notifier

Heartbeat data flow:

../_images/data-flow4.svg

AMT Isolation

AMT Isolation

This package provides tools and listeners for monitoring and managing the isolation state of AMT routers.

Processes in this package listen for relevant network events, evaluate AMT router status, and broadcast isolation state messages via RabbitMQ.

This section documents the amt-isolation-checker command-line tool.

Command-Line Interface

amt-isolation-checker

Usage

amt-isolation-checker [OPTIONS]

Options

--hostname <hostname>

RabbitMQ hostname [‘test-noc-alarms01.geant.org’, ‘test-noc-alarms02.geant.org’, ‘test-noc-alarms03.geant.org’]

--monitoring <monitoring>

monitoring requests exchange name [mon.requests]

--broker-exchange <broker_exchange>

alarm state broadcast exchange name [dashboard.alarms.broadcast]

--broker-queue <broker_queue>

amt isolation queue name [dashboard.amt_isolation]

--broadcast <broadcast>

isolation state broadcast exchange name [dashboard.deduplicated]

--username <username>

RabbitMQ username [dashboard]

--password <password>

RabbitMQ user password [password]

--vhost <vhost>

RabbitMQ vhost [/dashboard]

--watchdog <watchdog>

watchdog check frequency in seconds (no watchdog if unset)

--timeout <timeout>

number of seconds without correlator state messages to indicate error [240]

--timeout_reconnect, --no-timeout_reconnect

reconnect to rmq if no traps received in ‘timeout’ seconds [True]

--inventory <inventory>

Inventory Provider base uris [’https://test-inprov01.geant.org/’, ‘https://test-inprov02.geant.org/’, ‘https://test-inprov03.geant.org/’]

--inventory_token <inventory_token>

API Token for accessing the Inventory Provider service [default is None]

--sentry-dsn <sentry_dsn>

Sentry DSN

--sentry-environment <sentry_environment>

Sentry environment

Data Flow

../_images/data-flow5.svg

IX Events

Private and IX Public Peering monitoring

Processes in this package listen to the correlator state broadcast and publish messages about IXP reachability and Peering provider status.

IXP Reachability Checker

ixp-reachability-checker

Consumes correlator state messages and checks for IXP reachability.

An IXP is considered reachable if the Link endpoint for the router and that location is up and if the peering for >50% of the public peers for that IXP are up.

The IXP reachability status is published to the classifier.

Usage

ixp-reachability-checker [OPTIONS]

Options

--hostname <hostname>

RabbitMQ hostname [‘test-noc-alarms01.geant.org’, ‘test-noc-alarms02.geant.org’, ‘test-noc-alarms03.geant.org’]

--monitoring <monitoring>

monitoring requests exchange name [mon.requests]

--broker-exchange <broker_exchange>

alarm state broadcast exchange name [dashboard.alarms.broadcast]

--broker-queue <broker_queue>

IXP listener queue name [dashboard.ix]

--broadcast <broadcast>

outbound message exchange name [dashboard.deduplicated]

--username <username>

RabbitMQ username [dashboard]

--password <password>

RabbitMQ user password [password]

--vhost <vhost>

RabbitMQ vhost [/dashboard]

--watchdog <watchdog>

watchdog check frequency in seconds (no watchdog if unset)

--timeout <timeout>

number of seconds without correlator state messages to indicate error [240]

--timeout_reconnect, --no-timeout_reconnect

reconnect to rmq if no state received in ‘timeout’ seconds [True]

--inventory <inventory>

Inventory Provider base uris [’https://test-inprov01.geant.org/’, ‘https://test-inprov02.geant.org/’, ‘https://test-inprov03.geant.org/’]

--inventory_token <inventory_token>

API Token for accessing the Inventory Provider service [default is None]

--sentry-dsn <sentry_dsn>

Sentry DSN

--sentry-environment <sentry_environment>

Sentry environment

IX Group Listener

ix-group-listener

Consumes correlator state messages and publishes per-ASN IX group reachability.

All pub/priv peerings are fetched from inventory-provider and grouped by ASN. Each group is considered up if any individual peering in the group is up. One ix-group message is published per ASN group.

Usage

ix-group-listener [OPTIONS]

Options

--hostname <hostname>

RabbitMQ hostname [‘test-noc-alarms01.geant.org’, ‘test-noc-alarms02.geant.org’, ‘test-noc-alarms03.geant.org’]

--monitoring <monitoring>

monitoring requests exchange name [mon.requests]

--broker-exchange <broker_exchange>

alarm state broadcast exchange name [dashboard.alarms.broadcast]

--broker-queue <broker_queue>

IX group listener queue name [dashboard.ix.groups]

--broadcast <broadcast>

outbound message exchange name [dashboard.deduplicated]

--username <username>

RabbitMQ username [dashboard]

--password <password>

RabbitMQ user password [password]

--vhost <vhost>

RabbitMQ vhost [/dashboard]

--watchdog <watchdog>

watchdog check frequency in seconds (no watchdog if unset)

--timeout <timeout>

number of seconds without correlator state messages to indicate error [240]

--timeout_reconnect, --no-timeout_reconnect

reconnect to rmq if no state received in ‘timeout’ seconds [True]

--inventory <inventory>

Inventory Provider base uris [’https://test-inprov01.geant.org/’, ‘https://test-inprov02.geant.org/’, ‘https://test-inprov03.geant.org/’]

--inventory_token <inventory_token>

API Token for accessing the Inventory Provider service [default is None]

--sentry-dsn <sentry_dsn>

Sentry DSN

--sentry-environment <sentry_environment>

Sentry environment