Correlation Alarms

class dashboard.correlation.alarm.Alarm(db_id, uuid, endpoints, assessor, phase, severity, dirty, published, description, previous_state=None, final_severity=None, alarm_group=None, devoured=None, publish_update=<function publish_alarm_update>)
COALESCE_WINDOW_HOURS = 24

Similar alarms that occur within the coalescing window may be grouped together

FLOOD_SLACK_TIME = 300

don’t accept new endpoints that are created longer than this number of seconds after the earliest endpoint in this alarm

KILL_PERIOD = 30

after the alarm is finalized, if the alarm is up for this amount of time then discard it

PENDING_PERIOD = 60

the period during which the alarm description can change

SHORT_LIVED_ALARM_THRESHOLD = 120

if the alarm has been down for less than this value when finalizing, automatically close it

apply_changes(severity=None, description=None, state=None, session=None)
begin_lifecycle(session)

Begin the Alarms lifecycle by initializing its phase handler. This method must be called (and only called once per Alarm) after the Alarm has been created and correct initial phase has been set.

Parameters:

session – a SQLAlchemy Session object

can_accept_endpoint(endpoint)

Check if this alarm’s state allows the endpoint to be added.

Namely:

  • it’s within FLOOD_SLACK_TIME

  • if the alarm isn’t PENDING, then the severity doesn’t increase and the description doesn’t change

This is only called after the endpoint has been correlated with one or more endpoints already present in this alarm.

Returns:

true if this alarm can accept the endpoint

cleanup_after_being_devoured(session)
clear_current_timer(name=None)

Stops and removes the current timer, if it exists

Parameters:

name – Optional, only stop the current timer if its timer_name matches name

property contacts
property dashboard_service
property description
devour_alarms(alarms_to_devour)

Take over all endpoints in alarms_to_devour, and free all resources from those alarms (e.g. stop timers, etc.).

Parameters:

alarms_to_devour

discard_endpoint(endpoint)

Discard endpoint from self.endpoints.

Parameters:

endpoint

Returns:

True iff the endpoint was found and removed

dumpd()

Create a json-serializable dict that can be used to re-create this object.

Returns:

a json-serializable dict

property duration

Generalised method, should be overridden if there is a better field to be sorted on e.g. when they will always come from the same device use systemUpTime

WARNING: this method will fail in KILL_ME state

Returns:

the time difference in seconds between the first trap and the current time or the last traps if the alarm is closed

classmethod from_cache(endpoint_or_cached_dict)
classmethod from_endpoint(endpoint_or_cached_dict)
handle(**kwargs)
property init_time
initialize_phase(new_phase, session)
property is_euro_hpc
kill_alarm()
learn_from_db(session)

workaround for the first time we reload the cache after introduction of phase (DBOARD3-242)

this method should only be called when loading the cache

property locations
notify_endpoint_state_changed()
property phase
phase_handler
property project
publish_state()
property published
release_all()

Release any resources that will prevent this object from being garbage collected.

This method leaves this object in a corrupt state.

property resource_ids
restart_kill_timer()
restart_pending_timer()
restart_timer(name, timeout_seconds, handler)
save_if_dirty(session)
property services
set_published()
property severity
property state
timer = None
update_alarm()

Assessors

There’s one Assessor subclass for each type of alarm that is generated and managed by the dashboard.

class dashboard.correlation.assessors.amt_isolation.AmtIsolationAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.bgp_peering.BGPPeeringAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.bgp_vpn_rr.BGPVPNRAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.coriant_los.CoriantLOSAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.default_los.DefaultLOSAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.eurohpc_site.EuroHPCSiteAssessor(endpoints)

Assessor for EuroHPC site alarms.

A connection (EA-xxxxx) is considered down if EITHER its link endpoint OR its BGP peering endpoint has an active alarm. Both endpoint types are accepted so a single alarm can represent mixed link+peering failures.

Severity: - MAJOR — all connections to the site are down - MINOR — some (not all) connections are down

get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.nren_isolation.NRENIsolationAlarmAssessor(endpoints)

Assessor for NREN isolation alarms.

An NREN isolation alarm is raised when the nren-isolation-checker process determines that a specific NREN (identified by ASN) is isolated — i.e. all of its BGP peering sessions to the GÉANT core are down.

The classifier forwards nren-isolation messages to the correlator only when the NREN’s ASN is known to the HPC dashboard. The correlator creates or updates a NRENIsolationEndpoint for the affected ASN and groups any related EuroHPC link or peering endpoints under the same alarm.

The alarm groups the NRENIsolationEndpoint together with any EuroHPCLinkEndpoint and EuroHPCSitePeeringEndpoint instances that belong to the same NREN ASN.

Severity is always CRITICAL; the description names the NREN and its ASN.

get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.fiberlink.FiberLinkAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.infinera_cross.InfineraCrossAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.infinera_los.InfineraLOSAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.ix_group.IXGroupAssessor(endpoints)

IX Group alarms report on the reachability of a group of IX peering endpoints.

generate_description()

The description distinguishes between a complete or degraded outage.

generate_severity()

Severity is CRITICAL if the provider is completely unreachable because all peerings are down.

Otherwise, severity is MAJOR or MINOR if the percentage of down peerings is above or below IX_GROUP_MAJOR_THRESHOLD_PERCENT (50%).

get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.ixp.IXPReachabilityAssessor(endpoints)

IXP Reachability alarms report on the reachability of an IXP peering endpoint.

generate_description()

The description distinguishes between a complete or degraded IX site.

generate_severity()

Severity is CRITICAL if the link with the IXP is down.

If more than PUBLIC_PEERING_MIN_THRESHOLD_PERCENT (50%) or IX peerings at this site then the alarm is CRITICAL, otherwise it’s MAJOR.

get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

Method for base class access to a/the list of endpoints. :return:

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

return only physical ae interfaces, if present if not, then only physical interfaces, if present otherwise return all endpoints (cf. DBOARD3-263)

Returns:

iterable of endpoints

class dashboard.correlation.assessors.mtc_los.MTCLOSAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.root_dns.RootDnsAssessor(endpoints)

Root DNS alarms report on the reachability of root DNS servers.

All failing root DNS servers are correlated into a single alarm. Severity is MAJOR if exactly one server is unreachable, CRITICAL if more than one are unreachable.

generate_description()

The description lists the unreachable server name(s).

generate_severity()

Severity is CRITICAL if more than one root DNS server is unreachable, MAJOR if exactly one is unreachable.

get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible

class dashboard.correlation.assessors.router_isolation.RouterIsolationAlarmAssessor(endpoints)
get_endpoints()

Method for base class access to a/the list of endpoints. :return:

is_compatible_endpoint_type(endpoint)

Quick check on alarm type

Used as an assertion in contructors, and as a filter in Alarm.can_accept_endpoint.

Returns:

True if the endpoint type is compatible