Message Processing

Thread Architecture

../_images/architecture2.svg

Main Thread

  1. Receives message from RabbitMQ via dashboard.notifications.common.run

  2. Calls should_create_ticket(message) to validate

  3. If valid: adds retry_count=0 and enqueues to queue.Queue

  4. If invalid: discards message

Processing Thread

For each message:

  1. DB Check: Query DashboardAlarm by id

    • If not found → raise TicketError (non-transient) → discard

  2. TTS Check: Find overlapping maintenance tickets

    • If found → update existing ticket

    • If not found → create new ticket

  3. DB Update: Set DashboardAlarm.ticket_ref to ticket number

  4. Error Handling:

    • TicketError → log error, discard immediately

    • Other exceptions → retry with exponential backoff

Retry Mechanism

../_images/architecture3.svg

  • Exponential Backoff: delay = min(BASE_DELAY * (2^retry_count), MAX_DELAY)

    • BASE_DELAY = 1.0 second

    • MAX_DELAY = 60.0 seconds

  • Max Retries: MAX_RETRIES = 5

  • Implementation: Uses threading.Timer to re-queue after delay

  • After Max Retries: Log error and discard message

Retry Schedule:

Retry

Delay

1

1s

2

2s

3

4s

4

8s

5

16s

After 5

Discard

Thread Safety

  • Queue: queue.Queue is thread-safe

  • Timers: Tracked in a lock-protected list

  • DB Session: Each message gets its own session via session_scope()

Signal Handling

The service handles graceful shutdown on:

  • SIGTERM (standard termination)

  • SIGINT (Ctrl+C)

Shutdown Behavior:

  1. Set stop_event

  2. Main thread exits (RabbitMQ consumer stops)

  3. Processing thread terminates (daemon thread)

Logging

Standard Python logging with format:

%(asctime)s - %(name)s - %(levelname)s - %(message)s

Log Levels

  • INFO: Service startup/shutdown, ticket creation/updates

  • WARNING: Transient errors with retry information

  • ERROR: TicketError, max retries exceeded

  • DEBUG: Message enqueue/dequeue, validation results

Example Logs

2026-04-02 10:15:30 - tts_api_notifier - INFO - Starting TTS API notifier service
2026-04-02 10:15:31 - tts_api_notifier - INFO - Processing thread started
2026-04-02 10:15:32 - tts_api_notifier - INFO - Created ticket TTS-12345 for alarm 67890
2026-04-02 10:16:05 - tts_api_notifier - WARNING - Error processing alarm 67891 (retry 1): Connection timeout
2026-04-02 10:16:05 - tts_api_notifier - WARNING - Scheduling retry 2 for alarm 67891 in 2.0s

Dependencies

  • dashboard.notifications.common - RabbitMQ integration

  • dashboard.correlation.db - Database access

  • click - CLI framework

  • Python standard library: threading, queue, signal, logging, json