STOMP-Based Event Notification System

System Overview

The STOMP-based event notification system allows Waldur to communicate changes to resources, orders, and user roles to the waldur-site-agent that runs on a remote cluster. This eliminates the need for constant polling and enables immediate reactions to events.

The key components include:

1. STOMP Publisher (Waldur side): Located in the waldur_core/logging/utils.py file, this component publishes messages to STOMP queues when specific events occur.

2. Event Subscription Service: Manages subscriptions to events by creating unique topics for each type of notification. Related file: event subscription management via API: waldur_core/logging/views.py

3. STOMP Consumer (Agent side): The waldur-site-agent running on the resource provider's infrastructure that subscribes to these topics and processes incoming messages. Related files:

4. Event subscription registration: waldur_site_agent/event_processing/utils.py
5. STOMP message handlers: waldur_site_agent/event_processing/handlers.py
6. STOMP listener: waldur_site_agent/event_processing/listener.py

Event Flow

1. An event occurs in Waldur (e.g., a new order is created, a user role changes, or a resource is updated)
2. Waldur publishes a message to the appropriate STOMP queue(s)
3. The site agent receives the message and processes it based on the event type
4. The agent communicates with the backend (e.g., SLURM) to execute the necessary actions

Message Types

The system handles three primary types of events:

1. Order Messages: Notifications about marketplace orders (create, update, terminate)
2. User Role Messages: Changes to user permissions in projects
3. Resource Messages: Updates to resource configuration or status

Implementation Details

Publishing Messages (Waldur Side)

When events like order creation occur, Waldur prepares and publishes STOMP messages: code link

These messages are then sent via: publish_stomp_messages

Subscription Management (Agent Side)

The EventSubscriptionManager class handles creation of event subscriptions and setup of STOMP consumers:

• get_or_create_event_subscription - create an event subscription in Waldur if doesn't exist yet
• start_stomp_connection - setup STOMP client, connect agent to the broker and subscribe consumer to a queue

Message Processing (Agent Side)

When a message arrives, it's routed to the appropriate handler based on the event type:

• on_order_message_stomp - create or update resources on backend
• on_user_role_message_stomp - create or update access permissions on backend
• on_resource_message_stomp - create or update resource configuration on backend

Technical Components

1. WebSocket Transport: The system uses STOMP over WebSockets for communication
2. TLS Security: Connections can be secured with TLS
3. User Authentication: Each subscription has its own credentials and permissions in RabbitMQ
4. Queue Structure: Queue names follow the pattern /queue/subscription_{subscription_uuid}_offering_{offering_uuid}_{affected_object}

Error Handling and Resilience

The system includes:

• Graceful connection handling
• Signal handlers for proper shutdown
• Retry mechanisms for order processing
• Error logging and optional Sentry integration

Benefits of the STOMP Approach

1. Real-time Processing: Actions are triggered immediately when events occur
2. Reduced Network Traffic: No constant polling needed
3. Decoupling: The agent doesn't need direct access to Waldur's database
4. Scalability: Multiple agents can subscribe to different events
5. Reliability: The STOMP protocol provides queue persistency to ensure message delivery and different acknowledgement options on the agent side

This event-driven architecture significantly improves the responsiveness and efficiency of the order processing system compared to traditional polling approaches.