Skip to content

Plugin Development Guide

This guide covers everything needed to build a custom backend plugin for Waldur Site Agent. It is written for both human developers and LLM-based code generators.

Waldur Mastermind concepts

Before implementing a plugin, understand how Waldur Mastermind concepts map to plugin operations.

Waldur concept Description Plugin relevance
Offering Service catalog entry Config block per offering; picks backend plugin
Resource Allocation from an offering CRUD via BaseBackend; keyed by backend_id
Order Create/update/terminate request Triggers order_process mode
Component Measurable dimension (CPU, RAM) Defined in backend_components config
OfferingUser User linked to an offering Username backend generates usernames
accounting_type usage, limit or one Metered, quota, or one-time accounting per component
backend_id Resource ID on the backend See Resource identifiers

Architecture overview

A resource-management plugin consists of two main classes:

  • Backend (inherits BaseBackend): Orchestrates high-level operations (create resource, collect usage, manage users).
  • Client (inherits BaseClient): Handles low-level communication with the external system (CLI commands, API calls).

A separate plugin family covers username management (AbstractUsernameManagementBackend) — see the dedicated section below. A single distribution may register both, but the entry point groups are distinct.

graph TB
    WM[Waldur Mastermind<br/>REST API] <-->|Orders, Resources,<br/>Usage, Keys| SA[Site Agent Core<br/>Processor]
    SA -->|"user_context<br/>(ssh_keys, plan_quotas)"| BE[YourBackend<br/>BaseBackend]
    BE --> CL[YourClient<br/>BaseClient]
    CL --> EXT[External System<br/>CLI / API]
    BE -.->|backend_metadata| SA

    classDef waldur fill:#1E3A8A,stroke:#3B82F6,stroke-width:2px,color:#FFFFFF
    classDef core fill:#065F46,stroke:#10B981,stroke-width:2px,color:#FFFFFF
    classDef plugin fill:#581C87,stroke:#8B5CF6,stroke-width:2px,color:#FFFFFF
    classDef external fill:#92400E,stroke:#F59E0B,stroke-width:2px,color:#FFFFFF

    class WM waldur
    class SA core
    class BE,CL plugin
    class EXT external

Steps to implement a plugin

This is the step-by-step path for a developer building a new backend plugin from scratch. It links out to the detailed reference sections below for the specifics of each step.

  1. Study a reference plugin. Pick the closest match to your target system: plugins/slurm/ for a CLI-based backend, plugins/mup/ for an HTTP API-based one, or plugins/waldur/ if you need async orders or federation-style hooks. Read its backend.py, client.py, and tests/ together — the tests show the expected call shapes.

  2. Scaffold the package from the template. Copy docs/plugin-template/ to a new directory (e.g. plugins/mycustom/) and rename the package (waldur_site_agent_mycustom → waldur_site_agent_<yours>) and the class names throughout.

  3. Design the config. Decide what goes in backend_settings (backend behaviour, e.g. account prefixes) versus backend_components (billable dimensions, e.g. cpu/storage). See the annotated YAML configuration below, and pick unit_factor values carefully — see Common pitfalls.

  4. Implement BaseClient. This is the thin layer that talks to the external system. Start with get_resource, create_resource, delete_resource, and list_resources — see the BaseClient method reference.

  5. Implement BaseBackend. Call super().__init__(backend_settings, backend_components) first, then implement the abstract methods in this order: ping, _pre_create_resource, _collect_resource_limits, _get_usage_report. Fill in the remaining abstract methods, and only override hook methods (post_create_resource, _pre_delete_resource, etc.) if your backend needs the extra behaviour — see the BaseBackend method reference and Agent mode method matrix for what each mode calls.

  6. Opt into capability flags as needed. supports_decreasing_usage, supports_async_orders, supports_user_homedirs, supports_periodic_settings, handled_resource_states, etc. — see Capability flags and class attributes. Leave them at their defaults unless your backend actually needs the behaviour change. If your system assigns its own resource IDs, read Resource identifiers before writing any create code, and decide which failures should mark a resource ERRED — see Error handling and ERRED.

  7. Add a username management backend, if needed. Only required if your system needs its own username generation/lookup logic instead of the shared base backend — see Username management backends.

  8. Register entry points in pyproject.toml. At minimum waldur_site_agent.backends; add username_management_backends, component_schemas, or backend_settings_schemas groups as needed — see Entry point registration.

  9. Write tests. Mock BaseClient, and assert the usage-report shape, the limit-conversion math, and error handling per mode — see Testing guidance.

  10. Install and verify.

    1
2
3
4
    uv sync --all-packages
cd plugins/mycustom && uv run pytest tests/
uv run waldur_site_diagnostics   # confirms the entry point resolves
uvx prek run --all-files

    Plugin tests only resolve their entry points when run from inside the plugin directory — see the "Run plugin tests from inside the plugin dir" gotcha in the repo's CLAUDE.md.

  11. Wire it into a config file. Add an offerings entry pointing order_processing_backend / reporting_backend / membership_sync_backend at your entry point name, and run the agent against a real or staging offering to confirm end-to-end behaviour.

Before opening a merge request, skim Common pitfalls once more — most plugin review comments trace back to one of those six.

BaseBackend method reference

Abstract methods (must implement)

ping(raise_exception: bool = False) -> bool

  • Mode: All (health check)
  • Purpose: Verify backend connectivity.
  • No-op: Return False.

diagnostics() -> bool

  • Mode: Diagnostics CLI
  • Purpose: Log diagnostic info and return health status.
  • No-op: Log a message, return True.

list_components() -> list[str]

  • Mode: Diagnostics
  • Purpose: Return component types available on the backend.
  • No-op: Return [].

_get_usage_report(resource_backend_ids: list[str]) -> dict

  • Mode: report, membership_sync
  • Purpose: Collect usage data for resources.
  • Return format:
1
2
3
4
5
6
7
{
    "resource_backend_id_1": {
        "TOTAL_ACCOUNT_USAGE": {"cpu": 1000, "mem": 2048},
        "user1": {"cpu": 500, "mem": 1024},
        "user2": {"cpu": 500, "mem": 1024},
    }
}
  • Key rules:
  • Component keys must match backend_components config keys.
  • Values must be in Waldur units (after unit_factor conversion).
  • TOTAL_ACCOUNT_USAGE is required and must equal the sum of per-user values.
  • No-op: Return {}.

_collect_resource_limits(waldur_resource) -> tuple[dict, dict]

  • Mode: order_process (resource creation)
  • Purpose: Convert Waldur limits to backend limits and back.
  • Returns: (backend_limits, waldur_limits) where backend_limits has values multiplied by unit_factor.
  • No-op: Return ({}, {}).

_pre_create_resource(waldur_resource, user_context=None) -> None

  • Mode: order_process (resource creation)
  • Purpose: Set up prerequisites before resource creation (e.g., parent accounts).
  • user_context contains pre-resolved data: ssh_keys (UUID → public key), plan_quotas (component → value), team, offering_users.
  • Runs once for every candidate backend ID the uniqueness retry tries, including the ones the backend then rejects. Do not create anything named after the resource's own backend ID here; use _on_backend_resource_created.
  • No-op: Use pass.

downscale_resource(resource_backend_id: str) -> bool

  • Mode: membership_sync
  • Purpose: Restrict resource capabilities (e.g., set restrictive QoS).
  • No-op: Return True.

pause_resource(resource_backend_id: str) -> bool

  • Mode: membership_sync
  • Purpose: Prevent all usage of the resource.
  • No-op: Return True.

restore_resource(resource_backend_id: str) -> bool

  • Mode: membership_sync
  • Purpose: Restore resource to normal operation.
  • No-op: Return True.

get_resource_metadata(resource_backend_id: str) -> dict

  • Mode: membership_sync
  • Purpose: Return backend-specific metadata for Waldur.
  • No-op: Return {}.

Hook methods (override as needed)

These have default implementations in BaseBackend. Override only when your backend needs custom behavior.

Method Default When to override
_on_backend_resource_created No-op Create what is addressed by the backend ID later (group, directory, QoS)
post_create_resource No-op Post-creation setup; set resource.backend_metadata to push data to Waldur
_pre_delete_resource No-op Pre-deletion cleanup (cancel jobs)
post_delete_resource No-op Post-deletion cleanup (e.g., remove child offerings linked to the resource)
_pre_delete_user_actions No-op Per-user cleanup before removal
process_existing_users No-op Process existing users (homedirs)
check_pending_order Returns True Non-blocking order creation (see below)
evaluate_pending_order Returns ACCEPT Custom approval logic for pending orders (see below)
setup_target_event_subscriptions Returns [] STOMP subscriptions to target systems
get_usage_report_for_period Returns {} Historical usage queries for past billing periods
has_prepaid_components Returns False Enable duration-aware limit calculation for prepaid billing
sync_resource_end_date No-op Synchronise end_date between source and target Waldur instances
sync_project_end_date No-op Synchronise the project's end_date (receives the pre-fetched source project)
sync_resource_limits Pulls backend limits into Waldur Change the limits source of truth
sync_resource_effective_id No-op Reflect downstream backend_id as effective_id on the source resource
sync_resource_project No-op Push project metadata to backends that manage their own projects
update_user_attributes No-op Forward OFFERING_USER attribute updates to the backend
sync_offering_user_usernames Returns False Pull backend-assigned usernames into Waldur (federation)
create_user_homedirs Provided Customise homedir quota or path logic (see supports_user_homedirs)
apply_periodic_settings Reports failure Apply periodic usage-policy settings (see supports_periodic_settings)
run_preflight ping(), else BackendNotReadyError Probe endpoints before an order cycle
create_resource_with_id Creates under the given id Required if the backend assigns ids (see below)
recreate_missing_resource Re-creates under the old id Return False if ids cannot be reused (see below)
remote_waldur_base_urls Returns [] Waldur-to-Waldur: return the target's base URL (see below)
expects_target_event_subscriptions Returns False True if you override setup_target_event_subscriptions

For create_resource_with_id, recreate_missing_resource and remote_waldur_base_urls see Resource identifiers and Error handling and ERRED. sync_resource_limits backs the waldur_sync_resource_limits command. expects_target_event_subscriptions lets the event-mode watchdog retry a missing target subscription.

These hooks receive the agent's waldur_rest_client: evaluate_pending_order, sync_resource_end_date, sync_project_end_date, sync_resource_limits, sync_resource_effective_id and sync_offering_user_usernames, and on username backends reconcile_offering, reconcile_project_groups and release_users. No other method does — sync_resource_project and sync_user_profiles included.

Non-blocking order creation (optional)

Backends that create resources via remote APIs can use non-blocking order creation. Instead of blocking until the remote operation completes, the backend returns immediately with a pending_order_id in BackendResourceInfo.

To opt in, set the supports_async_orders class attribute to True. The processor only inspects order.backend_id for async tracking when this flag is enabled, which prevents conflicts with external systems (e.g. SharePoint) that may set order.backend_id for unrelated purposes.

1
2
class MyAsyncBackend(BaseBackend):
    supports_async_orders = True

When enabled, the core processor:

  1. Sets the source order's backend_id to the pending_order_id
  2. Keeps the order in EXECUTING state
  3. On subsequent polling cycles, calls check_pending_order(backend_id) to check completion
  4. When check_pending_order() returns True, marks the source order as DONE

Backends that opt in will typically also override handled_resource_states to include ResourceState.CREATING so that user/limit sync runs while the remote order is still in flight.

check_pending_order(order_backend_id: str) -> bool

  • Default: Returns True (no async orders, always "complete")
  • Override when: Your backend uses non-blocking resource creation
  • Returns: True if the remote order completed, False if still pending
  • Raises: BackendError if the remote order failed or was cancelled

Example (Waldur federation plugin):

1
2
3
4
5
6
7
def check_pending_order(self, order_backend_id: str) -> bool:
    target_order = self.client.get_order(UUID(order_backend_id))
    if target_order.state == OrderState.DONE:
        return True
    if target_order.state in {OrderState.ERRED, OrderState.CANCELED}:
        raise BackendError(f"Target order failed: {target_order.state}")
    return False  # Still pending

setup_target_event_subscriptions(source_offering, user_agent, global_proxy) -> list

  • Default: Returns [] (no target subscriptions)
  • Override when: Your backend supports STOMP events from a target system
  • Returns: List of StompConsumer tuples for lifecycle management
  • Called by: event_process mode during STOMP setup

Pending order evaluation (optional)

When an order arrives in PENDING_PROVIDER state, the agent calls evaluate_pending_order on the backend before taking any action. The default implementation returns ACCEPT, which preserves the existing auto-approve behaviour. Override this method to implement custom approval logic.

evaluate_pending_order(order, waldur_rest_client) -> PendingOrderDecision

  • Default: Returns PendingOrderDecision.ACCEPT
  • Override when: You need to inspect or gate orders before approval
  • Parameters:
  • order (OrderDetails) — full order data including project_uuid, customer_uuid, created_by_*, attributes, and consumer_message / provider_message fields.
  • waldur_rest_client (AuthenticatedClient) — authenticated client for fetching additional data from the Waldur API (e.g., project members and roles).
  • Returns one of:
  • PendingOrderDecision.ACCEPT — approve the order
  • PendingOrderDecision.REJECT — reject the order
  • PendingOrderDecision.PENDING — keep waiting; the order will be re-evaluated on the next polling cycle

Note: Only a few other hooks receive waldur_rest_client; see the list under Hook methods. Resource lifecycle methods get Waldur data via user_context instead.

Use cases

Scenario Approach
Wait for a PI Query project members, return PENDING until a PI role exists
Reject unprocessable orders Inspect order.attributes, return REJECT
Require a signed agreement Set provider_message, return PENDING until consumer_message is set

Example: wait for a PI before approving

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
from waldur_api_client.api.marketplace_provider_resources import (
    marketplace_provider_resources_team_list,
)
from waldur_site_agent.backend.backends import BaseBackend, PendingOrderDecision


class MyBackend(BaseBackend):
    def evaluate_pending_order(self, order, waldur_rest_client):
        team = marketplace_provider_resources_team_list.sync(
            client=waldur_rest_client,
            uuid=order.marketplace_resource_uuid.hex,
        )
        has_pi = any(
            member.role_name == "PI" for member in (team or [])
        )
        if not has_pi:
            return PendingOrderDecision.PENDING
        return PendingOrderDecision.ACCEPT

Example: reject orders that lack a required attribute

1
2
3
4
5
6
7
8
9
from waldur_site_agent.backend.backends import BaseBackend, PendingOrderDecision


class MyBackend(BaseBackend):
    def evaluate_pending_order(self, order, waldur_rest_client):
        attrs = getattr(order, "attributes", None) or {}
        if not attrs.get("project_justification"):
            return PendingOrderDecision.REJECT
        return PendingOrderDecision.ACCEPT

Username management backends

Username management is handled by a separate plugin family inheriting from AbstractUsernameManagementBackend (defined in waldur_site_agent/backend/backends.py). These backends generate or look up local-IDP usernames for OfferingUser records and are wired in the config via username_management_backend.

Required methods

Method Purpose
generate_username(offering_user) -> str Create a new local username. Return "" if generation is not supported.
get_username(offering_user) -> Optional[str] Look up the existing local username for the user, or None.

get_or_create_username is provided by the base class and calls get_username first, then generate_username only if no username is found.

Optional hook methods

Method Default When to override
sync_user_profiles(offering_users) No-op Push user profiles to the IDP before membership sync runs
deactivate_users(usernames) No-op Remove departed users from the external system
reconcile_offering(waldur_rest_client) No-op Offering-wide reconcile, once per periodic cycle
reconciles_project_groups() Returns False The backend writes the provider's project groups
reconcile_project_groups(waldur_rest_client) No-op Write project groups on project-group events
release_users(offering_users, waldur_rest_client) No-op Release accounts of offering users whose access ended
  • reconcile_offering runs even when the offering has no offering users, so state that must be cleaned up after the last account is gone has a place.
  • When reconciles_project_groups() returns True, event mode subscribes the offering to project-group events and calls reconcile_project_groups on them. It must tolerate running concurrently with reconcile_offering.
  • release_users is called after the users' associations were removed. Raise on failure, so core does not acknowledge the deletion to Waldur.

reconcile_offering, reconcile_project_groups and release_users receive the agent's waldur_rest_client so the backend can ask Waldur before acting (for example, whether the person still holds another project).

Class attribute is_username_authoritative

Defaults to True: the backend mints usernames and the value it returns is written back to Waldur. Set it to False (or override as a property) for backends where Waldur is the source of truth; core then skips username generation and the write-back for that offering.

Error signalling for user-action gates

When username generation requires the user to take an action (e.g. link an existing IdP account, complete a validation form), raise one of these exceptions from waldur_site_agent.backend.exceptions:

  • OfferingUserAccountLinkingRequiredError(comment, comment_url=None) — user must link an existing account.
  • OfferingUserAdditionalValidationRequiredError(comment, comment_url=None) — additional validation is required.

The processor moves the offering user into a PENDING_ACCOUNT_LINKING / PENDING_ADDITIONAL_VALIDATION state and surfaces the comment (and URL) to the operator. See docs/offering-users.md for the full state machine.

Entry point group

Register username management plugins under a different entry point group than resource backends:

1
2
[project.entry-points."waldur_site_agent.username_management_backends"]
mycustom = "waldur_site_agent_mycustom.username_backend:MyCustomUsernameBackend"

The fallback entry point name is base, provided by the waldur-site-agent-basic-username-management plugin.

BaseClient method reference

Implement every method below. They are marked @abstractmethod, but BaseClient is a plain class rather than an abc.ABC, so the marker is not enforced: a client missing a method still instantiates, and the inherited stub returns None when called. The core and BaseBackend call all of them.

Method Signature Purpose
list_resources () -> list[ClientResource] List all resources on backend
get_resource (resource_id) -> ClientResource or None Get single resource or None
create_resource (name, description, organization, parent_name=None) -> str Create resource
delete_resource (name) -> str Delete resource
set_resource_limits (resource_id, limits_dict) -> str or None Set limits (backend units)
get_resource_limits (resource_id) -> dict[str, int] Get limits (backend units)
get_resource_user_limits (resource_id) -> dict[str, dict[str, int]] Per-user limits
set_resource_user_limits (resource_id, username, limits_dict) -> str Set per-user limits
get_association (user, resource_id) -> Association or None Check user-resource link
create_association (username, resource_id, default_account=None) -> str Create user-resource link
delete_association (username, resource_id) -> str Remove user-resource link
get_usage_report (resource_ids, timezone=None) -> list Raw usage data from backend
list_resource_users (resource_id) -> list[str] List usernames for resource

Important: BaseClient also provides:

  • execute_command(command, silent=False) for running CLI commands with error handling — use it for CLI-based backends.
  • create_linux_user_homedir(username, umask="") which shells out to /sbin/mkhomedir_helper. Override only if your backend creates home directories some other way; otherwise it works as-is for SLURM-style Linux deployments.

Agent mode method matrix

This table shows which BaseBackend methods are called by each agent mode.

Method order_process report membership_sync event_process
ping (via run_preflight) each cycle ¹ - - each order cycle ¹
create_resource_with_id CREATE order - - CREATE event
recreate_missing_resource - - - forced resource sync (OFFERING_RESOURCES_SYNC event)
_pre_create_resource CREATE order - - CREATE event
post_create_resource CREATE order - - CREATE event
_collect_resource_limits CREATE order - - CREATE event
check_pending_order CREATE order (async) - - CREATE event (async)
evaluate_pending_order pending-provider orders - - -
set_resource_limits UPDATE order - - UPDATE event
delete_resource TERMINATE order - - TERMINATE event
_pre_delete_resource TERMINATE order - - TERMINATE event
pull_resource / pull_resources CREATE order usage pull sync cycle various events
_get_usage_report - usage pull sync cycle -
add_users_to_resource post-create - user sync role events
remove_users_from_resource - - user sync role events
add_user / remove_user - - role changes role events
downscale_resource - - status sync -
pause_resource - - status sync -
restore_resource - - status sync -
get_resource_metadata - - status sync -
setup_target_event_subscriptions - - - STOMP setup
list_resources - import - import event
get_resource_limits - import - import event
get_resource_user_limits - - limits sync -
set_resource_user_limits - - limits sync -
process_existing_users - - user sync -

¹ Only for backends with supports_cycle_preflight = True.

Reporting user add/remove failures

add_user and remove_user share one contract: return True when the user is associated (or removed) afterwards, False only when there was nothing to do (blank username, feature disabled), and raise BackendError when the backend refused. A failure folded into False reads as a skip and is never reported, so nobody learns the user is missing.

Raise UserNotProvisionedError (a BackendError subclass) when the user does not exist in the identity provider yet, e.g. no first Keycloak sign-in. The base add_users_to_resource skips such users and retries them on the next pass.

The base add_users_to_resource / remove_users_from_resource try every user and record each failure with its cause. Membership sync logs them at ERROR (the users, capped at ten, and the first cause) and retries them on the next pass. The resource state is not changed and last_sync is still refreshed: one member's problem is not a failure of an allocation that works for everyone else. A backend that overrides these bulk methods reports nothing unless it returns AddedUsers / RemovedUsers with failed filled in.

Usage report format specification

The _get_usage_report method must return data in this exact structure:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
{
    "<resource_backend_id>": {
        "TOTAL_ACCOUNT_USAGE": {
            "<component_key>": <int_value>,  # Sum of all per-user values
            ...
        },
        "<username_1>": {
            "<component_key>": <int_value>,
            ...
        },
        "<username_2>": {
            "<component_key>": <int_value>,
            ...
        },
    },
    "<another_resource_backend_id>": { ... },
}

Rules

  1. Component keys must exactly match those in the backend_components YAML config.
  2. Values must be integers in Waldur units (i.e., divide raw backend values by unit_factor).
  3. TOTAL_ACCOUNT_USAGE is a required key and must equal the sum of all per-user values for each component.
  4. If a resource has no usage, return {"TOTAL_ACCOUNT_USAGE": {"cpu": 0, "mem": 0, ...}}.
  5. If usage reporting is not supported, return {} (empty dict).

Example: SLURM CPU and memory

Given config:

1
2
3
4
5
6
7
backend_components:
  cpu:
    unit_factor: 60000
    measured_unit: "k-Hours"
  mem:
    unit_factor: 61440
    measured_unit: "gb-Hours"

If SLURM reports 120000 cpu-minutes and 122880 MB-minutes for user1:

1
2
3
4
5
6
{
    "hpc_my_allocation": {
        "TOTAL_ACCOUNT_USAGE": {"cpu": 2, "mem": 2},
        "user1": {"cpu": 2, "mem": 2},
    }
}

Calculation: 120000 / 60000 = 2, 122880 / 61440 = 2.

Capability flags and class attributes

BaseBackend exposes class-level capability flags that change how the core processor treats your backend.

supports_decreasing_usage: bool = False

Set to True if usage values can decrease between reports (e.g., a storage backend reporting current disk usage rather than accumulated compute time).

1
2
class MyStorageBackend(BaseBackend):
    supports_decreasing_usage = True

When False (default), the reporting processor skips updates where the new usage value is lower than the previously reported value, treating it as a data anomaly.

supports_cycle_preflight: bool = False

Set to True for backends that call a remote API during order processing. The order processor runs run_preflight() once per offering per cycle before listing orders. The default implementation calls ping() and raises BackendNotReadyError on failure so orders stay pending instead of ERRED.

Override run_preflight() to probe specific endpoints. Opt in from plugins such as Waldur federation and other HTTP backends can enable it when needed.

supports_async_orders: bool = False

Set to True for backends that complete order creation asynchronously on a remote system and report progress via pending_order_id. See the "Non-blocking order creation" section above for the full flow.

1
2
class MyAsyncBackend(BaseBackend):
    supports_async_orders = True

Resource API keys

Three flags cover API keys that a resource hands to its consumers, such as gateway keys or S3 access keys. The agent mints every key. It applies the key to the backend first and reports it to Waldur second, so Waldur only stores a key the backend already accepts.

Flag What the backend implements
supports_resource_api_keys generate_resource_keys at provisioning, and rotate_resource_key
supports_resource_api_key_lifecycle Single keys: mint_, pause_, resume_, delete_, update_resource_key
supports_resource_api_key_usage Reporting backend: get_resource_key_usage_report

Each per-key method changes the backend and returns. The core then sends Waldur the acknowledgement for that command: set_key, set_paused, set_ok or set_deleted. If the method raises, the core reports set_erred instead. The methods must be safe to call again after they succeeded, because the agent replays a command whose acknowledgement never reached Waldur.

Key commands are not orders, but the agent picks them up alongside orders, and a plugin does not choose how. order_process mode applies every pending command on each cycle, after the orders. event_process mode applies each command as its STOMP message arrives. Its reconciliation sweep replays a command left pending for 30 minutes, in case the message was lost.

  • Pausing a key is not pausing the resource. A key paused on its own must stay paused when pause_resource and restore_resource run around it. The membership sync calls restore_resource on every cycle for a resource that is not paused.
  • Never reuse a client_id. mint_resource_key receives every client_id Waldur holds for the resource, including those of deleted keys. Waldur attributes usage by client_id and refuses one that was already used.
  • Refuse what you cannot enforce. If a backend is given a setting it cannot apply, such as a model allowlist on a gateway with no per-key model rule, it must raise. Accepting the setting silently would show the key in Waldur as restricted when it is not.

A backend without supports_resource_api_key_lifecycle keeps generation and rotation only. The core answers any other command for its keys with set_erred. ceph-s3 is such a backend: its access key rotates together with the secret, and pausing a key there is a no-op.

Per-key usage is reported only when the offering sets the Waldur plugin option enable_api_key_provisioning. The report maps each resource to {client_id: {component: usage}}, and those figures must add up to the resource total. Map a resource to None when its usage cannot be attributed to keys; the core then skips per-key reporting for that resource. Waldur pauses a key once its reported usage reaches the key's limit.

supports_user_homedirs: bool = False

Set to True for backends that can create POSIX home directories for their users. The standalone waldur_site_create_homedirs command iterates over every configured offering and calls create_user_homedirs on each backend that declares support — the gate is the capability, not the backend type. The per-offering enable_user_homedir_account_creation setting can still opt out.

1
2
class MyHpcBackend(BaseBackend):
    supports_user_homedirs = True

A backend that opts in should also inherit HomedirSettingsSchema from waldur_site_agent.common.plugin_schemas in its settings schema, so enable_user_homedir_account_creation, default_homedir_umask, homedir_base_path and homedir_quota validate identically everywhere:

1
2
3
4
from waldur_site_agent.common.plugin_schemas import HomedirSettingsSchema

class MyHpcBackendSettingsSchema(HomedirSettingsSchema):
    model_config = ConfigDict(extra="allow")

SLURM declares the flag today. Leave it False for API-only backends: the inherited create_linux_user_homedir shells out to the local /sbin/mkhomedir_helper, which is meaningless where the users have no account on the agent's host.

Note that enabling the flag on an existing backend is a behaviour change for its deployments — enable_user_homedir_account_creation defaults to True, so operators already running waldur_site_create_homedirs will start getting home directories with no config change of their own. Land it with a release note.

supports_periodic_settings: bool = False

Set to True for backends that implement apply_periodic_settings, the hook Waldur calls with periodic usage-policy settings (fairshare, limits, usage resets). Override the method alongside the flag:

1
2
3
4
5
6
class MyBackend(BaseBackend):
    supports_periodic_settings = True

    def apply_periodic_settings(self, resource_id, settings, config=None):
        ...
        return {"success": True, "commands_executed": [...]}

The return value is relayed to Waldur's report-command-result endpoint, so it must carry a success key and, on failure, an error message. The default implementation returns an explicit failure — an unsupported backend still reports a verdict rather than leaving the policy without an answer.

handled_resource_states: list = [ResourceState.OK, ResourceState.ERRED]

Controls which resource states the membership processor fetches and processes. Override when your backend needs to manage users on resources that are still being provisioned (e.g., async backends that include CREATING).

1
2
3
4
from waldur_api_client.models.resource_state import ResourceState

class MyAsyncBackend(BaseBackend):
    handled_resource_states = [ResourceState.OK, ResourceState.ERRED, ResourceState.CREATING]

team_fetch_attempts: int = 1 and team_fetch_delay: float = 3.0

Control retry behaviour when _fetch_user_context_for_resource gets back an empty team list. The retry only fires on a completely empty response — it covers the race where a create-order event arrives before Waldur has committed the first membership row to the database.

The default of 1 (no retries) is correct for synchronous backends where the Waldur API is always consistent by the time the agent reads it. Raise team_fetch_attempts for backends that receive work over STOMP, where the event and the membership row may arrive out of order:

1
2
3
class MyStompBackend(BaseBackend):
    team_fetch_attempts = 4   # retry up to 4 times
    team_fetch_delay = 3.0    # seconds between retries

These are backend class attributes, not YAML configuration keys. They cannot be set per-offering in the configuration file.

requires_source_project: bool = False

Set to True for backends that mirror project metadata. Membership sync then pre-fetches the source project from Waldur and passes it to sync_resource_project and sync_project_end_date, so the plugin never calls Waldur for it.

shared_project_membership: bool = False

Set to True when every resource federated from one source project lands in one shared backend project (Waldur-to-Waldur federation). Membership sync then reconciles against the union of consented teams across all the project's resources, so a user is kept while they consent to any offering with a resource in the project.

Attributes read without a base-class default

The membership processor also reads these attributes with getattr, so they work only when set on your backend class:

  • skip_resource_team_diff = True — the backend manages membership below the resource level; core skips the team-vs-backend diff, so it adds and removes nobody.
  • user_resolve_method = "identity_bridge" — the diff is keyed on the user's CUID (ProjectUser.username) instead of the offering username (federation).
  • fetch_consented_users_only = True — the resource team is fetched with consenting users only.

Decision matrix for no-op implementations

If your backend does not support a certain operation, use these return values:

Method No-op return Meaning
ping False Backend has no health check
diagnostics True Diagnostics not implemented but OK
list_components [] No component discovery
_get_usage_report {} No usage reporting
_collect_resource_limits ({}, {}) No limits support
_pre_create_resource pass No pre-creation setup
downscale_resource True No downscaling concept
pause_resource True No pausing concept
restore_resource True No restore concept
get_resource_metadata {} No metadata

Annotated YAML configuration

Every ``yaml block that containsofferings:in the repository's Markdown, and every file underexamples/andplugins//examples/, is loaded bytests/test_docs_config_examples.pyin the core test job. A block fails when it does not load through the real configuration loader, uses a key the configuration model does not define (the loader would ignore it silently), names a backend that is not a registered entry point, sets no_backend` at all, or produces a settings-schema warning. A deliberately partial snippet opts out with a comment on the line before its opening fence (a blank line in between is fine):

1
<!-- docs-check: skip -->

A top-level key starting with . (for example .ldap: &ldap_settings) is accepted as a YAML anchor holder.

The mycustom entry point below is the template plugin's; it resolves only once you register your own, so the block is marked to be skipped.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
offerings:
  - name: "My Custom Offering"          # Human-readable name for logging

    # Waldur Mastermind connection
    waldur_api_url: "https://waldur.example.com/api/"
    waldur_api_token: "your-api-token"   # Service provider token
    waldur_offering_uuid: "uuid-here"    # UUID from Waldur offering page

    # Backend selection (entry point names from pyproject.toml)
    order_processing_backend: "mycustom"       # For create/update/terminate
    reporting_backend: "mycustom"              # For usage reporting
    membership_sync_backend: "mycustom"        # For user sync
    username_management_backend: "base"        # Username generation

    # Required. Selects the plugin schemas that validate backend_settings
    # and backend_components; it does not choose the backend for a mode.
    backend_type: "mycustom"

    # Event processing (optional)
    stomp_enabled: false

    # Backend-specific settings (passed to __init__ as backend_settings)
    backend_settings:
      default_account: "root"            # DefaultAccount= on user associations
      customer_prefix: "cust_"           # Prefix for customer-level accounts
      project_prefix: "proj_"            # Prefix for project-level accounts
      allocation_prefix: "alloc_"        # Prefix for allocation-level accounts

    # Component definitions (passed to __init__ as backend_components)
    backend_components:
      cpu:
        limit: 100                       # Default limit in Waldur units
        measured_unit: "k-Hours"         # Display unit in Waldur UI
        unit_factor: 60000               # Waldur-to-backend conversion factor
        accounting_type: "usage"         # "usage" = metered, "limit" = quota
        label: "CPU"                     # Display label in Waldur UI
        # Optional Waldur offering component fields:
        # description: "CPU time"        # Component description
        # min_value: 0                   # Minimum allowed value
        # max_value: 10000               # Maximum allowed value
        # max_available_limit: 5000      # Maximum available limit
        # default_limit: 100             # Default limit value
        # limit_period: "month"          # "annual", "month", "quarterly", "total"
        # article_code: "CPU-001"        # Billing article code
        # is_boolean: false              # Boolean (on/off) component
        # is_prepaid: false              # Prepaid billing
      storage:
        limit: 1000
        measured_unit: "GB"
        unit_factor: 1
        accounting_type: "limit"
        label: "Storage"

unit_factor explained

The unit_factor converts between Waldur display units and backend-native units:

  • backend_value = waldur_value * unit_factor
  • waldur_value = backend_value / unit_factor

Examples:

  • CPU k-Hours to SLURM cpu-minutes: unit_factor = 60000 (60 min x 1000)
  • GB-Hours to SLURM MB-minutes: unit_factor = 61440 (60 min x 1024 MB)
  • GB to GB (no conversion): unit_factor = 1

Entry point registration

Register your plugin in pyproject.toml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
[project]
name = "waldur-site-agent-mycustom"
version = "0.1.0"
dependencies = ["waldur-site-agent>=0.7.0"]

[project.entry-points."waldur_site_agent.backends"]
mycustom = "waldur_site_agent_mycustom.backend:MyCustomBackend"

# Optional: register a username management backend
[project.entry-points."waldur_site_agent.username_management_backends"]
mycustom = "waldur_site_agent_mycustom.username_backend:MyCustomUsernameBackend"

# Optional: component schema validation
[project.entry-points."waldur_site_agent.component_schemas"]
mycustom = "waldur_site_agent_mycustom.schemas:MyCustomComponentSchema"

# Optional: backend settings schema validation
[project.entry-points."waldur_site_agent.backend_settings_schemas"]
mycustom = "waldur_site_agent_mycustom.schemas:MyCustomBackendSettingsSchema"

The entry point name (e.g., mycustom) is what users put in backend_type, order_processing_backend, or username_management_backend in the config YAML. The four entry-point groups are independent — a single distribution may register some or all of them.

Processor-plugin data flow

Plugins generally do not have direct access to the Waldur API client. The core processor pre-resolves any Waldur data the plugin might need and passes it via user_context. Plugins return metadata to Waldur by setting resource.backend_metadata.

Exceptions: evaluate_pending_order receives waldur_rest_client directly, because the order has not been approved yet and no resource context exists at that point. The hooks listed under Hook methods receive it too, because they exist to reconcile state with Waldur.

Pre-resolved data in user_context

The processor enriches the user_context dict before calling backend methods. Plugins read from it without making API calls:

Key Type Contents
team list[ProjectUser] Team members of the resource's project (SDK objects)
offering_users list[OfferingUser] Offering users of the team members (SDK objects)
user_mappings dict[UUID, ProjectUser] Team members keyed by user UUID
offering_user_mappings dict[UUID, OfferingUser] The team's offering users keyed by user UUID
ssh_keys dict[str, str] Mapping of SSH key UUID → public key text
plan_quotas dict[str, int] Plan component quotas (component key → value)

team, offering_users and both mappings are empty when the team could not be fetched; never assume a key is present in unit tests.

Returning metadata via backend_metadata

To push metadata back to Waldur (e.g., access credentials, connection endpoints), set resource.backend_metadata in post_create_resource:

1
2
3
4
5
6
7
8
def post_create_resource(self, resource, waldur_resource, user_context=None):
    # ... create credentials, gather endpoints ...
    resource.backend_metadata = {
        "username": "admin",
        "password": generated_password,
        "endpoint": "https://service.example.com",
    }
    # The processor pushes this to Waldur automatically

Data flow

sequenceDiagram
    participant P as Processor
    participant B as YourBackend
    participant W as Waldur API
    participant E as External System

    P->>P: Fetch service provider
    P->>B: Set service_provider_uuid

    Note over P,B: Resource creation order arrives

    P->>W: Resolve SSH keys, plan quotas
    W-->>P: Pre-resolved data
    P->>B: _pre_create_resource(resource, user_context)
    B->>B: Read ssh_keys, plan_quotas from user_context
    B->>E: Create resource with resolved data
    E-->>B: Resource created
    B->>B: _on_backend_resource_created(backend_id, resource)

    P->>B: post_create_resource(resource, waldur_resource, user_context)
    B->>B: Set resource.backend_metadata
    B-->>P: Return
    P->>W: Push backend_metadata to Waldur

Example: resolving an SSH key UUID from user_context

When a resource attribute contains a UUID reference (e.g., an SSH key UUID from the order form), look it up in the pre-resolved ssh_keys dict:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
from uuid import UUID

class MyBackend(BaseBackend):
    @staticmethod
    def _resolve_ssh_key(key_value: str, ssh_keys: dict[str, str]) -> str:
        """Resolve SSH key from pre-resolved context.

        If key_value is a UUID, look it up. Otherwise treat as raw key text.
        """
        try:
            key_uuid = UUID(key_value.strip())
        except ValueError:
            return key_value  # Raw public key text, use as-is

        return ssh_keys.get(str(key_uuid), "") or ssh_keys.get(key_uuid.hex, "")

    def _pre_create_resource(self, waldur_resource, user_context=None):
        user_context = user_context or {}
        ssh_keys = user_context.get("ssh_keys", {})
        raw_key = waldur_resource.attributes.get("ssh_public_key", "")
        resolved_key = self._resolve_ssh_key(raw_key, ssh_keys)
        # Use resolved_key for resource setup ...

Design principles

  • Plugins should avoid importing waldur_api_client for runtime API calls. All Waldur data should come via user_context or BaseBackend attributes. The exceptions are the hooks that are handed waldur_rest_client (listed under Hook methods); use that client rather than building your own, so authentication (static token or OIDC) and global_proxy apply.
  • service_provider_uuid is still set on BaseBackend by the processor and can be read by plugins for constructing backend-side identifiers.
  • Handle missing context gracefully — user_context may be None or missing keys in unit tests. Always default to {} or empty values.

Resource identifiers

The order processor never calls create_resource. Every creation goes through create_resource_with_id(waldur_resource, resource_backend_id, user_context), from three places:

  1. CREATE order for a resource without a backend_id. Core passes a suggestion from _get_resource_backend_id(slug), retries with the next suggestion on DuplicateResourceError, and stores the returned BackendResourceInfo.backend_id in Waldur.
  2. CREATE/RESTORE order whose backend_id is set but pull_resource finds nothing. Core passes the existing backend_id and does not store a new id — it expects the resource back under the same one.
  3. Forced resource sync (OFFERING_RESOURCES_SYNC event), through recreate_missing_resource. Core passes the existing backend_id and discards the result.

The base implementation creates the resource under the id it is given, which is right for backends where the agent names things (SLURM accounts, LDAP groups, Harbor projects).

If your system assigns its own ids (cloud VMs, droplets, remote orders), override both:

  • create_resource_with_id: ignore a suggested id and return the id the system assigned. When it is called with the resource's own existing backend_id, raise BackendError instead of creating — a new object would get a new id that core never stores.
  • recreate_missing_resource: return False (with a log line) instead of creating a replacement that Waldur would never reference.

Skipping either leaks orphaned, possibly billed, objects on every retry or forced sync. plugins/digitalocean implements both; plugins/azure overrides both but does not refuse the existing-id case.

Error handling and ERRED

Core decides from the exception whether a failure marks a resource or order ERRED.

Membership sync classifies each exception in this order:

  1. Transient — any httpx.TransportError, or an HTTP 5xx or 429 raised through the Waldur client library (UnexpectedStatus): the resource is skipped this pass, its state unchanged. There is no URL check, so this also covers a network error or 5xx from your backend if it uses httpx or the Waldur client (SLURM REST, a Waldur-to-Waldur target).
  2. A 4xx from the agent's own Waldur API (wrong permission, missing endpoint): the step is skipped and logged, the other steps still run, the resource state is unchanged and last_sync is not refreshed.
  3. Anything else — BackendError, a CLI failure, a 4xx from a remote Waldur: the resource is marked ERRED with the error; the other resources still run.

An ERRED resource is set back to OK once every backend step runs again. So raise when the backend failed; never wrap a Waldur API failure in BackendError, or a Waldur hiccup marks healthy allocations ERRED. If a backend outage should mark resources ERRED, raise BackendError rather than letting a raw httpx error escape.

Waldur-to-Waldur backends call a remote Waldur with the same client library. Return its base URL from remote_waldur_base_urls(), so a 4xx from it stays a backend error (ERRED) even when the remote Waldur shares the agent's host. Its 5xx, 429 and network errors are transient (rule 1).

Order processing:

  • A transient error (rule 1 above) never errs an order: it is retried and the order keeps its state.
  • Before provisioning starts (approval, pending-order evaluation), any other failure leaves the order pending for the next cycle.
  • After provisioning starts, a backend failure marks the order ERRED — unless Waldur already shows it finished (done, canceled, rejected, erred), which core re-reads first.
  • Once an order is DONE, failures in post-processing (adding users, accounts) are logged and left to membership sync; the order stays DONE.
  • Raise BackendNotReadyError (for example from run_preflight) to keep orders pending while the backend is unreachable.

Common pitfalls

1. Unit factor direction

The unit_factor converts from Waldur units to backend units by multiplication. When reporting usage back, you must divide by unit_factor. Getting this backwards causes limits to be set at 1/60000th of the intended value or usage to be reported 60000x too high.

2. Missing TOTAL_ACCOUNT_USAGE

The _get_usage_report return dict must include a "TOTAL_ACCOUNT_USAGE" key for each resource. If the key is missing, the reporter logs No TOTAL_ACCOUNT_USAGE for <resource> … skipping and submits nothing for that resource and period — no total and no per-user usage. (Zeros are filled in only when the whole resource entry is missing from the report.)

3. Entry point not discovered

Common causes:

  • Package not installed (uv sync --all-packages)
  • Entry point group name misspelled. Resource backends use "waldur_site_agent.backends"; username management uses "waldur_site_agent.username_management_backends" (note the plural and the suffix). Validation schemas use "waldur_site_agent.component_schemas" / "waldur_site_agent.backend_settings_schemas".
  • Entry point value points to wrong class or module

Debug with:

1
2
3
from importlib.metadata import entry_points
print(list(entry_points(group="waldur_site_agent.backends")))
print(list(entry_points(group="waldur_site_agent.username_management_backends")))

4. Forgetting super().init()

Your backend __init__ must call super().__init__(backend_settings, backend_components). This sets up self.backend_settings, self.backend_components, and self.client. Then assign your own client:

1
2
3
4
def __init__(self, backend_settings, backend_components):
    super().__init__(backend_settings, backend_components)
    self.backend_type = "mycustom"
    self.client = MyCustomClient()

5. Returning wrong types from client methods

  • get_resource must return None (not raise) when resource is absent.
  • get_association must return None (not raise) when no association exists.
  • list_resources must return list[ClientResource], not raw dicts.

6. Component key mismatch

Component keys in _get_usage_report must exactly match the keys in backend_components config. If config has "cpu" but you report "CPU", the usage will be silently ignored.

Testing guidance

What to test per mode

Mode Test focus
order_process create_resource_with_id (the method core calls), delete_resource, limit conversion
report _get_usage_report format, unit conversion math
membership_sync add_user, remove_user, pause/restore
All ping, error handling, edge cases

Mock patterns

Mock the client to avoid needing a real backend:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from unittest.mock import MagicMock, patch
from waldur_site_agent.backend.structures import ClientResource, Association

def test_create_resource():
    backend = MyCustomBackend(
        backend_settings={"default_account": "root", "allocation_prefix": "test_"},
        backend_components={"cpu": {"unit_factor": 60000, "limit": 10}},
    )
    backend.client = MagicMock()
    backend.client.get_resource.return_value = None  # Resource doesn't exist yet
    backend.client.create_resource.return_value = "created"

    # ... test resource creation

Fixtures

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
import pytest

@pytest.fixture
def backend_settings():
    return {
        "default_account": "root",
        "customer_prefix": "c_",
        "project_prefix": "p_",
        "allocation_prefix": "a_",
    }

@pytest.fixture
def backend_components():
    return {
        "cpu": {
            "limit": 10,
            "measured_unit": "k-Hours",
            "unit_factor": 60000,
            "accounting_type": "usage",
            "label": "CPU",
        },
    }

@pytest.fixture
def backend(backend_settings, backend_components):
    b = MyCustomBackend(backend_settings, backend_components)
    b.client = MagicMock()
    return b

Key assertions

1
2
3
4
5
6
7
8
9
# Usage report format
report = backend._get_usage_report(["alloc_1"])
assert "TOTAL_ACCOUNT_USAGE" in report["alloc_1"]
assert all(k in report["alloc_1"]["TOTAL_ACCOUNT_USAGE"]
           for k in backend.backend_components)

# Limit conversion
backend_limits, waldur_limits = backend._collect_resource_limits(mock_resource)
assert backend_limits["cpu"] == waldur_limits["cpu"] * 60000

LLM implementation checklist

When implementing a new backend plugin with an LLM, follow these steps in order:

  1. Read existing plugins: Study plugins/slurm/ and plugins/mup/ for patterns.
  2. Copy the template: Start from docs/plugin-template/ and rename.
  3. Implement __init__: Call super().__init__(), set backend_type, create client.
  4. Implement BaseClient methods: Start with get_resource, create_resource, delete_resource, list_resources.
  5. Implement BaseBackend abstract methods: Start with ping, then _pre_create_resource, then _collect_resource_limits, then _get_usage_report.
  6. Handle unit conversion: Verify unit_factor math in both directions.
  7. Write tests: Mock the client, test each abstract method.
  8. Register entry points: Add to pyproject.toml.
  9. Test integration: Install with uv sync --all-packages and run waldur_site_diagnostics.
  10. Verify: Run uv run pytest and uvx prek run --all-files.

Files to study

  • waldur_site_agent/backend/backends.py — BaseBackend (resource backends) and AbstractUsernameManagementBackend (username plugins), plus PendingOrderDecision enum.
  • waldur_site_agent/backend/clients.py — Base client class.
  • waldur_site_agent/backend/structures.py — Data structures (ClientResource, Association, BackendResourceInfo with fields backend_id, parent_id, effective_id, users, usage, limits, pending_order_id, backend_metadata, endpoints — a list of {"name": …, "url": …} access endpoints pushed to Waldur).
  • waldur_site_agent/backend/exceptions.py — BackendError, DuplicateResourceError, and the OfferingUser*RequiredError exceptions used by username backends.
  • waldur_site_agent/common/plugin_schemas.py — PluginComponentSchema and PluginBackendSettingsSchema base classes for optional config validation entry points.
  • plugins/slurm/waldur_site_agent_slurm/backend.py — Reference implementation (CLI-based).
  • plugins/mup/waldur_site_agent_mup/backend.py — Reference implementation (API-based).
  • plugins/waldur/waldur_site_agent_waldur/backend.py — Reference for supports_async_orders, handled_resource_states, and the sync_resource_* hooks.
  • plugins/basic_username_management/ — Minimal reference for AbstractUsernameManagementBackend.

Common mistakes to avoid

  • Do not forget super().__init__(backend_settings, backend_components).
  • Do not return raw dicts from list_resources; return ClientResource objects.
  • Do not raise exceptions from get_resource when resource is absent; return None.
  • Do not forget the "TOTAL_ACCOUNT_USAGE" key in usage reports.
  • Do not confuse Waldur units with backend units in _collect_resource_limits.
  • Do not hardcode component keys; read them from self.backend_components.
  • Do not override only create_resource: core calls create_resource_with_id. If your system assigns ids, override recreate_missing_resource too.
  • Do not raise BackendError for a failed Waldur API call; it marks a healthy resource ERRED.