Skip to content

Upgrading the SLURM Plugin

This page covers SLURM-specific considerations when upgrading waldur-site-agent-slurm. Read the general upgrade guide first.

Required backend_settings keys

The SLURM backend reads the following keys from backend_settings. Required keys must be present or the agent will fail to start.

Key Required Notes
default_account Yes DefaultAccount= set on user associations; must exist in the cluster
default_account_policy No common (default), individual, or none — see below
root_account No Parent of the top-tier customer account. Defaults to default_account, then root
customer_prefix Yes Prefix for customer-level SLURM accounts
project_prefix Yes Prefix for project-level SLURM accounts
allocation_prefix Yes Prefix for allocation accounts
cluster_name No Must match the offering's backend_id in Waldur; required in multi-cluster setups
slurm_bin_path No Default /usr/bin
parent_account No Set for flat hierarchies (no customer tier); omit for nested hierarchy
default_partition No Fallback SLURM partition
enforce_offering_partitions No Default false
enable_user_homedir_account_creation No Default true
default_homedir_umask No Default 0077

Check the CHANGELOG for any new required keys before upgrading.

default_account_policy

Controls which account is passed as DefaultAccount= when the agent creates a user→account association (sacctmgr add user …). The default account is where a user's jobs charge when they submit without an explicit -A/--account.

  • common (default) — DefaultAccount=<default_account> on every association. Stable: always references the configured, backend-verified account.
  • individual — DefaultAccount=<resource_id> (the per-resource account). Keeps users off the org-wide root by default, but when that resource is terminated and its account deleted, the user's DefaultAccount dangles and SLURM rejects their job submissions until an operator repairs it.
  • none — DefaultAccount= omitted entirely; sacctmgr auto-assigns for new users. Relies on the deployment's sacctmgr auto-assignment for brand-new users; for an existing user whose prior default account was deleted, the stale default is left unchanged (they may be unable to submit until repaired).

common is the safe default and is what most deployments should use. Only switch to individual or none if you understand the dangling-DefaultAccount failure modes above and have an operational process to handle them. An invalid value (e.g. a typo) raises an error at agent startup rather than silently falling back to common.

QoS configuration

QoS state is driven by the paused and downscaled flags set by Waldur Mastermind (via policy or manual action). The agent maps these flags to SLURM QoS names:

1
2
3
4
backend_settings:
  qos_default: "normal"      # Applied when resource is active
  qos_downscaled: "low"      # Applied when Mastermind sets downscaled=true
  qos_paused: "pause"        # Applied when Mastermind sets paused=true

Optional per-account QoS creation during resource provisioning is available via qos_management:

1
2
3
4
5
6
7
backend_settings:
  qos_management:
    enabled: true
    # Opt-in; both default false so existing enabled-only configs are unchanged.
    # skip_qos_swap: true         # pause/downscale/restore use GrpSubmitJobs, never set qos=
    # apply_limits_to_qos: true   # GrpTRESMins on the QoS (requires enabled + skip_qos_swap)
    # ...other QoS management keys

skip_qos_swap cannot be combined with qos_default / qos_paused / qos_downscaled — SlurmBackend construction raises BackendError (plugin schema validation alone is soft-fail and only logs a warning). Drop those keys on the offering that uses skip_qos_swap.

Trade-off: with skip_qos_swap, Waldur paused / downscaled no longer change the account QoS list. Instead the agent sets GrpSubmitJobs=0 (block new submissions) and clears it with GrpSubmitJobs=-1 on restore — the same orthogonal lever used by QoS enforcement. Budget exhaustion is still enforced by DenyOnLimit on the dedicated QoS when apply_limits_to_qos is also on. This is compatible with QoS enforcement: enabling both does not silently disable pause.

apply_limits_to_qos prerequisites:

  • Requires enabled and skip_qos_swap. Misconfiguration (apply_limits_to_qos without enabled) is rejected at backend construction — otherwise the agent would write GrpTRESMins to a QoS that was never created and SLURM would silently leave the allocation uncapped.
  • The agent creates a QoS with the same name as the allocation account and writes GrpTRESMins there. Do not enable this on an offering where a QoS with that name already exists and is shared across accounts — creation is skipped when the name exists; GrpTRESMins is still written, but RawUsage is not reset on collision (only on a freshly created QoS).
  • When backend_components use target_components (one Waldur component mapped to several SLURM TRES, e.g. node_hours → billing + gres/gpu), pushing limits from Waldur to SLURM works. Pulling limits from SLURM back into Waldur during periodic sync does not yet reverse-map multi-target caps — the sync step no-ops and Waldur keeps the limits from the order. Usage reporting (sacct) is unaffected; only the limit echo is incomplete until a single reverse source is configured (follow-up).

Accounts with a DefaultQOS

slurmdbd requires every association's effective DefaultQOS to be in its effective QoS list and rolls back any sacctmgr modify that would break that (These associations don't have access to their default qos). The pause/downscale/restore swap reads the account's current default and, when it is set and not in the new list, writes qos=<x> defaultqos=<x> in one command. Accounts without a DefaultQOS are handled exactly as before (qos=<x> only); no configuration change is needed.

The agent only writes the account-level default. A user association that carries its own explicit DefaultQOS outside the new list still makes slurmdbd reject the swap — the error lists the offending U = <user> rows. Either clear those user-level defaults so they inherit the account's, or move them by hand:

1
sacctmgr modify user where account=<acct> cluster=<cluster> set defaultqos=<qos_paused>

Account hierarchy and sync_resource_project

When a project is moved to a different customer in Waldur, the SLURM account's parent must be updated to reflect the new customer account. The agent handles this via sync_resource_project, called at:

  • Polling mode: every order_process or membership_sync cycle.
  • Event-process (STOMP) mode: on incoming RESOURCE events, and periodically every reconciliation interval. Mastermind also pushes a RESOURCE event immediately when a project moves, so the hierarchy is corrected without waiting for the next cycle.

sync_resource_project is skipped when parent_account is set (flat hierarchy).

Validating after upgrade

Run diagnostics

waldur_site_diagnostics calls SlurmBackend.diagnostics(), which prints the configured prefixes, default_account, and SLURM version, and returns an error if sinfo is unreachable:

1
waldur_site_diagnostics -c /etc/waldur/waldur-site-agent-config.yaml 2>&1 | grep -E "SLURM|slurm|ERROR"

A healthy output looks like:

1
2
3
4
5
SLURM customer prefix          = hpc_c_
SLURM project prefix           = hpc_p_
SLURM allocation prefix        = hpc_a_
SLURM default account          = root
Slurm version: slurm 23.11.4

Verify account hierarchy for recently moved projects (STOMP mode)

If any project was moved between customers while the agent was not running or was on an older version, the SLURM account parent may be stale. Trigger a sync by restarting the agent or waiting for one reconciliation interval.

To check a specific account's parent directly:

1
sacctmgr show account <allocation_account> format=Account,ParentName -P

Confirm QoS names exist in SLURM

If you use qos_downscaled or qos_paused, verify the referenced QoS objects exist in the cluster:

1
sacctmgr show qos format=Name -P

A QoS name in backend_settings that does not exist in SLURM will cause the downscale or pause action to fail with a BackendError.

Filesystem quotas

If you use homedir_quota or project_directory (Lustre/CephFS/XFS quotas), no migration steps are required — quota configuration is read fresh on each resource creation or user add. After upgrading, run diagnostics and create a test resource to confirm quota-setting commands succeed.

See SLURM Storage Quotas for full configuration reference.