Skip to content

Waldur Site Agent - K8s UT Namespace Plugin

This plugin enables integration between Waldur Site Agent and Kubernetes clusters for managing ManagedNamespace custom resources (CRD: provisioning.hpc.ut.ee/v1) with optional Keycloak RBAC group integration.

Features

  • ManagedNamespace Lifecycle: Creates, updates, and deletes ManagedNamespace custom resources
  • Resource Quotas: Sets CPU, memory, storage, and GPU limits as namespace quotas
  • Role-Based Access Control: Creates 3 Keycloak groups per namespace (admin, readwrite, readonly)
  • Waldur Role Mapping: Maps Waldur roles to namespace access levels automatically
  • User Management: Adds/removes users from Keycloak groups, reconciles role changes
  • Usage Reporting: Reports actual resource consumption from K8s ResourceQuota or quota allocations
  • Namespace Labels & Annotations: Configurable labels and annotations propagated to created namespaces
  • Status Monitoring: Parses operator Ready condition and exposes readiness in Waldur metadata
  • Configurable User Identity: Choose which user attribute (email, civil_number, etc.) populates CR user fields
  • Namespace Name Validation: Validates generated names against RFC 1123 before CR creation
  • Status Operations: Supports downscale (minimal quota), pause (zero quota), and restore

Architecture

The plugin follows the Waldur Site Agent plugin architecture and consists of:

  • K8sUtNamespaceBackend: Main backend implementation that orchestrates namespace and user management
  • K8sUtNamespaceClient: Handles Kubernetes API operations for ManagedNamespace CRs
  • KeycloakClient: Manages Keycloak groups and user memberships (shared package)

Role Mapping

Waldur roles are mapped to namespace access levels. The default mapping is:

Waldur Role Namespace Role
manager admin
admin admin
member readwrite

This mapping is configurable via the role_mapping setting in backend_settings. Custom entries are merged with the defaults, so you only need to specify overrides or additions:

1
2
3
4
backend_settings:
  role_mapping:
    observer: "readonly"
    member: "readonly"  # override the default

Users whose Waldur role is not in the mapping fall back to default_role (default: readwrite).

Component Mapping

Waldur component keys are mapped to Kubernetes quota fields. The default mapping is:

Waldur Component K8s Quota Field Unit Format
cpu cpu Integer
ram memory {value}Gi
storage storage {value}Gi
gpu gpu Integer

This mapping is configurable via the component_quota_mapping setting in backend_settings. Custom entries are merged with the defaults:

1
2
3
backend_settings:
  component_quota_mapping:
    vram: "nvidia.com/vram"

Installation

Install the plugin using uv:

1
uv sync --all-packages

The plugin will be automatically discovered via Python entry points.

Setup Requirements

Kubernetes Cluster Setup

  1. Kubernetes Cluster: Accessible cluster with the ManagedNamespace CRD installed (provisioning.hpc.ut.ee/v1)
  2. Access Method: Either a kubeconfig file or in-cluster service account
  3. CR Namespace: A namespace where ManagedNamespace CRs will be created (default: waldur-system)

Keycloak Setup (Optional)

Required for RBAC group integration:

  1. Keycloak Server: Accessible Keycloak instance
  2. Target Realm: Where user accounts and groups will be managed
  3. Service User: User with group management permissions

Creating Keycloak Service User

  1. Login to Keycloak Admin Console
  2. Select Target Realm
  3. Create User:
  4. Username: waldur-site-agent-k8s
  5. Email Verified: Yes
  6. Enabled: Yes
  7. Set Password: In Credentials tab (temporary: No)
  8. Assign Roles: In Role Mappings tab
  9. Client Roles -> realm-management
  10. Add: manage-users (sufficient for group operations)

Waldur Marketplace Setup

  1. Marketplace Offering: Created with appropriate type (e.g., Marketplace.Basic)
  2. Components: Configured via waldur_site_load_components
  3. Offering State: Must be Active for order processing

Configuration

Minimal Configuration (K8s Only)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
offerings:
  - name: "k8s-namespaces"
    waldur_api_url: "https://your-waldur.com/"
    waldur_api_token: "your-waldur-api-token"
    waldur_offering_uuid: "your-offering-uuid"

    backend_type: "k8s-ut-namespace"
    order_processing_backend: "k8s-ut-namespace"
    membership_sync_backend: "k8s-ut-namespace"
    reporting_backend: "k8s-ut-namespace"

    backend_settings:
      kubeconfig_path: "/path/to/kubeconfig"
      cr_namespace: "waldur-system"
      namespace_prefix: "waldur-"
      keycloak_enabled: false
      sync_users_to_cr: true
      cr_user_identity_field: "email"
      namespace_labels:
        tenant: "waldur"
      namespace_annotations:
        description: "Managed by Waldur"

    backend_components:
      cpu:
        type: "cpu"
        measured_unit: "cores"
        accounting_type: "limit"
        label: "CPU Cores"
        unit_factor: 1
      ram:
        type: "ram"
        measured_unit: "GB"
        accounting_type: "limit"
        label: "Memory (GB)"
        unit_factor: 1
      storage:
        type: "storage"
        measured_unit: "GB"
        accounting_type: "limit"
        label: "Storage (GB)"
        unit_factor: 1

Full Configuration (with Keycloak)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
offerings:
  - name: "k8s-namespaces"
    waldur_api_url: "https://your-waldur.com/"
    waldur_api_token: "your-waldur-api-token"
    waldur_offering_uuid: "your-offering-uuid"

    backend_type: "k8s-ut-namespace"
    order_processing_backend: "k8s-ut-namespace"
    membership_sync_backend: "k8s-ut-namespace"
    reporting_backend: "k8s-ut-namespace"

    backend_settings:
      kubeconfig_path: "/path/to/kubeconfig"
      cr_namespace: "waldur-system"
      namespace_prefix: "waldur-"
      default_role: "readwrite"

      keycloak_enabled: true
      keycloak_use_user_id: true
      keycloak:
        keycloak_url: "https://your-keycloak.com/"
        keycloak_realm: "your-realm"
        keycloak_user_realm: "your-realm"
        keycloak_username: "waldur-site-agent-k8s"
        keycloak_password: "your-keycloak-password"
        keycloak_ssl_verify: true

    backend_components:
      cpu:
        type: "cpu"
        measured_unit: "cores"
        accounting_type: "limit"
        label: "CPU Cores"
        unit_factor: 1
      ram:
        type: "ram"
        measured_unit: "GB"
        accounting_type: "limit"
        label: "Memory (GB)"
        unit_factor: 1
      storage:
        type: "storage"
        measured_unit: "GB"
        accounting_type: "limit"
        label: "Storage (GB)"
        unit_factor: 1
      gpu:
        type: "gpu"
        measured_unit: "units"
        accounting_type: "limit"
        label: "GPU"
        unit_factor: 1

Configuration Reference

Backend Settings

Parameter Type Required Default Description
kubeconfig_path string No - Path to kubeconfig file (omit for in-cluster config)
cr_namespace string No waldur-system Namespace where ManagedNamespace CRs are created
namespace_prefix string No waldur- Prefix for created namespace names
default_role string No readwrite Default namespace role for users without explicit role
role_mapping object No See Role Mapping Custom Waldur role to namespace role mapping (merged with defaults)
component_quota_mapping object No See Component Mapping Custom component to K8s quota field mapping
keycloak_use_user_id boolean No true Use Keycloak user ID for lookup (false = use username)
sync_users_to_cr boolean No false Sync user identities to CR adminUsers/rwUsers/roUsers fields
cr_user_identity_field string No email User attribute for CR user fields
cr_user_identity_lowercase bool No false Lowercase the identity value before writing to CR
namespace_labels object No {} Labels to set on created namespaces (e.g., tenant: waldur)
namespace_annotations object No {} Annotations to set on created namespaces

Keycloak Settings (Optional)

Parameter Type Required Default Description
keycloak_enabled boolean No false Enable Keycloak RBAC integration
keycloak.keycloak_url string Conditional - Keycloak server URL
keycloak.keycloak_realm string Conditional - Keycloak realm name
keycloak.keycloak_user_realm string Conditional - Keycloak user realm for auth
keycloak.keycloak_username string Conditional - Keycloak admin username
keycloak.keycloak_password string Conditional - Keycloak admin password
keycloak.keycloak_ssl_verify boolean No true Whether to verify SSL certificates

Usage

Running the Agent

Start the agent with your configuration file:

1
uv run waldur_site_agent -c k8s-namespace-config.yaml -m order_process

Diagnostics

Run diagnostics to check connectivity:

1
uv run waldur_site_diagnostics -c k8s-namespace-config.yaml

Supported Agent Modes

  • order_process: Creates and manages ManagedNamespace CRs based on Waldur resource orders
  • membership_sync: Synchronizes user memberships between Waldur and Keycloak groups
  • report: Reports namespace quota allocations to Waldur

Resource Lifecycle

Namespace Creation

When a Waldur resource order is processed:

  1. Resource slug is validated (required for naming)
  2. Three Keycloak groups are created: ns_{slug}_admin, ns_{slug}_readwrite, ns_{slug}_readonly
  3. A ManagedNamespace CR is created with quota and group references in the spec
  4. The namespace name is {namespace_prefix}{slug} (e.g., waldur-my-project)
  5. If CR creation fails, Keycloak groups are cleaned up (compensating transaction)

Namespace Deletion

When a Waldur resource termination order is processed:

  1. The ManagedNamespace CR is deleted
  2. All 3 Keycloak groups are deleted

Limit Updates

When resource limits are updated in Waldur:

  1. Limits are converted to K8s resource quantities
  2. The CR's spec.quota is patched with the new values

User Management

When users are added to a Waldur resource:

  1. Each user's Waldur role is mapped to a namespace role (admin/readwrite/readonly)
  2. User is looked up in Keycloak
  3. User is removed from any incorrect role groups (role reconciliation)
  4. User is added to the correct role group

Direct CR User Sync

When sync_users_to_cr is enabled, user identities from Waldur are written directly to the ManagedNamespace CR's adminUsers, rwUsers, and roUsers fields. The managed-namespace-operator then creates RoleBindings with these identities as User subjects (optionally prefixed with CONTROLLER_USER_PREFIX on the operator side).

The cr_user_identity_field setting controls which user attribute is used as the identity value. The default is email, but any attribute exposed by the offering's user attribute config can be used (e.g., civil_number, username).

Each user's Waldur role is mapped to a namespace role using the same role_mapping configuration (see Role Mapping), and the identity value is placed in the corresponding CR field:

Namespace Role CR Field
admin adminUsers
readwrite rwUsers
readonly roUsers

On each membership sync cycle, the full current set of team members from Waldur is written to the CR. Users removed from the Waldur project team are automatically removed from the CR on the next sync, because empty lists are sent for roles with no members.

This can be used alongside Keycloak groups (both mechanisms populate the same RoleBindings) or without Keycloak (keycloak_enabled: false) for deployments that rely solely on OIDC-based authentication.

1
2
3
4
5
backend_settings:
  sync_users_to_cr: true
  cr_user_identity_field: "civil_number"  # or "email", "username", etc.
  cr_user_identity_lowercase: true        # optional, lowercase the value
  keycloak_enabled: false  # optional, can also be true for dual mode

The chosen field must be enabled in the offering's user attribute config (expose_civil_number: true) in Waldur. Users missing the configured attribute are skipped with a warning log.

When cr_user_identity_lowercase is enabled, the identity value is lowercased before writing to the CR (e.g., EE12345678901 becomes ee12345678901). This is useful when OIDC subject matching is case-sensitive and the identity source has mixed case.

When users are removed:

  1. User is removed from all 3 Keycloak groups

Usage Reporting

The plugin reports actual resource consumption by reading ResourceQuota.status.used from the managed namespace. The K8s service account needs get permission on resourcequotas in the target namespaces for this to work. If the ResourceQuota is not accessible, usage is reported as zeros.

Usage values are converted back to Waldur component units using the reverse of the component quota mapping (e.g., K8s limits.memory: 4Gi → Waldur ram: 4).

Namespace Labels & Annotations

Labels and annotations configured in backend_settings are included in the ManagedNamespace CR spec. The operator propagates them to the actual namespace. This is useful for cluster policies (e.g., Kyverno requiring a tenant label):

1
2
3
4
5
6
backend_settings:
  namespace_labels:
    tenant: "waldur"
    cost-center: "HPC-001"
  namespace_annotations:
    description: "Managed by Waldur Site Agent"

Status Operations

Operation Effect
Downscale Quota set to minimal: cpu=1, memory=1Gi, storage=1Gi
Pause Quota set to zero: cpu=0, memory=0Gi, storage=0Gi
Restore No-op (limits should be re-set via a separate update order)

Error Handling

  • Kubernetes connectivity issues are logged and raised as BackendError
  • Keycloak initialization failure logs a warning; user management operations become no-ops
  • CR creation failure triggers automatic Keycloak group cleanup
  • Missing users in Keycloak are logged as warnings and skipped
  • Missing backend ID on deletion is logged and skipped gracefully

Development

Running Tests

1
.venv/bin/python -m pytest plugins/k8s-ut-namespace/tests/

Code Quality

1
pre-commit run --all-files