waldur-site-agent-rancher-kc-crd

CRD-driven Rancher + Keycloak membership-sync plugin for Waldur Site Agent. Translates Waldur Resource + ResourceProject + UserRole state into ManagedRancherProject Custom Resources; the rancher-keycloak-operator running inside the target Kubernetes cluster owns the actual Rancher project + Keycloak group lifecycle.

At a glance

Entry point	Group	Role
rancher-kc-crd	waldur_site_agent.backends	membership sync only

Modes: membership_sync, event_process.

Operation	Behaviour
Create / terminate resource	Not handled — clusters pre-exist; use another backend for orders
Add / remove members	Writes one ManagedRancherProject CR per ResourceProject; the operator binds Keycloak groups
Pause / downscale / restore	No-op — returns True without changing anything
Usage reporting	No-op — reports nothing

Expected operator

This plugin only writes CRs — it relies on waldur/rancher-keycloak-operator to be running in the target Kubernetes cluster. That operator owns the ManagedRancherProject and RancherProjectInventory CRD definitions and is the only thing that talks to Rancher and Keycloak APIs.

Minimum operator version compatible with the current plugin: 0.3.0. Recommended: 0.3.1+ — earlier 0.3.x bug-fixes a metadata-sync gap where description / organization / projectSlug changes never reached Rancher after the initial create. The plugin no longer emits spec.namespace, which 0.2.x operators relied on for namespace creation; earlier versions also miss audit fields and the stale-project-ID cleanup fallback that the plugin assumes.

Attribute lookup (keycloak_user_lookup: attribute, see Matching users across Keycloaks by civil code) needs operator 0.5.0+: older CRDs reject the lookupAttribute member field, so the plugin emits it only when attribute lookup is configured.

The operator's helm chart lives in its own repo under helm/rancher-keycloak-operator/; install instructions are in the Setup section below.

Scope

membership_sync_backend only. This plugin handles user-to-project role bindings. It does not provision or terminate Rancher clusters, does not process orders, and does not report usage. Order processing and reporting are out of scope; if the offering needs them they have to be wired through other backends.

Cluster pre-exists. Each Waldur Resource's backend_id is its Rancher cluster ID; this plugin does not stand up Rancher clusters.

Opt-in per offering. Set membership_sync_backend: rancher-kc-crd on the offering's site-agent config to route its membership sync through this plugin.

Membership is per-ResourceProject, not a flat per-Resource team. Unlike SLURM-style backends, there's no single "team" for the whole Resource — each ResourceProject has its own UserRole list, and that's what drives Keycloak group membership (see User identity matching for exactly which field). The site agent's generic team-vs-backend membership diff (used by flat-team backends to detect new/stale users) doesn't apply here and is skipped for this backend — see skip_resource_team_diff in waldur_site_agent_rancher_kc_crd/backend.py.

---

Architecture

flowchart LR
    subgraph Waldur ["Waldur Mastermind"]
        WR[Resource]
        WRP[ResourceProject]
        WUR[UserRole]
        WR --- WRP --- WUR
    end

    subgraph Agent ["Site Agent / this plugin"]
        PR[pull_resource]
        TR[translator.build_cr_spec]
        CC[CrdClient.apply / list / delete]
        SR[status_reader]
    end

    subgraph K8s ["Kubernetes cluster"]
        CR[(ManagedRancherProject CR)]
        OP[rancher-keycloak-operator]
        CR -. watched by .-> OP
    end

    subgraph External ["External APIs"]
        RA[Rancher Mgmt API]
        KC[Keycloak Admin API]
    end

    Waldur -- "SDK GET<br/>(provider-resource-projects,<br/>list_users)" --> PR
    PR --> TR --> CC -- "kube apply" --> CR
    CR --> SR
    SR -- "status.* + drift" --> Waldur

    OP -- "v3 REST" --> RA
    OP -- "Admin REST" --> KC

The plugin's responsibilities stop at writing CRs and reading their status. The operator owns:

• creating / adopting Rancher projects, namespaces, and resource quotas
• creating Keycloak parent + child groups and binding them to the Rancher project via ProjectRoleTemplateBinding (PRTB) with groupPrincipalId: keycloakoidc_group://<group-name>
• adding/removing users to/from Keycloak groups based on spec.keycloak.roleBindings[].members[]
• cascading cleanup on CR delete (@kopf.on.delete)

---

Sequence: end-to-end membership sync

This is what happens on a single pull_resource call (i.e. one membership-sync cycle per Resource the offering owns).

sequenceDiagram
    autonumber
    participant SA as Site Agent
    participant PG as plugin
    participant W as Waldur API
    participant K as Kubernetes API
    participant OP as Operator
    participant R as Rancher
    participant KC as Keycloak

    SA->>PG: pull_resource(WaldurResource)
    PG->>W: GET /api/marketplace-provider-resource-projects/?resource_uuid=…
    W-->>PG: [RP1, RP2, …]
    loop per ResourceProject
        PG->>W: GET /api/marketplace-provider-resource-projects/<uuid>/list_users/
        W-->>PG: [UserRole, …]
        PG->>PG: build_cr_spec(resource, RP, users)
        PG->>K: kube apply ManagedRancherProject (server-side)
        K-->>OP: watch event (create/update)
        OP->>R: ensure project (create or adopt by name)
        OP->>R: ensure namespace + ResourceQuota
        OP->>KC: ensure parent group, child groups
        OP->>R: ensure PRTB(s) bound to KC groups
        OP->>KC: GET /users/{id} or /users?username=X<br/>(per member)
        OP->>KC: add/remove user from KC group
        OP->>K: patch status.conditions + status.keycloakRoleBindings.syncedMembers
        PG->>K: kube get ManagedRancherProject
        K-->>PG: status.* (synced members per binding)
    end
    PG->>K: list ManagedRancherProject by label waldur.io/resource-uuid
    K-->>PG: [all CRs for this resource]
    PG->>K: kube delete (CRs whose RP no longer in Waldur)
    K-->>OP: watch event (delete)
    OP->>R: delete PRTBs + project (with stale-ID find-by-name fallback)
    OP->>KC: delete child groups, parent group if empty
    PG-->>SA: BackendResourceInfo(users=union of synced members)

Key invariants:

• Idempotent at every step. Re-applying the same CR yields the same Rancher project and Keycloak groups. Re-syncing with the same user set is a no-op on the Rancher/Keycloak side.
• One CR per ResourceProject, named <resource.slug>-<rp.uuid[:8]>. Stable across renames.
• Orphan pruning by label. CRs are stamped with metadata.labels.waldur.io/resource-uuid; on each sync the plugin computes the set of expected CR names from the current Waldur RP list, then deletes any label-matching CR outside that set.
• Lookup-only for users. The operator never creates Keycloak users; it only binds existing ones. Users absent from Keycloak get a WARNING User <id> not found in Keycloak and are skipped (the PRTB and the group are still created — they're just empty for that user).

---

Sequence: orphan-CR pruning

sequenceDiagram
    autonumber
    participant SA as Site Agent
    participant W as Waldur
    participant K as Kubernetes
    participant OP as Operator
    participant R as Rancher
    participant KC as Keycloak

    Note over SA: pull_resource loop completes — 2 CRs applied for RP1, RP2
    SA->>W: GET resource-projects?resource_uuid=…
    W-->>SA: [RP1] (RP2 was deleted in Waldur)
    SA->>K: list mrp -l waldur.io/resource-uuid=…
    K-->>SA: [CR_RP1, CR_RP2]
    Note over SA: expected={CR_RP1}<br>found={CR_RP1, CR_RP2}<br>orphans={CR_RP2}
    SA->>K: kube delete CR_RP2
    K-->>OP: watch event (delete + finalizer hold)
    OP->>R: delete PRTBs for RP2's project
    OP->>KC: remove members from KC groups
    OP->>KC: delete KC child groups
    OP->>KC: delete KC parent group (if empty)
    OP->>R: delete Rancher project
    alt stored project ID is stale (e.g. project recreated externally with new ID)
        OP->>R: DELETE /v3/projects/<stored_id> -> 404
        OP->>R: GET /v3/projects?clusterId=…&name=<projectName>
        R-->>OP: {id: <current_id>}
        OP->>R: DELETE /v3/projects/<current_id> -> 200
    end
    OP->>K: remove finalizer (CR fully gone)

Why pruning lives in the plugin, not the operator: the operator doesn't know about Waldur — the Waldur RP list is the source of truth, and only the plugin sees both sides. The label selector keeps pruning safe: CRs without the label (manually-created, or from a different source) are never touched.

---

Sequence: cleanup on stale rancherProjectId

When a Rancher project is externally deleted and recreated between two operator reconciles, the CR's status.rancherProjectId points at a non-existent ID. The cleanup falls back to finding the live project by clusterId+projectName:

sequenceDiagram
    autonumber
    participant U as User / Test
    participant K as Kubernetes
    participant OP as Operator
    participant R as Rancher

    Note over U: status.rancherProjectId = p-OLD<br>(actual Rancher project: p-NEW, same name)
    U->>K: kubectl delete mrp <name>
    K-->>OP: on_delete handler fires
    OP->>R: DELETE /v3/projects/p-OLD
    R-->>OP: 404 Not Found
    Note over OP: delete_project returned False<br>-> stored ID is stale
    OP->>R: GET /v3/projects?clusterId=…&name=<projectName>
    R-->>OP: [{id: p-NEW, …}]
    OP->>OP: log WARNING — stored projectId p-OLD was stale, deleting current p-NEW found by name
    OP->>R: DELETE /v3/projects/p-NEW
    R-->>OP: 200 OK
    OP->>K: cleanup complete — release finalizer

This was the failure mode that left orphan Rancher projects after external recreation; fix landed in operator 0.2.2.

---

Setup

1. Operator: install in target cluster

The waldur/rancher-keycloak-operator must be running in the cluster you point this plugin at. One operator instance handles all ManagedRancherProject CRs in its namespace; one operator can manage Rancher projects across multiple downstream Rancher clusters (each CR specifies its own clusterId).

1a. CRDs

# Clone the operator repo (separate from waldur-site-agent):
git clone https://github.com/waldur/rancher-keycloak-operator.git
cd rancher-keycloak-operator

kubectl apply -f helm/rancher-keycloak-operator/templates/crds/
kubectl get crds | grep waldur.io
# expect:
#   managedrancherprojects.waldur.io
#   rancherprojectinventories.waldur.io

1b. Helm install (published image)

The published image is opennode/rancher-keycloak-operator:<version> on Docker Hub. Pin a specific version (don't use :latest):

kubectl create namespace waldur-system

helm upgrade --install rko \
  ./helm/rancher-keycloak-operator \
  --namespace waldur-system \
  --set image.repository=opennode/rancher-keycloak-operator \
  --set image.tag=0.3.1 \
  --set image.pullPolicy=IfNotPresent \
  --set "config.rancher.url=https://rancher.example.com" \
  --set "config.rancher.bearerToken=<rancher-bearer-token>" \
  --set "config.rancher.verifySsl=true" \
  --set "config.keycloak.url=https://keycloak.example.com" \
  --set "config.keycloak.realm=<realm>" \
  --set "config.keycloak.userRealm=master" \
  --set "config.keycloak.username=<kc-admin-user>" \
  --set "config.keycloak.password=<kc-admin-password>" \
  --set "config.keycloak.verifySsl=true"

kubectl rollout status deploy/rko-rancher-keycloak-operator -n waldur-system

1c. Smoke-test the operator (optional but recommended)

The operator repo ships a Tier-1 runbook at docs/tier-1-runbook.md that walks through applying a hand-crafted CR end-to-end against the configured Rancher and Keycloak before wiring in the site-agent. Run it once per cluster to catch credential / connectivity issues early.

1d. Required Rancher + Keycloak permissions

System	Role / scope
Rancher token	unscoped admin OR cluster-owner across all clusters this operator instance will manage
Keycloak admin user	realm-admin on the target realm (group create/delete, group member add/remove, user lookup)

---

2. Plugin: install on the site-agent host

The plugin is a workspace member of the waldur-site-agent repo and is installed automatically when you run uv sync --all-packages at the repo root. To verify it's discovered:

uv run python -c "from waldur_site_agent_rancher_kc_crd import backend; print(backend.RancherKcCrdBackend)"

2a. Site-agent host needs

• Network access to: Waldur Mastermind API, the Kubernetes API of the cluster running the operator.
• A kubeconfig file (or in-cluster service-account credentials if you run the agent inside the operator's cluster).
• The Kubernetes API user must be allowed to get/list/create/update/delete/patch managedrancherprojects.waldur.io in the chosen namespace.

2b. Configure the offering

Add a stanza to waldur-site-agent-config.yaml. Full reference at examples/rancher-kc-crd-config.yaml.

offerings:
  - name: "my-rancher-offering"

    waldur_api_url: "https://waldur.example.com/api/"
    waldur_api_token: "${WALDUR_API_TOKEN}"
    waldur_offering_uuid: "<offering-uuid>"

    backend_type: "rancher-kc-crd"
    membership_sync_backend: "rancher-kc-crd"

    backend_settings:
      # Repeats the top-level waldur_api_url/waldur_api_token on purpose.
      # This plugin fetches ResourceProjects and UserRoles directly from
      # Waldur inside pull_resource, but backends aren't handed the
      # framework's own Waldur client -- that's deliberately scoped to
      # the processor, not backends in general. So the plugin builds its
      # own AuthenticatedClient from these settings instead. Most other
      # backends don't need this because they don't talk to the Waldur
      # API themselves.
      waldur_api_url: "https://waldur.example.com/api/"
      waldur_api_token: "${WALDUR_API_TOKEN}"
      waldur_verify_ssl: true

      # Where the operator listens.
      kubeconfig_path: "~/.kube/config"   # or omit for in-cluster
      context: "my-cluster"
      namespace: "waldur-system"

      # Cluster ID comes from each Resource's `backend_id` (1:1 with
      # a Rancher downstream cluster). There is no offering-level
      # cluster_id setting -- if a resource's backend_id is empty,
      # CR build raises a clear error rather than emitting an
      # invalid CR.

      # Keycloak group naming -- see note A below for the full variable
      # list, including ${customer_slug}, ${project_slug}, ${resource_slug},
      # and ${rp_uuid_short}. The default keeps the names compact and
      # immutable; override `group_name_template` for human-readable
      # group names.
      parent_group_name: "c_${cluster_id}"
      group_name_template: "c_${cluster_id}_${rp_uuid}_${role_name}"  # default

      # Map Waldur offering role names -> Rancher role template IDs.
      # Roles absent from this map are skipped (operator can't bind them).
      role_map:
        project_member: "project-member"
        project_admin: "project-owner"
        create_ns: "create-ns"

      # User-identity link to Keycloak. See "User identity matching" below.
      keycloak_user_identity_source: username  # default
      keycloak_user_lookup: username           # default

2c. Run

rancher-kc-crd is a membership_sync_backend only. Order processing and reporting still need other backends (or none).

uv run waldur_site_agent --mode membership_sync \
    --config-file waldur-site-agent-config.yaml

Observability:

# Watch CRs being created/updated/deleted
kubectl get mrp -n waldur-system -L waldur.io/resource-uuid -w

# Inspect any CR's reconciliation state
kubectl describe mrp <name> -n waldur-system

# Operator logs (cleanup, member sync, drift)
kubectl logs deploy/rko-rancher-keycloak-operator -n waldur-system --tail=100 -f

---

Importing access from existing clusters

The sync is one-way: Waldur is the source of truth, and the agent never reads the bindings a cluster already has. Pointed at a cluster that is already in use, it adopts each Rancher project whose name matches a ResourceProject and grants only what Waldur knows. Existing project and cluster role bindings keep working, because the operator only touches bindings for its own groups, but Waldur doesn't show them. Import them before the agent's first sync with waldur-rancher-kc-crd-import.

This assumes the offering's Resources already exist in Waldur with backend_id set to the Rancher cluster ID. The command reads the agent config for the offering, role_map, cluster_role_map and the identity settings, and takes the Rancher and Keycloak credentials from the environment variables the operator uses:

export RANCHER_URL=https://rancher.example.com RANCHER_BEARER_TOKEN=token-...
export KEYCLOAK_URL=https://keycloak.example.com KEYCLOAK_REALM=rancher \
       KEYCLOAK_USERNAME=... KEYCLOAK_PASSWORD=...   # also KEYCLOAK_USER_REALM, *_VERIFY_SSL
export WALDUR_API_TOKEN=...   # a staff token; see below

# 1. Read every cluster of the offering: projects, PRTBs, CRTBs, and the Keycloak
#    users behind them (group principals expand to the group's direct members).
uv run waldur-rancher-kc-crd-import inventory -c waldur-site-agent-config.yaml \
    -o inventory.json

# 2. Review the plan (dry run), then apply it.
uv run waldur-rancher-kc-crd-import apply -c waldur-site-agent-config.yaml -i inventory.json
uv run waldur-rancher-kc-crd-import apply -c waldur-site-agent-config.yaml -i inventory.json \
    --execute --report import-report.json

To try the import on part of the offering first, narrow both commands with the same filters (each can be repeated):

Filter	Matches
--cluster c-m-abc12345	Rancher cluster ID (Resource.backend_id)
--resource <uuid-or-name>	Waldur Resource
--project <name-or-id>	Rancher project, by name, c-m-…:p-… ID or bare p-… ID

--project imports project-scope bindings only. Cluster role bindings don't belong to any one project, so they are left out.

uv run waldur-rancher-kc-crd-import inventory -c waldur-site-agent-config.yaml \
    --cluster c-m-abc12345 -o inventory-c-m-abc12345.json
uv run waldur-rancher-kc-crd-import apply -c waldur-site-agent-config.yaml \
    -i inventory-c-m-abc12345.json --project analytics

apply creates a ResourceProject for each Rancher project, named exactly like the project so that the operator adopts it, and records the Rancher project ID as its backend_id. It then grants each user the Waldur role that maps back to their Rancher role template: role_map for project bindings, cluster_role_map for cluster bindings. It can be re-run safely: existing ResourceProjects and roles are left as they are, and a missing backend_id is filled in.

The Waldur token must be a staff token. The import lists the offering's resources as their provider, records backend IDs through the provider API, and looks users up across organizations; neither the agent's own token nor an organization owner's token can do all three.

Quotas. The operator replaces a project's Rancher quota with the ResourceProject's limits on every sync, so a ResourceProject without limits clears the quota. apply therefore copies each project's resourceQuota.limit into the new ResourceProject: limitsCpu becomes cpu (millicores), limitsMemory becomes memory, and requestsStorage becomes storage. Other quota keys (pods, requests.cpu, and so on) have no Waldur equivalent. The plan flags them with WARN, because the first sync removes them. It also flags an existing ResourceProject that has no limits while its Rancher project has a quota. The offering must accept these limits, or the creation fails and is reported.

The plan lists everything it skips, with the reason:

• the Default and System projects (--include-system imports them);
• a Rancher project already linked, by backend_id, to a ResourceProject with a different name: the operator adopts by name, so a second ResourceProject would put two CRs on one project;
• several projects on one cluster sharing a name, because the operator would adopt an arbitrary one of them;
• role templates that no Waldur role maps to, and Waldur roles the offering doesn't define;
• principals that aren't Keycloak OIDC users or groups (local Rancher users, other auth providers);
• Keycloak users with no matching active Waldur user under the configured identity settings, for example a user who has never logged in to Waldur.

Rancher makes whoever creates a project its project-owner. Projects created by one admin therefore all carry that admin's binding, and the import grants the matching Waldur role on every one of them if the admin has a Waldur account. Drop those entries from the inventory if that isn't wanted.

The inventory is a plain JSON file, so you can drop entries from it before running apply. It is written with mode 0600, because it names users and, in civil-code mode, contains civil codes.

After the agent's first sync, confirm the CRs are Ready and that users have access through the new groups. Then delete the old bindings in Rancher; the operator never removes them. Keep in mind that an adopted project is treated like any other: removing its ResourceProject from Waldur deletes the Rancher project.

---

User identity matching

Each CR member is {userIdentifier, lookupByID[, lookupAttribute]}. Two settings decide what goes in it:

• keycloak_user_identity_source — the Waldur field that supplies userIdentifier: username (default), uuid, or civil_number.
• keycloak_user_lookup — how the operator finds that value in the Rancher Keycloak: username (default), id (Keycloak user ID), or attribute (a user attribute named by keycloak_lookup_attribute).

keycloak_use_user_id: true is the deprecated spelling of source: uuid + lookup: id; it still works (with a warning in the agent log) and produces the same CRs as before. Combining it with conflicting new settings is rejected.

Username (default) — source: username, lookup: username

Plugin sends UserRole.user_username. Operator does GET /admin/realms/<realm>/users?username=X&exact=true and uses the resulting user.id for group membership operations. Works in both OIDC and self-hosted Waldur as long as Waldur usernames align with Keycloak usernames — the typical OIDC mapping does this via the preferred_username claim.

UUID — source: uuid, lookup: id

Plugin sends UserRole.user_uuid. Operator does GET /admin/realms/<realm>/users/{uuid} (matches the Keycloak internal user.id). Use this only when Waldur was OIDC-provisioned AND its user UUIDs were seeded from the Keycloak sub claim, so that Waldur.user.uuid == Keycloak.user.id. The username path is preferred because it tolerates UUID divergence and works in more topologies.

This is not OfferingUser.username. SLURM-style backends key membership off OfferingUser.username — a per-offering, often auto-generated identifier (e.g. a POSIX account name like test-rancher-00) meant for the target system, not Keycloak. This plugin never looks at that field. Its membership unit is each ResourceProject's UserRole, and the identity it hands to the operator is UserRole.user_username (or user_uuid) — the person's actual Waldur account identity. If a user was added to Keycloak under a name that doesn't match their real Waldur username, the operator logs WARNING User <id> not found in Keycloak and silently skips them (see Troubleshooting) even though everything else — CR, group, PRTB — looks fully synced.

The operator never creates users. A user that doesn't exist in Keycloak under the chosen identifier gets logged as WARNING User <id> not found in Keycloak and is skipped — the PRTB and the group are still created and bound, the user just isn't a member yet. They become a member on the next reconcile after the user appears in Keycloak (e.g. their first OIDC login).

Civil code — source: civil_number

For a Rancher Keycloak that is not the one Waldur logs in with; see the next section.

The CR identifier and the identity reported back to Waldur are kept apart: the membership sync report is keyed by username in username mode and by user_uuid in the uuid and civil-number modes.

---

Matching users across Keycloaks by civil code

When Waldur and Rancher authenticate against different Keycloaks, Waldur usernames and UUIDs mean nothing on the Rancher side. What both sides share is the person's civil code (personal identification code), which Waldur stores as civil_number. Set keycloak_user_identity_source: civil_number and pick the lookup that matches how the Rancher Keycloak stores the code.

Key	Values (default first)	Meaning
keycloak_user_identity_source	username, uuid, civil_number	Waldur field that becomes userIdentifier
keycloak_user_lookup	username, id, attribute	How the operator looks it up; id needs source uuid
keycloak_lookup_attribute	string	Attribute to match; required with (only with) lookup attribute
keycloak_user_identity_template	${value}, e.g. EE${value}	Rendered around the source value
keycloak_user_identity_lowercase	true for lookup username, else false	Lowercase the result

Layout 1 — the username is the civil code. E.g. a Keycloak brokering TARA with the sub claim (EE38001010000) mapped to the username. Use keycloak_user_lookup: username (the default) and a template for any country prefix:

keycloak_user_identity_source: civil_number
keycloak_user_identity_template: "EE${value}"
# keycloak_user_lookup: username   (default)

Keycloak stores usernames lowercased and the operator compares the spec string with the stored username, so the identifier is lowercased by default (ee38001010000). A mixed-case value would be added and removed again on every reconcile.

Layout 2 — the civil code is a user attribute. Use attribute lookup and name the attribute. Requires operator 0.5.0+. The attribute must be declared in the realm's user profile with edit permission admin only, or filled by an identity-provider or LDAP mapper. Attribute lookup gives the grant to whoever holds the value, so a user-editable attribute would let anyone copy someone else's civil code into their own profile. The operator refuses attribute lookups in that case and logs an error:

keycloak_user_identity_source: civil_number
keycloak_user_lookup: attribute
keycloak_lookup_attribute: personalCode
# keycloak_user_identity_template: "EE${value}"   if the stored value is prefixed

The value is matched verbatim (no lowercasing unless keycloak_user_identity_lowercase: true), and syncedMembers in the CR status echoes it.

The offering must expose the civil code. Waldur includes user_civil_number in the list_users responses the plugin reads only when the offering's user attribute config has expose_civil_number enabled. Without it no member can be matched: the plugin logs a warning each cycle naming the setting, and every grant is reported as missing_in_idp with "Civil code not available for this user". A user whose civil code is empty is skipped the same way, individually.

Privacy. Civil codes never leave the agent towards Waldur — sync reports are keyed by user_uuid and the plugin maps confirmed members back to UUIDs — and agent logs show them masked (3800…00). They are, however, stored in plaintext in each CR's spec and status (userIdentifier, syncedMembers). Keep get/list/watch on managedrancherprojects in the CR namespace restricted to the agent, the operator and cluster administrators.

---

Configuration reference

waldur_api_url/waldur_api_token below intentionally repeat the offering's top-level fields of the same name — see the setup example above for why.

Key	Type	Required	Description
waldur_api_url	string	no	Mastermind API root with /api/; see note B.
waldur_api_token	string	no	Long-lived token from /api/users/<uuid>/keys/. Don't use a session token.
waldur_verify_ssl	bool	no (default true)	TLS verify for Waldur calls.
kubeconfig_path	string	no	Path to a kubeconfig file. Omit to use in-cluster credentials.
context	string	no	kubeconfig context to use when kubeconfig_path is set.
namespace	string	no (default waldur-system)	Namespace for ManagedRancherProject CRs.
parent_group_name	string	no	Top-level KC group; var ${cluster_id}. Default c_${cluster_id}.
group_name_template	string	no	Per-role child KC group, vars in note A.
role_map	dict	no (default {})	Waldur role → Rancher project role template ID; see note C.
cluster_role_map	dict	no	Resource-level role → Rancher cluster role template ID; see note C.
cluster_group_name_template	string	no	KC group per cluster role.
keycloak_user_identity_source	string	no (default username)	username, uuid or civil_number.
keycloak_user_lookup	string	no (default username)	username, id or attribute.
keycloak_lookup_attribute	string	with lookup attribute	User attribute to match (operator 0.5.0+).
keycloak_user_identity_template	string	no (default ${value})	E.g. EE${value}.
keycloak_user_identity_lowercase	bool	no	Default true for lookup username, false otherwise.
keycloak_use_user_id	bool	no	Deprecated: true = source uuid + lookup id.

Defaults: group_name_template is c_${cluster_id}_${rp_uuid}_${role_name}, cluster_group_name_template is c_${cluster_id}_cluster_${role_name}.

Note B — Waldur API credentials. waldur_api_url / waldur_api_token are optional to the loader, but without them pull_resource does nothing and membership is never synced. The plugin strips a trailing /api for the SDK; use a long-lived token, not a session token.

Note C — role maps. Roles outside role_map are skipped, so an empty map binds nobody. When cluster_role_map is set, the Resource-level user roles are written to every CR's spec.cluster block for cluster-wide bindings; when it is unset there are no cluster bindings.

The settings are validated by waldur_site_agent_rancher_kc_crd.schemas.RancherKcCrdBackendSettingsSchema; a misspelt key is logged as a warning when the agent loads its configuration. An invalid member-identity combination (e.g. source civil_number with lookup id) is logged the same way and also stops the backend from starting, since it would grant access to nobody or to the wrong users.

spec.clusterId is resolved from each Resource's backend_id (1:1 with a Rancher cluster) — there is no offering-level cluster_id setting. An empty backend_id raises a clear KeyError rather than silently emitting an invalid CR.

Note A — group_name_template variables. Available substitutions:

Variable	Source	Notes
${cluster_id}	Resource.backend_id	Rancher cluster ID, opaque
${role_name}	UserRole.role_name	Pre-mapping (before role_map)
${rp_uuid}	ResourceProject.uuid	Full 32-char hex
${rp_uuid_short}	first 8 chars of ${rp_uuid}	~4B combos, collision-free; same as cr_name
${customer_slug}	Resource.customer_slug	Waldur Customer (organization) slug
${project_slug}	Resource.project_slug	Waldur Project slug (parent Project; RPs have no slug)
${resource_slug}	Resource.slug	Waldur Resource slug (1:1 with cluster)
${project_name}	ResourceProject.name	Human-readable; may contain spaces

Default is c_${cluster_id}_${rp_uuid}_${role_name} — one Keycloak group per (cluster × project × role), matching Rancher's per-project-PRTB access model. The default uses ${rp_uuid} (immutable) for stability; override only if you have a strong reason.

Recommended human-readable opt-in template:

c_${cluster_id}_${customer_slug}_${project_slug}_${rp_uuid_short}_${role_name}

Renders e.g. c_c-m-glwxdksp_hpc-demo-org_genomics-2026_8706dd1a_project_member. Stays unique per RP via ${rp_uuid_short} while the slugs make the group name self-explaining in the Keycloak admin UI.

Custom-template constraints.

1. MUST include a per-project discriminator (${rp_uuid}, ${rp_uuid_short}, or ${project_name}); without it, multiple projects share one group and a user added to project A also gains access to B, C, … via the shared group's PRTBs.
2. Slugs (customer_slug, project_slug, resource_slug, project_name) are mutable -- renaming the entity in Waldur creates a new Keycloak group on the next reconcile and orphans the old one (the operator adopts groups by name and never renames adopted groups). Memberships in the old group become stale.
3. Switching the template after deployment has the same effect as a bulk rename: every existing CR re-renders, the operator creates fresh groups, the old groups linger with their stale members. Plan a one-time manual migration if you change the template against an existing deployment.
4. Keycloak's GROUP.NAME column is varchar(255). The plugin guards this at render time and raises a ValueError (with a hint about ${rp_uuid_short}) if the rendered name would exceed 255 chars, so the operator never tries to apply a CR that Keycloak would reject with HTTP 500. Stay well under by preferring ${rp_uuid_short} over ${rp_uuid} in long templates and keeping Waldur slugs reasonably short (say ≤ 50 chars each).

---

Troubleshooting

Plugin logs HTTP/1.1 401 Unauthorized from Waldur on every iteration

waldur_api_token is a session token from /api-auth/password/ (rotates on each call). Use a long-lived API token from /api/users/<uuid>/keys/.

pull_resource succeeds but no CRs are created

The resource has an empty backend_id. The agent's resource fetcher (waldur_site_agent/common/processors.py:_get_waldur_resources) drops resources without one. Fix: POST /api/marketplace-provider-resources/<uuid>/set_backend_id/.

Plugin logs IndexError: list index out of range in processor __init__

Customer is not registered as a service provider. Fix: POST /api/marketplace-service-providers/ with the customer URL.

Plugin logs GET .../api/api/marketplace-provider-resource-projects/... (doubled /api/)

You're on a build older than the URL fix bundled with the orphan-pruning commit. Pull latest plugin code, or as a workaround drop the trailing /api/ from waldur_api_url.

Operator logs WARNING User X not found in Keycloak for every user

Identity mismatch — the chosen identifier (username by default; UUID or civil code per keycloak_user_identity_source) isn't resolvable in Keycloak. With the default username path: ensure Waldur usernames map to existing Keycloak usernames. With the UUID path: align Waldur user UUIDs with the Keycloak OIDC sub. With civil codes: check the template (country prefix) and, for attribute lookup, that the attribute is declared in the realm user profile with edit permission admin only (the operator logs users can edit their own attribute and refuses lookups otherwise).

kubectl delete mrp succeeds in 0s but the Rancher project remains

status.rancherProjectId is stale; operator versions before 0.2.2 treated 404 on delete as success. Upgrade the operator to 0.2.2+ — cleanup now falls back to find-by-name.

Operator status.conditions to check for any CR (kubectl describe mrp <name>):

Condition	What status=False means
RancherProjectReady	Rancher create/adopt failed. Check operator log for httpx.HTTPStatusError.
ResourceQuotaReady	spec.resourceQuotas apply failed (only emitted by operator 0.3.0+).
KeycloakGroupsReady	Couldn't create/find parent or child KC groups. Check KC admin credentials.
RancherBindingsReady	PRTB creation failed — usually invalid rancherRole in role_map.
MembershipSynced	Per-user add/remove failed — see User X not found in Keycloak warnings.

---

Development

# Workspace install
cd <repo-root>
uv sync --all-packages

# Run unit tests for this plugin only
cd plugins/rancher-kc-crd && uv run pytest tests/

# Integration tests (requires a real K8s cluster with operator + CRDs)
K8S_CRD_TEST=1 \
RANCHER_CLUSTER_ID=c-m-abc12345 \
KUBE_CONTEXT=docker-desktop \
  uv run pytest tests/test_backend_integration.py -v

# Lint + format
uvx prek run --all-files

tests/ holds pure unit tests (translator, status reader, sync-report derivation, unmapped-role warnings, settings schema) and test_backend_integration.py, which only runs with K8S_CRD_TEST=1 against a real cluster with the operator and CRDs installed.

---

Companion components

Component	Role
rancher-keycloak-operator (separate repo)	Reconciles ManagedRancherProject CRs.
ManagedRancherProject CRD (in operator helm chart)	API surface the plugin writes to.