LiteLLM Plugin for Waldur Site Agent

This plugin integrates a LiteLLM proxy with Waldur so that LLM inference access can be sold, provisioned, metered, and billed through the Waldur marketplace. It provisions per-customer virtual keys from marketplace orders and reports token usage and spend back to Waldur for metering and billing.

It is the same shape as the sibling envoy-ai-gateway plugin, but talks to LiteLLM's management REST API instead of Kubernetes Secrets — so there is no cluster access and no separate usage warehouse to deploy.

Features

Virtual key lifecycle: provision, rotate, pause, restore, and terminate LiteLLM keys directly from Waldur marketplace orders
Native block/unblock pause: suspend access with POST /key/block; a blocked key returns 401 at the proxy and unblocking restores it without new key material
Usage reporting: report per-resource input/output tokens and USD spend from LiteLLM's own daily spend table — no sidecar collector
Enforcement backstop: mirror Waldur's limits onto each key (max_budget, tpm_limit, rpm_limit) so a burst between reporting cycles cannot outrun metering
Model allowlist: restrict every key of an offering to a set of models

Overview

The plugin exposes two backends that are normally paired on a single composed offering:

Management backend (litellm): the order_processing_backend and membership_sync_backend. Owns the key lifecycle.
Usage reporting backend (litellm-usage): the reporting_backend. Reads LiteLLM's spend data and submits it to Waldur.

How keys are identified

LiteLLM stores keys sha256-hashed and returns the plaintext sk-… exactly once, in the /key/generate response. The agent applies the key and immediately reports it to Waldur, which holds the only copy (encrypted) — the plugin keeps nothing.

Everything after provisioning is therefore addressed by one of two handles:

Handle	What it is	Used for
`key_alias`	`<resource_backend_id>-<n>`, agent-chosen, stable across rotation	Waldur's `client_id`; key lookup
`token`	LiteLLM's sha256 of the key	`/key/block`, `/key/unblock`, `/key/update`, `/key/delete`

The resource's backend_id is its UUID hex, and each key is a numbered slot beneath it. Keys are minted non-expiring (no duration): Waldur owns the resource lifecycle, and an expiry it does not know about would revoke access behind its back.

Limit enforcement

Primary path — Waldur is the authority:

litellm-usage reports usage to Waldur
a LIMIT component reaches its limit and the offering sets plugin_options.action_on_usage_limit: pause
mastermind pauses the resource
the agent calls pause_resource(), which blocks every key of the resource → 401 at the proxy

Backstop — set_resource_limits() mirrors the resource's limits onto each key (max_budget, tpm_limit, rpm_limit). Each key carries the resource's full budget, not a share of it: the keys are alternatives for one consumer, and splitting the budget would throttle a consumer using a single key to half its entitlement.

The backstop is a full target state, not a patch: every one of the three fields is written on every reconciliation, null where there is to be no cap. Dropping tpm from a resource's limits therefore clears tpm_limit on its keys (or returns it to the offering-wide tpm_limit default); writing only the fields the resource currently carries would leave a removed cap on the key forever, with no later cycle able to clear it. null really does clear on LiteLLM's side — see Key management for the verified behaviour.

Which limits reach the key

set_resource_limits() reads the resource's Waldur limits by name, so a field is only driven per-resource when the offering declares a component of that name:

Waldur limit	LiteLLM field	Falls back to
`token_cost`	`max_budget`	no cap (there is no offering-wide budget setting)
`tpm`	`tpm_limit`	the offering's `tpm_limit` backend setting
`rpm`	`rpm_limit`	the offering's `rpm_limit` backend setting

waldur_resource.limits only ever carries components the offering declares. An offering that declares just input_tokens, output_tokens and token_cost — the common case, and the example below — therefore drives max_budget only; tpm_limit and rpm_limit stay at whatever the tpm_limit / rpm_limit backend settings say, the same value for every resource on the offering.

To make rate limits per-resource instead, declare tpm and/or rpm as LIMIT components — see the commented lines in the example below. They are rate caps, not meters: nothing reports usage against them, so they exist purely to carry a number onto the key. budget_duration is not per-resource under any configuration; it is an offering-wide setting and only applies alongside a max_budget.

Set budget_duration whenever token_cost is a limit

A LiteLLM max_budget with no budget_duration is a lifetime budget: it never resets, and the key's spend keeps accumulating against it. So on an offering that limits token_cost but leaves budget_duration unset, the first billing period that exhausts the budget blocks the key permanently — Waldur's period rolls over and un-pauses the resource, but the proxy keeps answering 401, and nothing in this plugin resets max_budget or clears spend. A backstop that fails closed forever is worse than no backstop, so set budget_duration to match the offering's billing period.

Even then the two clocks are not aligned: LiteLLM resets a key's budget on its own schedule, counted from when the key was created, not from the start of Waldur's billing month. A key minted on the 20th under budget_duration: "30d" resets on the 20th. This only affects the backstop — Waldur's own metering and its report → pause path stay month-aligned — but it means a rotated or late-minted key can carry a reset date its siblings do not share.

Backend Types

Management Backend (`litellm`)

Agent method	LiteLLM call
`ping()`	`GET /health/readiness` (requires `db: connected`)
`create_resource_with_id()`	register only — no key minted here; surfaces endpoint `{api_url}/v1`
`generate_resource_keys()`	`POST /key/generate` per missing slot, yielding one at a time
`list_resource_client_ids()`	`GET /key/list`, narrowed by substring then filtered to `^<backend_id>-\d+$`
`rotate_resource_key()`	`POST /key/{token}/regenerate`, falling back to delete + generate
`prune_unknown_resource_keys()`	`POST /key/delete` for slots Waldur does not hold
`pause_resource()`	`POST /key/block` for every key of the resource
`restore_resource()`	`POST /key/unblock` for every key
`downscale_resource()`	same as pause (a key has no partial-capacity state)
`delete_resource()`	`POST /key/delete` for every key
`set_resource_limits()`	`POST /key/update` — full target state of the four backstop fields
`get_resource_metadata()`	`active` = any key not blocked
`recreate_missing_resource()`	returns `False` + warning — the agent cannot restore a key it never kept

soft_delete does not apply. delete_resource() is overridden, so the core's soft-delete path (zeroing limits instead of removing the account) is never reached and the setting is ignored: a terminate always deletes the keys. There is no soft form of it — a key left in place with a zero budget is still a live credential, and the agent cannot hand back the plaintext of one it kept.

Usage Reporting Backend (`litellm-usage`)

Read-only. Reports whichever of input_tokens, output_tokens and token_cost the offering declares, so an offering that prices tokens Waldur-side and one that bills LiteLLM's upstream cost both work without a mode flag.

Source: GET /user/daily/activity, walking results[].breakdown.api_keys — see Why not /global/spend/report below.

One fetch per pass: /user/daily/activity cannot be filtered by key, so a call returns the whole proxy's month and this backend picks out the rows it wants. The processor pulls one resource at a time (and once more per historical period), so the fetched rows are reused across the resources of a pass — otherwise each resource would walk the entire table again. usage_cache_ttl bounds that reuse so it cannot outlive the pass. It defaults to half WALDUR_SITE_AGENT_REPORT_PERIOD_MINUTES (so 15 minutes on the default 30-minute period): a whole period would keep an entry alive into the next pass, which is exactly when fresh numbers are due.

Rollup: usage is attributed per resource, not per key. The -<n> slot suffix is stripped from the alias and the rows are summed, so rotating a key does not split a tenant's bill and a resource used across several days and several keys is billed for all of it.

Configuration

Both backends are normally combined on one composed offering and share a single backend_settings block.

Management backend settings (`litellm`)

Setting	Required	Default	Description
`api_url`	yes	—	LiteLLM proxy base URL
`api_token`	yes	—	Master or admin key for the management API
`models`	no	all	Model allowlist pushed onto every key
`budget_duration`	no	—	Reset period for `max_budget`, e.g. `30d`. Set it whenever `token_cost` is a limit
`tpm_limit`	no	—	Default tokens-per-minute cap on each key
`rpm_limit`	no	—	Default requests-per-minute cap on each key
`verify_ssl`	no	`true`	Verify the proxy's TLS certificate
`timeout`	no	`30`	Per-request timeout in seconds

Usage reporting backend settings (`litellm-usage`)

Setting	Required	Default	Description
`api_url`	yes	—	LiteLLM proxy base URL
`api_token`	yes	—	Master or admin key for the spend API
`verify_ssl`	no	`true`	Verify the proxy's TLS certificate
`timeout`	no	`30`	Per-request timeout in seconds
`usage_cache_ttl`	no	half report period	Seconds a fetched month of rows is reused in one pass; `0` disables

Composed offering (both backends)

offerings:
  - name: "LLM Inference"
    order_processing_backend: "litellm"
    membership_sync_backend: "litellm"     # required — pause/restore blocks the keys
    reporting_backend: "litellm-usage"

    backend_settings:
      api_url: "https://litellm.example.com"
      api_token: "sk-master-..."
      models: ["gpt-4o", "llama-3.3-70b"]   # optional allowlist
      budget_duration: "30d"                # required in practice whenever token_cost is a limit
      tpm_limit: null
      rpm_limit: null
      verify_ssl: true

    backend_components:
      input_tokens:  { measured_unit: "tokens", accounting_type: "usage", label: "Input tokens" }
      output_tokens: { measured_unit: "tokens", accounting_type: "usage", label: "Output tokens" }
      token_cost:    { measured_unit: "USD",    accounting_type: "usage", label: "Inference cost" }

      # Optional: declare these only to drive tpm_limit / rpm_limit per resource rather
      # than from the offering-wide backend_settings above. They carry a cap onto the
      # key; no usage is ever reported against them.
      # tpm:         { measured_unit: "tokens/min",   accounting_type: "limit", label: "Tokens per minute" }
      # rpm:         { measured_unit: "requests/min", accounting_type: "limit", label: "Requests per minute" }

Enforcement is configured on the Waldur offering, not here: the component must be billing_type: LIMIT and the offering must set plugin_options.action_on_usage_limit: pause.

LiteLLM API behaviour

Verified against ghcr.io/berriai/litellm:main-stable (open-source, no enterprise licence) on 2026-08-25. These are the answers the implementation is built on; re-check them when targeting a much older or newer proxy.

Enterprise-gated endpoints

Three endpoints refuse to run without a LITELLM_LICENSE, and they do not agree on a status code — they only agree on the prose, which is why the client discriminates on the message text:

Endpoint	Status	Consequence
`POST /key/{key}/regenerate`	500	rotation falls back to delete + generate
`GET /global/spend/report`	400	not usable as the usage source
`GET /key/spend/report`	403	not usable

Why not `/global/spend/report`

The issue specified GET /global/spend/report?group_by=api_key as the usage source. It is enterprise-only, so on an open-source proxy it returns 400 and reports nothing. GET /user/daily/activity was chosen instead:

not gated
pre-aggregated per day, so a month's report never walks per-request rows
results[].breakdown.api_keys.<hash> carries metadata.key_alias alongside metrics.prompt_tokens, metrics.completion_tokens and metrics.spend — so no separate hash → alias map is needed
paginated via metadata.page / total_pages / has_more

Only breakdown.api_keys is read. The same numbers reappear under breakdown.models and breakdown.model_groups split by model and by model group; walking more than one breakdown would count the same spend twice.

GET /spend/logs/v2 also works without a licence and is the per-request fallback if the daily aggregate ever proves insufficient — but its rows carry only the key hash, not the alias, so using it would reintroduce the hash → alias map.

Key management

/key/generate accepts blocked: true and it persists — a key minted blocked returns 401 at the proxy immediately. The response carries key (plaintext, once) and token (the sha256).
/key/block, /key/unblock, /key/delete and /key/update all accept the hash as key; /key/delete takes a list, as {"keys": [...]}. An unknown hash returns 404 Key not found., which the client maps to "no such key" rather than to a failure.
/key/list supports key_alias (exact) and substring_matching=true. Substring matching is a server-side narrowing only: res1- also returns res1-extra-1, so the exact ^<backend_id>-\d+$ pattern still has to be applied client-side. return_full_object=true is required to get token and blocked. Paginated via page / size / total_pages.
/key/update treats an explicit null as clear, not ignore. Posting {"key": <hash>, "max_budget": null, "budget_duration": null, "tpm_limit": null, "rpm_limit": null} against a key that carried all four returns 200 and leaves every one of them null on the following /key/info and /key/list. This is what makes the full-target-state backstop above work; if the proxy ignored nulls instead, a removed cap would stay on the key forever.
/key/list returns total_pages and current_page at the top level of the payload (alongside keys and total_count), not nested under a metadata object — unlike /user/daily/activity, which nests them. Paging past the last page returns an empty keys list rather than an error. The two clients read the field from different places for this reason.
blocked comes back as null, not false, for a key that was never blocked, so it must be read as truthy/falsy rather than compared against False.
Aliases are globally unique: a duplicate key_alias is rejected with 400. A delete frees the alias, which is what makes the delete-then-generate rotation fallback work.
/health/readiness needs no auth and returns {"status": "healthy", "db": "connected"}. The key API is useless without the database, so db != "connected" is treated as down.
Spend rows are flushed in batches, so usage lags a live request by a few seconds.

Tests

cd plugins/litellm && uv run pytest tests/

The HTTP layer is mocked throughout — no live proxy is required.

Running against a real proxy

docker network create litellm-net
docker run -d --name litellm-db --network litellm-net \
  -e POSTGRES_PASSWORD=litellm -e POSTGRES_USER=litellm -e POSTGRES_DB=litellm postgres:17
docker run -d --name litellm --network litellm-net -p 4000:4000 \
  -v $PWD/config.yaml:/app/config.yaml \
  -e DATABASE_URL=postgresql://litellm:litellm@litellm-db:5432/litellm \
  -e LITELLM_MASTER_KEY=sk-master-local \
  ghcr.io/berriai/litellm:main-stable --config /app/config.yaml --port 4000

A config.yaml with a mock_response model exercises the whole path without an upstream provider or an API key:

model_list:
  - model_name: fake-gpt
    litellm_params:
      model: openai/gpt-4o
      api_key: sk-fake
      mock_response: "Hello from the mock model."
general_settings:
  master_key: sk-master-local