Skip to content

Provisioning

Some resources aren't created or deleted through a direct REST endpoint. Their lifecycle runs through Waldur's marketplace order flow: you submit an order against an offering, Waldur processes it asynchronously, and a resource appears when it completes. The CLI exposes this as provision and terminate verbs.

There are two ways in:

  • Typed OpenStack commands — openstack tenant, openstack instance, openstack volume get provision/terminate with a typed --generate-skeleton (flavor, image, ports, …), covered below.
  • Generic marketplace resource — provisions any offering type (SLURM, VMware, Azure, custom, …), with a free-form attributes body. See Provisioning other offerings.

The flow is the same either way: find an offering → submit an order → wait for it → get the resource.

1. Find an offering

Provisioning is always against an offering (a specific provider's OpenStack service). Browse them under marketplace offering, filtering by type:

1
2
waldur-cli marketplace offering list --filter type=OpenStack.Tenant --format json \
  --jmespath '[].{name: name, uuid: uuid, url: url}'

You'll need the offering's url, one of its plans' url, and the target project's url.

2. Build the order

provision takes the whole order body with the same --request / --request-file / --generate-skeleton options as create. The skeleton is the order envelope — offering, project, plan, limits, accepting_terms_of_service — with the resource's typed attributes filled in under attributes, derived from that offering type's own schema (so an instance skeleton has flavor, image, ports, … and a volume skeleton has size, type, …):

1
waldur-cli openstack instance provision --generate-skeleton yaml > vm.yaml

Fill in offering, project, and the attributes you want. As with create, any field left null — including nested attributes fields — is omitted, so you only fill what you need. accepting_terms_of_service defaults to true in the skeleton.

Ambient project scope

If you've set a current project, you can leave project out of the order body entirely — it's filled in from the scope. An explicit project in the body still overrides it.

A tenant (VPC) order, for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
{
  "offering": "https://waldur.example.com/api/marketplace-public-offerings/<uuid>/",
  "project": "https://waldur.example.com/api/projects/<uuid>/",
  "plan": "https://waldur.example.com/api/marketplace-public-offerings/<uuid>/plans/<uuid>/",
  "attributes": {
    "name": "my-vpc",
    "skip_connection_extnet": true,
    "skip_creation_of_default_router": true
  },
  "limits": {"cores": 4, "ram": 4096, "storage": 51200},
  "accepting_terms_of_service": true
}

3. Provision

1
waldur-cli openstack tenant provision --request-file vpc.yaml

By default provision submits the order and polls it to completion, then prints the resulting resource. The order moves through pending/executing states to done; a failed order (erred, rejected, or canceled) surfaces its error_message as an error and a non-zero exit.

  • --no-wait — submit and return the order immediately, without polling. Useful in scripts that track orders themselves.
  • --timeout N — how long to wait for a terminal state before giving up, in seconds (default 600). Timing out doesn't cancel the order; it just stops waiting.
  • --interval N — how often to poll, in seconds (default 3). Raise it for an order you expect to take a while, to cut down on request volume; lower it if you want faster feedback on a quick one.
1
2
3
waldur-cli openstack tenant provision --request-file vpc.yaml --timeout 300
waldur-cli openstack instance provision --request-file vm.yaml --no-wait
waldur-cli openstack tenant provision --request-file vpc.yaml --interval 10

The provisioned resource object includes both resource_uuid (the OpenStack tenant/instance/ volume itself) and uuid — its marketplace resource UUID, which is what you'll need to terminate it.

Preview first

Add --dry-run to print the exact order that would be submitted (with the project and any defaults already filled in) without actually creating anything — see --dry-run.

Terminating

terminate tears a resource down through the same order flow. It takes the resource's marketplace_resource_uuid — the marketplace resource, not the plugin resource's own UUID. You'll find it as the uuid/marketplace_resource_uuid field on the provision result, or via get/list:

1
waldur-cli openstack instance terminate <marketplace_resource_uuid>

Some resources accept termination options as JSON:

1
2
waldur-cli openstack instance terminate <marketplace_resource_uuid> \
  --request '{"delete_volumes": true, "release_floating_ips": true}'

Like provision, terminate waits for the termination order to finish by default; --no-wait, --timeout N, and --interval N apply the same way.

Provisioning other offerings

marketplace resource provisions any offering type — SLURM allocations, VMware or Azure VMs, or a provider's custom offering — using the same order flow, just with a free-form attributes body instead of a typed one (the accepted attributes are offering-specific and not all modeled in the schema):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# find the offering
waldur-cli marketplace offering list --filter type=Marketplace.Slurm

# provision (offering + project + whatever attributes/limits that offering wants)
waldur-cli marketplace resource provision --request '{
  "offering": "https://waldur.example.com/api/marketplace-public-offerings/<uuid>/",
  "project":  "https://waldur.example.com/api/projects/<uuid>/",
  "plan":     "https://waldur.example.com/api/marketplace-public-offerings/<uuid>/plans/<uuid>/",
  "attributes": {"name": "my-allocation"},
  "limits":     {"cpu": 100, "ram": 200, "gpu": 4}
}'

marketplace resource provision --generate-skeleton gives the order envelope with a generic {name, description} attributes stub — fill in the offering-specific fields yourself (from the offering in the Waldur portal). Everything the typed commands support applies here too: --no-wait/--timeout/--interval, --dry-run, and the ambient --project (leave project out of the body and it's filled in).

marketplace resource list/get browse provisioned resources (handy for finding the marketplace_resource_uuid to terminate), and marketplace resource terminate <uuid> tears any of them down:

1
2
waldur-cli marketplace resource list --filter offering_uuid=<uuid>
waldur-cli marketplace resource terminate <marketplace_resource_uuid>

The typed openstack ... provision commands are just a convenience for the three OpenStack types — anything they can do, marketplace resource can do generically.

Waiting on anything with real async state, not just orders

Resources with genuine async, server-side state to reach — an OpenStack tenant/instance/ volume, a network, subnet, security group, router, port or floating IP, a generic marketplace resource, and the order itself — get a wait command: polling on an interval until a --jmespath condition against the fetched object is met, or timing out. Waldur's API has no server-side push/watch mechanism, so this is a client-side poll, not an instant notification -- but it saves you from writing your own poll loop. Plain CRUD resources (a customer, a role, a catalog entry like a flavor) reach their final state synchronously on create/update, so there's nothing to wait for and no wait command -- check --help on a given resource to see whether it has one.

The condition can be a boolean comparison:

1
waldur-cli openstack instance wait <uuid> --jmespath "state=='OK'"

or a plain presence check -- anything the expression returns other than false/null counts as met:

1
waldur-cli marketplace order wait <order-uuid> --jmespath "resource_uuid"

--timeout N (default 600s) and --interval N (default 3s) control how long to wait and how often to poll.

This is what makes --no-wait genuinely useful for fire-and-forget provisioning: submit without blocking, keep the printed order uuid, and wait on it later — from the same script, a different process, or a different machine entirely:

1
2
3
4
ORDER=$(waldur-cli openstack tenant provision --request-file vpc.yaml --no-wait \
  --format json --jmespath 'uuid')
# ... do other things ...
waldur-cli marketplace order wait "$ORDER" --jmespath "state=='done'" --format json

marketplace order get <uuid> is the one-shot version — no polling, just the current state (the same object provision's own polling already checks internally, exposed standalone for when you submitted with --no-wait and only have the order UUID, before any resource exists to check instead).

Networking inside a tenant

Tenants, instances and volumes are ordered through the marketplace. Everything an instance plugs into inside a tenant is not: networks, subnets, security groups, routers, ports and floating IPs are created directly, and each is created asynchronously (Creating → OK). Most of them have no REST create of their own — you create them through an action on their parent:

To create Command Parent
a network openstack tenant create-network <tenant-uuid> tenant
a subnet openstack network create-subnet <network-uuid> network
a security group (with rules) openstack tenant create-security-group <tenant-uuid> tenant
a floating IP openstack tenant create-floating-ip <tenant-uuid> tenant
a server group openstack tenant create-server-group <tenant-uuid> tenant
a router openstack router create (the body names the tenant) —
a port openstack port create (the body names the network) —

Every one of these takes a request body and has a --generate-skeleton template, which you can print without a UUID:

1
2
waldur-cli openstack network create-subnet --generate-skeleton yaml
waldur-cli openstack tenant create-security-group --generate-skeleton

The body is validated against the API's schema before anything is sent, and --dry-run shows the exact request.

Wait between dependent steps

A network isn't usable until it's OK, and create-subnet refuses a network that isn't. The create commands return the new object straight away (still Creating), so capture its uuid and wait on it:

1
2
3
4
5
6
NET=$(waldur-cli openstack tenant create-network "$TENANT" \
  --request '{"name": "app-net"}' --format json | jq -r '.uuid // empty')
[ -n "$NET" ] || exit 1     # on an API error the JSON output has no uuid
waldur-cli openstack network wait "$NET" --jmespath "state=='OK'"
waldur-cli openstack network create-subnet "$NET" \
  --request '{"name": "app-subnet", "cidr": "10.10.0.0/24"}'

Guard the captured UUID as above: with --format json, an API error is printed as a JSON {"error": ...} object, so jq -r .uuid alone yields null (or, with // empty, nothing) and the next command would run against the wrong path.

A new subnet is attached to a router of the tenant unless the body sets "skip_router_connection": true or names a specific router. subnet connect/disconnect attach or detach it later.

Security groups and their rules

tenant create-security-group creates a group together with its rules. A rule is ethertype (IPv4/IPv6), direction (ingress/egress), protocol (tcp, udp, icmp, an IANA protocol number such as 58 for ICMPv6, or empty for any), a port range, and either a cidr or a remote_group:

1
2
3
4
5
6
7
8
9
waldur-cli openstack tenant create-security-group "$TENANT" --request '{
  "name": "ssh-v6",
  "rules": [
    {"ethertype": "IPv6", "direction": "ingress", "protocol": "tcp",
     "from_port": 22, "to_port": 22, "cidr": "::/0"},
    {"ethertype": "IPv6", "direction": "ingress", "protocol": "58",
     "from_port": -1, "to_port": -1, "cidr": "::/0"}
  ]
}'

For ICMP (and ICMPv6), the port fields hold the ICMP type and code: use -1 for any. Leaving them out is rejected (to_port: Empty value is not allowed).

To change the rules of an existing group, security-group set-rules <uuid> takes a JSON array of rules and replaces the whole rule set, so send every rule you want to keep, not just the new one:

1
2
waldur-cli openstack security-group set-rules "$SG" --generate-skeleton   # one-rule array template
waldur-cli openstack security-group set-rules "$SG" --request-file rules.json

An array body is validated as one: a single rule object instead of an array is rejected before anything is sent. The default group is managed by OpenStack itself and can't be renamed or deleted.

To attach groups to an instance at creation, list them under security_groups in the instance order (below). For an existing instance's port, port update-security-groups.

Routers and ports

Most tenants need neither: a tenant's default router connects its subnets, and an instance order creates its own ports. Reach for them when you need to control the topology:

  • router create / delete, add-router-interface / remove-router-interface (a subnet or port), set-external-gateway / remove-external-gateway, and set-routes for static routes.
  • port create / update / delete, enable-port / disable-port, enable-port-security / disable-port-security, set-allowed-address-pairs, update-port-ip, and update-security-groups.

Port security and allowed address pairs matter for a VM that sends or receives traffic for addresses other than its own — a router or VPN gateway, or a VM running its own nested cloud. Neutron's anti-spoofing drops that traffic until you widen the pairs or disable port security on the port.

Flavors, images and SSH keys

An instance order references a flavor, an image and (optionally) an SSH key by URL. Look the first two up in the tenant's catalog, and register your key once:

1
2
3
4
5
6
7
FLAVOR=$(waldur-cli openstack flavor list --filter tenant_uuid="$TENANT" \
  --filter cores__gte=8 --filter ram__gte=16384 --order cores,ram --limit 1 \
  --format json --jmespath '[0].url')
IMAGE=$(waldur-cli openstack image list --filter tenant_uuid="$TENANT" \
  --filter name_exact="Ubuntu 24.04 LTS" --format json --jmespath '[0].url')
KEY=$(waldur-cli auth ssh-key create --request "$(jq -n --arg k "$(cat ~/.ssh/id_ed25519.pub)" \
  '{name: "laptop", public_key: $k}')" --format json | jq -r .url)

A flavor's own disk is often small. Set system_volume_size (in MB) in the order to boot from a larger volume instead, and system_volume_type to pick its type.

Example: an IPv6-only VM

This puts a VM on a tenant network that has only an IPv6 subnet (here 2001:db8:1::/64, routed from the internet to the tenant's router), reachable over SSH from any IPv6 host. It uses the commands above end to end; $TENANT is the tenant's own UUID, and $A the API base URL (https://waldur.example.com/api).

1. Find the subnet, and check how it assigns addresses.

1
2
waldur-cli openstack subnet list --filter tenant_uuid="$TENANT" --format json \
  --jmespath '[?ip_version==`6`].{uuid: uuid, cidr: cidr, url: url, dns: dns_nameservers}'

Waldur can't create IPv6 subnets yet, so this one was created outside it — and Waldur doesn't show the subnet's IPv6 address mode (SLAAC, DHCPv6, or none) either. If it's SLAAC, the instance configures its address from the router's advertisements. If it's unset, Neutron hands out no address at all, and the instance learns it only from its config drive. Set "config_drive": true in the order either way: it works in both cases, and you can't tell from Waldur which one you have.

2. Allow SSH and ICMPv6 over IPv6. The groups a tenant starts with usually allow only IPv4. Create ssh-v6 as in Security groups and their rules.

3. Set DNS in cloud-init if the subnet has none. An IPv6-only VM also can't reach IPv4-only hosts (github.com has no IPv6 address). A DNS64 resolver, such as the public nat64.net service, answers for those hosts with addresses its NAT64 gateway translates:

1
2
3
4
5
6
7
8
9
#cloud-config
write_files:
  - path: /etc/systemd/resolved.conf.d/dns64.conf
    content: |
      [Resolve]
      DNS=2a00:1098:2b::1 2a01:4f8:c2c:123f::1
      Domains=~.
runcmd:
  - systemctl restart systemd-resolved

4. Order the instance.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
{
  "offering": "<$A>/marketplace-public-offerings/<instance-offering-uuid>/",
  "project": "<$A>/projects/<project-uuid>/",
  "accepting_terms_of_service": true,
  "attributes": {
    "name": "v6-host",
    "flavor": "<$FLAVOR>",
    "image": "<$IMAGE>",
    "system_volume_size": 102400,
    "config_drive": true,
    "ports": [{"subnet": "<the IPv6 subnet's url>"}],
    "security_groups": [{"url": "<default group url>"}, {"url": "<ssh-v6 url>"}],
    "ssh_public_key": "<$KEY>",
    "user_data": "<the cloud-init above, as a string>"
  }
}

1
waldur-cli openstack instance provision --request-file v6-host.json --timeout 1200 --interval 10

The instance offering is the one scoped to your tenant: marketplace offering list --filter type=OpenStack.Instance --filter project_uuid=<uuid>.

5. Connect.

1
ssh ubuntu@"$(waldur-cli openstack instance get <instance-uuid> --format json | jq -r '.internal_ips[0]')"

On a routed IPv6 subnet the instance's own address is public, so there's no floating IP; internal_ips holds it. (get has no --jmespath, so pipe its --format json output through jq.)

The bigger picture

An instance needs a tenant (and usually a network/subnet) to exist first, and its flavor and image are looked up within that tenant — so a from-scratch VM is a chain: provision a tenant, create a network and subnet in it, look up the tenant's flavor/image, then provision the instance referencing all of the above. Recipes & tips walks through a complete example.