Backups
Waldur keeps state in 2 components:
- Database - main persistency layer.
- Message queue - contains transient data mostly about scheduled jobs and cache.
Of these, only the database needs to be backed up. Uploaded files — organization logos,
attachments and the like — are stored in the database too
(waldur_core.media.storage.DatabaseStorage), so a database dump captures them as well.
A typical approach to a backup is:
1. Create a DB dump
- An entire db dump
1 | |
- An entire db dump with cleanup commands:
1 | |
- A db dump containing only data
1 | |
2. Copy backup to a remote location
Using rsync / scp or more specialised tools.
3. Restore the created backup
The dumps above are gzipped, so decompress them on the way in:
1 | |
We suggest to make sure that backups are running regularly, e.g. using cron.
Keep the encryption key
A dump does not hold the key Waldur encrypts credentials in the database with:
FIELD_ENCRYPTION_KEY, or, while that is unset, SECRET_KEY (GLOBAL_SECRET_KEY
with docker-compose, waldur.secretKey with Helm), which the key is then derived from.
During a key rotation, FIELD_ENCRYPTION_KEY_FALLBACKS holds the old ones. Back
them up as carefully as the dumps, but separately: restored without them, the
encrypted columns cannot be read, and nothing recovers them. See
Field encryption at rest.
Matrix chat
With Matrix chat on, the homeserver is a third component to back up: it keeps
the chat rooms, their messages and the users' Matrix accounts in its own
storage, the tuwunel_data volume with docker-compose and the
data-matrix-homeserver-0 PVC with Helm. Back it up together with Waldur's
database, so a restore brings both back to the same point, and stop the
homeserver while you copy it (with Helm, scale the matrix-homeserver
StatefulSet to zero). Also back up the tokens Waldur and the homeserver
share: with docker-compose, the waldur_matrix_secrets volume; with Helm, the
matrix-appservice-secret Secret, unless setup.existingSecret keeps it in
your secret manager. Both also hold the bootstrap password.
Chat depends on the encryption key above too. Users' escrowed recovery keys, the bot's encryption keys and access token, every history export's key and the appservice tokens in Waldur's settings are encrypted with it. Restored with another key, the bot refuses to start, the appservice stops working, and no recovery key or history export can be read. The Helm and docker-compose guides have the full backup and restore steps.
After restoring an older Waldur dump, or resetting Waldur's database, against
the same homeserver, the users provisioned since have a Matrix account that
Waldur no longer knows, and Waldur refuses to take it over: their chat stays
unavailable. Link them all with waldur link_matrix_account --all; see
Existing Matrix accounts.