Backup and disaster recovery

The database is the product. Everything Marolum has ever observed or been told is in it, and nothing else can regenerate the parts your backends have since pruned. This page is the operating procedure for keeping it.

What a restore gives back, and what it does not

Restoring a backup rewinds Marolum's own record to the moment the backup was taken.

• Current projections converge forward. Backends are re-read on the normal cadence, so Stacks, resources, outputs, and locks are current again within minutes.
• Durable history from before the backup is whatever the backup held. Operations Marolum accepted between the backup and the failure are gone. If the backend still holds them, the next update-history cycle re-observes them as fresh evidence; if the backend has pruned them, they are lost.
• Accounts, tokens, retention policy, audit rows and session records are restored as they were. An unexpired session present in the backup can remain valid after restore; logout or revocation after the backup is also rewound. A restore does not automatically sign everyone out.
• Transcript and state bodies decrypt only with the same key that encrypted them. A database backup without its key holds ciphertext.

Your backup interval is therefore your recovery point objective for the durable record. Daily is a reasonable default; tighten it if you rely on Marolum as the record for stacks whose backends prune aggressively.

What to back up

Protect backups exactly as you protect the live volume. A backup contains every account's credential hash, every token digest, every observed output, and every retained operation.

SQLite backup

Backups are cold: stop the replica, copy the database and its licensing directory, start the replica. A copy taken while the process is running is not a valid backup unless SQLite's own online backup mechanism produced it.

Copy both items into one directory per backup. The licensing directory sits beside the database (/data/licensing by default, or MAROLUM_CONNECTED_LICENSE_DIR) and holds the installation ID even on Community; a restore without it starts as a new installation. cp -p keeps UID 1000 as the owner, which the verification below and a restore both need.

backup="/srv/marolum/backups/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p -m 700 /srv/marolum/backups
mkdir -m 700 "$backup"
docker stop -t 30 marolum
cp -p /srv/marolum/data/marolum.db "$backup/marolum.db"
cp -pR /srv/marolum/data/licensing "$backup/licensing"
docker start marolum

From a named Docker volume, copy out with the image's own runtime. The image runs as UID 1000, so the backup directory must be writable by that user:

backup="$PWD/backups/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p -m 700 "$backup"
sudo chown 1000:1000 "$backup"   # Linux hosts only
docker stop -t 30 marolum
docker run --rm --entrypoint node \
  --mount type=volume,source=marolum-data,target=/data \
  --mount type=bind,source="$backup",target=/backup \
  "$MAROLUM_IMAGE" \
  -e "const fs = require('fs'); fs.copyFileSync('/data/marolum.db', '/backup/marolum.db'); fs.cpSync('/data/licensing', '/backup/licensing', { recursive: true });"
docker start marolum

Verify every copy

Two checks, every time:

1. The byte size matches the source.
2. PRAGMA integrity_check reports ok. SQLite cannot open a database in WAL mode from a read-only location, so the check copies the backup into the container's own /tmp first and never writes beside the backup:

docker run --rm --network none --entrypoint node \
  --mount type=bind,source="$backup/marolum.db",target=/backup/marolum.db,readonly \
  "$MAROLUM_IMAGE" \
  -e "const fs = require('fs'); fs.copyFileSync('/backup/marolum.db', '/tmp/marolum.db'); console.log(require('better-sqlite3')('/tmp/marolum.db', { readonly: true, fileMustExist: true }).pragma('integrity_check')[0].integrity_check);"

Docker Compose and systemd packages both checks into a nightly timer.

Copy it off the machine

A backup on the same disk as the database protects against nothing that takes the disk with it. Copy to a versioned object-store bucket with its own access policy, or into your existing backup system, and give it a retention period you have written down; it bounds any deletion promise you make about the data inside.

What about snapshots?

An EBS snapshot, Azure disk snapshot, or VM snapshot of a running Marolum is crash-consistent. It will usually recover, and it is a worthwhile second layer, but it is not a verified backup and no restore drill should depend on it alone. Snapshot the volume while the container is stopped if you want a snapshot you can trust.

PostgreSQL backup

You own the server; use its tooling. A physical base backup with WAL archiving gives point-in- time recovery; pg_dump of the Marolum database gives a logical copy. Take logical backups while Marolum is stopped or accept that in-flight writes may straddle the dump. The SQLite backup you took before adoption is not a PostgreSQL backup; keep it, but do not confuse the two.

Connected-licensing state lives in the shared database and is part of the backup; protect the backup as a credential. Never copy a database into a second live installation and run both.

Restore

SQLite

1. Stop and remove the running replica.
2. Replace /data/marolum.db with the backup's marolum.db, keeping UID 1000 as its owner and mode 0600 (cp -p from a backup taken as above).
3. Delete any leftover marolum.db-wal, marolum.db-shm, and marolum.db.owner files.
4. Restore the matching connected-license directory or offline license assets, plus transcript and state keys when used. Preserve the configured SQLite installation ID; do not create a second live installation using a copied identity.
5. Start the replica and confirm /api/v1/readiness returns 200.
6. Sign in, open Operations, and confirm Evidence coverage shows the expected earliest retained evidence.
7. If the restore follows lost or compromised access, use the documented administrator recovery procedure to rotate the bootstrap administrator password and revoke all active browser sessions. Review restored account and integration-token state separately; browser-session revocation does not revoke integration tokens.

PostgreSQL

Restore with your server's tooling, then restart every Marolum replica. Replicas cache nothing that survives a restart, but they must re-read the schema lineage and re-elect job ownership against the restored state.

Restoring an older release's backup

A backup is tied to the schema revision it was taken at. Start the image that took it, or a newer one, never an older one. If you need to move to a newer release after a restore, follow the normal upgrade procedure from the restored file.

Rebuilding the whole installation

When the host is gone:

1. Provision the new host per the environment guide, with an empty data volume.
2. For SQLite, copy the verified backup to the configured database path (/data/marolum.db by default), owned by UID 1000, mode 0600. For PostgreSQL, restore the server database using the PostgreSQL backup procedure.
3. Restore the matching connected-license directory or offline license assets, external keys, backends file and deployment configuration, including the selected database provider. SQLite's installation ID is in its connected-license directory; PostgreSQL's is in the restored shared database. Keep a single live installation for that identity.
4. Start the same image digest the backup was taken under.
5. Confirm readiness, sign in, and check Backends and Evidence coverage.
6. Update DNS or the load balancer target to the new host.

Do not start an empty database "to check that it works" and then swap the backup in. An empty database bootstraps a new administrator from the password file and assigns a new installation identity; connected licensing binds to that identity. Restore first, start once.

Drill it

A backup you have never restored is a hypothesis. Once a quarter, restore the latest backup onto a scratch host, start Marolum against it, sign in, and confirm the coverage panel shows what you expect. Write down how long it took; that number is your recovery time objective, and it is the only honest one you will get.

Preserving evidence after an incident

If you need to preserve the record itself:

1. Stop the replica cleanly rather than killing it, so the WAL checkpoints.
2. Take and verify a cold copy of marolum.db before doing anything else.
3. Do not edit or delete operation-history rows by hand. Marolum's integrity controls detect divergence, and hand-editing destroys the evidence and the ability to reason about it.
4. Use Data & Retention → Deletion history to establish what was removed, by which action, and when.
5. For an integrity failure reported by the expiry worker, preserve the database and its sidecars for investigation before restoring.