Troubleshooting
Find the symptom, apply the fix. Every Lumaft failure ends with one sanitized log line that names a closed reason; it never includes SQL, file paths, object keys, credentials, or raw exception text, so the line is safe to paste into a ticket.
How to read the log
docker logs lumaft 2>&1 | tail -n 20 # docker run
docker compose logs lumaft | tail -n 20 # Compose
aws logs tail /lumaft/application --since 10m # ECS with awslogs
A refused start is not a crash. Lumaft checks its configuration, storage, and schema in order and stops at the first thing it will not accept, before touching the database. Fix the named cause and start again.
The container exits at startup
| Log line contains | Cause | Fix |
|---|---|---|
| A backends-file reason code | LUMAFT_BACKENDS_FILE is invalid: bad JSON, an unknown field, a duplicate id, a missing layout on a Pulumi record, or a layout on a non-Pulumi record |
Validate the document against the schema; python3 -m json.tool backends.json catches syntax |
| A password-file reason code | The file is missing, empty, shorter than 12 characters, or fails the file rules | Recreate it; see the file rules below |
| A file-rule refusal (symlink, permissions, parent directory) | The file is a symlink, is writable by group or others, or its parent directory is | chmod 700 the directory, chmod 600 the file, chown 1000:1000 both, and mount a real file, not a projected symlink |
storage is unusable |
/data is not mounted, is owned by the wrong user, or the path is invalid |
Mount the volume, chown 1000:1000 and chmod 700 it, then restart |
another Lumaft replica already owns this database |
Two containers point at the same database or shared volume | Stop the other one, or give this one its own volume. SQLite is one replica, always |
of them does not run on an ordinary restart |
The new image carries a gated migration | Expected on some upgrades; follow Gated migrations |
is newer than this image supports |
You started an older image against a volume a newer one migrated | Start the newer image, or restore the pre-upgrade backup |
this image does not upgrade a database at schema revision |
Too large a version jump | Run the intermediate releases the line names |
applied from different migration content than this image carries |
Content pin mismatch | Do not retry; start the image that migrated the volume and report it |
| Conflicting database sources | Both LUMAFT_SQLITE_PATH and a PostgreSQL URL are set, or both URL forms are |
Set exactly one source |
LUMAFT_ALLOW_HTTP or LUMAFT_LOCAL_ADMIN_PASSWORD_ROTATE refused |
The value is not exactly true or false |
Use the literal strings; 1, yes, and TRUE are refused |
The file rules, once. Every file Lumaft reads from a path — the password, the backends
document, a PostgreSQL URL file, a license, a transcript key — must be an absolute path to a
regular file, not a symlink, not writable by group or others, below a directory that is not
writable by group or others, and readable by UID 1000. On Linux hosts the last part means
chown 1000:1000. Docker Desktop on macOS and Windows maps ownership automatically.
Readiness stays 503
/api/v1/readiness returns 200 only after a bounded database query succeeds.
| Situation | Meaning | Fix |
|---|---|---|
| 503 for the first few seconds after start | Migrations are still applying | Wait; the image's health check allows a 10-second start period |
| 503 that never clears, container still running | The database cannot be queried — usually PostgreSQL unreachable | Check the PostgreSQL URL, network path, and TLS mode; check the log line |
| 503 on a standby replica (Enterprise) | The Enterprise entitlement lapsed; standbys stop serving | Expected; the primary keeps serving. Relicense or scale to one replica |
Load balancer reports unhealthy, curl locally says ready |
Wrong health-check path or port on the target group | Path /api/v1/readiness, port 8080, success code 200 |
Readiness never contacts your bucket. A bad S3 credential does not make readiness fail; see Backend problems for that.
Sign-in problems
| Symptom | Cause | Fix |
|---|---|---|
| Sign-in succeeds and immediately returns to the sign-in screen | The browser dropped the Secure session cookie because the page is plain HTTP | On a laptop over http://127.0.0.1, set LUMAFT_ALLOW_HTTP=true. Anywhere else, put TLS in front and leave it unset |
| Same, but the deployment sits behind a load balancer | The browser reached the instance directly over HTTP instead of the load balancer's HTTPS hostname | Use the https:// hostname; keep port 8080 closed to everything except the load balancer |
| "Too many attempts" or a silent refusal after several tries | Five failures from one address, or for one username, block for fifteen minutes | Wait fifteen minutes, or restart the container (the throttle is process-local) |
| The bootstrap password does not work on a restarted container | The password file is read only when no account exists. A database with an account ignores the file | Use the existing password, or follow Recover a lost administrator password |
| A new account cannot sign in | The account was created disabled, or the password is under 12 characters | Check status on Users & Access; reset the password |
| OIDC or SAML sign-in fails with a closed refusal | Missing or wrong LUMAFT_OIDC_<KEY>_* / LUMAFT_SAML_<KEY>_* variables, or LUMAFT_PUBLIC_ORIGIN unset |
Set them on every replica; the redirect URI must match exactly |
Sessions last twelve hours absolutely. Being signed out after twelve hours is not a fault.
Backend problems
Open Administration → Backends. Each record shows its diagnostic result and last successful observation.
| Symptom | Cause | Fix |
|---|---|---|
| Diagnostic: credential failure | No credentials reached the container | EC2/ECS: attach the role. Laptop: export credentials into .env. Azure/VMware: check the env file |
| Diagnostic: authorization failure | The identity lacks s3:ListBucket on the bucket or s3:GetObject on the prefix, or KMS denied |
Compare with IAM setup on AWS; run the three-command boundary test |
| Diagnostic succeeded, Stacks is empty after two minutes | Wrong layout or prefix |
List the bucket per Find your backend details |
| Some stacks appear, others do not | Two layouts in one prefix, and only one record declared | Add a second record for the other layout |
| A stack shows Current stack observation failed | One checkpoint failed to read or parse; earlier data is shown as last-successful | Check the object exists and the identity can read it; an unsupported checkpoint version fails closed |
Locks and update history read engine-unsupported |
The backend is not a Pulumi backend; those tiers are Pulumi-only | Expected. See Connect a backend |
| Collector coverage is stale on Operations | The backend has not observed successfully for a while | Check the diagnostic and /api/v1/observation-polling for backoff |
| Observation slows down, backoff climbs | Intermittent network failure or S3 throttling | Check NAT and egress; an S3 gateway endpoint removes NAT from the path on AWS |
I edited backends.json and nothing changed |
The file is read once at startup | Restart the container |
| I removed a backend and its data is still there | Disabling or removing a record keeps its cached projection by design | Use Data & Retention purge or forget if the data must go |
Nothing in Operations yet
| Symptom | Meaning |
|---|---|
| History not observed | Lumaft has never successfully read update history for this scope. Wait for the first five-minute cycle, or check the backend diagnostic |
| Observed, no retained operations | Lumaft read the backend and it holds no update history for these stacks |
| Coverage exists outside these filters | Your filters are hiding it |
| Before coverage began | You are looking at a window before Lumaft first observed. That interval is unknowable |
| Stacks show but no operations for a non-Pulumi backend | Expected; update history is a Pulumi-only tier |
Coverage starts when Lumaft starts. History the backend pruned before that is gone and is never reconstructed. See Reading the operation timeline.
Storage and disk
| Symptom | Cause | Fix |
|---|---|---|
/api/v1/storage-health reports writeProbe.outcome: failed |
The database can no longer accept writes — usually a full volume | Free space or grow the volume; readiness deliberately does not flap on this |
| A large purge did not free disk | Deletion frees bytes inside the SQLite file; the file never shrinks by itself | Run the offline VACUUM in Database integration |
lumaft.db-wal and lumaft.db-shm exist after a stop |
The stop was not clean (killed, or the grace period was too short) | Stop with docker stop -t 30; the next clean start checkpoints the WAL |
lumaft.db-wal appears after a read-only diagnostic command |
A read-only connection cannot checkpoint on close | Harmless; the next clean start removes it |
| The database is on EFS/NFS/Azure Files and behaves erratically | Network filesystems are outside the supported SQLite topology | Move /data to a block volume, or adopt PostgreSQL |
Runner integration
| Symptom | Cause | Fix |
|---|---|---|
lumaft exec: evidence delivery failed … network-unreachable |
The runner cannot reach LUMAFT_URL. The deployment ran unchanged |
Check DNS, TLS trust (NODE_EXTRA_CA_CERTS for a private CA), and egress from the runner |
The wrapper exits 78 before running anything |
Misconfiguration: missing LUMAFT_URL, both or neither token sources, http:// off loopback, token file outside the home directory |
Fix the setting the message names |
The wrapper exits 64 before running anything |
The invocation could not be parsed | Check the -- separator and the supported commands |
401 unauthorized from the ingestion API |
Revoked, expired, or mistyped token | Issue a successor token and revoke the old one |
404 not-found for a stack you can see in the console |
The token's scope does not cover this backend/project/stack | Issue a token scoped to it; out-of-scope and nonexistent look identical by design |
The operation shows incomplete |
The runner announced the run and never finalized; its lease expired | Check the runner's own logs; the outcome is honest, not an error |
| Two rows for one deployment | One from the backend, one from the runner | Expected in this build |
Upgrade refusals
Every refusal is described with its fix in Startup failure messages. The two you will see most:
- A gated migration. Take a cold backup, start once with
LUMAFT_SCHEMA_UPGRADE_APPLYset to the named revision, remove it afterwards. - An older image against a newer volume. Start the newer image. If you truly need the older one, restore the pre-upgrade backup.
Reporting a problem
Safe to include:
- The Lumaft-authored log line, verbatim.
- The image version and digest.
- The schema revision from the version stamp.
/api/v1/health,/api/v1/readiness, and the typed outcomes from/api/v1/storage-health.- The shape of your backends file with bucket, prefix, and endpoint values removed.
Never include the database or a backup, an export document, Pulumi state or outputs, any credential or token, or infrastructure identifiers you would not publish. Lumaft produces no support bundle; collect the items above by hand.