Connect a backend

Marolum has two integration points with your infrastructure:

The two never share a credential. The identity that reads state cannot write evidence, and the token that writes evidence cannot read anything.

State backends

Choose the engine and storage type, then confirm the actual namespace and layout. For S3, collect bucket, prefix, region and Pulumi layout using Find your backend details. The additional storage sections below explain the other providers and native workspace layouts. An incorrect layout or namespace can make a healthy backend show no stacks.

Add a cloud backend in the console

1. Open Administration → Backends → Add backend with backends:manage.
2. Choose the engine and S3, Google Cloud Storage or Azure Blob storage. Enter the storage coordinates and engine-specific options using the existing read-only installation identity.
3. Select Test connection, then Add and observe. Changing the form requires a new test; saving repeats the test before committing the backend.
4. Check observation health after the first cycle. A successful bounded list/read test does not prove every object is readable, and an empty namespace can pass.

Console additions persist in Marolum's database and become available without a restart. Back up that database along with startup configuration and external credentials. The form cannot accept credentials, grant cloud permissions or select arbitrary endpoints. Custom S3 endpoints must already be trusted by startup configuration. File-owned records cannot be replaced in the UI; this flow adds records and does not provide general editing or removal.

PostgreSQL state sources remain operator-configured in the protected startup file. Filesystem sources can also use the allowed-directory setup described below when the installed release provides it. The combined file/database limit is 100 records, independent of edition; different configurations cannot reuse an existing ID.

The backends file

Backends are declared in one JSON file named by MAROLUM_BACKENDS_FILE. Marolum reads it exactly once at startup, validates the whole document, and refuses to start if any record is invalid. Changing this file requires a restart. When it is unset, the file-owned set is empty; durable console additions remain available.

{
  "schemaVersion": 1,
  "backends": [
    {
      "kind": "s3",
      "id": "production",
      "displayName": "Production infrastructure",
      "bucket": "pulumi-state-bucket",
      "prefix": "team/platform",
      "layout": "project-scoped",
      "region": "us-east-1",
      "forcePathStyle": false,
      "enabled": true
    }
  ]
}

Limits: at most 100 records, file size at most 1 MiB, UTF-8, no unknown fields, no repeated id. Credentials of any kind in this file are rejected.

Layouts

Each Pulumi backend names exactly one layout. Marolum does not guess.

With prefix team/platform, the Pulumi directory is team/platform/.pulumi/. A bucket that holds both layouts, or several prefixes, needs one backend record per layout and prefix.

Pulumi has deprecated the legacy layout and intends to remove it. Marolum reads legacy state directly and is unaffected, but treat a legacy backend as something to migrate.

S3-compatible stores

Add an endpoint and, usually, forcePathStyle: true:

{
  "kind": "s3",
  "id": "private-cloud",
  "displayName": "Private cloud infrastructure",
  "bucket": "pulumi-state-bucket",
  "layout": "legacy",
  "region": "local",
  "endpoint": "https://s3.internal.example",
  "forcePathStyle": true,
  "enabled": true
}

The endpoint must be HTTPS with no path, query string, fragment, or embedded credentials. Plain HTTP is accepted only for localhost, 127.0.0.1, and [::1].

Terraform and OpenTofu backends

A terraform or opentofu record has no layout. For S3, Marolum observes version-4 state at env:/<workspace>/<project>/terraform.tfstate and <project>/terraform.tfstate beneath the prefix, and presents each as a project-scoped stack with its resources, non-sensitive outputs, serial, and engine version.

{
  "kind": "s3",
  "engine": "terraform",
  "id": "terraform-platform",
  "displayName": "Terraform platform state",
  "bucket": "terraform-state-bucket",
  "prefix": "platform",
  "region": "us-east-1",
  "readVersions": true,
  "enabled": true
}

• Always set a prefix. Without one the IAM list condition cannot be narrowed and the grant covers the whole bucket.
• readVersions: true walks every retained object version the bucket still holds, oldest first, so each serial becomes a witnessed transition instead of one collapsed diff. It requires the version-read permissions below.
• The state key must be <project>/terraform.tfstate, optionally under the default env:/<workspace>/ prefix. A custom workspace_key_prefix or a multi-segment key is not observed.

Engine support bounds. State observation is verified against Terraform 1.15.9 and OpenTofu 1.12.6; other builds are unverified. Terraform and OpenTofu can write native S3 lock files when locking is enabled, but Marolum does not currently observe them. Backend-native update history remains unavailable for those engines, so lock and update-history observation are Pulumi-only and these backends answer engine-unsupported for both rather than showing empty results.

Google Cloud Storage and Azure Blob

All three engines have adapters for both providers; live provider qualification remains outstanding. Use controlled evaluation sources. For GCS use kind: gcs and the existing bucket; for Azure use kind: azure-blob, account and bucket (the container name). Both accept an optional relative prefix. Omit S3-only region, endpoint and forcePathStyle fields. Pulumi requires its actual layout and must omit readVersions.

Keep credentials on the installation. Google uses application default credentials with only storage.objects.list and storage.objects.get on the intended bucket. Azure uses DefaultAzureCredential and a container-scoped Storage Blob Data Reader role. Sovereign clouds and custom/emulator endpoints are outside these adapters.

For Terraform/OpenTofu, set a stable projectName and omit layout. GCS reads [prefix/]workspace.tfstate. Azure also requires stateKey: default state is [prefix/]stateKey, and named workspaces use [prefix/]stateKeyenv:workspace. Retained object versions are opt-in with readVersions; reads are pinned to the provider generation/version.

Linux filesystem state

Use kind: filesystem with an absolute normalized root directory mounted read-only into Marolum. bucket is a logical label, not the path. For Pulumi, the root contains .pulumi and the record supplies the real layout. For Terraform/OpenTofu, set projectName, omit layout, and match the relative stateKey and workspaceDirectory to the engine configuration; defaults are terraform.tfstate and terraform.tfstate.d. Named workspace files are [prefix/]workspaceDirectory/workspace/terraform.tfstate.

The source is separate from Marolum's writable data volume. Symlinks, traversal, multiply linked state files and special files are refused. Network filesystems have no acceptance evidence. readVersions: true is refused; Marolum does not create, acquire or remove engine locks.

Add an allowed filesystem directory in the console

Use this path only when your installed release offers Allowed state directory in Add backend. Older images retain the startup-file route above; follow the frozen guide matching your image.

The installation operator first mounts the required state directories read-only and sets MAROLUM_MANAGED_FILESYSTEM_ROOTS to a JSON array of exact absolute Linux paths, for example ["/srv/state/team-a", "/srv/state/team-b"]. No directory is allowed by default. The setting accepts at most 32 paths within 16 KiB, is read at startup, and must match across replicas. Changing it requires a restart. The setting does not create a mount or grant OS access.

With backends:manage, open Add backend, choose Linux filesystem and select one allowed directory. Enter a display name, unique backend ID, logical storage label, and the engine's layout or workspace fields. A descendant is not automatically an allowed new root; use the relative prefix within a selected root or ask the operator to allow a separate directory. Test the connection, then add and observe. A changed root or other form entry requires a new test, and a new addition rechecks access before saving. No password is entered or stored by this flow.

Saved definitions live in Marolum's database and become available to replicas without restart. Allowed path choices require management permission; ordinary backend views do not expose host paths. Back up the database and retain the deployment's root policy and mounts. A restored managed definition is rechecked against that policy. Removing an allowed root while it is still referenced fails configuration closed; this version provides no general UI editing or deletion.

If the save response is lost, retry the same entries. An identical UI-owned filesystem definition is returned without a duplicate source or add audit. A changed definition or a startup-owned ID still conflicts. This retry confirms saved configuration; it does not claim a new connection check or fresh observation. Check observation health separately.

PostgreSQL state sources

This connection reads infrastructure state and is separate from the database storing Marolum's own history and accounts. Configure kind: postgres, host, optional port (5432), database, schema, table, credentialsFile and a logical bucket label in the protected startup file. Use a dedicated identity with CONNECT on that database, USAGE on the schema and SELECT on the existing state table. Marolum uses read-only transactions, never the engine's writer credentials.

The external credential file contains user, password and optionally ca with a PEM trust certificate. It must be a protected regular file, mode 0600 or stricter, without symlinks or a group/world-writable parent. Verified TLS is the default; ssl: false is limited to explicit loopback testing. Do not paste connection credentials into the UI or backend JSON.

Pulumi defaults to public.pulumi_state; set its actual layout and native key prefix. A native Pulumi URL ending in /infrastructure also writes keys beneath infrastructure/, so the Marolum prefix must match. Terraform defaults to terraform_remote_state.states; its table must remain states. OpenTofu also permits a configured table. For both, supply projectName, omit layout and prefix, and use the existing engine-initialized table. Workspaces come from its name column and state from data. Retained-version walking is unavailable on PostgreSQL.

File rules

MAROLUM_BACKENDS_FILE is interpreted literally: no expansion, no interpolation. The path must:

• be absolute and normalized;
• name a regular file, opened without following a final symlink, with a non-symlinked parent;
• have a parent directory that is not writable by group or others;
• not be writable by group or others.

Mode 0600 owned by UID 1000 is the recommended shape. Kubernetes-style projected volumes, which expose files through symlinks, are not supported; write the file into a task-local volume from an init step instead. Each environment guide shows this.

Credentials

Marolum reads S3 through the standard AWS SDK credential provider chain. It does not have its own credential setting. In order, the SDK looks for:

1. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN
2. A web identity token (AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN)
3. The ECS task role (container credentials endpoint)
4. The EC2 instance profile (instance metadata)

Prefer the platform identity: an ECS task role, an EC2 instance profile, or a federated role. Use static keys only where no workload identity exists, scope them to the policy below, and rotate them. Never place any of these values in the backends file or in the image.

For S3-compatible stores, the same environment variables carry the store's access key pair.

IAM policy

Marolum sends ListObjectsV2 and GetObject. That is the whole request surface for base observation; no configuration makes it write.

Pulumi backend. Replace BUCKET_NAME and OPTIONAL_PREFIX/; remove the prefix placeholder entirely when the backend has none.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListPulumiState",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET_NAME",
      "Condition": { "StringLike": { "s3:prefix": ["OPTIONAL_PREFIX/.pulumi/*"] } }
    },
    {
      "Sid": "ReadPulumiState",
      "Effect": "Allow",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::BUCKET_NAME/OPTIONAL_PREFIX/.pulumi/*"
    }
  ]
}

Terraform or OpenTofu backend. State objects sit directly beneath the prefix, so the grant scopes to the prefix itself.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListEngineState",
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:ListBucketVersions"],
      "Resource": "arn:aws:s3:::BUCKET_NAME",
      "Condition": { "StringLike": { "s3:prefix": ["OPTIONAL_PREFIX/*"] } }
    },
    {
      "Sid": "ReadEngineState",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::BUCKET_NAME/OPTIONAL_PREFIX/*"
    }
  ]
}

Drop s3:ListBucketVersions and s3:GetObjectVersion when readVersions is not set.

Additional grants

• SSE-KMS buckets: kms:Decrypt on that specific key, and the key policy must permit it.
• Several buckets or disjoint prefixes: one scoped statement pair per namespace.
• S3-compatible stores: map the equivalent bucket-list and object-read operations for the configured namespace.

This observer policy is a permanent boundary. Marolum never needs, and never asks for, bucket creation, object write, overwrite, delete, or any deployment permission.

Activation and observation

An enabled backend starts observation immediately after bootstrap. Marolum runs three schedulers per backend:

• Every cadence carries 10 % jitter. A failed cycle backs off exponentially, from twice the base delay to a 15-minute ceiling, and resets after a clean cycle.
• All three schedulers share one installation-wide concurrency budget of two backends at a time.
• A credential, network, or authorization failure is isolated to its backend. It never affects readiness or other backends.
• A disabled record is visible with enabled: false and is skipped without contacting S3. Disabling or removing a record keeps its cached projection; Marolum does not purge it automatically. What every later change does (disable, remove, re-add, rename, migrate a layout) is in Changing your backends safely.

Verify the connection

1. Open Administration → Backends → Connection diagnostics. Reachable means the connection check reached the configured state namespace. It does not prove that a checkpoint parsed successfully or that every object is readable. This view requires backends:diagnostics:read.
2. Open Overview → Backends to inspect observation health. Expand a backend to see its engine and storage details, including its Pulumi layout where applicable. Healthy, Partial and Failed describe observed stack health; they are separate from connection diagnostics. GET /api/v1/backends returns configured metadata, including disabled records, and requires backends:read; it is not a diagnostic or observation result.
3. Open Stacks after the first checkpoint cycle commits. Open the intended stack and check its observation time and whether the displayed snapshot is current or retained from the last successful read.

Check each configured engine and namespace independently. The controls illustrated in Your first hour do not establish acceptance of a different engine or storage provider.

The diagnostic runs once after bootstrap, asynchronously, and does not gate readiness. There is no live reload of the startup file; file edits require restart. Console additions are durable and observed without restart, as described above.

Observation health

Administrators can read the process-local scheduler report at GET /api/v1/observation-polling (requires storage:health:read). It reports completed-cycle counts, sanitized failure classes, current backoff depth, and the actual jittered delay per scheduler, with no backend identity or provider payload. Counters reset on restart.

Troubleshooting

Runner evidence (optional)

A backend record can tell Marolum that an update completed. It cannot tell Marolum that a run started, was cancelled, or was a preview. Runner integration closes that gap: your pipeline or developer machine reports the operation to Marolum's ingestion API as it runs, using a separate write-only token.

It is optional, it never changes a Pulumi command's exit code, and it is documented on its own page: Runner integration.