Operations

1. Deployment modes

Mode	Database	Objects	Replicas	Use
Single node, embedded (default)	Bundled Postgres started by strataplane	Directory	1	Evaluation, labs, small and medium fleets (≤ ~10k agents). One package or container, one data directory.
Single node, external	DATABASE_URL to an existing Postgres	Directory or S3	1	When the organisation already runs Postgres with backups and monitoring.
HA	External Postgres with its own HA (Patroni, CloudNativePG, managed)	S3	2+ behind a load balancer	Large fleets, no single point of failure in the control plane.

All modes run the same binary, schema and code paths, and every mode can host several tenants (architecture.md §12). A multi-tenant service runs in HA mode.

2. Embedded Postgres

Implementation: postgresql_embedded with the bundled feature (stack.md). The Postgres archive for the target platform is embedded in the binary at build time (38–98 MiB depending on target), so nothing is downloaded at runtime.

Aspect	Behaviour
Layout	/var/lib/strataplane/pg/<major>/ (binaries extracted once), /var/lib/strataplane/pgdata/ (cluster).
Network	Loopback only, on a port chosen at each start; password authentication.
Credentials	Superuser password generated at initdb, created 0600 in the data directory (itself 0700); the app connects as an owner role of the strataplane database, not as superuser.
Lifecycle	strataplane serve runs initdb on first start, then starts the postmaster, waits until it accepts connections, runs migrations, then opens listeners. On SIGTERM: stop accepting, let in-flight requests finish (at most 30 s), write buffered agent liveness, pg_ctl stop -m fast.
Crash safety	strataplane holds a lock on strataplane.lock in the data directory while it runs; a second process on the same directory aborts start. With the lock held, a stale postmaster.pid is removed and a live postmaster, the orphan of a killed process, is stopped. Running under systemd with KillMode=mixed keeps Postgres in the service cgroup; in the container it is a child of PID 1 and stops with it.
User	Postgres refuses to run as root; the service runs as strataplane, the container image as a non-root user.
Tuning	At each start, shared_buffers (a quarter of the memory, 128 MB–8 GB), max_connections (memory / 16 MB, 50–400) and work_mem derived from the available memory (the cgroup limit if lower than the machine's); database.postgres sets these or any other server setting.
Monitoring	/metrics exports the database size, connections and checkpoint stats from pg_stat_*.

Backups (embedded):

• strataplane db backup --to FILE runs the bundled pg_dump -Fc over the socket; --encrypt-to AGE_RECIPIENT encrypts the dump with age.
• Scheduled backups via config (backup.schedule, an interval such as 24h; backup.dir, backup.keep, backup.recipients for age encryption). In HA, one replica takes each scheduled backup (claimed in the database). A dump is written as .partial and renamed when complete; keep counts complete dumps only.
• The container image has pg_dump 18, so backups also work against an external database.
• strataplane db restore FILE restores into an empty database (service stopped) without migrating it first; the next serve migrates a dump of an older version, a dump of a newer version is refused.
• Secret values in dumps stay encrypted. Restores need the secret key, which is not part of the dump: back it up separately (§6).
• The dump holds no objects. Back up blobs.dir (or the bucket) as well; a restore with older objects shows the missing evidence as unavailable (§9).

Postgres major version. 1.0 bundles Postgres 18. The binary refuses to start on a data directory whose cluster has another major version; such a cluster moves with db backup and db restore.

Moving to external Postgres:

strataplane db backup --to sp.dump                    # embedded, service running
pg_restore -d "postgres://strataplane@db.example.com/strataplane" sp.dump
# set DATABASE_URL in the service environment, restart

3. External Postgres and HA

• Postgres ≥ 15. No extensions, no superuser. The app role owns the strataplane schema; migrations run under a Postgres advisory lock, so replicas can start concurrently. strataplane db status lists applied, pending and unknown migrations; db migrate applies them, for deployments that start replicas with serve --no-migrate.
• Connection: DATABASE_URL with sslmode=verify-full recommended. Pool size per replica configurable (default 20).
• PgBouncer: session pooling only. Transaction pooling breaks LISTEN/NOTIFY and session advisory locks. Alternatively, give the listener and the shard locks their own direct connection (DATABASE_DIRECT_URL). The tenant scope is set with SET LOCAL per transaction, which works with either pooling mode.
• Controller shards: each tenant belongs to one of controllers.shards shards (default 16), assigned at creation to the shard with the fewest agents. A replica runs a shard's controllers (reconciler, rollout, status scheduler, placement, jobs) while it holds pg_advisory_lock(<shard>) on a dedicated connection, records on it that the shard runs (controller_shard), and takes at most ceil(shards / live replicas) shards, so shards spread over replicas. If that connection drops, the replica stops the shard's work immediately and the lock passes to another replica. /api/v1/health/ready reports the shards a replica runs. The shard count is stored in the database by strataplane init (or the first start) and fixed from then on; a different configured value is ignored with a warning. Installation-wide jobs (blob sweeper, pruning of operations and sessions) run on the replica holding shard 0.
• All replicas serve traffic. Agent and API requests are stateless apart from per-replica caches: the agent token cache (entries live 30 s; revoking or rotating a credential and forgetting an instance drop them on every replica by NOTIFY, and a listener reconnect drops all) and the desired_since cache, which answers Telegraf's watcher only while the database is unreachable. Elastic checkin long-polls can land on any replica.
• Load balancer: TCP passthrough on the agent listener when agents use mTLS (client certs must reach Strataplane); sticky sessions are not required. Health check /api/v1/health/ready. Agent source addresses drive network zones and Telegraf class mode: a load balancer in front of the agent listener passes them by PROXY protocol v2, or an HTTP proxy (no mTLS through it) by X-Forwarded-For, from agent_proxy_from.
• Host names: one for the management listener (public_url) and one for the agent listener (agent_url), for all tenants.

4. Configuration

Configuration comes from /etc/strataplane/strataplane.toml, overridable by STRATAPLANE_* environment variables.

Platforms. The server runs on Linux only: deb/rpm with a systemd unit, or the container image. Agents, runners and stratactl also run on Windows.

[server]
management_listen = "0.0.0.0:8443"
management_proxy_from = []           # CIDRs of reverse proxies whose X-Forwarded-For gives the client address (audit source_ip)
management_tls    = true             # false: plain HTTP on the management listener behind a TLS-terminating ingress
agent_listen      = "0.0.0.0:8444"
public_url        = "https://strataplane.example.com"        # UI and management API of every tenant; OIDC redirect URI base
agent_url         = "https://strataplane.example.com:8444"   # rendered into agent configs, all tenants
tls_cert = "/etc/strataplane/tls/server.pem"   # re-read when the files change
tls_key  = "/etc/strataplane/tls/server.key"
agent_client_auth = "request"        # none | request | require
agent_proxy_from  = []               # CIDRs whose PROXY protocol v2 header or X-Forwarded-For gives the agent's address

[database]
# url = "postgres://…"               # unset → embedded; or DATABASE_URL
# direct_url = "postgres://…"        # listener and shard locks past a pooler; or DATABASE_DIRECT_URL
pool_size = 20                       # per replica
data_dir = "/var/lib/strataplane"
# postgres = { shared_buffers = "2GB" }  # embedded only: server settings over the derived ones

[auth]
oidc_issuer = "https://id.example.com"
oidc_client_id = "strataplane"
oidc_client_secret_file = "/etc/strataplane/oidc-secret"
oidc_audiences = []                  # other audiences an ID token may carry besides the client id (Zitadel: the project id); service mode adds billing.project_id
groups_claim = "groups"              # self-hosted: teams synced at login
admin_group = "strataplane-admins"   # group of a new tenant's admins team
local_users = true                   # break-glass admin
session_hours = 12                   # web session lifetime
require_verified_email = false       # refuse OIDC logins without email_verified: true; always on in service mode
platform_admins = []                 # OIDC subjects that are platform admins; when set, each OIDC login sets the flag from it

[tenants]
default_limits = { agents = 10000, object_bytes = "50GiB", chat_bytes = "5GiB", api_rate = 50 }

[controllers]
shards = 16                          # fixed at init
workers = 4                          # operation workers per replica
tenant_operations = 2                # operations one tenant runs at once

[egress]
deny = ["127.0.0.0/8", "::1/128", "169.254.0.0/16", "fe80::/10"]
own_networks = true                  # also deny the networks of the server's interfaces

[blobs]
backend = "fs"                       # fs | s3
dir = "/var/lib/strataplane/blobs"   # fs
# s3 = { endpoint = "https://s3.eu-central-1.amazonaws.com", region = "eu-central-1", bucket = "strataplane", prefix = "" }
# S3 credentials: STRATAPLANE_S3_ACCESS_KEY_ID / _SECRET_ACCESS_KEY, or the platform's workload identity

[secrets]
key_file = "/etc/strataplane/secret.key"   # or STRATAPLANE_SECRET_KEY; generated by `strataplane init`

[catalog]
cache_dir = "/var/lib/strataplane/catalog"   # local copies of validator binaries
trusted_keys = ["/etc/strataplane/catalog-signing.pub"]

[backup]
# schedule = "24h"                   # interval; unset → no scheduled backups
# dir = "/var/lib/strataplane/backups"
# keep = 7
# recipients = ["age1…"]             # age-encrypt the dumps

# [billing]                          # service mode (architecture.md §12); absent: self-hosted, one tenant
# url = "https://account.example.com"   # billing service: API under /api/v1/, account UI for people
# project_id = "…"                   # Strataplane's Zitadel project id: the audience of billing's pushes
# provisioning_subject = "…"         # sub of billing's provisioning machine user, the only caller of /internal/billing
# client_id = "strataplane-billing-client"     # Strataplane's machine user for the billing API (client credentials)
# client_secret_file = "/etc/strataplane/billing-client-secret"
# billing_project_id = "…"           # the billing service's Zitadel project id (requested audience)

Every key can be set by environment: STRATAPLANE_<SECTION>_<KEY>, the value parsed as a TOML value when it is one (STRATAPLANE_AUTH_PLATFORM_ADMINS='["1234"]', STRATAPLANE_BLOBS_S3='{ endpoint = "https://…", region = "fsn1", bucket = "strataplane" }') and as a string otherwise.

Kubernetes. The container image runs unprivileged (uid 10001) with a read-only root filesystem when /tmp and database.data_dir are writable volumes (emptyDir with external Postgres and S3). A typical service-mode deployment:

• configuration entirely from the environment: DATABASE_URL, STRATAPLANE_SECRET_KEY (from a Secret; the same on every replica), STRATAPLANE_SERVER_PUBLIC_URL, STRATAPLANE_SERVER_AGENT_URL, STRATAPLANE_AUTH_*, STRATAPLANE_BLOBS_* with STRATAPLANE_S3_ACCESS_KEY_ID/_SECRET_ACCESS_KEY, and the [billing] keys; secrets read from files (oidc_client_secret_file, billing.client_secret_file) come from mounted Secrets; STRATAPLANE_AUTH_LOCAL_USERS=false with auth.platform_admins naming the staff who administer the platform;
• the management listener serves plain HTTP (STRATAPLANE_SERVER_MANAGEMENT_TLS=false) behind the ingress, which terminates TLS for public_url and sets X-Forwarded-For (server.management_proxy_from = the ingress's address range). The ingress does not route /internal/billing/*; the billing service calls the service inside the cluster. The agent listener terminates TLS itself (tls_cert/tls_key from a certificate Secret) behind a TCP or TLS passthrough, so agent client certificates reach Strataplane;
• strataplane init as a Job (or init container) before the first rollout: it checks the secret key against the database, fixes the shard count and, in service mode, creates no tenant. It is idempotent; serve runs the migrations itself under an advisory lock;
• probes: liveness /api/v1/health/live, readiness /api/v1/health/ready on the management port; a rolling update with maxUnavailable: 0 keeps agents served.

The container image's config (packaging/strataplane.toml) keeps the secret key out of the data volume: pass STRATAPLANE_SECRET_KEY or mount the key file, and mount the catalog signing key at trusted_keys.

5. Air-gapped installs and catalog bundles

Nothing is fetched at runtime. What would otherwise be downloaded arrives as a catalog bundle per agent kind and version:

telegraf-1.40.2.spbundle (tar.zst, reproducible, signed)
  manifest.json                 # format, kind, version, files (path, size, sha256), plugins, wasm modules, validators
  signature                     # ed25519 over the bytes of manifest.json, with the key id
  schemas/<section>/<plugin>.json   # JSON Schema 2020-12 per plugin, generated by telegraf-schema
  samples/<section>/<plugin>/sample.conf, docs/<section>/<plugin>/README.md
  wasm/telegraf-parsers.wasm, wasm/telegraf-parsers-parquet.wasm   # parser modules ("parsers", "parsers-parquet")
  validator/linux-{amd64,arm64}/telegraf   # stock binaries for `telegraf config check`

elastic-agent-9.5.5.spbundle
  manifest.json, signature, policy/schema.json (policy core structure),
  components/*.spec.json (input types per component, converted from the
  agent package's spec YAML), supported versions, validator image reference

vector-0.60.0.spbundle
  manifest.json, signature,
  schemas/<section>/<type>.json, schemas/global/options.json, schemas/definitions.json
                                # split from `vector generate-schema`, docs/<section>/<type>.md
  wasm/vector-vrl.wasm          # VRL module ("vrl")
  validator/linux-{amd64,arm64}/vector   # static stock binaries for `vector validate`

strataplane-agent-1.0.0.spbundle
  manifest.json, journey API (L2 checks, assistant .d.ts),
  packages/ (MSI, deb, rpm, macOS tarball for local runs), image/ (OCI tarball with a pinned headless Chromium)

• strataplane catalog import FILE (or a platform admin in the UI) verifies the signature against trusted_keys, stores schemas in the database and wasm modules and validator binaries as objects (§9). Every replica copies the validators it runs into catalog.cache_dir. Bundles serve all tenants.
• Bundles are built in Strataplane's CI from upstream release tags. Operators can build their own with tools/build-bundle for patched agents.
• Without a bundle for an agent's version, Strataplane uses the nearest older one and flags the agent (catalog_mismatch).
• Journey runners are installed by the operator's own tooling from the strataplane-agent bundle's packages. The Linux package uses a system Chromium or the bundle's image; nothing is downloaded at runtime. Runners run as an ordinary service; no interactive session is needed.

6. Secrets

• Key hierarchy. The deployment key encrypts nothing but the tenants' data keys (tenant_key). Each tenant's data key encrypts its sealed values (secret versions, LLM provider API keys, audit sink credentials; sealed_value) and objects (AES-256-GCM, with the tenant id in the associated data, so ciphertext copied into another tenant never decrypts). Destroying a tenant's data keys makes everything it left in backups and buckets unreadable.
• strataplane init generates the deployment's 256-bit secret key and writes it to secrets.key_file (0400, outside the data directory: /etc/strataplane/secret.key), or prints it for STRATAPLANE_SECRET_KEY (containers, HA replicas). All HA replicas need the same key.
• The key file is a small keyring, one <id> <base64 key> line per key: the highest id wraps, every key unwraps. The database stores only key fingerprints and, per data key, the deployment key id. Losing the key loses all secret values.
• Back the key up separately from database backups: strataplane secrets export-key prints it once and records the export; readiness warns until that happened.
• A wrong or missing key is detected at start by fingerprint; the server then reports not ready, neither delivers nor accepts secret values and creates no tenants, since a new tenant's data key could not be wrapped.
• Rotation of the deployment key: strataplane secrets rotate-key adds a new key to the keyring file, re-wraps all data keys in one transaction and retires the old key. In HA, --add-only only adds the key (or, with STRATAPLANE_SECRET_KEY, prints the line to add to the variable); after the new keyring is on every replica, rotate-key re-wraps under that added key instead of adding another.
• Rotation of a data key: strataplane secrets rotate-key --tenant SLUG adds a data key, re-encrypts the tenant's sealed values in one transaction and retires the old key once the blob sweeper has re-encrypted the tenant's objects.
• Rotating a value is writing a new version; it is a render input, so the agents that reference it get a new artifact without a rollout (architecture.md §4).

7. Upgrades of Strataplane

• Minor versions: rolling restart of replicas. Migrations are backward-compatible for one minor version (expand/contract), so old and new replicas can run side by side during the roll: an old replica starts against a database migrated by a newer one only if each of its unknown migrations is marked compatible.
• The agent wire protocols are stable within a major version.
• Upgrades change no rendered bytes; new renderer behaviour arrives as a render level that tenants adopt by change set (architecture.md §4). A major upgrade starts only when no release pointer, pin or active rollout of any tenant is on a level the new major drops.

8. Sizing (initial targets)

Fleet	Mode	CPU / RAM	Notes
≤ 2,000 agents	embedded	2 vCPU / 4 GB
≤ 10,000 agents	embedded or external	4 vCPU / 8 GB	heartbeat 1 min, Elastic poll timeout 4 min
≤ 50,000 agents	HA, external	3 × 4 vCPU / 8 GB, Postgres 8 vCPU / 32 GB

Journey runners add little: 200 runners check in every 60 s (≈ 3 requests/s) and post one result per run; 500 journeys at 5-minute intervals on four runners each are ≈ 7 results/s.

Load per 10,000 agents: Telegraf ≈ 170 heartbeats/s and ≈ 170 HEAD/s at 1-min intervals (both coalesced or cache-served); Elastic Agent ≈ 40 checkins/s with 4-min long-polls and ≈ 10,000 open connections across replicas.

Tenants share replicas and Postgres: size for the sum of their agents. Tenants add little by themselves (a few rows each); the controller shards bound how many tenants one replica reconciles.

Table volumes for 10,000 agent instances, 500 spec files and one spec change per hour:

Table	Rows	Notes
tenant, tenant_key	tenants
host, agent_instance	10k	updated in batches (heartbeats coalesced)
class_served	≤ class client addresses	written only when the artifact served to an address changes
host_label	~200k	~20 labels per host and source
spec_blob	grows ~1–5 per change	small TOML files
artifact	~distinct renders × retention	typically far fewer than agents, since hosts with the same layers and agent version render identically
change_set_impact	up to 10k per open change set	deleted when the change set is merged or closed
agent_status_event	transitions only	partition or prune by age
agent_log_entry	≤ 200 per agent
secret_delivery	one per agent reload per referenced secret	prune by age after export
journey_state	journeys × matching runners (e.g. 500 × 4)	updated per run
journey_receipt	results of the last 24 h (≈ 600k at 7/s)	pruned after 24 h
journey_run, journey_evidence	≤ 20 failed and 200 passed runs per journey (defaults), about 1–5 KiB per row: ~0.5 GiB for 500 journeys	evidence is objects, failed runs only, ≤ 2 MiB per run, ~20 GiB worst case for 500 journeys, typically far less (one screenshot and snapshot per run)

9. Object storage

Large binaries are objects, not rows:

Object	Key	Written by
Journey evidence	tenants/<tenant>/evidence/<run id>/<idx>	runner result upload (journeys.md §6)
Parser modules	catalog/<bundle hash>/wasm/<module> (parsers, parsers-parquet)	catalog import (§5)
Validator binaries	catalog/<bundle hash>/validator/<os>-<arch>	catalog import

• Backends: fs, a directory on the server (single node), and s3, any S3-compatible API (AWS S3, MinIO, Ceph RGW, …), required for HA. Both need only put, get, delete and list by prefix.
• Rows are the index. Each object has a row with size and SHA-256 (catalog_bundle, journey_evidence). The object is written before its row commits; keys derive from row ids, so a retried upload overwrites its own object. Deleting rows enqueues their keys in blob_delete, and the blob sweeper deletes them; hourly it also deletes objects under tenants/ and catalog/ older than a day that no row references.
• Encryption. Tenant objects are encrypted with the tenant's data key (AES-256-GCM per object, the key id and nonce in the object header), so a bucket alone exposes nothing and a deleted tenant's objects stay unreadable. Catalog objects are signed, not encrypted.
• Reads go through Strataplane: downloads stream from the store after the API's authorization check; no presigned URLs.
• Quota. Object bytes count against the tenant's object_bytes limit (architecture.md §12).
• Air-gapped installs use fs or an on-premises S3 such as MinIO.