Telegraf module

Verified against Telegraf 1.40.2 (2026-10-22) with the spike in spikes/telegraf-delivery, unless noted.

1. What stock Telegraf gives us

Mechanism	Behaviour (verified)	Since
--config https://…	GET with Accept: application/toml and User-Agent: Telegraf/<version> Go/<version>. Sends Authorization: Bearer $TELEGRAF_CONTROLLER_TOKEN, or Token $INFLUX_TOKEN as fallback. Retries --config-url-retry-attempts times after the first attempt (default 3: four attempts 10 s apart), then exits; this holds for the fetch on reload too. No request timeout.	URL config: old; controller token: 1.39.0
--config-url-watch-interval	Every interval a HEAD per URL with the same auth (no Accept). Compares the Last-Modified string with the value of the watcher's first HEAD after start or reload, not the GET's; any difference, in either direction, sends SIGHUP (full reload). The status code is ignored. A response without Last-Modified resets the reference: the next value becomes the new one, so a change during the gap is missed. Connection errors keep the reference. No request timeout.	1.31 (2024-06)
Reload	Stops the running plugins, then re-reads all config sources and re-initialises secret stores. An invalid or unreachable config makes Telegraf exit, which systemd or the Windows service manager then restarts.	–
[[outputs.heartbeat]]	POST JSON (schema_v1.json): id, version, schema, optional hostname (os.Hostname()), statistics (errors, warnings, metrics since last success), configurations (redacted sources), logs, status (OK, WARN, FAIL, UNDEFINED from CEL conditions), last (unix seconds, only after a failed send). include defaults to ["hostname"]. First sent one interval after start or reload; Retry-After is ignored. Bearer token (secret-capable), custom headers (secret-capable), full HTTP client TLS (mTLS with PEM).	1.37.0; logs and status: 1.38
[[secretstores.http]]	One GET per @{…} reference at load and reload, returning {key: value} JSON; mTLS with PEM, bearer/basic, optional AES-CBC. A non-200 response or a missing key aborts loading, and Telegraf exits.	1.27
telegraf config check	Parses and loads a config without running it.	1.32 (2024-08)
Env substitution	${VAR} and $VAR in the config are replaced from the process environment before parsing; an unset variable stays literal text.	–

Minimum supported Telegraf: 1.39 (controller token header). Strataplane does not use the INFLUX_TOKEN fallback; agents reporting an older version are flagged unsupported_version and receive no config.

2. Bootstrap on the host

# One-time enrollment (installer, Puppet, Ansible, MSI custom action):
curl -sf --cert /etc/telegraf/identity/cert.pem --key /etc/telegraf/identity/key.pem \
  -H "Authorization: Bearer $ENROLLMENT_TOKEN" -H "Accept: text/plain" \
  -d '{"hostname":"web01.example.com"}' \
  https://strataplane.example.com:8444/agent/v1/telegraf/enroll \
  > /etc/telegraf/strataplane.env        # TELEGRAF_CONTROLLER_TOKEN=spt_…, STRATAPLANE_INSTANCE_ID=…

# Service:
telegraf --config https://strataplane.example.com:8444/agent/v1/telegraf/config \
         --config-url-watch-interval 1m --config-url-retry-attempts -1

• --config-url-retry-attempts -1 keeps an agent retrying instead of exiting when the server is briefly unreachable at start or reload.

• stratactl agent enroll telegraf (cli.md) produces the same env file, so installers don't need curl logic.

• The env file is 0640 root:telegraf. It holds the per-agent token and the instance id; with mTLS also STRATAPLANE_TLS_CERT and STRATAPLANE_TLS_KEY, the paths of the identity files (readable by the telegraf user), which the rendered config refers to. The enrollment request names them (tls_cert, tls_key); otherwise they are identity/cert.pem and identity/key.pem under /etc/telegraf/, or under C:\Program Files\Telegraf\ when os is windows. No local telegraf.conf is needed.

• Enrollment runs once per machine. Machines built from one image use class mode instead.

Class mode

For machines built from one image (Windows notebooks distributed as snapshots or by Intune, labs). Every machine of a class runs the same command line with the same enrollment token of mode class as TELEGRAF_CONTROLLER_TOKEN. Nothing in the image is per machine and there is no enrollment step:

# In the image, identical on every clone:
sc.exe create telegraf start= auto binPath= "\"C:\Program Files\Telegraf\telegraf.exe\" --service-name telegraf --config https://strataplane.example.com:8444/agent/v1/telegraf/config --config-url-watch-interval 1m --config-url-retry-attempts -1"
reg add HKLM\SYSTEM\CurrentControlSet\Services\telegraf /v Environment /t REG_MULTI_SZ /d TELEGRAF_CONTROLLER_TOKEN=spn_…

• telegraf service install does not pass --config-url-watch-interval to the service, hence sc.exe.
• Identity comes from the heartbeat. Class artifacts set the heartbeat instance_id to the token id and always include hostname. The first heartbeat with a new hostname creates the host (host_kind and labels from the token) and its instance, optionally pending approval.
• One config for the class. A config request carries no identity, so it gets the class default: the artifact rendered from the token's labels and the Telegraf version in User-Agent, at the release pointer. Every class artifact contains the heartbeat block, so the first heartbeat always follows. Pending and quarantined members get the class default too.
• Members with their own config. Class hosts are ordinary hosts. A label such as role=netprobe on a few of them, set by hand or by a Placement that picks N online members (spec-format.md §3.9), or a network zone label, puts them into further layers, so their desired artifact differs from the class default. A config request gets such a member's artifact when its source address resolves to the member: the member heartbeated last from that address, and no other member of the class heartbeated from it within 10 × heartbeat.interval. Otherwise it gets the class default, and the member shows sync = out_of_sync with reason ambiguous_address (several clients behind one NAT). A notebook waking up on its old address keeps its config. Only these members need a lookup; replicas keep them in memory per token, rebuilt from agent_instance and invalidated by NOTIFY like the other agent caches, so the rest of the class costs one artifact. Layers for the whole class select on the token's labels: a layer that selects on os or arch gives every member its own config.
• Source address. The TCP peer, or the address that a proxy in server.agent_proxy_from passes by PROXY protocol v2 or X-Forwarded-For (operations.md §4).
• Convergence. Class requests follow the watcher rule of §6 per address: the artifact last served to an address (class_served) is compared with what the address resolves to. A new member or a client on a reused address runs the class default or the previous member's config until its first heartbeat changes the resolution, then reloads within one watch interval. A heartbeat that reports another member's artifact shows a reused address, which is one more reason class instances get no secrets.
• Limits. Class instances are low and receive no secrets (delivery rule); outputs need sinks that accept the class's writes without per-client credentials. The class token is readable by every user of the image; it grants the class's configs and heartbeats under any host name of the class. An enrolled instance owns its host name: class heartbeats for it are rejected, and enrolling a class host converts it. Revoking the token stops config delivery for the whole class. Instances follow rollouts and pins like enrolled ones; the class default follows the release pointer.

3. Endpoints

All on the agent listener. See openapi.yaml, tag agent-telegraf.

Endpoint	Auth	Behaviour
POST /agent/v1/telegraf/enroll	enrollment token; optional client cert	Creates or re-binds host and instance; returns token and instance id. With a client cert from a trust anchor of the token's tenant, the cert SAN must match the host name and binds the instance to it (assurance); re-enrollment with the same cert rotates the token.
GET /agent/v1/telegraf/config	agent token, or class token (class mode)	Rendered TOML of the desired artifact. Sets Last-Modified = desired_since, ETag = artifact hash, records served_artifact.
HEAD /agent/v1/telegraf/config	agent token, or class token	Same headers, no body. Always answers with Last-Modified when the token is valid, including during degraded operation (§6).
POST /agent/v1/telegraf/heartbeat	agent token (+ the bound cert, if the instance has one), or class token	Accepts schema_v1 payloads. Updates health, statistics, version, applied artifact and source address; with a class token it finds or creates the host by hostname. 202 on accept; 503 + Retry-After on overload; never silently drops.
GET /agent/v1/telegraf/secrets	agent token (+ the bound cert, if the instance has one: high)	secretstores.http endpoint: JSON map of exactly the secrets this agent may receive (delivery rule). Every key delivered is audited.

4. Rendering

The merged tree becomes TOML with toml_edit, deterministically:

# Rendered by Strataplane. Do not edit; changes are made in the spec.

[agent]
  interval = "10s"
  flush_interval = "10s"

[global_tags]
  owner = "team-middleware"
  stage = "prod"

# secrets: metrics_gw_token@v3
[[secretstores.http]]
  id = "strataplane"
  url = "https://strataplane.example.com:8444/agent/v1/telegraf/secrets"
  token = "${TELEGRAF_CONTROLLER_TOKEN}"

# instance inputs.jolokia2_agent.tomcat
[[inputs.jolokia2_agent]]
  alias = "jolokia2_agent.tomcat"
  urls = ["http://localhost:8778/jolokia"]
  ...

# instance outputs.http.metrics-prod
[[outputs.http]]
  alias = "http.metrics-prod"
  url = "https://metrics-gw.example.com/api/v1/push"
  [outputs.http.headers]
    Authorization = "Bearer @{strataplane:metrics_gw_token}"

Rules:

• Order: [agent], [global_tags], secret stores, then inputs, processors, aggregators, outputs; within each section by instance key. Options are in lexical order, scalars before tables, so the catalog never shapes the bytes.
• alias is always <plugin>.<name>, so heartbeat log sources (inputs.jolokia2_agent::jolokia2_agent.tomcat) map back to instance keys.
• Secret refs become @{strataplane:<key>}; with format, the reference is embedded in the template string ("Bearer @{strataplane:metrics_gw_token}"). The secretstores.http block is added only if the artifact references secrets. A comment above it lists the secret versions the artifact was rendered against (# secrets: metrics_gw_token@v3), so a new version changes the artifact and Last-Modified, and the reload re-reads the store (render inputs).
• Per-agent values stay out of the body. The agent token and instance id arrive as environment variables, so hosts of a role share one artifact. Telegraf adds the host tag itself.
• raw is written verbatim into the instance's own table, after its scalar options and before its sub-tables. It must parse as TOML and may only set keys of that table: dotted keys and inline tables are allowed, table headers ([x], [[x]]) are not, since they would open tables outside the instance. A key that repeats a typed option or alias, or a secret store id = "strataplane", is also a render error. Option names and values inside raw are checked by L3 only.
• No inputs, no config: Telegraf refuses a config without inputs, so such a render is a render error (render_error) and the agent keeps what it runs.

Module-owned heartbeat block

Rendered last into every artifact. Its values come from the telegraf Settings object (spec-format.md §3.8), so a Settings change is an ordinary change with an impact set:

[[outputs.heartbeat]]
  alias = "strataplane"
  url = "https://strataplane.example.com:8444/agent/v1/telegraf/heartbeat"
  instance_id = "${STRATAPLANE_INSTANCE_ID}"
  token = "${TELEGRAF_CONTROLLER_TOKEN}"
  interval = "1m"                             # Settings telegraf: heartbeat.interval
  include = ["hostname", "statistics", "configs", "logs", "status"]
  [outputs.heartbeat.headers]
    X-Strataplane-Artifact = "3f2a…"          # set when served
  [outputs.heartbeat.logs]
    level = "warn"                            # heartbeat.logLevel
    limit = 50                                # heartbeat.logLimit
  [outputs.heartbeat.status]
    fail = "…"                                # status.fail (CEL)
    warn = "…"

The artifact is stored with an empty X-Strataplane-Artifact value, and its hash is taken over those bytes. Serving fills in the hash, so the served bytes are deterministic per artifact and ETag is stable.

Applied detection. A heartbeat carrying X-Strataplane-Artifact = H proves the agent loaded the config with artifact H, because the header is only part of that config. applied_artifact is set to H. If the agent fetched a new artifact (served_artifact) but heartbeats still carry the old header after 2 × watch interval + heartbeat interval, or heartbeats stop, the reload failed: sync = out_of_sync with reason reload_failed, and health not_reporting if the process is crash-looping on an invalid config.

5. Validation

Level	Implementation
L2 schema	JSON Schema per plugin and Telegraf version, generated by telegraf-schema (stack.md §4). Unknown options are errors, matching Telegraf's own strictness.
L3 sandbox	telegraf config check of the matching version, run as a subprocess against the served bytes with secret references replaced by dummy values and the Strataplane secret store removed, so it needs no network. It runs in a network namespace (unshare -rn) where the platform permits one. An E! line or a non-zero exit fails the check. Linux servers only; elsewhere L3 reports itself skipped.
Parser preview	The telegraf-parse wasm (§8) for data_format options, against user-supplied samples. Informational, not a gate.

Validator binaries and schemas come from the per-version catalog bundle (operations.md §5).

6. Watcher safety

Telegraf's watcher compares each HEAD's Last-Modified with the value of its first HEAD after a reload, not with the GET that loaded the config. A change between that GET and the first HEAD, or during a gap in which a proxy answered without Last-Modified, is never picked up. Strataplane therefore answers by what the agent actually got:

1. Served differs from desired: a changing value. While the artifact last served by GET (served_artifact; for class requests the address's class_served row) differs from the desired one, HEAD answers with the current time, so the next tick reloads. Otherwise it answers with desired_since, which is stored in the database and identical on every replica, so a load balancer switching replicas never triggers reloads.
2. Every authenticated HEAD carries Last-Modified, even when the database is briefly unavailable: replicas cache desired_since and the served state per instance and per class address; on a cache miss during an outage they answer with the last value the agent was served (from the token cache), never with a new one.
3. Load balancer guidance: route /agent/v1/telegraf/config with health checks so that a replica restart never answers a HEAD with an error page, pass Last-Modified through, and keep idle and connect timeouts short: Telegraf's requests have no timeout, so a held connection stalls its watcher silently.
4. Detection: an agent whose served_at stops advancing while heartbeats continue has a stalled watcher. It gets sync = out_of_sync with reason watcher_stalled once desired ≠ applied for longer than 2 × watchInterval. Class members are served by address, so only members with their own config are checked. Restarting Telegraf is left to the operator's tooling.

Secrets never crash an agent by design. Telegraf exits when the secret store lacks a referenced key, so the reconciler renders a secret reference only for an instance the delivery rule can admit (architecture.md §11); otherwise the render fails (render_error) and the agent keeps its config.

7. Heartbeat ingest

• Parsed with strict serde types for schema_v1 (unknown fields tolerated for forward compatibility, schema > 1 logged). An id that is still a literal ${…} (unset variable) is rejected.
• Coalesced per instance in memory; flushed every second or 5,000 updates as one UPDATE … FROM UNNEST(…). Status transitions and log entries are inserted in the same transaction.
• Health from status (OK → healthy, WARN → degraded, FAIL → failed, absent or UNDEFINED → healthy unless statistics show errors), plus staleness (agent_health).
• statistics.errors and warnings are exported as counters on /metrics per agent kind, owner and revision (not per agent, to bound cardinality).

8. Parser wasm

Purpose. Answer "which metrics does this parser config make from this sample?" in milliseconds, deterministically, without a Telegraf sandbox. The assistant iterates on json_v2, xpath, csv, grok, logfmt, … configs for inputs.http, inputs.exec, inputs.mqtt_consumer, inputs.tail, inputs.kafka_consumer. The UI uses the same module for a live sample → metrics preview in the plugin form.

Build (tools/telegraf-wasm, Go):

• GOOS=wasip1 GOARCH=wasm, one module per Telegraf release tag, built in CI and shipped in the catalog bundle.
• Imports the parser packages and the influx serializer only. Decodes the parser's TOML table with influxdata/toml into the registered parser struct, not through config.Config, so the secret store and input/output registry stay out.
• Verified 2026-10-21: json_v2 + influx serializer, 7.6 MB stripped, correct output under Node's WASI and under wasmtime 49.
• Telegraf fork: parsers/influx (and dropwizard) import config.Duration, which drags in memguard (mlock/mmap, not available in wasm). tools/telegraf-wasm/fork.sh VERSION derives the fork from each release tag: it moves Duration and Size into a package of their own, so all parsers are available, and fails loudly when a release no longer fits.
• Two modules per version: parsers (25 parsers, about 29 MB) and parsers-parquet (about 37 MB), loaded on demand.

ABI. WASI command module, JSON in on stdin, JSON out on stdout, no filesystem, no network, no clock (time comes from the request). config_toml holds the bare parser options or a plugin table; metric_name defaults to that plugin's name, as in Telegraf, else to the parser's. {"abi": 1, "op": "info"} lists the module's parsers; a request the module refuses as a whole (unknown parser, invalid TOML) answers with a top-level error:

{ "abi": 1, "parser": "json_v2", "config_toml": "[[inputs.http.json_v2]]\n…",
  "metric_name": "http", "default_time": "2026-10-21T12:00:00Z",
  "samples": [{ "name": "a.json", "data_b64": "…" }] }

{ "abi": 1, "telegraf_version": "1.40.1",
  "results": [{ "sample": "a.json",
                "metrics": [{ "name": "http", "tags": {…}, "fields": {…}, "time": "…" }],
                "line_protocol": "http,room=kitchen temp=21.5 1792598400000000000\n",
                "warnings": [], "error": null }] }

Hosts:

Where	Runtime
Browser (assistant shell and UI preview)	Dedicated Web Worker with @bjorn3/browser_wasi_shim; module fetched from the catalog by version, compiled once and cached (IndexedDB). Terminated on timeout.
stratactl telegraf-parse	wasmtime embedded in the CLI; modules cached locally per version hash.
Server (POST /api/v1/tools/telegraf/parse)	wasmtime with fuel/epoch and memory limits; precompiled modules cached. For clients without local wasm.

With a fixed default_time, results depend only on (version, parser, sha256(config), sha256(samples)).

9. Import

stratactl import telegraf DIR turns an existing Telegraf config directory into layers for migration: one instance per plugin table, keyed <section>.<plugin>.<name> with the name taken from alias where set. The result is written into the working copy and goes through a change set like any other edit. The UI's import of a single file uses the same converter (POST /api/v1/tools/telegraf/import; the file is not stored).