Exchange the bootstrap token before any agent module starts (#2520)

## Problem

An agent installed by the cluster deploy flow starts with `--enroll` and
a one-time bootstrap token. Only the host module exchanged that token
for the runtime token. Everything else `main.go` builds at startup kept
the bootstrap token for the whole first process lifetime:

- the remote config client (initial fetch and the one-minute refresh)
- the updater
- the Docker / Podman and Kubernetes modules and their primary report
targets

The bootstrap token carries only the enrollment scope and the server
removes it on exchange, so each of those failed with 401 until the
service restarted. On a freshly deployed Proxmox node this showed up as
a remote config 401 every minute. Those modules also kept the
pre-enrollment agent ID instead of the server-assigned one.

## Fix

`main.go` now enrolls once, before it builds any authenticated client:

- `hostagent.Enroll` is a standalone entry point over the existing
enrollment loop. Payload, headers, retry and backoff, status handling,
and `runtime.token` / `agent-id` persistence are unchanged.
- It runs after self-test and agent-ID preparation and before the first
remote config fetch. Every module is then built from the runtime token
and the canonical agent ID.
- The ID is persisted to the configured agent-ID file, and the host
module's own enrollment is switched off so there is no second exchange.
- A persisted runtime token is reused without a request or host
discovery. A `403` keeps the configured token as a non-bootstrap
credential. A permanent rejection stops startup before any module is
built.
- Windows is gated off and keeps its previous behaviour. Startup there
runs before the service control dispatcher, and an enrollment retry
could outlast its 30-second deadline. No Windows installer passes
`--enroll`.
- Enrollment and reporting now derive hostname, OS and architecture
through one `resolveHostIdentity`, so the runtime token is bound to the
hostname the agent reports under.

## Tests

- `TestRunEnrollsBeforeBuildingAuthenticatedModules`: enrollment happens
before any request to Pulse. Remote config, the updater, and the host,
Docker and Kubernetes configs and their targets all carry the runtime
token and canonical ID, and the agent-ID file is written. With the new
step disabled, this test fails.
- `TestRunStopsBeforeModulesWhenEnrollmentIsRejected`,
`TestRunWithoutEnrollDoesNotExchangeTokens` and
`TestRunLeavesEnrollmentToTheHostModuleWhenGatedOff`.
- `TestEnroll*` cases for `Enroll`: payload and identity, the
monitoring-only command ceiling, reuse of a persisted token (including
when host discovery fails), the 403 keep, the 409 rejection, and a
missing token.
- `TestEnrollPresentsTheIdentityNewReports` checks that enrollment
presents the hostname, OS and architecture that `New` reports, with
Proxmox detection and the architecture fallback.
- `go test ./cmd/pulse-agent ./internal/hostagent
./internal/remoteconfig` passes. `go vet` is clean for linux, windows
and freebsd.

## Not in this change

- The health listener still starts after enrollment, so `/readyz` is
unreachable while enrollment retries on network errors. The installer
only warns in that case. With the health listener enabled, its
registration check now always sees the runtime token. With it disabled,
the process-only check can still race enrollment.
- Distinguishing a deploy binding mismatch from a non-bootstrap token,
which both return 403.
- Two older durability gaps that this change doesn't make worse. If
writing a custom `--agent-id-file` fails, the next start has no fallback
to the canonical ID saved in the state directory. If persisting
`runtime.token` fails, enrollment still succeeds, and the next restart
is left with the spent bootstrap token.
This commit is contained in:
rcourtman
2026-10-05 17:05:17 +01:00
committed by GitHub
10 changed files with 722 additions and 30 deletions
@@ -797,6 +797,7 @@ class CanonicalCompletionGuardTest(unittest.TestCase):
"internal/hostagent/commands_host_update_test.go",
"internal/hostagent/commands_storage_cleanup_test.go",
"internal/hostagent/docker_lifecycle_test.go",
"internal/hostagent/enroll_test.go",
"internal/hostagent/issue1595_sas_collection_test.go",
"internal/hostagent/observer_delivery_test.go",
"internal/hostagent/package_updates_test.go",
@@ -4715,6 +4715,7 @@ class SubsystemLookupTest(unittest.TestCase):
"internal/hostagent/commands_host_update_test.go",
"internal/hostagent/commands_storage_cleanup_test.go",
"internal/hostagent/docker_lifecycle_test.go",
"internal/hostagent/enroll_test.go",
"internal/hostagent/issue1595_sas_collection_test.go",
"internal/hostagent/observer_delivery_test.go",
"internal/hostagent/package_updates_test.go",