Same class of race fixed for newTimerFn in 8e5ef365d: tests swapped the
package-level jsonMarshalFn hook while async goroutines leaked from
earlier tests (sendCommandAck ack retries via runAsync) could still be
reading it, tripping the race detector. Replace the global with a
per-Agent jsonMarshalFn seam (nil defaults to json.Marshal), make the
decode payload helpers Agent methods so they use it, and inject the
failing marshaller into the tests that previously swapped the global.
Verified with go test ./internal/dockeragent/ -race -count=20.
Contract-Neutral: test seam refactor to fix data race, no public contract delta
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tests swapped the package-level newTimerFn hook while async goroutines
leaked from earlier tests (backup-cleanup and stop-command paths) were
still reading it in waitForAsyncDelay, tripping the race detector on CI.
Replace the global with a per-Agent newTimerFn seam (nil defaults to
time.NewTimer), make waitForContextDelay an Agent method, and inject the
immediate timer into the tests that previously swapped the global.
Verified with go test ./internal/dockeragent/ -race -count=20.
Contract-Neutral: test seam refactor to fix data race, no public contract delta
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep manual check commands active until registry collection completes. Deduplicate replayed and concurrent commands, bound collection and acknowledgement retries, surface registry result counts, and prove timeout, rate-limit, and replay behavior.
Eight new branch-coverage tests taking thirty-one previously unreached
functions from zero to covered, with no source or existing test touched.
internal/kubernetesagent: twenty-one pure report helpers, including the pointer
converters proved non-aliasing in both directions, the ingress host and address
collectors across their trim, dedupe and insertion-order arms, the endpoint
slice readiness count where a nil Ready field counts as ready, and the target
role predicate.
internal/agentexec: the sudo long-option value gate over the real option list
including the inline equals form, and the approval grant verification error
unwrapped through errors.Is.
internal/alerts: the alert config alias normalization across the nil config
guard, the empty threshold early return, the blank type-key continue arm and
the legacy-delete versus supported-keep split, asserting both maps stay
independent.
internal/alerts/specs: the resource incident rollup evidence validation, each
failure arm asserted on its concrete error and the check order pinned when
several fields are invalid at once.
internal/cloudcp/docker: the not-found predicate through a wrapped error, the
route host label precedence, and the Traefik host rule parser across quoting
styles, combined matchers, multiple host clauses and malformed input.
internal/cloudcp/portal: the anonymous bootstrap builder, asserting no tenant
or user identity field is ever populated on the anonymous result.
internal/config: the legacy OIDC environment provider, including the arm where
an already-configured provider is present and the redirect derivation from a
public URL with a trailing slash.
internal/dockeragent: the update-all payload decode across wrong-typed and
missing fields, and the docker filter conversion.
Contract-Neutral: test-only branch coverage, no contract surface touched
Follow-up to the 2026-07-17 live docker-update exercise where a DiskUsage
roundtrip against a colima daemon parked 6+ minutes even though
dockerCallWithRetry wraps every call in a 20s context.WithTimeout.
Investigation result: context deadline propagation through moby client
v0.5.0 (request.go, API-version negotiation) and the otelhttp transport
wrapper is intact. Reproducing with a deliberately hung unix-socket
daemon aborts DiskUsage at the deadline in both hang shapes (pre-header
and mid-body), so there is no client-library bug to fix or file
upstream; the production stall's root cause remains environmental
(deadline timer never fired process-side).
Containment and diagnosis:
- buildReport now runs under dockerCollectCycleTimeout (5m) so a wedged
cycle can never stall the module indefinitely, plus an independent
watchdog timer that logs an error with a full goroutine dump if the
cycle outlives even that deadline - capturing exactly the evidence
that was missing from the original incident.
- hung_daemon_deadline_test.go pins that a context deadline aborts the
real moby client against a hung unix-socket daemon (pre-header and
mid-body stalls), guarding future moby/otelhttp upgrades.
The incident note referenced a dockerCollectCycleTimeout watchdog as
already added; it did not exist on any branch - this commit is that
containment, landed for real.
Contract-Neutral: dockeragent collect-cycle watchdog containment: timeout plumbing only, no collection-semantics or contract-surface delta
Update checks against registries without a hardcoded token endpoint
(lscr.io and other spec-compliant registries) failed with a blanket
"Check failed" because the manifest HEAD went out anonymously and the
401 was terminal. Parse the Bearer challenge on the 401, fetch a pull
token from the advertised realm, and retry once. Token endpoints that
answer with access_token instead of token are now accepted too.
Fixes#1583.
v6.1.0-rc.1 retired the legacy update endpoints before a replacement
existed, so the UI's Update button failed with an internal-jargon 410
(issue #1564). This lands the replacement end to end: update_container
is a typed agentexec operation with its own strict codec, durable
receipts, and a request digest bound to the image digest the plan
observed; the unified agent bridges execution to the Docker module's
existing pull/backup/recreate/verify/rollback implementation (which now
reports rollback attempt and outcome); and the container action
executor plans, dispatches, and reconciles the operation with declared
backup/rollback compensation truth. Containers advertise an
admin-approval update capability while an image update with a stated
current digest is detected. The legacy endpoints stay retired but
return actionable copy.
Proven live against a Colima daemon: single-container update, the
issue-1564 shared-network-namespace update, and the full UI journey
(Update button, governed review, approve, run) all completed with the
namespace preserved and the backup retained.
One-click updates recreated containers from the raw inspected config, so
Docker rejected the create for any container using network_mode:
container:<id> (compose service:<x>) or host with "conflicting options:
hostname and the network mode", leaving the workload stopped under its
_pulse_backup_ name until the rollback rename restored it. Docker fills
Config.Hostname with the namespace owner's ID on such containers, so the
verbatim copy always tripped the daemon validation (verified against a
real Docker 29.5.2 daemon: verbatim create 409s, sanitized create
succeeds and the replacement starts in the shared namespace).
Strip the namespace-owned settings before ContainerCreate: hostname and
domainname for container:/host modes, plus exposed/published ports,
links, DNS options, and extra hosts for container: mode, all of which
the daemon derives from the owning namespace. Bridge and user-defined
networks are untouched.
Refs #1564
getDockerCommandPayload returned dispatched commands on every report
fetch, causing the agent to re-execute check-updates on every poll
cycle. When the ack also failed, the report was buffered and retried,
creating an infinite loop.
- Only return command payload on the queued->dispatched transition;
subsequent fetches return nil (agent already received it).
- Don't propagate ack errors from handleCheckUpdatesCommand; the report
was delivered and check-updates is fire-and-forget. Command expires
if ack never succeeds.
Refs #1504
Back-port v5 fix 48bdfdc30 to v6. In node scope, collectSwarmDataFromManager
now derives each service's Desired/Running/Completed task counts from the
node-local task list and drops services with no tasks on this node, instead
of reusing the cluster-wide ServiceStatus and a 'keep all when none match'
fallback. Without this, a node-scoped Swarm view showed wrong counts and
services that aren't running on that node. Adds two regression tests.
Dead-code sweep. Functions flagged unreachable by golang.org/x/tools/cmd/deadcode
and confirmed unused across pulse, pulse-enterprise, pulse-pro and pulse-mobile by
adversarial cross-repo verification. Cross-module reachability was checked
explicitly (only pkg/ exported symbols are importable by other modules; internal/
packages and _test.go files are not). go build, go vet and test-compile all pass.
When the agent runs as root, os.Getuid() returns 0 so it only probes
/run/user/0/docker.sock. Rootless Docker installs live at
/run/user/1000/docker.sock (or similar). Glob /run/user/*/docker.sock
and /run/user/*/podman/podman.sock to discover sockets for all users.
Probe ~/.docker/run/docker.sock for RuntimeDocker and RuntimeAuto
before falling back to /var/run/docker.sock. This lets the agent
connect on macOS without requiring DOCKER_HOST to be set manually.
Ref #1200
Docker container URL preserved on update (#1054): container updates
recreate the container with a new runtime ID. The agent now includes
{oldContainerId, newContainerId} in the completion ACK payload; the
server uses this to copy persisted metadata (custom URLs, descriptions,
tags) to the new ID so nothing is lost. Migration is a copy, not a move,
so rollback scenarios still find metadata under the original ID.
Reduce metrics.db write amplification (#1124): add a UNIQUE index on
(resource_type, resource_id, metric_type, timestamp, tier) so rollup
reprocessing after a failed checkpoint uses INSERT OR IGNORE instead of
creating duplicate rows. Existing duplicates are deduplicated once on
startup if the index creation would otherwise fail. Also sets
wal_autocheckpoint(500) to checkpoint the WAL more frequently, preventing
unbounded WAL growth.
Fixes#1054Fixes#1124
FreeBSD auto-update (#1254): determineArch() now includes freebsd in its
OS switch, producing freebsd-amd64/arm64 instead of falling through to
a uname -m fallback that incorrectly returned linux-<arch>. FreeBSD agents
were downloading Linux ELF binaries and failing to exec them.
Docker rootless socket (#1200): buildRuntimeCandidates() now probes
/run/user/<uid>/docker.sock before the system-wide /var/run/docker.sock,
enabling auto-detection of Docker rootless installations.
Duplicate PVE/PBS hosts (#1245, #1252): handleSecureAutoRegister() now
deduplicates by host URL, updating the existing instance's token in-place
instead of appending a duplicate entry on each re-run of the setup script.
Fixes#1254Fixes#1200Fixes#1245Fixes#1252
(cherry picked from commit 0f1d9e9b9fea6c8b9e65872e8a78e25f93653eef)
Docker's one-shot stats API (stream=false) returns PreCPUStats from the
daemon's internal cache, which many Docker versions don't update between
non-streaming reads. This causes every call to return the same stale
PreCPUStats from container start, producing a constant lifetime-average
CPU% (e.g. 3.4%) instead of current usage.
Switch to always using manual delta tracking, which stores the previous
sample from our own reads and computes accurate deltas between collection
cycles. The first cycle returns 0 while establishing a baseline; all
subsequent cycles produce correct current CPU percentages.
The Docker agent was not passing the disk exclusion list to
hostmetricsCollect(), so excluded mounts appeared in the Docker tab
disk totals. Also add server-side fsfilters filtering to Docker
report processing for parity with the host agent path.
- Add PBS/PMG polling interval environment variable overrides in config.go
- Fix temp path expectation in detect_root_test.go using filepath.Join
- Use EvalSymlinks for symlink target comparison in self_update_test.go
- Add Linux-only skip for MAC fallback test in agent_new_test.go
- Add OS-aware RAID/SMART assertions in agent_metrics_test.go
The agent was crashing with 'fatal error: concurrent map writes' when
handleCheckUpdatesCommand spawned a goroutine that called collectOnce
concurrently with the main collection loop. Both code paths access
a.prevContainerCPU without synchronization.
Added a.cpuMu mutex to protect all accesses to prevContainerCPU in:
- pruneStaleCPUSamples()
- collectContainer() delete operation
- calculateContainerCPUPercent()
Related to #1063
When Docker daemon runs inside an LXC container, it may report 0 for
MemTotal because it can't read the cgroup memory limits correctly.
This caused the UI to show "0B / 7GB" and trigger false alerts with
overflow percentages (214748364799.6%).
The fix checks if Docker's info.MemTotal is 0 and falls back to
gopsutil's /proc/meminfo reading (snapshot.Memory.TotalBytes) which
works correctly in LXC environments.
Fixes#1075
When a user's reverse proxy redirects HTTP to HTTPS, Go's default HTTP
client behavior converts POST requests to GET on 301/302 redirects
(per HTTP specification). This causes the Pulse server to return 405
"Only POST is allowed" errors.
Added CheckRedirect to all agent HTTP clients (host, docker, kubernetes)
that returns a clear error message guiding users to use the correct
protocol in their --url flag instead of silently following redirects.
Related to #1058
When --docker-runtime=podman is explicitly set, the agent should try
Podman-specific sockets first before falling back to environment
defaults (which try /var/run/docker.sock).
Also adds /var/run/podman/podman.sock as a candidate socket path,
which is used by CoreOS and some Fedora configurations.
Related to #1045
When a Docker agent tries to register with a token that's already bound
to another agent, the error was logged generically as "Failed to send
docker report". Users had to dig into logs to understand the issue.
Now logs a prominent error message:
"DOCKER REGISTRATION FAILED: This API token is already used by another
Docker agent. Each Docker host requires its own unique token. Generate
a new token in Pulse Settings > Agents and reinstall with the new token."
Related to #1027
courtmanr@gmail.com