Expose confined, identity-bound filesystem observations through the shared
resource pipeline so investigations can distinguish an exhausted container
mount from unrelated host capacity. Keep unavailable measurements explicit.
Isolate alert-history reads from durable writes and reuse one chronological
fold across polling. Catch up through bounded durable event IDs so simultaneous
readers do not replay every retained snapshot. Retain expired actions when
investigation outcomes move back to needs attention, and keep attached
Assistant context focused.
Record live storage diagnosis, healthy and dependency controls, approved and
rejected Docker outcomes, source-bound browser proof and exact test limits.
Missing-access continuity, VM dispatch completion and remaining Assistant
orchestration defects stay open in the redesign plan.
Cache unchanged inactive container detail between live telemetry cycles so NAS hosts with large historical inventories do not re-inspect every stopped container every 30 seconds. Keep running state live, fail open to fresh inspection when lifecycle evidence is incomplete, and clear daemon-scoped caches on reconnect.
Refs #1729.
Contract-Neutral: Bounds internal daemon call cadence without changing report or operator contracts
Container update detection only ever negotiated anonymous pull tokens, so
containers from registries that reject anonymous digest HEADs pinned a
permanent "authentication required" badge (#1706). The agent already runs
on the Docker host, so the checker now resolves the same credential store
docker pull uses - config.json auths entries, credsStore/credHelpers
credential helpers (docker-credential-<name> get), and Podman's auth.json -
and presents the stored login: Basic auth on Bearer token negotiation and
on the hardcoded Docker Hub / ghcr.io token endpoints, direct answers to
Basic challenges, and the refresh-token grant for identity-token logins
such as Azure ACR.
Credentials never leave the host: they are only presented to the registry
or its token endpoint, helper output stays out of reported check errors,
and lookups are cached in memory for five minutes. Helper names are
validated before exec, and a stale login falls back to the anonymous path
so checks that used to work keep working. Set
PULSE_DISABLE_REGISTRY_CREDENTIALS=true (--disable-registry-credentials)
to keep detection anonymous-only. The agent-lifecycle and security-privacy
subsystem contracts pin the host-local credential boundary.
The installer's discover_rootless_container_runtime only globbed
/run/user/* sockets and never consulted the system Docker daemon, so a
socket-activated rootless Podman API socket (alive only for root's login
session on Debian/OMV) won discovery over a healthy rootful Docker and
pinned PULSE_DOCKER_RUNTIME=podman plus CONTAINER_HOST/PODMAN_HOST/
XDG_RUNTIME_DIR into the agent unit. The env-application block also ran
for explicit --enable-docker installs. Rootless discovery now defers to
system_docker_runtime_is_active (docker info with DOCKER_HOST stripped,
or a live /var/run/docker.sock probe) before touching rootless sockets.
On the agent, detectRuntime short-circuited to podman whenever the
preference said podman, mislabeling connections that actually fell
through to the Docker socket and disabling Swarm collection. The
preference is now an ordering hint: a podman-preferred connection landing
on a docker endpoint reports docker, while unlabeled endpoints with no
runtime signals still honor the pin. When the bound socket disappears
mid-run the agent re-runs runtime discovery after three consecutive
daemon-unavailable collects, swapping the connection behind a
swappableDockerClient so concurrent goroutines keep a stable handle.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same class of race fixed for newTimerFn in 8e5ef365d: tests swapped the
package-level jsonMarshalFn hook while async goroutines leaked from
earlier tests (sendCommandAck ack retries via runAsync) could still be
reading it, tripping the race detector. Replace the global with a
per-Agent jsonMarshalFn seam (nil defaults to json.Marshal), make the
decode payload helpers Agent methods so they use it, and inject the
failing marshaller into the tests that previously swapped the global.
Verified with go test ./internal/dockeragent/ -race -count=20.
Contract-Neutral: test seam refactor to fix data race, no public contract delta
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tests swapped the package-level newTimerFn hook while async goroutines
leaked from earlier tests (backup-cleanup and stop-command paths) were
still reading it in waitForAsyncDelay, tripping the race detector on CI.
Replace the global with a per-Agent newTimerFn seam (nil defaults to
time.NewTimer), make waitForContextDelay an Agent method, and inject the
immediate timer into the tests that previously swapped the global.
Verified with go test ./internal/dockeragent/ -race -count=20.
Contract-Neutral: test seam refactor to fix data race, no public contract delta
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep manual check commands active until registry collection completes. Deduplicate replayed and concurrent commands, bound collection and acknowledgement retries, surface registry result counts, and prove timeout, rate-limit, and replay behavior.
getDockerCommandPayload returned dispatched commands on every report
fetch, causing the agent to re-execute check-updates on every poll
cycle. When the ack also failed, the report was buffered and retried,
creating an infinite loop.
- Only return command payload on the queued->dispatched transition;
subsequent fetches return nil (agent already received it).
- Don't propagate ack errors from handleCheckUpdatesCommand; the report
was delivered and check-updates is fire-and-forget. Command expires
if ack never succeeds.
Refs #1504
When the agent runs as root, os.Getuid() returns 0 so it only probes
/run/user/0/docker.sock. Rootless Docker installs live at
/run/user/1000/docker.sock (or similar). Glob /run/user/*/docker.sock
and /run/user/*/podman/podman.sock to discover sockets for all users.
Probe ~/.docker/run/docker.sock for RuntimeDocker and RuntimeAuto
before falling back to /var/run/docker.sock. This lets the agent
connect on macOS without requiring DOCKER_HOST to be set manually.
Ref #1200
Docker container URL preserved on update (#1054): container updates
recreate the container with a new runtime ID. The agent now includes
{oldContainerId, newContainerId} in the completion ACK payload; the
server uses this to copy persisted metadata (custom URLs, descriptions,
tags) to the new ID so nothing is lost. Migration is a copy, not a move,
so rollback scenarios still find metadata under the original ID.
Reduce metrics.db write amplification (#1124): add a UNIQUE index on
(resource_type, resource_id, metric_type, timestamp, tier) so rollup
reprocessing after a failed checkpoint uses INSERT OR IGNORE instead of
creating duplicate rows. Existing duplicates are deduplicated once on
startup if the index creation would otherwise fail. Also sets
wal_autocheckpoint(500) to checkpoint the WAL more frequently, preventing
unbounded WAL growth.
Fixes#1054Fixes#1124
FreeBSD auto-update (#1254): determineArch() now includes freebsd in its
OS switch, producing freebsd-amd64/arm64 instead of falling through to
a uname -m fallback that incorrectly returned linux-<arch>. FreeBSD agents
were downloading Linux ELF binaries and failing to exec them.
Docker rootless socket (#1200): buildRuntimeCandidates() now probes
/run/user/<uid>/docker.sock before the system-wide /var/run/docker.sock,
enabling auto-detection of Docker rootless installations.
Duplicate PVE/PBS hosts (#1245, #1252): handleSecureAutoRegister() now
deduplicates by host URL, updating the existing instance's token in-place
instead of appending a duplicate entry on each re-run of the setup script.
Fixes#1254Fixes#1200Fixes#1245Fixes#1252
(cherry picked from commit 0f1d9e9b9fea6c8b9e65872e8a78e25f93653eef)
Docker's one-shot stats API (stream=false) returns PreCPUStats from the
daemon's internal cache, which many Docker versions don't update between
non-streaming reads. This causes every call to return the same stale
PreCPUStats from container start, producing a constant lifetime-average
CPU% (e.g. 3.4%) instead of current usage.
Switch to always using manual delta tracking, which stores the previous
sample from our own reads and computes accurate deltas between collection
cycles. The first cycle returns 0 while establishing a baseline; all
subsequent cycles produce correct current CPU percentages.
The Docker agent was not passing the disk exclusion list to
hostmetricsCollect(), so excluded mounts appeared in the Docker tab
disk totals. Also add server-side fsfilters filtering to Docker
report processing for parity with the host agent path.
The agent was crashing with 'fatal error: concurrent map writes' when
handleCheckUpdatesCommand spawned a goroutine that called collectOnce
concurrently with the main collection loop. Both code paths access
a.prevContainerCPU without synchronization.
Added a.cpuMu mutex to protect all accesses to prevContainerCPU in:
- pruneStaleCPUSamples()
- collectContainer() delete operation
- calculateContainerCPUPercent()
Related to #1063
When a user's reverse proxy redirects HTTP to HTTPS, Go's default HTTP
client behavior converts POST requests to GET on 301/302 redirects
(per HTTP specification). This causes the Pulse server to return 405
"Only POST is allowed" errors.
Added CheckRedirect to all agent HTTP clients (host, docker, kubernetes)
that returns a clear error message guiding users to use the correct
protocol in their --url flag instead of silently following redirects.
Related to #1058
When --docker-runtime=podman is explicitly set, the agent should try
Podman-specific sockets first before falling back to environment
defaults (which try /var/run/docker.sock).
Also adds /var/run/podman/podman.sock as a candidate socket path,
which is used by CoreOS and some Fedora configurations.
Related to #1045
When a Docker agent tries to register with a token that's already bound
to another agent, the error was logged generically as "Failed to send
docker report". Users had to dig into logs to understand the issue.
Now logs a prominent error message:
"DOCKER REGISTRATION FAILED: This API token is already used by another
Docker agent. Each Docker host requires its own unique token. Generate
a new token in Pulse Settings > Agents and reinstall with the new token."
Related to #1027
Add PULSE_DISABLE_DOCKER_UPDATE_CHECKS environment variable and
--disable-docker-update-checks flag to disable Docker image update
detection. This is useful for:
- Avoiding Docker Hub rate limits
- Users who don't want update notifications in their dashboard
Related to Discussion #982
- Separated metrics collection into internal/dockeragent/collect.go
- Added agent self-update pre-flight check (--self-test)
- Implemented signed binary verification with key rotation for updates
- Added batch update support to frontend with parallel processing
- Cleaned up agent.go and added startup cleanup for backup containers
- Updated documentation for Docker features and agent security
Fixed an issue where all Docker containers were showing 'click to update'
even when they were up to date. The root cause was comparing the wrong
digest types:
- Previously: Compared ImageID (local config hash) vs registry manifest digest
- Now: Uses RepoDigests from image inspect, which is the actual manifest
digest that Docker received from the registry when pulling the image
For multi-arch images, the registry returns a manifest list digest, while
Docker stores the platform-specific image config digest locally. These
will never match, causing false positives for all multi-arch images.
Changes:
- Added ImageInspectWithRaw to dockerClient interface
- Added getImageRepoDigest method to extract RepoDigest from image
- Added matchesImageReference helper for Docker Hub naming conventions
- Added tests for matchesImageReference
Fixes#955
- Add comprehensive test coverage for agent report, flush buffer, and deps
- Expand flow, HTTP, CPU, and swarm test coverage
- Refactor registry access to use deps interface for better testability
- Add container update and self-update test scenarios
Allows specifying which IP address the agent should report, useful for:
- Multi-homed systems with separate management networks
- Systems with private monitoring interfaces
- VPN/overlay network scenarios
Usage:
pulse-agent --report-ip 192.168.1.100
PULSE_REPORT_IP=192.168.1.100 pulse-agent
Content was being streamed twice:
1. During each iteration of the tool loop (intended for intermediate feedback)
2. Again after the loop ended with finalContent (redundant)
This caused duplicate responses when using Ollama and other providers.
- Add container update command handling to unified agent
- Agent can now receive update_container commands from Pulse server
- Pulls latest image, stops container, creates backup, starts new container
- Automatic rollback on failure
- Backup container cleaned up after 5 minutes
- Added comprehensive test coverage for container update logic
BREAKING CHANGE: AI Patrol now uses EXACT alert thresholds by default
instead of warning 5-15% before the threshold.
Changes:
- Default behavior: Patrol warns at your configured threshold (e.g., 96% = warns at 96%)
- New setting: 'use_proactive_thresholds' enables the old early-warning behavior
- API: Added use_proactive_thresholds to GET/PUT /api/settings/ai
- Backend: Added SetProactiveMode/GetProactiveMode to PatrolService
- Backend: Added GetThresholds to PatrolService for UI display
- Tests: Updated and added tests for both exact and proactive modes
- Also fixed unused imports in dockeragent/agent.go
When proactive mode is disabled (default):
- Watch: threshold - 5% (slight buffer)
- Warning: exact threshold
When proactive mode is enabled:
- Watch: threshold - 15%
- Warning: threshold - 5%
Related to #951
The GuestURL field was missing from NodeFrontend and its converter,
causing configured Guest URLs to be ignored when clicking on cluster
node names. The frontend would fall back to the auto-detected IP
instead of using the user-configured Guest URL.
Related to #940
- Add Env field to Container struct in pkg/agents/docker/report.go
- Extract env vars from inspect.Config.Env in Docker agent
- Mask sensitive values (password, secret, key, token, etc.) with ***
- Display env vars in container drawer with green badges (amber for masked)
- Add tests for maskSensitiveEnvVars function
Related to #916
Users can now exclude specific mount points from disk monitoring:
- Via CLI: --disk-exclude /mnt/backup --disk-exclude '/media/*'
- Via env: PULSE_DISK_EXCLUDE=/mnt/backup,*pbs*
Patterns support:
- Exact paths: /mnt/backup
- Prefix patterns: /mnt/ext*
- Contains patterns: *pbs*
This addresses the common case where external disks or
PBS datastores are being monitored but shouldn't be.
Related to #823
- Log payload size (in KB and bytes) at debug level
- Warn when payload approaches 400KB (512KB limit)
- Helps diagnose 'request body too large' errors
- Add unit tests for internal/buffer package
- Fix misleading "ring buffer" comment (it's a bounded FIFO queue)
- Remove unused BufferCapacity config field from both agents
- Rewrite flaky integration test to use polling instead of fixed sleeps
- Move normalizeVersion to utils.NormalizeVersion for single source of truth
- Update agentupdate and dockeragent packages to use shared function
- Add 14 test cases for version normalization
This prevents bugs like issue #773 where a fix applied to one copy
but not the other caused an update loop.
rcourtman