The guest Docker socket probe hung minipc hard enough to need a power
cycle (2026-08-20): ~100 orphaned pct exec children, load 133, sshd and
pveproxy starved. Three bugs chained, each fixed here:
1. Dispatcher re-issued a probe while the previous one was still
executing. The poll cycle's enrichment context had expired, so
ExecuteCommand dispatched, returned the context error 50ms later,
and the next 3s cycle sent the identical command again — unbounded
concurrency against a host that was slow to begin with. The
monitoring dispatcher now takes a per-guest in-flight claim before
dispatching probe or inventory commands (completed probes release
it; abandoned ones hold it for a 2-minute window), and both dispatch
paths bail out under a dead context.
2. The host agent never got the July process-leak fix: 45480a5cc
landed only on pulse/v6-release, so main-line agents killed just the
direct shell on timeout, orphaning pct exec → lxc-attach children
and blocking Wait on their inherited pipes (10s timeouts reported as
300s+ durations). Port it: run each command in its own process
group, SIGKILL the group on cancel, bound Wait with WaitDelay, and
treat ErrWaitDelay after a clean exit as success.
3. Server-side abandonment never reached the agent. ExecuteCommand and
ReadFile now refuse to dispatch under an already-expired context,
and send a best-effort cancel_command when they stop waiting; the
agent cancels the in-flight execution (killing its process group)
and reports "command canceled". Older agents ignore the unknown
message type.
Also add a per-node circuit breaker: three consecutive command failures
on one node suspend all Docker probe/inventory dispatch to it on the
existing 1m→30m backoff schedule, so a host-level stall (NFS flapping)
stops the probing entirely instead of failing guest by guest.
Regression tests simulate the storm without hardware: a never-returning
executor is not re-issued across poll cycles, an expired context
dispatches nothing and records no failure, abandoned probes hold their
claim, the breaker blocks new guests on a failing node, and the agent
kills the whole process group on timeout and on server-issued cancel.
Contract-Neutral: monitor.go delta is three private struct fields holding Docker probe dispatch state; host-agent deletion/re-enrollment lifecycle untouched — contracts and all other proofs are staged
Preserve tenant-scoped metadata through partial URL updates and project stable URLs across runtime identities. Use a safe adjacent launch control across overview tables with desktop and mobile regression coverage.
URLs saved through the resource drawer historically landed in the docker
store under the runtime container key, which any stable record (including
an intentionally empty cleared one) outranks in the unified customUrl
projection, and which orphans on container recreation. On report ingest,
copy those records into the stable app-container guest key when it is
missing (cleared links stay cleared), healing saves stranded before the
drawer moved to the stable identity. Also read the runtime key before its
copy-if-missing container-name snapshot so the freshest write wins among
the docker-store fallbacks.
Refs #1556
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
- Add DOMPurify sanitization for AI chat markdown rendering (XSS fix)
- Configure DOMPurify to add target=_blank and rel=noopener to links
- Update system prompt to align with command approval policy
- Clarify safe vs destructive commands in prompt
- Improve patrol auto-fix mode guidance with safe operation list
- Add verification requirements for auto-fix actions
- Update observe-only mode to be clearer about read-only restrictions
Tests cover nil/empty inputs, basic field copying, time fields
(CreatedAt/UpdatedAt with nil and zero value handling), update status
conversion, endpoint ports, and labels cloning.
Two critical fixes to prevent test timeouts:
1. Nil map panic in TestPollPVEInstanceUsesRRDMemUsedFallback:
- Test monitor was missing nodeLastOnline map initialization
- Panic occurred when pollPVEInstance tried to update nodeLastOnline[nodeID]
- Caused deadlock when panic recovery tried to acquire already-held mutex
- Added nodeLastOnline: make(map[string]time.Time) to test monitor
2. Alert manager goroutine leak in Docker tests:
- newTestMonitor() created alert manager but never stopped it
- Background goroutines (escalationChecker, periodicSaveAlerts) kept running
- Added t.Cleanup(func() { m.alertManager.Stop() }) to test helper
These fixes resolve the 10+ minute test timeouts in CI workflows.
Related to workflow run 19281508603.
Fixes three test failures that were blocking release workflow:
1. TestApplyDockerReportGeneratesUniqueIDsForCollidingHosts:
- Initialize dockerTokenBindings and dockerMetadataStore in test helper
- These maps were nil causing panic on first access
2. TestSendGroupedAppriseHTTP & TestSendTestNotificationAppriseHTTP:
- Configure allowlist to permit localhost (127.0.0.1) for test servers
- SSRF protection was blocking httptest.NewServer() URLs
- Tests need to allowlist the test server IP to bypass security checks
Related to workflow fix in 5fa78c3e3.
Extends Docker container monitoring with comprehensive disk and storage information:
- Writable layer size and root filesystem usage displayed in new Disk column
- Block I/O statistics (read/write bytes totals) shown in container drawer
- Mount metadata including type, source, destination, mode, and driver details
- Configurable via --collect-disk flag (enabled by default, can be disabled for large fleets)
Also fixes config watcher to consistently use production auth config path instead of following PULSE_DATA_DIR when in mock mode.
rcourtman