monitorFrontendStatus's "pod" branch was re-deriving status from
Kubernetes.PodPhase alone, so a CrashLoopBackOff pod (Phase=Running
with unready containers) was reported as "running" even though
statusFromKubernetesPod had already classified it as StatusWarning.
Drop the unconditional phase=running -> "running" mapping and fall
through to the unified-status switch, which surfaces StatusWarning as
"degraded". Phase-based mappings for pending/unknown/succeeded/failed
are kept since they still cover edge cases where the unified status
is StatusOnline but phase carries useful information.
Adds two regression cases in TestMonitorFrontendAndMetricHelpers:
- pod warning running degraded (CrashLoopBackOff)
- pod online running running (healthy baseline)
The Proxmox-side LXC Docker inventory poll occasionally produces an
ok=true report with zero CONTAINER lines but a valid PS_OK marker:
docker ps -a inside the LXC genuinely returned an empty list for
that one call, typically while watchtower is briefly recreating a
container or the dockerd is mid-restart. ApplyDockerReport then
wipes the host's container list, every connected WebSocket client
sees every row for that host flash blank, and the next successful
poll 10 s later restores it.
Before calling ApplyDockerReport, check whether the host already
has containers tracked in state. If yes and the new report has
zero containers, log at debug and skip the apply, counting the
poll as skipped rather than collected. A host that is genuinely
empty still applies on first sight (no previous containers means
no previous state to protect).
Add regression tests for both branches: a populated host receiving
an empty report keeps its container list; a freshly-seen empty
host still gets created.
The Proxmox-LXC Docker hosts row was blank (no CPU / MEM-used /
Disk) and the display name appended a noisy "(LXC <vmid>)" suffix,
because the inventory script only collects Docker daemon metadata,
not host resource usage. Pulse already polls the same data for the
underlying LXC via the PVE cluster/resources endpoint, so the
Docker host row should mirror what the rest of Pulse already knows.
Before applying each LXC inventory report, enrich the Host info
with CPU / Memory / Disk / Uptime taken from the source LXC's
existing models.Container. Drop the "(LXC <vmid>)" suffix from
the synthesized display name. On the frontend, detect the
proxmox-lxc-docker: hostSourceId prefix and emit a styled
"LXC <vmid>" badge into the System column instead of the
filtered-out runtime-only "docker" badge.
The Proxmox-side LXC Docker inventory script masked `docker ps`
failures with `|| true`, so when `pct exec` was truncated, slow, or
`docker ps` itself transiently failed, the parser produced an
ok=true report with `containers=[]` and `ApplyDockerReport` blanked
the host's container list. Containers then reappeared on the next
successful poll, producing the visible flicker on the Docker page.
Emit a `PS_OK` marker only when `docker ps` exits cleanly, and
refuse to apply the report when the marker is absent. A genuine
"host has zero containers" state still applies (PS_OK present,
zero CONTAINER lines), but a truncated or failed poll now skips
apply and lets the previous container list survive the inter-poll
gap. Add regression tests for both paths.
Two specific platform-page quality issues from the audit:
1. **/kubernetes/overview only had 1 cluster.** Bumping the K8s cluster
count past 1 had been deferred because the prior
monitor-broadcast equivalence test compared the raw snapshot count
to the broadcast count exactly, and broadcast's
`coalesceBroadcastResources` + second-pass coalesce inside
`convertResourcesForBroadcast` legitimately drops merge candidates
that the raw snapshot keeps. Switch the test to compare against
the canonical snapshot count within a ±5% tolerance so future
fixture bumps stay green without loosening any of the test's
exact-name and exact-identity assertions. With that in place, bump
`K8sClusterCount` 1 → 3 in `internal/mock/generator.go`,
`scripts/toggle-mock.sh`, and the matching
`scripts/tests/test-toggle-mock.sh` so the canonical mock estate
ships with production + staging + edge clusters end-to-end.
Live mock survey: k8s-cluster: 3, k8s-deployment: 42, pod: 120,
plus 15 K8s nodes merged onto their agent hosts.
2. **/vmware/storage looked empty under platform-page chrome.**
`resolveStorageSourceKey` was reading only `storage.type` (the
on-disk technology like `vsan`, `vmfs`, `nfs41`, `zfs-pool`) and
never consulted `storage.platform` (the canonical platform key
like `vmware-vsphere` or `truenas`). Source filter chip options
were therefore generated as `vsan`, `vmfs`, `nfs41`, etc., and
`forcedSourceFilter='vmware-vsphere'` had nothing to match.
Prefer the canonical `storage.platform` tag when set, so VMware
datastores group under `vmware-vsphere`, TrueNAS pools group under
`truenas`, PBS datastores under `proxmox-pbs`, etc., for both the
chip options and the embedded platform-page filter.
Browser verification (Playwright, chromium, live mock-mode dev runtime):
- 9 tests pass.
Targeted vitest:
- `src/features/storageBackups` + `src/utils/__tests__/sourcePlatforms.test.ts` +
`src/components/Storage/__tests__/storageSourceOptions.test.ts` (31
files / 141 tests) green.
Go tests:
- `go test ./internal/mock/... ./internal/monitoring/... ./internal/vmware/...`
all green.
Contracts updated:
- `monitoring.md` Shared Boundaries: new K8s multi-cluster default,
±5% tolerance for the broadcast equivalence assertion.
- `deployment-installability.md` Shared Boundaries: toggle-mock.sh /
DefaultConfig parity updated for the K8sClusterCount=3 baseline.
Mock pages were sparse: 3 Proxmox nodes × 3 VMs × 3 LXCs, 2 Docker
hosts × 5 containers, 1 K8s cluster × 3 nodes × 10 pods × 4
deployments. That populated platform pages with handfuls of rows
rather than table density that exercises sorting, grouping, drawers,
and responsive layout.
Bump `internal/mock/generator.go::DefaultConfig` to target a mature
small-to-mid homelab / SMB environment:
- NodeCount: 3 → 5 (matches the curated demo scenario's pve1..pve5
regional naming)
- VMsPerNode: 3 → 6
- LXCsPerNode: 3 → 8
- DockerHostCount: 2 → 5
- DockerContainersPerHost: 5 → 14
- GenericHostCount: 2 → 4
- K8sClusterCount: 1 (unchanged; the curated demo and broadcast
coalesce tests assume a single cluster identity)
- K8sNodesPerCluster: 3 → 5
- K8sPodsPerCluster: 10 → 40
- K8sDeploymentsPerCluster: 4 → 14
Resource survey under the new defaults (live mock backend):
- TOTAL 307 unique resources (was ~50-100)
- app-container: 75, storage: 55, system-container: 44, pod: 40,
vm: 31, physical_disk: 19, agent: 15, k8s-deployment: 14,
docker-host: 5, network-endpoint: 5, pmg: 2, pbs: 1, k8s-cluster: 1
Platform pages now feel populated under mock mode:
- /docker/overview: 5 hosts (was 2)
- /docker/containers: 75 containers (was 13)
- /kubernetes/nodes: 5 (was 3)
- /kubernetes/pods: 40 (was 10)
- /kubernetes/deployments: 14 (was 4)
`internal/mock/demo_scenarios.go` extended to season `local`,
`local-zfs`, and per-node iso/service-pool storage names for pve6 and
beyond, so future NodeCount bumps don't regress the curated demo into
generic "service-pool" labels (a test guard explicitly forbids that
alias). A new `TestDemoScenarioStorageNamingHandlesScaledNodeCount`
covers the scaled-NodeCount path.
`internal/monitoring/monitor_unified_state_test.go` updated to compare
the broadcast count against the coalesced snapshot count rather than
the raw snapshot count — the broadcast path merges resources that
share a canonical host key (K8s nodes onto linked agent hosts), so
larger fixture sizes legitimately produce more merge candidates, and
the prior raw-equality assertion would have broken on any future
fixture growth too. The test still asserts every canonical name and
mock identity it checked before.
`scripts/toggle-mock.sh` (`mock_default_entries`) and the matching
`scripts/tests/test-toggle-mock.sh` assertions are aligned with the
new defaults so `npm run mock:edit` and per-dev `.env` seeding match
the canonical baseline.
Contracts updated:
- `monitoring.md` Shared Boundaries: records the new DefaultConfig
target sizes and the requirement that demo-scenario seasoning stay
aligned with NodeCount changes.
- `deployment-installability.md` Shared Boundaries: records that
`mock_default_entries()` in toggle-mock.sh must stay aligned with
`internal/mock.DefaultConfig` so CLI/toggle/runtime mock densities
never drift apart.
Targeted Go tests:
- `go test ./internal/mock/...` green
- `go test ./internal/monitoring/...` green
Playwright (chromium, live mock-mode dev runtime):
- 9 tests, all pass; populated assertions now hit dense tables (5
hosts, 14+ containers, 40 pods, etc.).
Known remaining fixture gaps (canonical adapter, not config):
- VMware fixture inventory in `internal/vmware/fixtures.go` is
hardcoded at 4 hosts / 6 VMs / 4 datastores; not scaled in this
commit.
- TrueNAS fixture inventory in `internal/truenas/fixtures.go` is
similarly hardcoded; not scaled in this commit.
Ensure unified resource snapshots include recent standalone host-agent continuity so Infrastructure does not briefly undercount connected systems after login or restart.
Keep desired config fingerprints as response metadata derived from the signed command and settings payload.
Use merged agent profile settings when building remote config fingerprints.
A Proxmox host wedged on a ZFS deadlock yesterday took the cluster API poll
with it (context deadline exceeded). The unified connections aggregator
flipped the Connection from active to stale to unreachable, and the
Settings / Infrastructure page rendered the right badges, but no top-nav
alert ever fired because nothing was actively notifying off that derived
state. Patrol's deterministic triage flagged it every minute, but its LLM
investigation stage has been broken since 2026-02-26 so flags never
escalated into user-visible findings. Result: a 3 hour outage I only
noticed because I happened to open Settings.
This wires an active notification off the same connection state the
Settings badges already use:
- internal/alerts/connection.go: new CheckConnection +
clearConnectionDegradedAlert that fire connection-degraded after three
consecutive stale or unreachable observations. Severity scales: stale
warning, unreachable / unauthorized critical. Clear runs through the
same recovery-confirmation gate as clearNodeOfflineAlert so a single
flap back to active doesn't silently resolve a real outage. Paused,
disabled, and non-platform connections are no-ops.
- internal/api/connections_alerts.go: snapshot translator that turns
api.Connection into the narrow alerts.ConnectionSnapshot view. Keeping
the snapshot type inside the alerts package preserves the existing
api -> monitoring import direction; the monitor would have cycled if
it called back into api directly.
- internal/monitoring: new SetConnectionsSnapshotLister hook + a
per-tick checkConnectionAlerts call in the main poll loop, alongside
the existing evaluate*Agents passes.
- internal/api/router.go: register the lister closure on r.monitor so
the alerts loop sees the same Connection rows the HTTP handler does.
- internal/alerts/specs/types.go: add "connection" to the migration
bridge list of accepted ResourceTypes, alongside node / docker-host /
proxmox-disk / etc. The connection concept doesn't have a canonical
unified resource type yet; this matches the existing pattern for
alert-keyed resources that aren't first-class canonical.
Test coverage in internal/alerts/connection_test.go covers active never
fires, three stale observations escalate from pending to warning,
unreachable escalates warning to critical, unauthorized fires critical
cold, paused / disabled / agent never fire, recovery confirmation gate,
and a stale flap during recovery resets the gate.
TestResourceAlertSpecValidateAllowsConnectionMigrationBridgeType mirrors
the existing migration-bridge proof tests for the new type.
The PBS poller's version-failure exit at monitor_pbs_pmg.go did not
set pollErr before returning, so even after fixing the defer-arg
capture in bf6261adc the deferred recordTaskResult still saw nil and
recorded the poll as a success. PMG's analogous path already sets
pollErr correctly. Mirror that here so the per-instance pollStatusMap,
the connections aggregator, and the circuit breaker all see PBS auth
and version failures as failures.
Add assertions to the existing PBS and PMG auth-failure tests that
the per-instance pollStatusMap entry has a zero LastSuccess and a
non-zero ConsecutiveFailures. The original tests covered downstream
state but not the recorder, which is why two distinct cases of this
class of bug went unnoticed.
The deferred recordTaskResult call was passing pollErr as a function
argument, so it captured the value at defer-time (always nil) instead
of the value at execution time. Result: the per-instance pollStatusMap
treated every poll as a success — LastSuccess was set to "now" on
every cycle, ConsecutiveFailures stayed at zero, and the circuit
breaker never opened, even when the staleness tracker (which used a
proper closure) recorded the same poll as a failure.
The connections aggregator derives state from PollStatus.LastSuccess,
so the Connections UI reported broken PVE/PBS/PMG instances as
"active / verified / healthy" while no data was ingested. Wrap the
recordTaskResult call in a defer closure so it reads the live pollErr
at execution time, matching the pollMetrics and stalenessTracker
defers immediately above.
Add three monitoring broadcast guardrail tests covering platform
identity for resources that have both an API/appliance facet and a
linked Pulse host agent:
- Proxmox VE node + agent (Pi case) stays on platformType=proxmox-pve
- TrueNAS appliance + agent stays on platformType=truenas
- Single-source Unraid agent host stays on platformType=agent
These pin the contract that the API/appliance facet wins over the
agent facet during platform type derivation, and that single-source
agent hosts do not promote OSName strings to platform identifiers.
Retire runtime/API/UI monitored-system volume enforcement now that infrastructure monitoring is no longer capped.
Keep only legacy metadata scrubbing and purchase-start compatibility for old max_monitored_systems references.
Rename the remaining preview surface to monitored-system impact and make previews explanatory rather than save-blocking.
Update subsystem contracts and RA7 evidence for the caps-retired invariant.
Refs #1447
Use the canonical Docker host identity for token bindings after a matched reconnect, preserve previous/current identity aliases for recreated containers, and keep rejecting token reuse by genuinely different Docker hosts.
Record the RC3 release-gate evidence for the Docker-in-LXC reconnect path.
Parse the /proc/mdstat operation keyword for mdadm arrays and propagate it through host reports, models, unified resources, monitoring views, alert metadata, and AI storage summaries.
Treat recovery and reshape as rebuild signals while silencing routine check and resync maintenance, with fallback rebuild detection only when no mdstat operation is available.
Tests cover mdstat operation parsing plus recovery, check, and resync alert behavior.
Fixes#1446
Introduces GET /api/connections and POST /api/connections/probe as the
backend half of the one-ledger / one-editor connection redesign.
- GET /api/connections aggregates PVE/PBS/PMG/VMware/TrueNAS/agent rows
into a unified Connection shape with derived state (active, paused,
unauthorized, unreachable, stale, pending) computed from in-memory
scheduler health plus agent Host.LastSeen. No new persisted state.
- POST /api/connections/probe fingerprints a host across the five
supported products in parallel (2s dial + 1s read, 3s total, max 5
concurrent). Admin-gated (RequireAdmin + ScopeSettingsWrite) to block
unauthenticated SSRF against internal hosts.
- Disabled bool on PVEInstance/PBSInstance/PMGInstance (zero-value =
enabled, preserves existing nodes.json); pollers skip disabled
instances at client init, reconnect, and per-node iteration.
- NodeConfigRequest/Response gain Enabled; write path translates
*bool -> Disabled so omitted field leaves state untouched.
- ConnectionsAPI frontend client (list/probe) typed off the Go shape.
Contracts updated: api-contracts, monitoring, agent-lifecycle,
performance-and-scalability, storage-recovery. Proofs added:
contract_test.go JSON snapshot for Connection and ProbeResponse,
monitoring guardrails for the Disabled-skip behavior, and a vitest
mock-client test for ConnectionsAPI.
Frontend editor / drawer / table rewrite lands in a separate block.
rcourtman