fix: correct Proxmox guest rate sampling

This commit is contained in:
rcourtman
2026-07-24 10:17:21 +01:00
parent ac0fb263c2
commit 49217d284d
42 changed files with 1717 additions and 719 deletions
@@ -36,6 +36,11 @@ one compressed unified-agent report without a collector-side suffix cap.
websocket payloads. Carrying plural availability facets through that serializer
is an adjacent monitoring/API projection and does not change agent enrollment,
report admission, removal, update, profile, or command authority.
The JSON-excluded Proxmox VM/LXC I/O-rate validity fields carried by
`internal/models/models.go` are likewise monitoring-owned sidecar evidence.
They distinguish a valid idle interval from an unknown rate for history,
alerts, and unified-resource projection; they do not describe agent report
admission, heartbeat, enrollment, command reachability, or fleet liveness.
Docker / Podman report sizing is a two-dimensional transport contract:
`pkg/agents/docker/report_limits.go` owns the inclusive encoded HTTP-body and
decoded JSON byte ceilings, derives the 80% early-warning boundaries, and
@@ -43,6 +43,13 @@ used/free/total evidence must not open or clear a memory alert. If such a
sample follows an active alert, the alert remains active with its last trusted
value until a subsequent trusted sample proves recovery; missing evidence is
not evidence that pressure disappeared.
Proxmox guest disk-read, disk-write, network-in, and network-out thresholds
consume only monitoring-owned valid rate observations. A valid idle interval
is explicit zero and may prove recovery; a first sample, missing/null counter,
partial response, or rejected out-of-order sample is unknown and must not
start, clear, or match a custom metric filter. Alert units remain MiB/s at the
threshold boundary (`bytes/s / 1024 / 1024`); that display/threshold conversion
must not be applied to the upstream cumulative-counter divisor.
## Canonical Files
@@ -144,6 +144,31 @@ no core-count division and no in-guest host-agent substitution. A linked host
agent may supplement metrics the platform does not provide, but guest CPU and
its `vm` / `system-container` history target remain platform-owned so dashboard,
details, API state, alerts, and history cannot select different authorities.
Proxmox guest disk and network throughput has one cumulative-counter sampling
contract. `diskread`, `diskwrite`, `netin`, and `netout` are cumulative bytes;
the canonical rate is `(current counter - previous counter) / elapsed
observation seconds`, in bytes per second, with no 1024 divisor. Elapsed time
comes from the receipt time stamped immediately after the relevant Proxmox API
response is decoded, not from later guest-agent, filesystem, or metadata
enrichment. Each counter keeps an independent adjacent-sample baseline:
explicitly unchanged counters produce a valid zero, missing/null fields produce
unknown, out-of-order samples produce unknown without moving the baseline, and
a counter decrease caused by restart, reconnect, migration epoch change, or
wrap rebases that counter and produces a valid zero for the reset interval. A
source-uptime rollback rebases the complete counter epoch and leaves the first
post-restart rate unknown, including when a busy guest already surpassed its
pre-restart counter value before the next poll.
First-sample and missing-field unknowns remain internal validity state; the
legacy API/websocket guest number fields stay numeric, while history, unified
metrics, and alerts omit the unknown observation instead of manufacturing
zero. The rate-tracker identity is `(configured PVE instance, guest kind,
VMID)`: it survives node migration, separates QEMU from LXC, and prevents
duplicate configured cluster identities from sharing a concurrent baseline.
Idle and partial samples still refresh tracker liveness.
Proxmox row liveness uses the same cadence-derived threshold as source
freshness (`max(2 * configured poll interval, 60s)`). Node offline grace and
guest preservation must not expire between healthy 60- or 90-second polls, and
must not use a separate fixed 60-second timer.
Tenant monitor enumeration is monitoring-owned runtime topology, not a
reporting source of truth. `MultiTenantMonitor.ListOrganizationIDs` may expose
persisted organization IDs to API-owned background workers, but it must not
@@ -202,6 +227,8 @@ node-local Agent evidence.
20. `internal/dockeragent/collect.go`
21. `pkg/proxmox/ceph.go`
21a. `pkg/proxmox/cluster_client.go`
21b. `pkg/proxmox/client.go`
21c. `pkg/proxmox/io_counters.go`
22. `pkg/proxmox/zfs.go`
23. `internal/monitoring/guest_memory_sources.go`
24. `internal/monitoring/guest_memory_stability.go`
@@ -222,6 +249,7 @@ node-local Agent evidence.
37. `pkg/agents/docker/report.go`
38. `internal/models/models.go`
38a. `internal/models/proxmox_guest_state.go`
38b. `internal/models/metrics_types.go`
39. `internal/models/models_frontend.go`
40. `internal/models/converters.go`
41. `internal/models/deepcopy.go`
@@ -1076,14 +1104,17 @@ legacy PBS and PMG summaries on current `LastSeen` and health state each tick,
so long-lived infrastructure, workloads, storage, and recovery demos do not
decay into synthetic stale-state warnings while mock mode remains enabled.
That same Proxmox container monitoring boundary now also owns runtime counter
recovery when the lower-fidelity container list or cluster-resources payload
reports stale or zero I/O totals. `internal/monitoring/monitor_pve.go`,
authority when the lower-fidelity container list or cluster-resources payload
and the current-status payload differ. `internal/monitoring/monitor_pve.go`,
`internal/monitoring/monitor_pve_guest_lxc.go`, and
`internal/monitoring/monitor_polling_containers.go` must merge the current
`GetContainerStatus` counters through one canonical `mergeContainerRuntimeCounters`
path before LXC rate calculation and must reuse the same prefetched status
snapshot for metadata enrichment instead of paying disconnected metric and
metadata status reads that can diverge.
path before LXC rate calculation. A present status field is newer authority
even when it is zero or lower after a restart; an absent/null status field
retains the listing field and its presence state. The merge must retain the
status response receipt time and reuse the same prefetched status snapshot for
metadata enrichment instead of paying disconnected metric and metadata status
reads that can diverge.
That same Proxmox backup/snapshot boundary owns bounded concurrent guest
snapshot enumeration. `internal/monitoring/monitor_backups.go` must query VM
and LXC snapshot endpoints through one capped worker pool and preserve
@@ -1349,6 +1349,7 @@
"exact_files": [
"internal/api/host_agent_removal_lifecycle_integration_test.go",
"internal/config/host_continuity_test.go",
"internal/models/metrics_types_test.go",
"internal/monitoring/monitor_host_agent_removal_lifecycle_test.go",
"internal/monitoring/monitor_host_agents_test.go",
"scripts/installtests/agent_state_dir_lifecycle_test.go",
@@ -5184,6 +5185,7 @@
"internal/mock/platform_fixtures.go",
"internal/models/converters.go",
"internal/models/deepcopy.go",
"internal/models/metrics_types.go",
"internal/models/models.go",
"internal/models/models_frontend.go",
"internal/models/proxmox_guest_state.go",
@@ -5192,7 +5194,9 @@
"pkg/agents/host/report.go",
"pkg/agents/kubernetes/report.go",
"pkg/proxmox/ceph.go",
"pkg/proxmox/client.go",
"pkg/proxmox/cluster_client.go",
"pkg/proxmox/io_counters.go",
"pkg/proxmox/zfs.go"
],
"verification": {
@@ -5397,6 +5401,7 @@
"match_files": [
"internal/models/converters.go",
"internal/models/deepcopy.go",
"internal/models/metrics_types.go",
"internal/models/models.go",
"internal/models/models_frontend.go",
"internal/models/proxmox_guest_state.go",
@@ -5408,6 +5413,7 @@
"internal/dockeragent/swarm_coverage_test.go",
"internal/models/deepcopy_test.go",
"internal/models/host_transient_expiry_test.go",
"internal/models/metrics_types_test.go",
"internal/models/state_host_test.go",
"internal/monitoring/issue1595_collection_trust_test.go",
"internal/monitoring/monitor_full_coverage_test.go",
@@ -5418,6 +5424,24 @@
"internal/unifiedresources/registry_test.go"
]
},
{
"id": "proxmox-guest-counter-runtime",
"label": "Proxmox guest cumulative-counter presence and receipt-time proof",
"match_prefixes": [],
"match_files": [
"pkg/proxmox/client.go",
"pkg/proxmox/io_counters.go"
],
"allow_same_subsystem_tests": false,
"test_prefixes": [],
"exact_files": [
"internal/api/router_state_test.go",
"internal/monitoring/issue1613_contract_test.go",
"internal/monitoring/monitor_pve_guest_lxc_test.go",
"internal/monitoring/ratetracker_test.go",
"pkg/proxmox/io_counters_test.go"
]
},
{
"id": "proxmox-zfs-runtime",
"label": "proxmox zfs compatibility proof",
@@ -5587,6 +5611,7 @@
"internal/monitoring/canonical_guardrails_test.go",
"internal/monitoring/issue1485_unraid_lifecycle_test.go",
"internal/monitoring/issue1595_collection_trust_test.go",
"internal/monitoring/issue1613_contract_test.go",
"internal/monitoring/monitor_additional_test.go",
"internal/monitoring/monitor_alert_intent_test.go",
"internal/monitoring/monitor_alert_override_migration_test.go",
@@ -5595,6 +5620,8 @@
"internal/monitoring/monitor_host_agent_removal_lifecycle_test.go",
"internal/monitoring/monitor_host_agents_test.go",
"internal/monitoring/monitor_pve_cluster_refresh_test.go",
"internal/monitoring/monitor_pve_guest_lxc_test.go",
"internal/monitoring/ratetracker_test.go",
"internal/unifiedresources/code_standards_test.go"
]
}
@@ -1568,6 +1568,11 @@ recovery scope, or a storage/recovery-owned secret source.
remain absent optional timestamps on that shared facet; storage and recovery
must not reinterpret absence or a year-one zero-time serialization as
recovery age, missed backup cadence, or restore freshness.
The same shared type file keeps `MetricValue.value` numeric whenever a
metric object is present. Storage/recovery consumers must preserve explicit
zero as an observed idle value and use absence of the metric object for
unknown Proxmox guest I/O; they must not reinterpret zero as missing or
accept a JSON `null` value as recovery, backup, or storage-health evidence.
## Forbidden Paths
@@ -55,6 +55,14 @@ at the adapter boundary; a mutation that arrives during a rebuild must apply to
the newly published registry rather than a superseded pointer. Authoritative
snapshot omissions still remove resources and emit the normal canonical
history change.
Proxmox guest I/O projection preserves the monitoring-owned rate validity
contract. A valid idle interval publishes an explicit zero-valued `bytes/s`
metric; an unknown first, missing, partial, or out-of-order observation leaves
that metric absent. `MetricValue.value` is always a numeric JSON field when a
metric object exists, so `/api/resources`, `/api/state`, websocket resources,
history, alerts, and UI adapters never need to interpret `null` as either idle
or unknown. Unified resources must not infer validity from the numeric guest
compatibility field alone.
Physical-disk resources own cross-source disk identity. When Proxmox inventory
and host-agent SMART telemetry describe the same device, the merged resource
must retain Proxmox node/instance source payloads while carrying SMART