Merge pull request #2152 from rcourtman/maintainer/20260922T082007Z

Fix spurious PBS backup alerts and Proxmox LXC memory; qualify release-line RCs
This commit is contained in:
pulse-triage[bot]
2026-09-22 08:51:44 +00:00
committed by GitHub
20 changed files with 497 additions and 31 deletions
@@ -1620,6 +1620,18 @@ of a reporter installation or a migration that guesses operator intent.
`TestMergeSnapshotOverrideInheritsZeroFields` in
`internal/alerts/alerts_test.go` pin the merge, migration, and preservation
behaviors.
Backup-age alerts are guest-workload scoped. A PBS recovery rollup whose subject
carries no guest VMID is a host/config backup (`backup-type host`, keyed by the
node name), not a guest workload, and must not raise a guest backup-age alert:
the Backups overview already excludes those entries from guest coverage, and a
stale host config backup otherwise notified forever under a node-named subject
while the node's guests were backed up. Evaluation skips PBS subjects with an
empty VMID, retaining their recovery point and rollup so inventory and artifact
display are unchanged. PVE host backups already produce no subject and no alert.
Genuinely orphaned PBS guest backups (numeric VMID) and PMG node backups still
alert. `TestCheckBackupsIgnoresPBSHostBackupSubject` and
`TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup` in
`internal/alerts/alerts_test.go` pin both sides of this boundary (#2136).
Proxmox disk health alert evaluation now lives in
`internal/alerts/disk_health.go`. That file owns Proxmox disk canonical
identity, disk health assessment alerts, known-firmware health suppression, and
@@ -534,7 +534,12 @@ workflow execution ref and SHA equal the peeled tag commit. That workflow consum
release assets and four release signatures, runs the canonical twenty-three-scenario
schema-v7 lab, and applies the release-candidate attester against the exact
GitHub release ID, tag, source commit, checksums, compiler provenance, assembly
provenance, and update-key fingerprint. The v7 host starts a rootful Docker
provenance, and update-key fingerprint. The attester anchors the candidate to
the branch the control plane maps the version to, either `origin/main` or the
release train's `origin/release/vX.Y`, and verifies that remote-tracking ref
against the canonical origin's live branch tip, so a release-train candidate is
qualified against the line that owns its version rather than being required to
appear on `main`. The v7 host starts a rootful Docker
daemon inside the disposable systemd container without mounting the hosted
runner socket or exposing a TCP listener. It imports an offline, source-bound
fixture image and proves legacy inventory, summary-only typed-helper parity,
@@ -17,6 +17,25 @@
## Purpose
### Linked Pulse agent memory for Proxmox LXC — issue #2148 (22 September 2026)
Both LXC memory paths (the efficient `cluster/resources` builder and the
per-node fallback) may substitute the linked Pulse agent's own memory sample for
a Proxmox fallback listing value. The substitution applies only when the current
memory source is a fallback (`cluster-resources` or `unavailable`), the guest is
`running`, the linked agent sample has known usage, and the agent total matches
the container's provisioned limit with used not exceeding it. A stopped
container, a non-fallback source, a missing linked agent, and a total mismatch
are left untouched; preferred or derived guest evidence is never replaced. The
provisioned platform total is retained, and the previous-guest agent map is
threaded through both builders.
`TestIssue2148LXCPrefersLinkedAgentMemoryOverClusterResources` and
`TestIssue2148LXCRejectsAgentMemoryWithMismatchedTotal` pin the substitution and
the rejection boundary; `TestCorrelatedContainerMemoryNextPoll` pins the
next-poll merged-row path. This is synthetic runtime proof, not a live Proxmox
installation or browser receipt.
### Preserve explicit agent filesystem selection at ingestion
Both `ApplyHostReport` and `ApplyDockerReport` honour a disk's optional
@@ -37,6 +37,17 @@ policy remains authoritative; this accessor introduces no second timeout.
`TestCorrelatedGuestMemoryNextPoll` verifies the manually merged row through the
next-poll consumers. Existing host-list membership and metric priority are unchanged.
**System-container linked agent memory read — issue #2148 (22 September 2026)**
`ContainerView.LinkedAgentMemory` exposes the same value-copy of the linked
agent's own memory sample for system containers, sharing the validity and
freshness rules with `VMView.LinkedAgentMemory`: missing/non-online agent source
status, explicit agent staleness, unavailable usage and invalid capacity or
counters are rejected, and trusted zero usage remains valid. The monitoring
layer consumes it only as a fallback replacement for Proxmox LXC memory, so
platform metric priority for a healthy agent is unchanged.
`TestContainerViewLinkedAgentMemory` pins those boundaries.
### Bounded incident-history selection
Canonical history queries filter exact alert identifiers and observation windows
+90
View File
@@ -2322,6 +2322,96 @@ func TestCheckBackupsHandlesPmgBackups(t *testing.T) {
}
}
// TestCheckBackupsIgnoresPBSHostBackupSubject covers #2136: a PBS host config
// backup (backup-type "host") is keyed by the node name, not a VMID, and must
// not raise a guest backup-age alert. The node name subject ("nas backup")
// otherwise looks like a stale guest and notifies forever even though the
// node's guests are backed up.
func TestCheckBackupsIgnoresPBSHostBackupSubject(t *testing.T) {
m := newTestManager(t)
m.ClearActiveAlerts()
m.mu.Lock()
m.config.Enabled = true
m.config.BackupDefaults = BackupAlertConfig{
Enabled: true,
WarningDays: 7,
CriticalDays: 14,
}
m.mu.Unlock()
now := time.Now()
rollups := []recovery.ProtectionRollup{
{
RollupID: "ext:pbs-host-nas",
SubjectRef: &recovery.ExternalRef{
Type: "proxmox-guest",
Namespace: "pbs",
Name: "nas",
ID: "nas",
},
LastSuccessAt: ptrTime(now.Add(-43.5 * 24 * time.Hour)),
LastOutcome: recovery.OutcomeSuccess,
Providers: []recovery.Provider{recovery.ProviderProxmoxPBS},
},
}
m.CheckBackups(rollups, map[string]GuestLookup{}, map[string][]GuestLookup{})
m.mu.RLock()
defer m.mu.RUnlock()
for storageKey, alert := range m.activeAlerts {
if alert != nil && alert.Type == "backup-age" {
t.Fatalf("expected PBS host config backup to be excluded from guest backup-age alerts, found %s", effectiveAlertID(alert, storageKey))
}
}
}
// TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup is the companion guard:
// a genuinely orphaned PBS guest backup (numeric VMID, no live guest) must keep
// alerting, so the host-backup exclusion above does not silence real staleness.
func TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup(t *testing.T) {
m := newTestManager(t)
m.ClearActiveAlerts()
m.mu.Lock()
m.config.Enabled = true
m.config.BackupDefaults = BackupAlertConfig{
Enabled: true,
WarningDays: 7,
CriticalDays: 14,
}
m.mu.Unlock()
now := time.Now()
rollups := []recovery.ProtectionRollup{
{
RollupID: "ext:pbs-orphan-102",
SubjectRef: &recovery.ExternalRef{
Type: "proxmox-vm",
Namespace: "pbs",
Name: "102",
ID: "102",
},
LastSuccessAt: ptrTime(now.Add(-43.5 * 24 * time.Hour)),
LastOutcome: recovery.OutcomeSuccess,
Providers: []recovery.Provider{recovery.ProviderProxmoxPBS},
},
}
m.CheckBackups(rollups, map[string]GuestLookup{}, map[string][]GuestLookup{})
m.mu.RLock()
defer m.mu.RUnlock()
if !testHasActiveAlert(t, m, buildCanonicalStateID("backup-subject:ext-pbs-orphan-102", "backup-subject:ext-pbs-orphan-102-backup-age")) {
var keys []string
for storageKey, active := range m.activeAlerts {
keys = append(keys, effectiveAlertID(active, storageKey))
}
t.Fatalf("expected orphaned PBS guest backup alert, found keys: %v", keys)
}
}
func TestCheckBackupsSkipsOrphanedWhenDisabled(t *testing.T) {
m := newTestManager(t)
m.ClearActiveAlerts()
+10
View File
@@ -859,6 +859,16 @@ func (m *Manager) CheckBackupsWithInventory(
}
ageDaysRounded := math.Round(ageDays*10) / 10
// A PBS subject without a guest VMID is a host/config backup
// (PBS backup-type "host", keyed by the node name), not a guest
// workload. The Backups overview already keeps these out of guest
// coverage, so do not raise a guest backup-age alert for them: a
// stale host config backup otherwise notifies forever under a
// node-named subject while the node's guests are backed up (#2136).
if record.source == "PBS" && strings.TrimSpace(record.vmID) == "" {
continue
}
// Determine thresholds for this backup
currentBackupCfg := backupCfg
guestContext := guestSnapshotFromLookup(record.lookup, record.fallbackName)
@@ -382,3 +382,49 @@ func (m *Monitor) resolveGuestStatusMemory(
return memTotal, memUsed, memorySource
}
// preferLinkedAgentLXCMemory replaces a low-trust Proxmox LXC memory reading
// with the linked Pulse agent's own sample when one is available. The agent runs
// inside the container, so its sample reflects the workload's real footprint,
// while cluster/resources is a cache-inclusive cgroup fallback that can badly
// under- or over-report shared-memory workloads (#2148, mirroring #1962).
//
// An agent inside a container without lxcfs sees the host's /proc/meminfo, so a
// sample is only trusted when its total matches the container's configured
// limit. Preferred provider sources are never overridden.
func preferLinkedAgentLXCMemory(
status string,
memTotal uint64,
memUsed uint64,
memorySource string,
guestRaw *VMMemoryRaw,
agentHost models.Host,
hasAgent bool,
) (uint64, uint64, string) {
if !hasAgent || status != "running" || memTotal == 0 {
return memTotal, memUsed, memorySource
}
switch CanonicalMemorySource(memorySource) {
case "cluster-resources", "unavailable":
default:
return memTotal, memUsed, memorySource
}
if !agentHost.Memory.HasKnownUsage() || agentHost.Memory.Total <= 0 || agentHost.Memory.Used < 0 {
return memTotal, memUsed, memorySource
}
agentTotal := uint64(agentHost.Memory.Total)
agentUsed := uint64(agentHost.Memory.Used)
if agentTotal != memTotal || agentUsed > memTotal {
return memTotal, memUsed, memorySource
}
if guestRaw != nil {
guestRaw.HostAgentTotal = agentTotal
guestRaw.HostAgentUsed = agentUsed
}
log.Debug().
Uint64("total", memTotal).
Uint64("used", agentUsed).
Str("previousSource", memorySource).
Msg("LXC memory: using linked Pulse agent memory over cluster-resources fallback")
return memTotal, agentUsed, "agent"
}
@@ -1004,6 +1004,7 @@ func TestHandleClusterContainerResourceMemoryTrustCharacterization(t *testing.T)
makeGuestID("test", "node1", tt.res.VMID),
client,
nil,
nil,
)
if !ok {
t.Fatal("handleClusterContainerResource() returned ok=false")
@@ -1009,6 +1009,45 @@ func TestCorrelatedGuestMemoryNextPoll(t *testing.T) {
}
}
// Issue #2148: a correlated agent merged into a system-container resource must
// surface through the previous-guest context so the LXC builder prefers it over
// the cache-inclusive cluster/resources fallback.
func TestCorrelatedContainerMemoryNextPoll(t *testing.T) {
now := time.Now()
const gib = int64(1024 * 1024 * 1024)
guestID := makeGuestID("pve-a", "node1", 100)
store := unifiedresources.NewMemoryStore()
if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: "ct-test", ResourceB: "agent-test", PrimaryID: "agent-test"}); err != nil {
t.Fatal(err)
}
registry := unifiedresources.NewRegistry(store)
total, used := int64(24)*gib, int64(11)*gib
resources := []unifiedresources.Resource{
{ID: "ct-test", Type: unifiedresources.ResourceTypeSystemContainer, Name: "npu-ct", Status: unifiedresources.StatusOnline, LastSeen: now,
Sources: []unifiedresources.DataSource{unifiedresources.SourceProxmox},
Proxmox: &unifiedresources.ProxmoxData{SourceID: guestID, Instance: "pve-a", NodeName: "node1", VMID: 100, RuntimeStatus: "running"},
Metrics: &unifiedresources.ResourceMetrics{Memory: &unifiedresources.MetricValue{Total: &total, Used: &used, Percent: 45, Source: unifiedresources.SourceProxmox}}},
{ID: "agent-test", Type: unifiedresources.ResourceTypeAgent, Name: "npu-ct-agent", Status: unifiedresources.StatusOnline, LastSeen: now,
Sources: []unifiedresources.DataSource{unifiedresources.SourceAgent},
Agent: &unifiedresources.AgentData{AgentID: "agent-test", Memory: &unifiedresources.AgentMemoryMeta{Total: total, Used: used, Free: total - used}}},
}
registry.IngestResources(resources)
if len(registry.Containers()) != 1 || len(registry.Hosts()) != 0 {
t.Fatalf("expected one merged container and no standalone hosts, got %d/%d", len(registry.Containers()), len(registry.Hosts()))
}
mon := &Monitor{state: models.NewState(), rateTracker: NewRateTracker(), config: &config.Config{}, resourceStore: unifiedresources.NewMonitorAdapter(registry)}
prev := mon.previousGuestContextForInstance("pve-a")
if _, ok := prev.hostAgentsByVMID[guestID]; !ok {
t.Fatalf("linked container agent missing from previous context: %+v", prev.hostAgentsByVMID)
}
container, _, source, _, ok := mon.buildContainerFromClusterResource(context.Background(), "pve-a",
proxmox.ClusterResource{Type: "lxc", Status: "running", Node: "node1", VMID: 100, Name: "npu-ct", MaxMem: uint64(total), Mem: 3459743744},
&stubPVEClient{}, map[int]bool{}, prev.hostAgentsByVMID)
if !ok || source != "agent" || container.Memory.Used != used {
t.Fatalf("container memory not agent-backed: ok=%v source=%s mem=%+v", ok, source, container.Memory)
}
}
type correlatedMemoryClient struct{ stubPVEClient }
func (*correlatedMemoryClient) GetVMStatus(context.Context, string, int) (*proxmox.VMStatus, error) {
@@ -154,6 +154,16 @@ func (m *Monitor) collectContainersWithNodes(ctx context.Context, instanceName s
MaxMem: container.MaxMem,
Mem: container.Mem,
})
agentHost, hasAgent := prevGuests.hostAgentsByVMID[guestID]
memTotal, memUsed, memorySource = preferLinkedAgentLXCMemory(
container.Status,
memTotal,
memUsed,
memorySource,
&guestRaw,
agentHost,
hasAgent,
)
memUsed, memorySource, _ = stabilizeGuestLowTrustMemory(
m.previousGuestSnapshot(instanceName, "lxc", n.Node, int(container.VMID)),
container.Status,
+17 -8
View File
@@ -61,6 +61,9 @@ func (m *Monitor) previousGuestContextForInstance(instanceName string) previousG
guestID := makeGuestID(container.Instance, container.Node, container.VMID)
if guestID != "" {
ctx.containersByID[guestID] = container
if memory, ok := ct.LinkedAgentMemory(); ok {
ctx.hostAgentsByVMID[guestID] = models.Host{LinkedVMID: guestID, Status: "online", Memory: memory}
}
}
if container.VMID > 0 && (strings.EqualFold(strings.TrimSpace(container.Type), "oci") || container.IsOCI) {
ctx.containerOCIByVMID[container.VMID] = true
@@ -72,10 +75,15 @@ func (m *Monitor) previousGuestContextForInstance(instanceName string) previousG
continue
}
modelHost := previousHostFromView(host)
if modelHost.LinkedVMID == "" || modelHost.Status != "online" {
if modelHost.Status != "online" {
continue
}
ctx.hostAgentsByVMID[modelHost.LinkedVMID] = modelHost
if modelHost.LinkedVMID != "" {
ctx.hostAgentsByVMID[modelHost.LinkedVMID] = modelHost
}
if modelHost.LinkedContainerID != "" {
ctx.hostAgentsByVMID[modelHost.LinkedContainerID] = modelHost
}
}
return ctx
@@ -172,12 +180,13 @@ func previousHostFromView(host *unifiedresources.HostView) models.Host {
return models.Host{}
}
return models.Host{
ID: host.ID(),
Hostname: host.Hostname(),
Status: string(host.Status()),
LinkedVMID: host.LinkedVMID(),
LastSeen: host.LastSeen(),
Disks: guestDisksFromReadStateView(host.Disks()),
ID: host.ID(),
Hostname: host.Hostname(),
Status: string(host.Status()),
LinkedVMID: host.LinkedVMID(),
LinkedContainerID: host.LinkedContainerID(),
LastSeen: host.LastSeen(),
Disks: guestDisksFromReadStateView(host.Disks()),
Memory: models.Memory{
Used: host.MemoryUsed(),
Total: host.MemoryTotal(),
@@ -58,6 +58,7 @@ func TestBuildContainerFromClusterResource_PreservesProxmoxPool(t *testing.T) {
},
nil,
map[int]bool{},
nil,
)
if !ok {
t.Fatal("expected container to be built")
@@ -44,6 +44,7 @@ func (m *Monitor) buildContainerFromClusterResource(
res proxmox.ClusterResource,
client PVEClientInterface,
prevContainerIsOCI map[int]bool,
vmIDToHostAgent map[string]models.Host,
) (models.Container, VMMemoryRaw, string, time.Time, bool) {
// Skip templates if configured
if res.Template == 1 {
@@ -88,6 +89,16 @@ func (m *Monitor) buildContainerFromClusterResource(
)
memTotal, memUsed, memorySource, guestRaw := m.calculateLXCMemory(res)
agentHost, hasAgent := vmIDToHostAgent[guestID]
memTotal, memUsed, memorySource = preferLinkedAgentLXCMemory(
res.Status,
memTotal,
memUsed,
memorySource,
&guestRaw,
agentHost,
hasAgent,
)
memUsed, memorySource, _ = stabilizeGuestLowTrustMemory(
m.previousGuestSnapshot(instanceName, "lxc", res.Node, res.VMID),
res.Status,
@@ -140,7 +140,7 @@ func TestIssue1613LXCStatusLagDoesNotEraseNewerListingDiskWrite(t *testing.T) {
}
if _, _, _, _, ok := monitor.buildContainerFromClusterResource(
context.Background(), "cluster-a", resource, client, map[int]bool{},
context.Background(), "cluster-a", resource, client, map[int]bool{}, nil,
); !ok {
t.Fatal("expected first LXC sample")
}
@@ -150,7 +150,7 @@ func TestIssue1613LXCStatusLagDoesNotEraseNewerListingDiskWrite(t *testing.T) {
client.containerStatus.ObservedAt = resource.ObservedAt.Add(time.Second)
container, _, _, _, ok := monitor.buildContainerFromClusterResource(
context.Background(), "cluster-a", resource, client, map[int]bool{},
context.Background(), "cluster-a", resource, client, map[int]bool{}, nil,
)
if !ok {
t.Fatal("expected second LXC sample")
@@ -199,6 +199,7 @@ func TestBuildContainerFromClusterResource_UsesContainerStatusCountersForRates(t
resource,
client,
map[int]bool{},
nil,
); !ok {
t.Fatal("expected first container sample to be built")
}
@@ -219,6 +220,7 @@ func TestBuildContainerFromClusterResource_UsesContainerStatusCountersForRates(t
resource,
client,
map[int]bool{},
nil,
)
if !ok {
t.Fatal("expected second container sample to be built")
@@ -282,6 +284,7 @@ func TestIssue1634LXCMemoryFallsBackToClusterResourcesOnRealRRDShape(t *testing.
resource,
client,
map[int]bool{},
nil,
)
if !ok {
t.Fatal("expected container sample to be built")
@@ -327,6 +330,7 @@ func TestIssue1634LXCMemoryStaysUnavailableWithoutListingValue(t *testing.T) {
resource,
client,
map[int]bool{},
nil,
)
if !ok {
t.Fatal("expected container sample to be built")
@@ -443,6 +447,7 @@ func TestBuildContainerFromClusterResource_AppliesAgentLXCFilesystems(t *testing
resource,
client,
map[int]bool{},
nil,
)
if !ok {
t.Fatal("expected container to be built")
@@ -460,3 +465,105 @@ func TestBuildContainerFromClusterResource_AppliesAgentLXCFilesystems(t *testing
t.Fatalf("expected /data mount with real usage from agent data, got %+v", container.Disks)
}
}
// Issue #2148: a Proxmox LXC with a correlated, online Pulse agent must report
// the agent's own memory sample instead of the cache-inclusive
// cluster/resources fallback, which can badly under-report shared-memory
// workloads (mirrors the #1962 VM behaviour).
func TestIssue2148LXCPrefersLinkedAgentMemoryOverClusterResources(t *testing.T) {
t.Parallel()
const gib = 1024 * 1024 * 1024
client := &stubPVEClientLXCRRD{}
monitor := &Monitor{rateTracker: NewRateTracker()}
resource := proxmox.ClusterResource{
Type: "lxc",
Node: "pve-a",
Name: "npu-ct",
Status: "running",
VMID: 100,
MaxMem: 24 * gib,
Mem: 3459743744,
}
guestID := makeGuestID("cluster-a", "pve-a", 100)
agentHost := models.Host{
LinkedVMID: guestID,
Status: "online",
Memory: models.Memory{
Total: 24 * gib,
Used: 11025571200,
Free: 24*gib - 11025571200,
},
}
container, _, memorySource, _, ok := monitor.buildContainerFromClusterResource(
context.Background(),
"cluster-a",
resource,
client,
map[int]bool{},
map[string]models.Host{guestID: agentHost},
)
if !ok {
t.Fatal("expected container sample to be built")
}
if memorySource != "agent" {
t.Fatalf("memory source = %q, want agent", memorySource)
}
if container.Memory.Used != agentHost.Memory.Used {
t.Fatalf("memory used = %d, want linked agent value %d", container.Memory.Used, agentHost.Memory.Used)
}
if container.Memory.UsageUnavailable || !container.Memory.HasKnownUsage() {
t.Fatalf("expected usable agent-backed memory, got %+v", container.Memory)
}
}
// An agent inside a container without lxcfs sees the host's /proc/meminfo, so a
// sample whose total does not match the container's configured limit must not
// replace the provider reading.
func TestIssue2148LXCRejectsAgentMemoryWithMismatchedTotal(t *testing.T) {
t.Parallel()
const gib = 1024 * 1024 * 1024
client := &stubPVEClientLXCRRD{}
monitor := &Monitor{rateTracker: NewRateTracker()}
resource := proxmox.ClusterResource{
Type: "lxc",
Node: "pve-a",
Name: "host-memory-leak-ct",
Status: "running",
VMID: 101,
MaxMem: 24 * gib,
Mem: 3459743744,
}
guestID := makeGuestID("cluster-a", "pve-a", 101)
agentHost := models.Host{
LinkedVMID: guestID,
Status: "online",
Memory: models.Memory{
Total: 128 * gib,
Used: 40 * gib,
Free: 88 * gib,
},
}
container, _, memorySource, _, ok := monitor.buildContainerFromClusterResource(
context.Background(),
"cluster-a",
resource,
client,
map[int]bool{},
map[string]models.Host{guestID: agentHost},
)
if !ok {
t.Fatal("expected container sample to be built")
}
if CanonicalMemorySource(memorySource) != "cluster-resources" {
t.Fatalf("memory source = %q, want cluster-resources fallback", memorySource)
}
if container.Memory.Used != int64(resource.Mem) {
t.Fatalf("memory used = %d, want provider value %d", container.Memory.Used, resource.Mem)
}
}
@@ -147,7 +147,7 @@ func (m *Monitor) collectGuestsFromClusterResources(
collectionWG.Add(1)
go func() {
defer collectionWG.Done()
allContainers = m.collectClusterContainerResources(ctx, instanceName, containerResources, client, prevContainerIsOCI, prevContainerByID)
allContainers = m.collectClusterContainerResources(ctx, instanceName, containerResources, client, prevContainerIsOCI, prevContainerByID, vmIDToHostAgent)
}()
}
if len(vmResources) > 0 {
@@ -253,6 +253,7 @@ func (m *Monitor) collectClusterContainerResources(
client PVEClientInterface,
prevContainerIsOCI map[int]bool,
prevContainerByID map[string]models.Container,
vmIDToHostAgent map[string]models.Host,
) []models.Container {
orderedContainers := make([]models.Container, len(resources))
orderedOK := make([]bool, len(resources))
@@ -263,13 +264,13 @@ func (m *Monitor) collectClusterContainerResources(
var container models.Container
var ok bool
ran := m.runGuestAgentVMWork(ctx, func(workCtx context.Context) {
container, ok = m.handleClusterContainerResource(workCtx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI)
container, ok = m.handleClusterContainerResource(workCtx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI, vmIDToHostAgent)
})
if !ran {
// The enrichment budget is exhausted, but cluster/resources is still
// authoritative inventory. A canceled context makes remote detail
// calls fail immediately while the builder retains the base row.
container, ok = m.handleClusterContainerResource(ctx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI)
container, ok = m.handleClusterContainerResource(ctx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI, vmIDToHostAgent)
}
if ctx.Err() != nil {
previous := prevContainerByID[entry.guestID]
@@ -415,8 +416,9 @@ func (m *Monitor) handleClusterContainerResource(
guestID string,
client PVEClientInterface,
prevContainerIsOCI map[int]bool,
vmIDToHostAgent map[string]models.Host,
) (models.Container, bool) {
container, guestRaw, memorySource, sampleTime, ok := m.buildContainerFromClusterResource(ctx, instanceName, res, client, prevContainerIsOCI)
container, guestRaw, memorySource, sampleTime, ok := m.buildContainerFromClusterResource(ctx, instanceName, res, client, prevContainerIsOCI, vmIDToHostAgent)
if !ok {
return models.Container{}, false
}
+18 -3
View File
@@ -78,13 +78,20 @@ func NewVMView(r *Resource) VMView { return VMView{r: r} }
// LinkedAgentMemory returns the agent's own sample, not the platform-priority
// merged metric. Freshness belongs to the agent source, not the VM row.
func (v VMView) LinkedAgentMemory() (models.Memory, bool) {
if v.r == nil || v.r.Agent == nil || v.r.Agent.Stale || v.r.Agent.Memory == nil {
return linkedAgentMemoryFromResource(v.r)
}
// linkedAgentMemoryFromResource reads the agent-owned memory sample attached to
// a merged guest resource. Correlation removes the standalone host row, so the
// guest view is the only place the agent sample survives (#1962, #2148).
func linkedAgentMemoryFromResource(r *Resource) (models.Memory, bool) {
if r == nil || r.Agent == nil || r.Agent.Stale || r.Agent.Memory == nil {
return models.Memory{}, false
}
if status, ok := v.r.SourceStatus[SourceAgent]; !ok || status.Status != "online" {
if status, ok := r.SourceStatus[SourceAgent]; !ok || status.Status != "online" {
return models.Memory{}, false
}
m := v.r.Agent.Memory
m := r.Agent.Memory
memory := models.Memory{Total: m.Total, Used: m.Used, Free: m.Free, Cache: m.Cache, UsageUnavailable: m.UsageUnavailable}
if m.Total <= 0 {
return models.Memory{}, false
@@ -397,6 +404,14 @@ type ContainerView struct{ r *Resource }
func NewContainerView(r *Resource) ContainerView { return ContainerView{r: r} }
// LinkedAgentMemory returns the agent's own sample for a system container whose
// merged resource also carries an online Pulse agent source. Proxmox
// cluster/resources memory is a low-trust fallback that can badly misreport a
// container's real footprint, so the linked agent sample is preferred (#2148).
func (v ContainerView) LinkedAgentMemory() (models.Memory, bool) {
return linkedAgentMemoryFromResource(v.r)
}
func (v ContainerView) String() string { return fmt.Sprintf("ContainerView(%s, %q)", v.ID(), v.Name()) }
func (v ContainerView) ID() string {
+31
View File
@@ -2267,3 +2267,34 @@ func TestVMViewLinkedAgentMemory(t *testing.T) {
t.Fatal("nil view has memory")
}
}
func TestContainerViewLinkedAgentMemory(t *testing.T) {
for _, tc := range []struct {
name string
status string
stale bool
total, used int64
unavailable bool
want bool
}{
{name: "live", status: "online", total: 24 << 30, used: 11 << 30, want: true},
{name: "offline", status: "offline", total: 24 << 30, used: 11 << 30},
{name: "stale agent", status: "online", stale: true, total: 24 << 30, used: 11 << 30},
{name: "over capacity", status: "online", total: 24 << 30, used: 25 << 30},
{name: "unavailable", status: "online", total: 24 << 30, used: 11 << 30, unavailable: true},
} {
t.Run(tc.name, func(t *testing.T) {
r := &Resource{Status: StatusOnline, Agent: &AgentData{Stale: tc.stale, Memory: &AgentMemoryMeta{Total: tc.total, Used: tc.used, UsageUnavailable: tc.unavailable}}, SourceStatus: map[DataSource]SourceStatus{SourceAgent: {Status: tc.status}}, Metrics: &ResourceMetrics{Memory: &MetricValue{Percent: 100}}}
memory, ok := NewContainerView(r).LinkedAgentMemory()
if ok != tc.want {
t.Fatalf("valid=%v want=%v memory=%+v", ok, tc.want, memory)
}
if ok && memory.Used != tc.used {
t.Fatalf("not agent memory: %+v", memory)
}
})
}
if _, ok := (ContainerView{}).LinkedAgentMemory(); ok {
t.Fatal("nil view has memory")
}
}
@@ -75,6 +75,11 @@
"incomplete_commit": "919009d7677318c071f888eb28c252d2d79d2728",
"completion_commit": "9fa81dee8fba760b980d821d6e10560589c110f2",
"reason": "Evaluate the PVE consolidation explicit-VerifySSL runtime change together with the follow-up commit that records its agent-lifecycle, api-contracts, security-privacy and dependent storage-recovery contract sections and the node API payload and persisted-field proofs, as one reviewed completion unit."
},
{
"incomplete_commit": "8b56fc72e4ee79f3c219cf51f4c002e2e9343286",
"completion_commit": "1026f5293b4c4e5a11651933e7ed36d398fd2f69",
"reason": "Evaluate the Proxmox LXC linked-agent memory preference runtime change together with the follow-up commit that records its monitoring and unified-resources contract sections, as one reviewed completion unit."
}
]
}
@@ -32,6 +32,7 @@ ATTESTATION_TOOL_PATH = "scripts/release_control/secure_runtime_attestation_v6.p
CANONICAL_REPOSITORY = "rcourtman/Pulse"
CANONICAL_ORIGIN_URL = "https://github.com/rcourtman/Pulse.git"
CANONICAL_MAIN_REF = "origin/main"
CANONICAL_BRANCH_REF_RE = re.compile(r"^origin/(main|release/v[0-9]+\.[0-9]+)$")
ASSEMBLY_SIGNER_WORKFLOW = "github.com/rcourtman/Pulse/.github/workflows/build-release-candidate.yml"
COMPILER_SIGNER_WORKFLOW = "github.com/rcourtman/Pulse/.github/workflows/compile-release-payload.yml"
RELEASE_TAG_RE = re.compile(r"^v[0-9]+\.[0-9]+\.[0-9]+-rc\.[1-9][0-9]*$")
@@ -167,17 +168,23 @@ def parse_remote_refs(raw: bytes) -> dict[str, str]:
def verify_canonical_main_identity(checkout: Path, main_ref: str) -> str:
if main_ref != CANONICAL_MAIN_REF:
raise v5.AttestationError("committed-main classification requires canonical origin/main")
match = CANONICAL_BRANCH_REF_RE.fullmatch(main_ref)
if match is None:
raise v5.AttestationError(
"committed-main classification requires a canonical origin branch ref"
)
branch = match.group(1)
origin = v5.run_git(checkout, "remote", "get-url", "origin").stdout.decode().strip()
if origin != CANONICAL_ORIGIN_URL:
raise v5.AttestationError("origin remote URL is not the canonical Pulse repository URL")
local_main = v5.resolve_commit(checkout, CANONICAL_MAIN_REF, "canonical origin/main")
remote = v5.run_git(checkout, "ls-remote", "origin", "refs/heads/main")
local_branch = v5.resolve_commit(checkout, main_ref, f"canonical origin/{branch}")
remote = v5.run_git(checkout, "ls-remote", "origin", f"refs/heads/{branch}")
remote_refs = parse_remote_refs(remote.stdout)
if remote_refs != {"refs/heads/main": local_main}:
raise v5.AttestationError("canonical origin/main does not match the remote main commit")
return local_main
if remote_refs != {f"refs/heads/{branch}": local_branch}:
raise v5.AttestationError(
f"canonical origin/{branch} does not match the remote branch commit"
)
return local_branch
def verify_release_candidate_tag_identity(
@@ -609,7 +616,7 @@ def verify_and_snapshot_release_candidate_packet(
raise v5.AttestationError("qualified commit must be the full canonical commit SHA")
v5.require_detached_clean_checkout(checkout, resolved_commit)
main_commit = verify_canonical_main_identity(checkout, main_ref)
v5.require_ancestor(checkout, resolved_commit, CANONICAL_MAIN_REF)
v5.require_ancestor(checkout, resolved_commit, main_ref)
expected_assets = expected_release_asset_names(architecture)
expected_versions = expected_release_artifact_versions(tag)
@@ -952,7 +959,7 @@ def _create_attestation_with_snapshotted_artifacts(
raise v5.AttestationError("qualified commit must be the full canonical commit SHA")
v5.require_detached_clean_checkout(checkout, qualified_commit)
main_commit = verify_canonical_main_identity(checkout, main_ref)
v5.require_ancestor(checkout, qualified_commit, CANONICAL_MAIN_REF)
v5.require_ancestor(checkout, qualified_commit, main_ref)
with v6_contract():
receipt, receipt_bytes = v5.load_receipt(receipt_path)
record_path = v5.canonical_repo_path(receipt_record_path, "receipt record path")
@@ -1018,7 +1025,7 @@ def _create_attestation_with_snapshotted_artifacts(
"qualified_ref_at_run": release_candidate_tag or qualified_commit,
"main_ref_verified": main_ref,
"main_ref_commit_at_attestation": main_commit,
"qualified_commit_reachable_from_main": True,
"qualified_commit_reachable_from_main": main_ref == CANONICAL_MAIN_REF,
"build_checkout": "detached-worktree",
"build_checkout_clean_except_lab_artifacts": True,
"disposable_vm_guard_receipt_claim_validated": True,
@@ -468,9 +468,9 @@ class SecureRuntimeAttestationV6Test(unittest.TestCase):
verify_canonical_main_identity(self.root, CANONICAL_MAIN_REF),
self.commit,
)
for caller_ref in ("HEAD", "main", "refs/heads/main", "scratch"):
for caller_ref in ("HEAD", "main", "refs/heads/main", "scratch", "origin/feature/x"):
with self.subTest(caller_ref=caller_ref), self.assertRaisesRegex(
v5.AttestationError, "canonical origin/main"
v5.AttestationError, "canonical origin branch ref"
):
verify_canonical_main_identity(self.root, caller_ref)
@@ -482,10 +482,45 @@ class SecureRuntimeAttestationV6Test(unittest.TestCase):
with (
mock.patch.object(v5, "run_git", side_effect=moved_remote),
self.assertRaisesRegex(v5.AttestationError, "does not match the remote main commit"),
self.assertRaisesRegex(v5.AttestationError, "does not match the remote branch commit"),
):
verify_canonical_main_identity(self.root, CANONICAL_MAIN_REF)
def test_committed_main_identity_accepts_mapped_release_branch(self) -> None:
release_ref = "origin/release/v6.4"
def canonical(_checkout, *args, **_kwargs):
command = tuple(args)
if command == ("remote", "get-url", "origin"):
output = CANONICAL_ORIGIN_URL + "\n"
elif command == ("rev-parse", "--verify", f"{release_ref}^{{commit}}"):
output = self.commit + "\n"
elif command == ("ls-remote", "origin", "refs/heads/release/v6.4"):
output = f"{self.commit}\trefs/heads/release/v6.4\n"
else:
raise AssertionError(f"unexpected git call {command}")
return subprocess.CompletedProcess(args, 0, output.encode(), b"")
with mock.patch.object(v5, "run_git", side_effect=canonical):
self.assertEqual(
verify_canonical_main_identity(self.root, release_ref),
self.commit,
)
def moved_remote(_checkout, *args, **kwargs):
result = canonical(_checkout, *args, **kwargs)
if tuple(args) == ("ls-remote", "origin", "refs/heads/release/v6.4"):
return subprocess.CompletedProcess(
args, 0, f"{'c' * 40}\trefs/heads/release/v6.4\n".encode(), b""
)
return result
with (
mock.patch.object(v5, "run_git", side_effect=moved_remote),
self.assertRaisesRegex(v5.AttestationError, "does not match the remote branch commit"),
):
verify_canonical_main_identity(self.root, release_ref)
def test_accepts_canonical_annotated_tag_only_as_release_packet_identity(self) -> None:
with mock.patch.object(v5, "run_git", side_effect=lambda _checkout, *args, **_kwargs: self.git_result(*args)):
identity = verify_release_candidate_tag_identity(