mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-10-24 04:33:17 +00:00
Merge pull request #2152 from rcourtman/maintainer/20260922T082007Z
Fix spurious PBS backup alerts and Proxmox LXC memory; qualify release-line RCs
This commit is contained in:
@@ -1620,6 +1620,18 @@ of a reporter installation or a migration that guesses operator intent.
|
||||
`TestMergeSnapshotOverrideInheritsZeroFields` in
|
||||
`internal/alerts/alerts_test.go` pin the merge, migration, and preservation
|
||||
behaviors.
|
||||
Backup-age alerts are guest-workload scoped. A PBS recovery rollup whose subject
|
||||
carries no guest VMID is a host/config backup (`backup-type host`, keyed by the
|
||||
node name), not a guest workload, and must not raise a guest backup-age alert:
|
||||
the Backups overview already excludes those entries from guest coverage, and a
|
||||
stale host config backup otherwise notified forever under a node-named subject
|
||||
while the node's guests were backed up. Evaluation skips PBS subjects with an
|
||||
empty VMID, retaining their recovery point and rollup so inventory and artifact
|
||||
display are unchanged. PVE host backups already produce no subject and no alert.
|
||||
Genuinely orphaned PBS guest backups (numeric VMID) and PMG node backups still
|
||||
alert. `TestCheckBackupsIgnoresPBSHostBackupSubject` and
|
||||
`TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup` in
|
||||
`internal/alerts/alerts_test.go` pin both sides of this boundary (#2136).
|
||||
Proxmox disk health alert evaluation now lives in
|
||||
`internal/alerts/disk_health.go`. That file owns Proxmox disk canonical
|
||||
identity, disk health assessment alerts, known-firmware health suppression, and
|
||||
|
||||
@@ -534,7 +534,12 @@ workflow execution ref and SHA equal the peeled tag commit. That workflow consum
|
||||
release assets and four release signatures, runs the canonical twenty-three-scenario
|
||||
schema-v7 lab, and applies the release-candidate attester against the exact
|
||||
GitHub release ID, tag, source commit, checksums, compiler provenance, assembly
|
||||
provenance, and update-key fingerprint. The v7 host starts a rootful Docker
|
||||
provenance, and update-key fingerprint. The attester anchors the candidate to
|
||||
the branch the control plane maps the version to, either `origin/main` or the
|
||||
release train's `origin/release/vX.Y`, and verifies that remote-tracking ref
|
||||
against the canonical origin's live branch tip, so a release-train candidate is
|
||||
qualified against the line that owns its version rather than being required to
|
||||
appear on `main`. The v7 host starts a rootful Docker
|
||||
daemon inside the disposable systemd container without mounting the hosted
|
||||
runner socket or exposing a TCP listener. It imports an offline, source-bound
|
||||
fixture image and proves legacy inventory, summary-only typed-helper parity,
|
||||
|
||||
@@ -17,6 +17,25 @@
|
||||
|
||||
## Purpose
|
||||
|
||||
### Linked Pulse agent memory for Proxmox LXC — issue #2148 (22 September 2026)
|
||||
|
||||
Both LXC memory paths (the efficient `cluster/resources` builder and the
|
||||
per-node fallback) may substitute the linked Pulse agent's own memory sample for
|
||||
a Proxmox fallback listing value. The substitution applies only when the current
|
||||
memory source is a fallback (`cluster-resources` or `unavailable`), the guest is
|
||||
`running`, the linked agent sample has known usage, and the agent total matches
|
||||
the container's provisioned limit with used not exceeding it. A stopped
|
||||
container, a non-fallback source, a missing linked agent, and a total mismatch
|
||||
are left untouched; preferred or derived guest evidence is never replaced. The
|
||||
provisioned platform total is retained, and the previous-guest agent map is
|
||||
threaded through both builders.
|
||||
|
||||
`TestIssue2148LXCPrefersLinkedAgentMemoryOverClusterResources` and
|
||||
`TestIssue2148LXCRejectsAgentMemoryWithMismatchedTotal` pin the substitution and
|
||||
the rejection boundary; `TestCorrelatedContainerMemoryNextPoll` pins the
|
||||
next-poll merged-row path. This is synthetic runtime proof, not a live Proxmox
|
||||
installation or browser receipt.
|
||||
|
||||
### Preserve explicit agent filesystem selection at ingestion
|
||||
|
||||
Both `ApplyHostReport` and `ApplyDockerReport` honour a disk's optional
|
||||
|
||||
@@ -37,6 +37,17 @@ policy remains authoritative; this accessor introduces no second timeout.
|
||||
`TestCorrelatedGuestMemoryNextPoll` verifies the manually merged row through the
|
||||
next-poll consumers. Existing host-list membership and metric priority are unchanged.
|
||||
|
||||
**System-container linked agent memory read — issue #2148 (22 September 2026)**
|
||||
|
||||
`ContainerView.LinkedAgentMemory` exposes the same value-copy of the linked
|
||||
agent's own memory sample for system containers, sharing the validity and
|
||||
freshness rules with `VMView.LinkedAgentMemory`: missing/non-online agent source
|
||||
status, explicit agent staleness, unavailable usage and invalid capacity or
|
||||
counters are rejected, and trusted zero usage remains valid. The monitoring
|
||||
layer consumes it only as a fallback replacement for Proxmox LXC memory, so
|
||||
platform metric priority for a healthy agent is unchanged.
|
||||
`TestContainerViewLinkedAgentMemory` pins those boundaries.
|
||||
|
||||
### Bounded incident-history selection
|
||||
|
||||
Canonical history queries filter exact alert identifiers and observation windows
|
||||
|
||||
@@ -2322,6 +2322,96 @@ func TestCheckBackupsHandlesPmgBackups(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCheckBackupsIgnoresPBSHostBackupSubject covers #2136: a PBS host config
|
||||
// backup (backup-type "host") is keyed by the node name, not a VMID, and must
|
||||
// not raise a guest backup-age alert. The node name subject ("nas backup")
|
||||
// otherwise looks like a stale guest and notifies forever even though the
|
||||
// node's guests are backed up.
|
||||
func TestCheckBackupsIgnoresPBSHostBackupSubject(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.BackupDefaults = BackupAlertConfig{
|
||||
Enabled: true,
|
||||
WarningDays: 7,
|
||||
CriticalDays: 14,
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
rollups := []recovery.ProtectionRollup{
|
||||
{
|
||||
RollupID: "ext:pbs-host-nas",
|
||||
SubjectRef: &recovery.ExternalRef{
|
||||
Type: "proxmox-guest",
|
||||
Namespace: "pbs",
|
||||
Name: "nas",
|
||||
ID: "nas",
|
||||
},
|
||||
LastSuccessAt: ptrTime(now.Add(-43.5 * 24 * time.Hour)),
|
||||
LastOutcome: recovery.OutcomeSuccess,
|
||||
Providers: []recovery.Provider{recovery.ProviderProxmoxPBS},
|
||||
},
|
||||
}
|
||||
|
||||
m.CheckBackups(rollups, map[string]GuestLookup{}, map[string][]GuestLookup{})
|
||||
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
for storageKey, alert := range m.activeAlerts {
|
||||
if alert != nil && alert.Type == "backup-age" {
|
||||
t.Fatalf("expected PBS host config backup to be excluded from guest backup-age alerts, found %s", effectiveAlertID(alert, storageKey))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup is the companion guard:
|
||||
// a genuinely orphaned PBS guest backup (numeric VMID, no live guest) must keep
|
||||
// alerting, so the host-backup exclusion above does not silence real staleness.
|
||||
func TestCheckBackupsStillAlertsOnOrphanedGuestPBSBackup(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
|
||||
m.mu.Lock()
|
||||
m.config.Enabled = true
|
||||
m.config.BackupDefaults = BackupAlertConfig{
|
||||
Enabled: true,
|
||||
WarningDays: 7,
|
||||
CriticalDays: 14,
|
||||
}
|
||||
m.mu.Unlock()
|
||||
|
||||
now := time.Now()
|
||||
rollups := []recovery.ProtectionRollup{
|
||||
{
|
||||
RollupID: "ext:pbs-orphan-102",
|
||||
SubjectRef: &recovery.ExternalRef{
|
||||
Type: "proxmox-vm",
|
||||
Namespace: "pbs",
|
||||
Name: "102",
|
||||
ID: "102",
|
||||
},
|
||||
LastSuccessAt: ptrTime(now.Add(-43.5 * 24 * time.Hour)),
|
||||
LastOutcome: recovery.OutcomeSuccess,
|
||||
Providers: []recovery.Provider{recovery.ProviderProxmoxPBS},
|
||||
},
|
||||
}
|
||||
|
||||
m.CheckBackups(rollups, map[string]GuestLookup{}, map[string][]GuestLookup{})
|
||||
|
||||
m.mu.RLock()
|
||||
defer m.mu.RUnlock()
|
||||
if !testHasActiveAlert(t, m, buildCanonicalStateID("backup-subject:ext-pbs-orphan-102", "backup-subject:ext-pbs-orphan-102-backup-age")) {
|
||||
var keys []string
|
||||
for storageKey, active := range m.activeAlerts {
|
||||
keys = append(keys, effectiveAlertID(active, storageKey))
|
||||
}
|
||||
t.Fatalf("expected orphaned PBS guest backup alert, found keys: %v", keys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckBackupsSkipsOrphanedWhenDisabled(t *testing.T) {
|
||||
m := newTestManager(t)
|
||||
m.ClearActiveAlerts()
|
||||
|
||||
@@ -859,6 +859,16 @@ func (m *Manager) CheckBackupsWithInventory(
|
||||
}
|
||||
ageDaysRounded := math.Round(ageDays*10) / 10
|
||||
|
||||
// A PBS subject without a guest VMID is a host/config backup
|
||||
// (PBS backup-type "host", keyed by the node name), not a guest
|
||||
// workload. The Backups overview already keeps these out of guest
|
||||
// coverage, so do not raise a guest backup-age alert for them: a
|
||||
// stale host config backup otherwise notifies forever under a
|
||||
// node-named subject while the node's guests are backed up (#2136).
|
||||
if record.source == "PBS" && strings.TrimSpace(record.vmID) == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// Determine thresholds for this backup
|
||||
currentBackupCfg := backupCfg
|
||||
guestContext := guestSnapshotFromLookup(record.lookup, record.fallbackName)
|
||||
|
||||
@@ -382,3 +382,49 @@ func (m *Monitor) resolveGuestStatusMemory(
|
||||
|
||||
return memTotal, memUsed, memorySource
|
||||
}
|
||||
|
||||
// preferLinkedAgentLXCMemory replaces a low-trust Proxmox LXC memory reading
|
||||
// with the linked Pulse agent's own sample when one is available. The agent runs
|
||||
// inside the container, so its sample reflects the workload's real footprint,
|
||||
// while cluster/resources is a cache-inclusive cgroup fallback that can badly
|
||||
// under- or over-report shared-memory workloads (#2148, mirroring #1962).
|
||||
//
|
||||
// An agent inside a container without lxcfs sees the host's /proc/meminfo, so a
|
||||
// sample is only trusted when its total matches the container's configured
|
||||
// limit. Preferred provider sources are never overridden.
|
||||
func preferLinkedAgentLXCMemory(
|
||||
status string,
|
||||
memTotal uint64,
|
||||
memUsed uint64,
|
||||
memorySource string,
|
||||
guestRaw *VMMemoryRaw,
|
||||
agentHost models.Host,
|
||||
hasAgent bool,
|
||||
) (uint64, uint64, string) {
|
||||
if !hasAgent || status != "running" || memTotal == 0 {
|
||||
return memTotal, memUsed, memorySource
|
||||
}
|
||||
switch CanonicalMemorySource(memorySource) {
|
||||
case "cluster-resources", "unavailable":
|
||||
default:
|
||||
return memTotal, memUsed, memorySource
|
||||
}
|
||||
if !agentHost.Memory.HasKnownUsage() || agentHost.Memory.Total <= 0 || agentHost.Memory.Used < 0 {
|
||||
return memTotal, memUsed, memorySource
|
||||
}
|
||||
agentTotal := uint64(agentHost.Memory.Total)
|
||||
agentUsed := uint64(agentHost.Memory.Used)
|
||||
if agentTotal != memTotal || agentUsed > memTotal {
|
||||
return memTotal, memUsed, memorySource
|
||||
}
|
||||
if guestRaw != nil {
|
||||
guestRaw.HostAgentTotal = agentTotal
|
||||
guestRaw.HostAgentUsed = agentUsed
|
||||
}
|
||||
log.Debug().
|
||||
Uint64("total", memTotal).
|
||||
Uint64("used", agentUsed).
|
||||
Str("previousSource", memorySource).
|
||||
Msg("LXC memory: using linked Pulse agent memory over cluster-resources fallback")
|
||||
return memTotal, agentUsed, "agent"
|
||||
}
|
||||
|
||||
@@ -1004,6 +1004,7 @@ func TestHandleClusterContainerResourceMemoryTrustCharacterization(t *testing.T)
|
||||
makeGuestID("test", "node1", tt.res.VMID),
|
||||
client,
|
||||
nil,
|
||||
nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("handleClusterContainerResource() returned ok=false")
|
||||
|
||||
@@ -1009,6 +1009,45 @@ func TestCorrelatedGuestMemoryNextPoll(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Issue #2148: a correlated agent merged into a system-container resource must
|
||||
// surface through the previous-guest context so the LXC builder prefers it over
|
||||
// the cache-inclusive cluster/resources fallback.
|
||||
func TestCorrelatedContainerMemoryNextPoll(t *testing.T) {
|
||||
now := time.Now()
|
||||
const gib = int64(1024 * 1024 * 1024)
|
||||
guestID := makeGuestID("pve-a", "node1", 100)
|
||||
store := unifiedresources.NewMemoryStore()
|
||||
if err := store.AddLink(unifiedresources.ResourceLink{ResourceA: "ct-test", ResourceB: "agent-test", PrimaryID: "agent-test"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
registry := unifiedresources.NewRegistry(store)
|
||||
total, used := int64(24)*gib, int64(11)*gib
|
||||
resources := []unifiedresources.Resource{
|
||||
{ID: "ct-test", Type: unifiedresources.ResourceTypeSystemContainer, Name: "npu-ct", Status: unifiedresources.StatusOnline, LastSeen: now,
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceProxmox},
|
||||
Proxmox: &unifiedresources.ProxmoxData{SourceID: guestID, Instance: "pve-a", NodeName: "node1", VMID: 100, RuntimeStatus: "running"},
|
||||
Metrics: &unifiedresources.ResourceMetrics{Memory: &unifiedresources.MetricValue{Total: &total, Used: &used, Percent: 45, Source: unifiedresources.SourceProxmox}}},
|
||||
{ID: "agent-test", Type: unifiedresources.ResourceTypeAgent, Name: "npu-ct-agent", Status: unifiedresources.StatusOnline, LastSeen: now,
|
||||
Sources: []unifiedresources.DataSource{unifiedresources.SourceAgent},
|
||||
Agent: &unifiedresources.AgentData{AgentID: "agent-test", Memory: &unifiedresources.AgentMemoryMeta{Total: total, Used: used, Free: total - used}}},
|
||||
}
|
||||
registry.IngestResources(resources)
|
||||
if len(registry.Containers()) != 1 || len(registry.Hosts()) != 0 {
|
||||
t.Fatalf("expected one merged container and no standalone hosts, got %d/%d", len(registry.Containers()), len(registry.Hosts()))
|
||||
}
|
||||
mon := &Monitor{state: models.NewState(), rateTracker: NewRateTracker(), config: &config.Config{}, resourceStore: unifiedresources.NewMonitorAdapter(registry)}
|
||||
prev := mon.previousGuestContextForInstance("pve-a")
|
||||
if _, ok := prev.hostAgentsByVMID[guestID]; !ok {
|
||||
t.Fatalf("linked container agent missing from previous context: %+v", prev.hostAgentsByVMID)
|
||||
}
|
||||
container, _, source, _, ok := mon.buildContainerFromClusterResource(context.Background(), "pve-a",
|
||||
proxmox.ClusterResource{Type: "lxc", Status: "running", Node: "node1", VMID: 100, Name: "npu-ct", MaxMem: uint64(total), Mem: 3459743744},
|
||||
&stubPVEClient{}, map[int]bool{}, prev.hostAgentsByVMID)
|
||||
if !ok || source != "agent" || container.Memory.Used != used {
|
||||
t.Fatalf("container memory not agent-backed: ok=%v source=%s mem=%+v", ok, source, container.Memory)
|
||||
}
|
||||
}
|
||||
|
||||
type correlatedMemoryClient struct{ stubPVEClient }
|
||||
|
||||
func (*correlatedMemoryClient) GetVMStatus(context.Context, string, int) (*proxmox.VMStatus, error) {
|
||||
|
||||
@@ -154,6 +154,16 @@ func (m *Monitor) collectContainersWithNodes(ctx context.Context, instanceName s
|
||||
MaxMem: container.MaxMem,
|
||||
Mem: container.Mem,
|
||||
})
|
||||
agentHost, hasAgent := prevGuests.hostAgentsByVMID[guestID]
|
||||
memTotal, memUsed, memorySource = preferLinkedAgentLXCMemory(
|
||||
container.Status,
|
||||
memTotal,
|
||||
memUsed,
|
||||
memorySource,
|
||||
&guestRaw,
|
||||
agentHost,
|
||||
hasAgent,
|
||||
)
|
||||
memUsed, memorySource, _ = stabilizeGuestLowTrustMemory(
|
||||
m.previousGuestSnapshot(instanceName, "lxc", n.Node, int(container.VMID)),
|
||||
container.Status,
|
||||
|
||||
@@ -61,6 +61,9 @@ func (m *Monitor) previousGuestContextForInstance(instanceName string) previousG
|
||||
guestID := makeGuestID(container.Instance, container.Node, container.VMID)
|
||||
if guestID != "" {
|
||||
ctx.containersByID[guestID] = container
|
||||
if memory, ok := ct.LinkedAgentMemory(); ok {
|
||||
ctx.hostAgentsByVMID[guestID] = models.Host{LinkedVMID: guestID, Status: "online", Memory: memory}
|
||||
}
|
||||
}
|
||||
if container.VMID > 0 && (strings.EqualFold(strings.TrimSpace(container.Type), "oci") || container.IsOCI) {
|
||||
ctx.containerOCIByVMID[container.VMID] = true
|
||||
@@ -72,10 +75,15 @@ func (m *Monitor) previousGuestContextForInstance(instanceName string) previousG
|
||||
continue
|
||||
}
|
||||
modelHost := previousHostFromView(host)
|
||||
if modelHost.LinkedVMID == "" || modelHost.Status != "online" {
|
||||
if modelHost.Status != "online" {
|
||||
continue
|
||||
}
|
||||
ctx.hostAgentsByVMID[modelHost.LinkedVMID] = modelHost
|
||||
if modelHost.LinkedVMID != "" {
|
||||
ctx.hostAgentsByVMID[modelHost.LinkedVMID] = modelHost
|
||||
}
|
||||
if modelHost.LinkedContainerID != "" {
|
||||
ctx.hostAgentsByVMID[modelHost.LinkedContainerID] = modelHost
|
||||
}
|
||||
}
|
||||
|
||||
return ctx
|
||||
@@ -172,12 +180,13 @@ func previousHostFromView(host *unifiedresources.HostView) models.Host {
|
||||
return models.Host{}
|
||||
}
|
||||
return models.Host{
|
||||
ID: host.ID(),
|
||||
Hostname: host.Hostname(),
|
||||
Status: string(host.Status()),
|
||||
LinkedVMID: host.LinkedVMID(),
|
||||
LastSeen: host.LastSeen(),
|
||||
Disks: guestDisksFromReadStateView(host.Disks()),
|
||||
ID: host.ID(),
|
||||
Hostname: host.Hostname(),
|
||||
Status: string(host.Status()),
|
||||
LinkedVMID: host.LinkedVMID(),
|
||||
LinkedContainerID: host.LinkedContainerID(),
|
||||
LastSeen: host.LastSeen(),
|
||||
Disks: guestDisksFromReadStateView(host.Disks()),
|
||||
Memory: models.Memory{
|
||||
Used: host.MemoryUsed(),
|
||||
Total: host.MemoryTotal(),
|
||||
|
||||
@@ -58,6 +58,7 @@ func TestBuildContainerFromClusterResource_PreservesProxmoxPool(t *testing.T) {
|
||||
},
|
||||
nil,
|
||||
map[int]bool{},
|
||||
nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected container to be built")
|
||||
|
||||
@@ -44,6 +44,7 @@ func (m *Monitor) buildContainerFromClusterResource(
|
||||
res proxmox.ClusterResource,
|
||||
client PVEClientInterface,
|
||||
prevContainerIsOCI map[int]bool,
|
||||
vmIDToHostAgent map[string]models.Host,
|
||||
) (models.Container, VMMemoryRaw, string, time.Time, bool) {
|
||||
// Skip templates if configured
|
||||
if res.Template == 1 {
|
||||
@@ -88,6 +89,16 @@ func (m *Monitor) buildContainerFromClusterResource(
|
||||
)
|
||||
|
||||
memTotal, memUsed, memorySource, guestRaw := m.calculateLXCMemory(res)
|
||||
agentHost, hasAgent := vmIDToHostAgent[guestID]
|
||||
memTotal, memUsed, memorySource = preferLinkedAgentLXCMemory(
|
||||
res.Status,
|
||||
memTotal,
|
||||
memUsed,
|
||||
memorySource,
|
||||
&guestRaw,
|
||||
agentHost,
|
||||
hasAgent,
|
||||
)
|
||||
memUsed, memorySource, _ = stabilizeGuestLowTrustMemory(
|
||||
m.previousGuestSnapshot(instanceName, "lxc", res.Node, res.VMID),
|
||||
res.Status,
|
||||
|
||||
@@ -140,7 +140,7 @@ func TestIssue1613LXCStatusLagDoesNotEraseNewerListingDiskWrite(t *testing.T) {
|
||||
}
|
||||
|
||||
if _, _, _, _, ok := monitor.buildContainerFromClusterResource(
|
||||
context.Background(), "cluster-a", resource, client, map[int]bool{},
|
||||
context.Background(), "cluster-a", resource, client, map[int]bool{}, nil,
|
||||
); !ok {
|
||||
t.Fatal("expected first LXC sample")
|
||||
}
|
||||
@@ -150,7 +150,7 @@ func TestIssue1613LXCStatusLagDoesNotEraseNewerListingDiskWrite(t *testing.T) {
|
||||
client.containerStatus.ObservedAt = resource.ObservedAt.Add(time.Second)
|
||||
|
||||
container, _, _, _, ok := monitor.buildContainerFromClusterResource(
|
||||
context.Background(), "cluster-a", resource, client, map[int]bool{},
|
||||
context.Background(), "cluster-a", resource, client, map[int]bool{}, nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected second LXC sample")
|
||||
@@ -199,6 +199,7 @@ func TestBuildContainerFromClusterResource_UsesContainerStatusCountersForRates(t
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
nil,
|
||||
); !ok {
|
||||
t.Fatal("expected first container sample to be built")
|
||||
}
|
||||
@@ -219,6 +220,7 @@ func TestBuildContainerFromClusterResource_UsesContainerStatusCountersForRates(t
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected second container sample to be built")
|
||||
@@ -282,6 +284,7 @@ func TestIssue1634LXCMemoryFallsBackToClusterResourcesOnRealRRDShape(t *testing.
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected container sample to be built")
|
||||
@@ -327,6 +330,7 @@ func TestIssue1634LXCMemoryStaysUnavailableWithoutListingValue(t *testing.T) {
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected container sample to be built")
|
||||
@@ -443,6 +447,7 @@ func TestBuildContainerFromClusterResource_AppliesAgentLXCFilesystems(t *testing
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
nil,
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected container to be built")
|
||||
@@ -460,3 +465,105 @@ func TestBuildContainerFromClusterResource_AppliesAgentLXCFilesystems(t *testing
|
||||
t.Fatalf("expected /data mount with real usage from agent data, got %+v", container.Disks)
|
||||
}
|
||||
}
|
||||
|
||||
// Issue #2148: a Proxmox LXC with a correlated, online Pulse agent must report
|
||||
// the agent's own memory sample instead of the cache-inclusive
|
||||
// cluster/resources fallback, which can badly under-report shared-memory
|
||||
// workloads (mirrors the #1962 VM behaviour).
|
||||
func TestIssue2148LXCPrefersLinkedAgentMemoryOverClusterResources(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const gib = 1024 * 1024 * 1024
|
||||
|
||||
client := &stubPVEClientLXCRRD{}
|
||||
monitor := &Monitor{rateTracker: NewRateTracker()}
|
||||
resource := proxmox.ClusterResource{
|
||||
Type: "lxc",
|
||||
Node: "pve-a",
|
||||
Name: "npu-ct",
|
||||
Status: "running",
|
||||
VMID: 100,
|
||||
MaxMem: 24 * gib,
|
||||
Mem: 3459743744,
|
||||
}
|
||||
guestID := makeGuestID("cluster-a", "pve-a", 100)
|
||||
agentHost := models.Host{
|
||||
LinkedVMID: guestID,
|
||||
Status: "online",
|
||||
Memory: models.Memory{
|
||||
Total: 24 * gib,
|
||||
Used: 11025571200,
|
||||
Free: 24*gib - 11025571200,
|
||||
},
|
||||
}
|
||||
|
||||
container, _, memorySource, _, ok := monitor.buildContainerFromClusterResource(
|
||||
context.Background(),
|
||||
"cluster-a",
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
map[string]models.Host{guestID: agentHost},
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected container sample to be built")
|
||||
}
|
||||
if memorySource != "agent" {
|
||||
t.Fatalf("memory source = %q, want agent", memorySource)
|
||||
}
|
||||
if container.Memory.Used != agentHost.Memory.Used {
|
||||
t.Fatalf("memory used = %d, want linked agent value %d", container.Memory.Used, agentHost.Memory.Used)
|
||||
}
|
||||
if container.Memory.UsageUnavailable || !container.Memory.HasKnownUsage() {
|
||||
t.Fatalf("expected usable agent-backed memory, got %+v", container.Memory)
|
||||
}
|
||||
}
|
||||
|
||||
// An agent inside a container without lxcfs sees the host's /proc/meminfo, so a
|
||||
// sample whose total does not match the container's configured limit must not
|
||||
// replace the provider reading.
|
||||
func TestIssue2148LXCRejectsAgentMemoryWithMismatchedTotal(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const gib = 1024 * 1024 * 1024
|
||||
|
||||
client := &stubPVEClientLXCRRD{}
|
||||
monitor := &Monitor{rateTracker: NewRateTracker()}
|
||||
resource := proxmox.ClusterResource{
|
||||
Type: "lxc",
|
||||
Node: "pve-a",
|
||||
Name: "host-memory-leak-ct",
|
||||
Status: "running",
|
||||
VMID: 101,
|
||||
MaxMem: 24 * gib,
|
||||
Mem: 3459743744,
|
||||
}
|
||||
guestID := makeGuestID("cluster-a", "pve-a", 101)
|
||||
agentHost := models.Host{
|
||||
LinkedVMID: guestID,
|
||||
Status: "online",
|
||||
Memory: models.Memory{
|
||||
Total: 128 * gib,
|
||||
Used: 40 * gib,
|
||||
Free: 88 * gib,
|
||||
},
|
||||
}
|
||||
|
||||
container, _, memorySource, _, ok := monitor.buildContainerFromClusterResource(
|
||||
context.Background(),
|
||||
"cluster-a",
|
||||
resource,
|
||||
client,
|
||||
map[int]bool{},
|
||||
map[string]models.Host{guestID: agentHost},
|
||||
)
|
||||
if !ok {
|
||||
t.Fatal("expected container sample to be built")
|
||||
}
|
||||
if CanonicalMemorySource(memorySource) != "cluster-resources" {
|
||||
t.Fatalf("memory source = %q, want cluster-resources fallback", memorySource)
|
||||
}
|
||||
if container.Memory.Used != int64(resource.Mem) {
|
||||
t.Fatalf("memory used = %d, want provider value %d", container.Memory.Used, resource.Mem)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -147,7 +147,7 @@ func (m *Monitor) collectGuestsFromClusterResources(
|
||||
collectionWG.Add(1)
|
||||
go func() {
|
||||
defer collectionWG.Done()
|
||||
allContainers = m.collectClusterContainerResources(ctx, instanceName, containerResources, client, prevContainerIsOCI, prevContainerByID)
|
||||
allContainers = m.collectClusterContainerResources(ctx, instanceName, containerResources, client, prevContainerIsOCI, prevContainerByID, vmIDToHostAgent)
|
||||
}()
|
||||
}
|
||||
if len(vmResources) > 0 {
|
||||
@@ -253,6 +253,7 @@ func (m *Monitor) collectClusterContainerResources(
|
||||
client PVEClientInterface,
|
||||
prevContainerIsOCI map[int]bool,
|
||||
prevContainerByID map[string]models.Container,
|
||||
vmIDToHostAgent map[string]models.Host,
|
||||
) []models.Container {
|
||||
orderedContainers := make([]models.Container, len(resources))
|
||||
orderedOK := make([]bool, len(resources))
|
||||
@@ -263,13 +264,13 @@ func (m *Monitor) collectClusterContainerResources(
|
||||
var container models.Container
|
||||
var ok bool
|
||||
ran := m.runGuestAgentVMWork(ctx, func(workCtx context.Context) {
|
||||
container, ok = m.handleClusterContainerResource(workCtx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI)
|
||||
container, ok = m.handleClusterContainerResource(workCtx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI, vmIDToHostAgent)
|
||||
})
|
||||
if !ran {
|
||||
// The enrichment budget is exhausted, but cluster/resources is still
|
||||
// authoritative inventory. A canceled context makes remote detail
|
||||
// calls fail immediately while the builder retains the base row.
|
||||
container, ok = m.handleClusterContainerResource(ctx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI)
|
||||
container, ok = m.handleClusterContainerResource(ctx, instanceName, entry.resource, entry.guestID, client, prevContainerIsOCI, vmIDToHostAgent)
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
previous := prevContainerByID[entry.guestID]
|
||||
@@ -415,8 +416,9 @@ func (m *Monitor) handleClusterContainerResource(
|
||||
guestID string,
|
||||
client PVEClientInterface,
|
||||
prevContainerIsOCI map[int]bool,
|
||||
vmIDToHostAgent map[string]models.Host,
|
||||
) (models.Container, bool) {
|
||||
container, guestRaw, memorySource, sampleTime, ok := m.buildContainerFromClusterResource(ctx, instanceName, res, client, prevContainerIsOCI)
|
||||
container, guestRaw, memorySource, sampleTime, ok := m.buildContainerFromClusterResource(ctx, instanceName, res, client, prevContainerIsOCI, vmIDToHostAgent)
|
||||
if !ok {
|
||||
return models.Container{}, false
|
||||
}
|
||||
|
||||
@@ -78,13 +78,20 @@ func NewVMView(r *Resource) VMView { return VMView{r: r} }
|
||||
// LinkedAgentMemory returns the agent's own sample, not the platform-priority
|
||||
// merged metric. Freshness belongs to the agent source, not the VM row.
|
||||
func (v VMView) LinkedAgentMemory() (models.Memory, bool) {
|
||||
if v.r == nil || v.r.Agent == nil || v.r.Agent.Stale || v.r.Agent.Memory == nil {
|
||||
return linkedAgentMemoryFromResource(v.r)
|
||||
}
|
||||
|
||||
// linkedAgentMemoryFromResource reads the agent-owned memory sample attached to
|
||||
// a merged guest resource. Correlation removes the standalone host row, so the
|
||||
// guest view is the only place the agent sample survives (#1962, #2148).
|
||||
func linkedAgentMemoryFromResource(r *Resource) (models.Memory, bool) {
|
||||
if r == nil || r.Agent == nil || r.Agent.Stale || r.Agent.Memory == nil {
|
||||
return models.Memory{}, false
|
||||
}
|
||||
if status, ok := v.r.SourceStatus[SourceAgent]; !ok || status.Status != "online" {
|
||||
if status, ok := r.SourceStatus[SourceAgent]; !ok || status.Status != "online" {
|
||||
return models.Memory{}, false
|
||||
}
|
||||
m := v.r.Agent.Memory
|
||||
m := r.Agent.Memory
|
||||
memory := models.Memory{Total: m.Total, Used: m.Used, Free: m.Free, Cache: m.Cache, UsageUnavailable: m.UsageUnavailable}
|
||||
if m.Total <= 0 {
|
||||
return models.Memory{}, false
|
||||
@@ -397,6 +404,14 @@ type ContainerView struct{ r *Resource }
|
||||
|
||||
func NewContainerView(r *Resource) ContainerView { return ContainerView{r: r} }
|
||||
|
||||
// LinkedAgentMemory returns the agent's own sample for a system container whose
|
||||
// merged resource also carries an online Pulse agent source. Proxmox
|
||||
// cluster/resources memory is a low-trust fallback that can badly misreport a
|
||||
// container's real footprint, so the linked agent sample is preferred (#2148).
|
||||
func (v ContainerView) LinkedAgentMemory() (models.Memory, bool) {
|
||||
return linkedAgentMemoryFromResource(v.r)
|
||||
}
|
||||
|
||||
func (v ContainerView) String() string { return fmt.Sprintf("ContainerView(%s, %q)", v.ID(), v.Name()) }
|
||||
|
||||
func (v ContainerView) ID() string {
|
||||
|
||||
@@ -2267,3 +2267,34 @@ func TestVMViewLinkedAgentMemory(t *testing.T) {
|
||||
t.Fatal("nil view has memory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestContainerViewLinkedAgentMemory(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
status string
|
||||
stale bool
|
||||
total, used int64
|
||||
unavailable bool
|
||||
want bool
|
||||
}{
|
||||
{name: "live", status: "online", total: 24 << 30, used: 11 << 30, want: true},
|
||||
{name: "offline", status: "offline", total: 24 << 30, used: 11 << 30},
|
||||
{name: "stale agent", status: "online", stale: true, total: 24 << 30, used: 11 << 30},
|
||||
{name: "over capacity", status: "online", total: 24 << 30, used: 25 << 30},
|
||||
{name: "unavailable", status: "online", total: 24 << 30, used: 11 << 30, unavailable: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
r := &Resource{Status: StatusOnline, Agent: &AgentData{Stale: tc.stale, Memory: &AgentMemoryMeta{Total: tc.total, Used: tc.used, UsageUnavailable: tc.unavailable}}, SourceStatus: map[DataSource]SourceStatus{SourceAgent: {Status: tc.status}}, Metrics: &ResourceMetrics{Memory: &MetricValue{Percent: 100}}}
|
||||
memory, ok := NewContainerView(r).LinkedAgentMemory()
|
||||
if ok != tc.want {
|
||||
t.Fatalf("valid=%v want=%v memory=%+v", ok, tc.want, memory)
|
||||
}
|
||||
if ok && memory.Used != tc.used {
|
||||
t.Fatalf("not agent memory: %+v", memory)
|
||||
}
|
||||
})
|
||||
}
|
||||
if _, ok := (ContainerView{}).LinkedAgentMemory(); ok {
|
||||
t.Fatal("nil view has memory")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,6 +75,11 @@
|
||||
"incomplete_commit": "919009d7677318c071f888eb28c252d2d79d2728",
|
||||
"completion_commit": "9fa81dee8fba760b980d821d6e10560589c110f2",
|
||||
"reason": "Evaluate the PVE consolidation explicit-VerifySSL runtime change together with the follow-up commit that records its agent-lifecycle, api-contracts, security-privacy and dependent storage-recovery contract sections and the node API payload and persisted-field proofs, as one reviewed completion unit."
|
||||
},
|
||||
{
|
||||
"incomplete_commit": "8b56fc72e4ee79f3c219cf51f4c002e2e9343286",
|
||||
"completion_commit": "1026f5293b4c4e5a11651933e7ed36d398fd2f69",
|
||||
"reason": "Evaluate the Proxmox LXC linked-agent memory preference runtime change together with the follow-up commit that records its monitoring and unified-resources contract sections, as one reviewed completion unit."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -32,6 +32,7 @@ ATTESTATION_TOOL_PATH = "scripts/release_control/secure_runtime_attestation_v6.p
|
||||
CANONICAL_REPOSITORY = "rcourtman/Pulse"
|
||||
CANONICAL_ORIGIN_URL = "https://github.com/rcourtman/Pulse.git"
|
||||
CANONICAL_MAIN_REF = "origin/main"
|
||||
CANONICAL_BRANCH_REF_RE = re.compile(r"^origin/(main|release/v[0-9]+\.[0-9]+)$")
|
||||
ASSEMBLY_SIGNER_WORKFLOW = "github.com/rcourtman/Pulse/.github/workflows/build-release-candidate.yml"
|
||||
COMPILER_SIGNER_WORKFLOW = "github.com/rcourtman/Pulse/.github/workflows/compile-release-payload.yml"
|
||||
RELEASE_TAG_RE = re.compile(r"^v[0-9]+\.[0-9]+\.[0-9]+-rc\.[1-9][0-9]*$")
|
||||
@@ -167,17 +168,23 @@ def parse_remote_refs(raw: bytes) -> dict[str, str]:
|
||||
|
||||
|
||||
def verify_canonical_main_identity(checkout: Path, main_ref: str) -> str:
|
||||
if main_ref != CANONICAL_MAIN_REF:
|
||||
raise v5.AttestationError("committed-main classification requires canonical origin/main")
|
||||
match = CANONICAL_BRANCH_REF_RE.fullmatch(main_ref)
|
||||
if match is None:
|
||||
raise v5.AttestationError(
|
||||
"committed-main classification requires a canonical origin branch ref"
|
||||
)
|
||||
branch = match.group(1)
|
||||
origin = v5.run_git(checkout, "remote", "get-url", "origin").stdout.decode().strip()
|
||||
if origin != CANONICAL_ORIGIN_URL:
|
||||
raise v5.AttestationError("origin remote URL is not the canonical Pulse repository URL")
|
||||
local_main = v5.resolve_commit(checkout, CANONICAL_MAIN_REF, "canonical origin/main")
|
||||
remote = v5.run_git(checkout, "ls-remote", "origin", "refs/heads/main")
|
||||
local_branch = v5.resolve_commit(checkout, main_ref, f"canonical origin/{branch}")
|
||||
remote = v5.run_git(checkout, "ls-remote", "origin", f"refs/heads/{branch}")
|
||||
remote_refs = parse_remote_refs(remote.stdout)
|
||||
if remote_refs != {"refs/heads/main": local_main}:
|
||||
raise v5.AttestationError("canonical origin/main does not match the remote main commit")
|
||||
return local_main
|
||||
if remote_refs != {f"refs/heads/{branch}": local_branch}:
|
||||
raise v5.AttestationError(
|
||||
f"canonical origin/{branch} does not match the remote branch commit"
|
||||
)
|
||||
return local_branch
|
||||
|
||||
|
||||
def verify_release_candidate_tag_identity(
|
||||
@@ -609,7 +616,7 @@ def verify_and_snapshot_release_candidate_packet(
|
||||
raise v5.AttestationError("qualified commit must be the full canonical commit SHA")
|
||||
v5.require_detached_clean_checkout(checkout, resolved_commit)
|
||||
main_commit = verify_canonical_main_identity(checkout, main_ref)
|
||||
v5.require_ancestor(checkout, resolved_commit, CANONICAL_MAIN_REF)
|
||||
v5.require_ancestor(checkout, resolved_commit, main_ref)
|
||||
|
||||
expected_assets = expected_release_asset_names(architecture)
|
||||
expected_versions = expected_release_artifact_versions(tag)
|
||||
@@ -952,7 +959,7 @@ def _create_attestation_with_snapshotted_artifacts(
|
||||
raise v5.AttestationError("qualified commit must be the full canonical commit SHA")
|
||||
v5.require_detached_clean_checkout(checkout, qualified_commit)
|
||||
main_commit = verify_canonical_main_identity(checkout, main_ref)
|
||||
v5.require_ancestor(checkout, qualified_commit, CANONICAL_MAIN_REF)
|
||||
v5.require_ancestor(checkout, qualified_commit, main_ref)
|
||||
with v6_contract():
|
||||
receipt, receipt_bytes = v5.load_receipt(receipt_path)
|
||||
record_path = v5.canonical_repo_path(receipt_record_path, "receipt record path")
|
||||
@@ -1018,7 +1025,7 @@ def _create_attestation_with_snapshotted_artifacts(
|
||||
"qualified_ref_at_run": release_candidate_tag or qualified_commit,
|
||||
"main_ref_verified": main_ref,
|
||||
"main_ref_commit_at_attestation": main_commit,
|
||||
"qualified_commit_reachable_from_main": True,
|
||||
"qualified_commit_reachable_from_main": main_ref == CANONICAL_MAIN_REF,
|
||||
"build_checkout": "detached-worktree",
|
||||
"build_checkout_clean_except_lab_artifacts": True,
|
||||
"disposable_vm_guard_receipt_claim_validated": True,
|
||||
|
||||
@@ -468,9 +468,9 @@ class SecureRuntimeAttestationV6Test(unittest.TestCase):
|
||||
verify_canonical_main_identity(self.root, CANONICAL_MAIN_REF),
|
||||
self.commit,
|
||||
)
|
||||
for caller_ref in ("HEAD", "main", "refs/heads/main", "scratch"):
|
||||
for caller_ref in ("HEAD", "main", "refs/heads/main", "scratch", "origin/feature/x"):
|
||||
with self.subTest(caller_ref=caller_ref), self.assertRaisesRegex(
|
||||
v5.AttestationError, "canonical origin/main"
|
||||
v5.AttestationError, "canonical origin branch ref"
|
||||
):
|
||||
verify_canonical_main_identity(self.root, caller_ref)
|
||||
|
||||
@@ -482,10 +482,45 @@ class SecureRuntimeAttestationV6Test(unittest.TestCase):
|
||||
|
||||
with (
|
||||
mock.patch.object(v5, "run_git", side_effect=moved_remote),
|
||||
self.assertRaisesRegex(v5.AttestationError, "does not match the remote main commit"),
|
||||
self.assertRaisesRegex(v5.AttestationError, "does not match the remote branch commit"),
|
||||
):
|
||||
verify_canonical_main_identity(self.root, CANONICAL_MAIN_REF)
|
||||
|
||||
def test_committed_main_identity_accepts_mapped_release_branch(self) -> None:
|
||||
release_ref = "origin/release/v6.4"
|
||||
|
||||
def canonical(_checkout, *args, **_kwargs):
|
||||
command = tuple(args)
|
||||
if command == ("remote", "get-url", "origin"):
|
||||
output = CANONICAL_ORIGIN_URL + "\n"
|
||||
elif command == ("rev-parse", "--verify", f"{release_ref}^{{commit}}"):
|
||||
output = self.commit + "\n"
|
||||
elif command == ("ls-remote", "origin", "refs/heads/release/v6.4"):
|
||||
output = f"{self.commit}\trefs/heads/release/v6.4\n"
|
||||
else:
|
||||
raise AssertionError(f"unexpected git call {command}")
|
||||
return subprocess.CompletedProcess(args, 0, output.encode(), b"")
|
||||
|
||||
with mock.patch.object(v5, "run_git", side_effect=canonical):
|
||||
self.assertEqual(
|
||||
verify_canonical_main_identity(self.root, release_ref),
|
||||
self.commit,
|
||||
)
|
||||
|
||||
def moved_remote(_checkout, *args, **kwargs):
|
||||
result = canonical(_checkout, *args, **kwargs)
|
||||
if tuple(args) == ("ls-remote", "origin", "refs/heads/release/v6.4"):
|
||||
return subprocess.CompletedProcess(
|
||||
args, 0, f"{'c' * 40}\trefs/heads/release/v6.4\n".encode(), b""
|
||||
)
|
||||
return result
|
||||
|
||||
with (
|
||||
mock.patch.object(v5, "run_git", side_effect=moved_remote),
|
||||
self.assertRaisesRegex(v5.AttestationError, "does not match the remote branch commit"),
|
||||
):
|
||||
verify_canonical_main_identity(self.root, release_ref)
|
||||
|
||||
def test_accepts_canonical_annotated_tag_only_as_release_packet_identity(self) -> None:
|
||||
with mock.patch.object(v5, "run_git", side_effect=lambda _checkout, *args, **_kwargs: self.git_result(*args)):
|
||||
identity = verify_release_candidate_tag_identity(
|
||||
|
||||
Reference in New Issue
Block a user