mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-23 19:57:09 +00:00
Let guest metadata writes finish before the monitor stops
persistGuestIdentity spawned a detached goroutine per changed guest to write guest_metadata.json, with a comment noting it avoided blocking the monitor. Nothing tracked those goroutines, so neither Monitor.Stop nor MultiTenantMonitor.Stop could wait for them and a queued write could land after shutdown. In hosted mode that means a write into a tenant directory that offboarding is already removing, and a stray guest_metadata.json.tmp left behind when the atomic write is interrupted. The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and WaitForPendingWrites drains it under a bounded timeout matching tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown. Monitor.Stop drains before closing the metrics store. This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed with "directory not empty". The test itself is unchanged, because it was never a test bug. A goroutine dump at cleanup time showed the writers still live, created by persistGuestIdentity, blocked on the store mutex. Verified causally rather than by observation alone: the target test fails 0/4 with the drain removed and passes 8/8 with it, against 2/3 failures on the unmodified baseline. The regression tests fail if SetAsync stops tracking its goroutine. Note for a future pass, deliberately not changed here: each changed guest still triggers a full-file save, so one poll cycle over N changed guests does N marshals and N atomic writes that serialize on the store mutex anyway. Fixing that means coalescing at the call site and is a behavioural change beyond this defect.
This commit is contained in:
@@ -5811,3 +5811,10 @@ coarse activity class (`list`, `export`, `verify`, `summary`); the handler runs
|
||||
unchanged whether or not recording succeeds, and a router without persistence
|
||||
serves the request rather than failing it
|
||||
(`TestWithAuditReadActivity_NilPersistenceIsSafe`).
|
||||
### Monitor shutdown drains queued guest metadata writes
|
||||
|
||||
`Monitor.Stop` waits for in-flight `GuestMetadataStore` writes before closing
|
||||
the metrics store, so a tenant monitor that has been stopped is guaranteed not
|
||||
to write into its data directory afterwards. Tenant offboarding and any caller
|
||||
that removes a tenant directory can rely on `Stop` having quiesced disk writes,
|
||||
rather than racing a detached goroutine.
|
||||
|
||||
@@ -2767,3 +2767,24 @@ host dataset evidence is copied onto the matching provider-owned ZFS pool;
|
||||
provider health, scan, device, and error fields remain authoritative. When the
|
||||
provider cannot return pool detail, monitoring may synthesize only a minimal
|
||||
`UNKNOWN` pool so valid dataset evidence is still inspectable.
|
||||
### Guest metadata writes are owned by the store and drained on shutdown
|
||||
|
||||
`persistGuestIdentity` no longer detaches its own goroutine per changed guest.
|
||||
It calls `GuestMetadataStore.SetAsync`, which tracks the write on a WaitGroup
|
||||
so `GuestMetadataStore.WaitForPendingWrites` can drain it. `Monitor.Stop` drains
|
||||
before closing the metrics store, under a bounded timeout matching
|
||||
`tenantMonitorShutdownTimeout` so a wedged store cannot hold up tenant teardown.
|
||||
|
||||
Untracked writes were observable, not theoretical: a queued write could land
|
||||
after the monitor stopped and after a tenant directory was being removed,
|
||||
leaving a stray `guest_metadata.json.tmp` from the interrupted atomic write.
|
||||
That is what made `TestHostedTenantAgentInstallTokenCannotReportToOtherTenant`
|
||||
fail its `t.TempDir` cleanup with "directory not empty".
|
||||
`TestGuestMetadataStore_WaitForPendingWritesDrainsQueuedWrites` and
|
||||
`TestGuestMetadataStore_DataDirIsRemovableAfterDrain` pin the drain and fail if
|
||||
`SetAsync` stops tracking its goroutine.
|
||||
|
||||
Known and deliberately unchanged: each changed guest still triggers a full-file
|
||||
save, so one poll cycle over N changed guests performs N marshals and N atomic
|
||||
writes that serialize on the store mutex. Coalescing them is a behavioural
|
||||
change beyond the shutdown defect.
|
||||
|
||||
Reference in New Issue
Block a user