Let guest metadata writes finish before the monitor stops

persistGuestIdentity spawned a detached goroutine per changed guest to write
guest_metadata.json, with a comment noting it avoided blocking the monitor.
Nothing tracked those goroutines, so neither Monitor.Stop nor
MultiTenantMonitor.Stop could wait for them and a queued write could land after
shutdown. In hosted mode that means a write into a tenant directory that
offboarding is already removing, and a stray guest_metadata.json.tmp left
behind when the atomic write is interrupted.

The store now owns the goroutine. SetAsync tracks the write on a WaitGroup and
WaitForPendingWrites drains it under a bounded timeout matching
tenantMonitorShutdownTimeout, so a wedged store cannot hold up tenant teardown.
Monitor.Stop drains before closing the metrics store.

This is what made TestHostedTenantAgentInstallTokenCannotReportToOtherTenant
flaky: t.TempDir cleanup raced a queued write into orgs/client-b and failed
with "directory not empty". The test itself is unchanged, because it was never
a test bug. A goroutine dump at cleanup time showed the writers still live,
created by persistGuestIdentity, blocked on the store mutex.

Verified causally rather than by observation alone: the target test fails 0/4
with the drain removed and passes 8/8 with it, against 2/3 failures on the
unmodified baseline. The regression tests fail if SetAsync stops tracking its
goroutine.

Note for a future pass, deliberately not changed here: each changed guest still
triggers a full-file save, so one poll cycle over N changed guests does N
marshals and N atomic writes that serialize on the store mutex anyway. Fixing
that means coalescing at the call site and is a behavioural change beyond this
defect.
This commit is contained in:
rcourtman
2026-08-05 18:52:21 +01:00
parent 37a8f4a6ff
commit f3dd544ce2
8 changed files with 253 additions and 6 deletions
@@ -5811,3 +5811,10 @@ coarse activity class (`list`, `export`, `verify`, `summary`); the handler runs
unchanged whether or not recording succeeds, and a router without persistence
serves the request rather than failing it
(`TestWithAuditReadActivity_NilPersistenceIsSafe`).
### Monitor shutdown drains queued guest metadata writes
`Monitor.Stop` waits for in-flight `GuestMetadataStore` writes before closing
the metrics store, so a tenant monitor that has been stopped is guaranteed not
to write into its data directory afterwards. Tenant offboarding and any caller
that removes a tenant directory can rely on `Stop` having quiesced disk writes,
rather than racing a detached goroutine.
@@ -2767,3 +2767,24 @@ host dataset evidence is copied onto the matching provider-owned ZFS pool;
provider health, scan, device, and error fields remain authoritative. When the
provider cannot return pool detail, monitoring may synthesize only a minimal
`UNKNOWN` pool so valid dataset evidence is still inspectable.
### Guest metadata writes are owned by the store and drained on shutdown
`persistGuestIdentity` no longer detaches its own goroutine per changed guest.
It calls `GuestMetadataStore.SetAsync`, which tracks the write on a WaitGroup
so `GuestMetadataStore.WaitForPendingWrites` can drain it. `Monitor.Stop` drains
before closing the metrics store, under a bounded timeout matching
`tenantMonitorShutdownTimeout` so a wedged store cannot hold up tenant teardown.
Untracked writes were observable, not theoretical: a queued write could land
after the monitor stopped and after a tenant directory was being removed,
leaving a stray `guest_metadata.json.tmp` from the interrupted atomic write.
That is what made `TestHostedTenantAgentInstallTokenCannotReportToOtherTenant`
fail its `t.TempDir` cleanup with "directory not empty".
`TestGuestMetadataStore_WaitForPendingWritesDrainsQueuedWrites` and
`TestGuestMetadataStore_DataDirIsRemovableAfterDrain` pin the drain and fail if
`SetAsync` stops tracking its goroutine.
Known and deliberately unchanged: each changed guest still triggers a full-file
save, so one poll cycle over N changed guests performs N marshals and N atomic
writes that serialize on the store mutex. Coalescing them is a behavioural
change beyond the shutdown defect.