Live funded qualification found hidden tool results and misleading action
submission outcomes. Share the result-bearing transcript across stored chat
and product history, render the retained evidence, and distinguish captured
proposals from broker acceptance. Keep review usable while Patrol is paused.
Record Gemini route pricing and exact qualification limits. Integrate current
main and repeat browser proof for the incoming login flow. Approved/rejected
recovery remains unqualified without the development command agent.
Release qualification crashed in historical baseline SQLite ingestion during the workloads-summary seed. Preserve a metrics-only diagnostic for the same 84,000-row batch shape, with row-count and integrity checks before and after reopen, without the HTTP or reflection fixtures. This does not reproduce or fix the unexplained crash.
Validation: ten focused runs on Go 1.26.7 and one on Go 1.26.8 passed; a one-repeat race run passed. The three-repeat race run timed out at 180 seconds during its final integrity check and remains retained evidence. Omitting the final seed batch makes the row-count assertion fail.
Change-source: pulse-maintainer
Missing block I/O and container image sizes could become false evidence
for diagnosis. Preserve per-direction counter presence and measured zero
through collection, resource conversion and browser rendering. Separate
new observed history from ambiguous retained disk series without deleting
old rows or changing public metric names.
Keep partial host rates distinct and persist a newly enabled Disk I/O
column across the first preference reload.
The 500-node dashboard query triggered repeated ordinal string conversions
in the SQLite driver while matching numbered parameters. Use alphabetic
named bindings to preserve current values and shared query branches without
that allocation cost.
Cover large cached scopes with changed resource families, identities,
metric filters and windows, including IDs that resemble SQL syntax.
Refs #1928
Canonical tier reconciliation rebuilt SQL and probed absent preferred tiers
for each fallback point, regressing batch reads and allocation costs. Reuse
bounded query shapes with current bindings and snapshot-scoped absence
checks, then append consecutive points directly to their output series.
Preserve coverage and ordering semantics and verify fresh bindings after
new preferred observations arrive. Integrate current main test additions.
Execute plain retained reconciliation in one current SQLite snapshot and
reuse bounded compiled statements. Preserve per-series chronology without
a metric sort while keeping display aggregation ordering explicit.
Exact-base worker comparisons cover the prior PR benchmark failures. Full
metrics/database and focused concurrent race checks pass. Final CI and
real diagnostic outcome qualification remain open.
Keep canonical disk risk, source freshness and retained history intact when
Assistant and Patrol gather evidence. Proposal acceptance validates an action
contract and must not rewrite uncertain conclusions as established root cause.
Preserve complete subscription tool batches without exposing routing envelopes
as answers. Keep wide answer tables readable and keyboard-scrollable on mobile.
Optimize retained tier reconciliation without discarding gaps or newer samples.
Record failed real-model diagnoses and outstanding autonomous qualification
separately from passing data-path and interface checks.
Non-empty aggregate tiers hid recent raw samples and missing metric series.
Unify single and batch reads with indexed overlap resolution before
downsampling, and preserve recorded extrema through subsequent rollups.
Use canonical Proxmox storage coordinates in summaries and node history.
Discovery routing does not establish an installed Agent or agent history.
Record live evidence freshness and the remaining diagnosis qualification gaps.
The persistence profile labelled /proc/self/io write_bytes as physical writes, which could mislead release qualification into attributing process accounting to device wear. Report process writes and cancellation separately, preserving unavailable counters as unknown rather than implying zero.
Add focused accounting coverage for malformed, missing, overflowing and decreasing counters. This is diagnostic-only: it neither changes runtime persistence nor establishes installed write cost or release readiness.
Validation: focused TestIssue1124ProcessIOAccounting and serialised persistence profile passed before this message-only repair; git diff --check passed. The tested tree is unchanged.
Change-source: pulse-maintainer
Incorporate the metrics startup hook capture from PR #1868 while retaining
the reviewed causal cleanup and barrier-ordering coverage. This advances the
open publication proposal without rewriting any accepted commit.
Change-source: pulse-maintainer
Contract-Neutral: Integration reconciliation only; no additional public contract delta.
TestNewStoreDefersStartupMaintenance bounded NewStore at 200ms. On a
slow CI disk that bound tripped, the test failed, and its NewStore
goroutine kept running: the maintenance worker it spawned read the
package-level startupMaintenanceHook after the next test had installed
its own closure, closed that test's started channel a second time, and
panicked the whole rest-1 shard (run 33630289317, attempt 1).
Capture the hook once in NewStore so a store can only ever call the
hook that was installed when it was built. Prove deferral by ordering
instead of wall clock: the hook parks the worker, and NewStore must
return while it is parked. A regression to inline maintenance now
blocks that receive until the package timeout instead of flaking.
Cleanup releases the worker and waits for NewStore before restoring
the hook, so a failed run cannot leak a parked store either.
Contract-Neutral: behavioral test-flake fix with no public contract delta
The race-enabled suite can take longer than the tests' fixed sleeps while opening SQLite stores. Assert worker ordering through channels instead, and always release and join blocked maintenance workers so a failed assertion cannot contaminate the following test.
Change-source: pulse-maintainer
6b79aa997 bounded WriteBatchSync itself, which broke its read-your-writes
contract on slow disks: CI's metrics write-amplification and 500-node
load tests count committed rows after writing, and mock seeding reads
store coverage straight back, so the 2-second early return failed both
(runs 31475700902, 31494553977). Fast local disks masked it.
WriteBatchSync returns to a full commit wait. The monitoring pipeline's
four sync sites move to WriteBatchBounded, which carries the bounded
enqueue-plus-wait semantics, so the #1437 slow-disk stall fix stays
exactly where the hazard is. Both paths share prepareWriteBatch
validation, and a new regression test pins WriteBatchSync waiting past
the bounded budget.
Refs #1437
Contract-Neutral: behavioral fix: split bounded pipeline writes from synchronous batch writes, restores read-your-writes (#1437 follow-up), no public contract delta
WriteBatchSync waited unboundedly for the ingestion worker's commit. The
monitoring pipeline calls it inline from state broadcast, agent ingest,
and poll publish, so a metrics disk slow enough to back up the write
queue froze the monitor after its first cycle: polls stopped being
scheduled, PollStatus.LastSuccess never advanced, and healthy API
sources degraded to stale/agent-only while SQLite ground through
retention maintenance (107s cleanup, multi-second commits on the
reporter's instance).
enqueueAndWait now shares a single 2s budget across enqueue and commit.
A queue that cannot accept the batch within the budget drops it with a
warning, matching enqueueWrite's saturation behavior. A batch that
enqueues but has not committed stays queued and is not lost; the caller
moves on and a rate-limited warning records the backlog. Healthy disks
keep read-your-writes semantics.
Refs #1437
Contract-Neutral: behavioral fix: bound metrics store sync write wait (#1437), no public contract delta
Use a native absolute path in the SMART override test so Windows verifies the same contract as Unix. Move the metrics concurrency proof into the canonical verification artifact and document that live ingestion remains independent from lifecycle maintenance.
Run startup maintenance, rollups, and retention on a lifecycle-owned worker separate from buffered metric ingestion. SQLite still serializes write transactions, while maintenance CPU/read work can no longer stop writeCh from draining until it drops live batches. Join both workers before closing the shared database.\n\nFixes #1601
The metrics store capped its SQLite pool at one connection, so every UI
history read queued behind every buffered-write commit, and behind the
WAL checkpoints those commits pick up at the 4000-page threshold. On
write-heavy installs (many Docker agents with churning containers) that
serialization presented as sustained 120-260ms COMMIT warnings and an
unresponsive UI even with idle CPU and fast disks. Writes were never the
risk: flush, rollup, retention, and maintenance already funnel through
the single background worker goroutine, and the WriteBatchSync poller
path serializes on the WAL write lock via busy_timeout.
Raising the pool exposed a second bug: auto_vacuum(INCREMENTAL) in the
per-connection DSN pragmas replays as a database-header write whenever
the pool opens a new connection, which blocks connection creation behind
the active writer for up to the full 30s busy_timeout. auto_vacuum is a
persistent database property that migrateAutoVacuum already establishes
once at startup, so the per-connection copy is dropped.
Refs #1601
Contract-Neutral: behavioral fix: metrics store read concurrency and per-connection auto_vacuum pragma removal, no public contract delta (#1601)
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.
Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.
Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Follow-up to 2b8ce06c0: that commit's contract update states reclaim skips
the WAL checkpoint entirely when the freelist is empty, but the prior commit
captured the pre-refinement store.go (the re-stage was lost to a .git/index.lock
race). Make the code match the contract: return early on a freelist_count read
error and when freelist == 0, so a steady-state no-op cycle does not run an
extra hourly wal_checkpoint(TRUNCATE).
runRetention reclaimed only PRAGMA incremental_vacuum(5000) (~20MB) and
only when that hourly cycle deleted rows. On instances where an hourly
retention pass frees more than 5000 pages, the freelist grows net-positive
every cycle and the sqlite file bloats unboundedly (5GB+ of free pages over
~60MB of live data) even though row retention works. This is the documented
'50+ resources -> 5GB+' symptom the 5000-page batch was meant to fix.
Reclaim every cycle (so a pre-existing backlog drains even in a no-delete
hour) and size each reclaim to the current freelist, capped at 50000 pages
(~200MB) so a large backlog drains over several cycles without holding the
write lock for minutes. Skip the checkpoint when the freelist is empty so
steady-state WAL cadence is unchanged.
Surfaced by the demo server (pulse-relay): metrics.db had grown to 5.9GB
(5GB freelist), filling its 25GB disk and breaking the nightly backup.
Adds TestStoreRetentionReclaimsFreePages to the perf-and-scalability proof
set and documents the reclaim invariant in the subsystem contract.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Extract alert config types, normalization, and identity helpers into internal/alerts/config while preserving the existing alerts package API through aliases and wrappers.
Move Manager callback lifecycle state into a same-package callbackBus, keeping public Set/Subscribe methods unchanged.
Harden metrics SQLite artifacts to owner-only permissions and cover permissive umask behavior.
Proof: go test -json ./internal/api -count=1; go test ./internal/alerts/... ./internal/monitoring ./internal/ai/... ./internal/websocket ./internal/config ./pkg/metrics; go test ./internal/alerts/... ./pkg/metrics
rcourtman