GET /api/charts/workloads-summary has had no first-party caller since
the v6 import: ChartsAPI.getWorkloadsSummaryCharts arrived with it and
nothing ever called it. GET /api/charts/storage-summary lost its last
reader when storageSummaryTrendCache.ts was deleted on 2026-05-26.
Both still cost something. The workloads route carried a payload cache,
an SLO test in the release latency gate and a benchmark, and mock mode
rebuilt the 24-hour storage aggregate on every sampler tick for a cache
nothing read.
No stable release ever requested either route from the browser: v6.0.0
and later carry no caller, and the storage trend reader shipped only in
v6.0.0-rc.1 to rc.5. Neither route is in docs/API.md (a list of common
endpoints, not a complete one), and pulse-pro, pulse-mobile and
pulse-dev-infra never call them. A monitoring:read token could reach
them, so a private script may notice, but nothing known depends on
them. pulse-enterprise names the SLO test only as fixture data, and its
latency gate passes a test that is absent from the candidate as
removed. So the routes go with their clients:
handlers, response types, the monitor trend helpers, the mock aggregate
and its prewarm, the SLO budget, the benchmark, and the ChartsAPI
methods and types. Requests to either path now get 404.
buildMockWorkloadMetricHistorySeries goes too, because the
workloads-summary handler was its only production caller.
A TypeScript reference graph from src/index.tsx finds no production
reference to the removed methods or types. The production build loses
only the two static methods from the ChartsAPI chunk (326 B raw, 49 B
gzip); every other file differs only by chunk hashes and SRI values.
deadcode ./cmd/pulse and golangci unused report nothing newly
unreachable. On private mock stacks the Proxmox overview in Trends, a
guest History drawer and the storage page make the same chart requests
before and after at 1440 and 390 px, and neither build requested the
removed routes.
Contract prose that still described either route as live now records
the retirement, and the workload chart guard pins the per-type metric
sets the surviving /api/charts/workloads route requests.
Metric identity lived in idx_metrics_lookup, ordered (type, id, metric,
tier, timestamp). Once a series retains history, that order gives every
series its own insertion point, so each poll's commit rewrites one index
leaf page per series through the WAL and again at checkpoint. On a
three-node homelab running v6.4.5-rc.2 that came to 330-607 KB/s of disk
writes, about 10.6 KB per ~60-byte sample and 27-50 GB a day (#1966).
idx_metrics_query_all already holds the same five columns time-major, so
one poll's samples for a resource share pages. It now carries uniqueness
and serves every read, and the metric-major lookup index is dropped. With
an hour of retained history for 480 series, 30 polls write 6,022 WAL
frames instead of 25,310 (4.2x fewer); the empty-table issue-1124 estate
drops from 36,308 frames to 20,223, and its ceiling tightens from 40,000
to 23,000.
The tier-reconciliation overlap probe now names the identity index too;
left to the planner it seeks on (tier, timestamp) alone once the
metric-major tree is gone, which TestRetainedQueryPlansUseIndexes caught.
Metric-filtered reads now scan a resource's other metrics in the window:
a single-metric 500-node query went from 45 to 59 us and the 24-hour
50-disk smart_temp batch from about 0.25 to 0.49 s. All-metric dashboard
reads are unchanged. Continuous write volume is the user-visible harm, so
the contract records that trade explicitly.
The migration reuses the crash-safe swap: databases whose identity is
already enforced by a unique index defer the rebuild to startup
maintenance and swap in one transaction, and a database with no unique
index migrates synchronously. Keeping the name idx_metrics_query_all
means a downgrade rebuilds only the lookup index and leaves no orphan.
Refs #1966
Address the remaining verified findings from the pkg/metrics/store.go review
(#2079 finding 5 and the lower-impact nits):
- writeBatch now retries the whole transaction on a retryable lock error.
BEGIN is deferred, so contention is hit at the first INSERT or COMMIT; the
old loop retried only Begin() and skipped rows. While the one-time startup
maintenance (deferred identity-index rebuild and auto-vacuum VACUUM) is
active, an extended retry budget keeps upgrade-time history instead of
dropping it after the steady-state budget.
- migrateLegacyHostResourceType probes with an indexed EXISTS before opening a
write transaction, so an already-migrated store does not dirty the WAL on
every boot.
- QueryAll caps the per-series MetricPoint preallocation so a fine requested
step over a long range cannot reserve hundreds of thousands of slots.
Regressions: TestStoreStartupMaintenanceSignalsWriteRetryBudget,
TestStoreWriteBatchExtendsRetryDuringStartupMaintenance,
TestEstimateQueryAllBatchSeriesCapacityCapsPreallocation. The
performance-and-scalability contract records the invariants.
Change-source: pulse-maintainer
The canonical completion guard requires the metrics-store hot-path proof in
pkg/metrics/store_additional_test.go. Move the rollup-gap, upsert-spread,
busy-retry and auto-vacuum regressions there and point the
performance-and-scalability contract at that file. The runtime change and
contract section are in the preceding fix(metrics) commit.
Change-source: pulse-maintainer
Address the remaining verified findings from the pkg/metrics/store.go review
(#2079 findings 2-4 and 6):
- rollupTier no longer leaps its checkpoint over a source gap. Previously a
trailing or interspersed gap advanced the checkpoint to the rollup cutoff, so
a lower tier that backfilled the gap later had those rows stranded below the
upper checkpoint and purged by retention.
- Retry conditions now classify SQLITE_BUSY/LOCKED by driver result code. The
modernc driver reports "database is locked (5) (SQLITE_BUSY)", which the
previous exact-string comparison never matched, so a busy writer dropped its
batch without retrying.
- The metrics upsert preserves an existing rollup row's min/max spread with
COALESCE instead of overwriting it with NULL.
- migrateAutoVacuum pins the pragma and VACUUM to one connection and verifies
the persisted mode, so the one-time conversion cannot silently repeat a
full-file VACUUM on every restart.
Regressions: TestStoreRollupTierBackfillsInterspersedGap,
TestStoreWriteBatchPreservesRollupSpread, TestIsRetryableWriteError,
TestIsRetryableWriteErrorMatchesDriverBusyCode,
TestStoreAutoVacuumPersistsAcrossRestart. The performance-and-scalability
contract records the invariants.
Change-source: pulse-maintainer
The shared release preflight worker exports GITHUB_ACTIONS=true so isolated single-repository checkouts take their private-sibling skip. That flag also made the load/SLO latency guards hard-fail on a contended shared host, where the tests' own documented local-contention skip is the intended behaviour. Distinguish a real GitHub Actions run by GITHUB_RUN_ID, which only a hosted runner sets, so the worker keeps the cross-repo skip while latency overruns skip instead of failing. Hosted CI enforcement is unchanged.
Change-source: pulse-maintainer
The ingestion worker closed writeCh when it observed stopCh. A concurrent WriteWithTier that passed its stopping check, or a WriteBatchSync/WriteBatchBounded caller (which never checks stopping), could then send on the closed channel and panic the process during shutdown. Reproduced deterministically: enqueueWrite after Close panics with send on closed channel, and 8 concurrent monitoring writers racing Close panic in boundedEnqueueAndWait (store.go:1034).
Never close writeCh. Drain already-queued requests non-blockingly, then process them. Late writes land in the buffered channel and are discarded with the store rather than crashing it. Record the invariant in the performance-and-scalability contract and pin it in the accepted store proof file.
Change-source: pulse-maintainer
History requests can occupy every database connection and make cheap
statistics reads wait for seconds. Reserve one read-only connection for
current committed counts, owned and closed by the metrics store.
Keep the SQL, data model, load workload and latency budgets unchanged.
Add deterministic pool-saturation, committed-data and shutdown coverage.
Live funded qualification found hidden tool results and misleading action
submission outcomes. Share the result-bearing transcript across stored chat
and product history, render the retained evidence, and distinguish captured
proposals from broker acceptance. Keep review usable while Patrol is paused.
Record Gemini route pricing and exact qualification limits. Integrate current
main and repeat browser proof for the incoming login flow. Approved/rejected
recovery remains unqualified without the development command agent.
Release qualification crashed in historical baseline SQLite ingestion during the workloads-summary seed. Preserve a metrics-only diagnostic for the same 84,000-row batch shape, with row-count and integrity checks before and after reopen, without the HTTP or reflection fixtures. This does not reproduce or fix the unexplained crash.
Validation: ten focused runs on Go 1.26.7 and one on Go 1.26.8 passed; a one-repeat race run passed. The three-repeat race run timed out at 180 seconds during its final integrity check and remains retained evidence. Omitting the final seed batch makes the row-count assertion fail.
Change-source: pulse-maintainer
Missing block I/O and container image sizes could become false evidence
for diagnosis. Preserve per-direction counter presence and measured zero
through collection, resource conversion and browser rendering. Separate
new observed history from ambiguous retained disk series without deleting
old rows or changing public metric names.
Keep partial host rates distinct and persist a newly enabled Disk I/O
column across the first preference reload.
The 500-node dashboard query triggered repeated ordinal string conversions
in the SQLite driver while matching numbered parameters. Use alphabetic
named bindings to preserve current values and shared query branches without
that allocation cost.
Cover large cached scopes with changed resource families, identities,
metric filters and windows, including IDs that resemble SQL syntax.
Refs #1928
Canonical tier reconciliation rebuilt SQL and probed absent preferred tiers
for each fallback point, regressing batch reads and allocation costs. Reuse
bounded query shapes with current bindings and snapshot-scoped absence
checks, then append consecutive points directly to their output series.
Preserve coverage and ordering semantics and verify fresh bindings after
new preferred observations arrive. Integrate current main test additions.
Execute plain retained reconciliation in one current SQLite snapshot and
reuse bounded compiled statements. Preserve per-series chronology without
a metric sort while keeping display aggregation ordering explicit.
Exact-base worker comparisons cover the prior PR benchmark failures. Full
metrics/database and focused concurrent race checks pass. Final CI and
real diagnostic outcome qualification remain open.
Keep canonical disk risk, source freshness and retained history intact when
Assistant and Patrol gather evidence. Proposal acceptance validates an action
contract and must not rewrite uncertain conclusions as established root cause.
Preserve complete subscription tool batches without exposing routing envelopes
as answers. Keep wide answer tables readable and keyboard-scrollable on mobile.
Optimize retained tier reconciliation without discarding gaps or newer samples.
Record failed real-model diagnoses and outstanding autonomous qualification
separately from passing data-path and interface checks.
Non-empty aggregate tiers hid recent raw samples and missing metric series.
Unify single and batch reads with indexed overlap resolution before
downsampling, and preserve recorded extrema through subsequent rollups.
Use canonical Proxmox storage coordinates in summaries and node history.
Discovery routing does not establish an installed Agent or agent history.
Record live evidence freshness and the remaining diagnosis qualification gaps.
The persistence profile labelled /proc/self/io write_bytes as physical writes, which could mislead release qualification into attributing process accounting to device wear. Report process writes and cancellation separately, preserving unavailable counters as unknown rather than implying zero.
Add focused accounting coverage for malformed, missing, overflowing and decreasing counters. This is diagnostic-only: it neither changes runtime persistence nor establishes installed write cost or release readiness.
Validation: focused TestIssue1124ProcessIOAccounting and serialised persistence profile passed before this message-only repair; git diff --check passed. The tested tree is unchanged.
Change-source: pulse-maintainer
Incorporate the metrics startup hook capture from PR #1868 while retaining
the reviewed causal cleanup and barrier-ordering coverage. This advances the
open publication proposal without rewriting any accepted commit.
Change-source: pulse-maintainer
Contract-Neutral: Integration reconciliation only; no additional public contract delta.
TestNewStoreDefersStartupMaintenance bounded NewStore at 200ms. On a
slow CI disk that bound tripped, the test failed, and its NewStore
goroutine kept running: the maintenance worker it spawned read the
package-level startupMaintenanceHook after the next test had installed
its own closure, closed that test's started channel a second time, and
panicked the whole rest-1 shard (run 33630289317, attempt 1).
Capture the hook once in NewStore so a store can only ever call the
hook that was installed when it was built. Prove deferral by ordering
instead of wall clock: the hook parks the worker, and NewStore must
return while it is parked. A regression to inline maintenance now
blocks that receive until the package timeout instead of flaking.
Cleanup releases the worker and waits for NewStore before restoring
the hook, so a failed run cannot leak a parked store either.
Contract-Neutral: behavioral test-flake fix with no public contract delta
The race-enabled suite can take longer than the tests' fixed sleeps while opening SQLite stores. Assert worker ordering through channels instead, and always release and join blocked maintenance workers so a failed assertion cannot contaminate the following test.
Change-source: pulse-maintainer
6b79aa997 bounded WriteBatchSync itself, which broke its read-your-writes
contract on slow disks: CI's metrics write-amplification and 500-node
load tests count committed rows after writing, and mock seeding reads
store coverage straight back, so the 2-second early return failed both
(runs 31475700902, 31494553977). Fast local disks masked it.
WriteBatchSync returns to a full commit wait. The monitoring pipeline's
four sync sites move to WriteBatchBounded, which carries the bounded
enqueue-plus-wait semantics, so the #1437 slow-disk stall fix stays
exactly where the hazard is. Both paths share prepareWriteBatch
validation, and a new regression test pins WriteBatchSync waiting past
the bounded budget.
Refs #1437
Contract-Neutral: behavioral fix: split bounded pipeline writes from synchronous batch writes, restores read-your-writes (#1437 follow-up), no public contract delta
WriteBatchSync waited unboundedly for the ingestion worker's commit. The
monitoring pipeline calls it inline from state broadcast, agent ingest,
and poll publish, so a metrics disk slow enough to back up the write
queue froze the monitor after its first cycle: polls stopped being
scheduled, PollStatus.LastSuccess never advanced, and healthy API
sources degraded to stale/agent-only while SQLite ground through
retention maintenance (107s cleanup, multi-second commits on the
reporter's instance).
enqueueAndWait now shares a single 2s budget across enqueue and commit.
A queue that cannot accept the batch within the budget drops it with a
warning, matching enqueueWrite's saturation behavior. A batch that
enqueues but has not committed stays queued and is not lost; the caller
moves on and a rate-limited warning records the backlog. Healthy disks
keep read-your-writes semantics.
Refs #1437
Contract-Neutral: behavioral fix: bound metrics store sync write wait (#1437), no public contract delta
Use a native absolute path in the SMART override test so Windows verifies the same contract as Unix. Move the metrics concurrency proof into the canonical verification artifact and document that live ingestion remains independent from lifecycle maintenance.
Run startup maintenance, rollups, and retention on a lifecycle-owned worker separate from buffered metric ingestion. SQLite still serializes write transactions, while maintenance CPU/read work can no longer stop writeCh from draining until it drops live batches. Join both workers before closing the shared database.\n\nFixes #1601
The metrics store capped its SQLite pool at one connection, so every UI
history read queued behind every buffered-write commit, and behind the
WAL checkpoints those commits pick up at the 4000-page threshold. On
write-heavy installs (many Docker agents with churning containers) that
serialization presented as sustained 120-260ms COMMIT warnings and an
unresponsive UI even with idle CPU and fast disks. Writes were never the
risk: flush, rollup, retention, and maintenance already funnel through
the single background worker goroutine, and the WriteBatchSync poller
path serializes on the WAL write lock via busy_timeout.
Raising the pool exposed a second bug: auto_vacuum(INCREMENTAL) in the
per-connection DSN pragmas replays as a database-header write whenever
the pool opens a new connection, which blocks connection creation behind
the active writer for up to the full 30s busy_timeout. auto_vacuum is a
persistent database property that migrateAutoVacuum already establishes
once at startup, so the per-connection copy is dropped.
Refs #1601
Contract-Neutral: behavioral fix: metrics store read concurrency and per-connection auto_vacuum pragma removal, no public contract delta (#1601)
A full local go test run on 2026-06-11 failed pkg/metrics while vite
builds were saturating the machine. Reproduced under synthetic 8-core
load: only the TestSLO_* wall-clock p95 assertions failed, never the
functional tests. A latency budget measured on a shared dev machine
cannot distinguish host contention from a code regression; load storms
inflated medians up to ~4x with no code change, so no absolute
threshold separates the two.
Route the latency SLO assertions (pkg/metrics, internal/api,
internal/monitoring) and the load/stress perf assertions (internal/api)
through helpers that keep strict enforcement on GitHub Actions runners
(controlled environment, existing hosted-runner envelopes unchanged)
and skip locally on overrun, printing the full p50/p95/p99
distribution in the skip message. A local pass still means the budget
was genuinely met. Error-response and correctness checks remain hard
failures. CI -race runs are unaffected: these tests already skip under
the race detector.
Verified: pkg/metrics, internal/api, and internal/monitoring all pass
with 8 CPU burners saturating the host, the scenario that previously
turned TestSLO_RollupTierBatchedFleet and four other budget tests red.
Mock mode seeded rich in-memory chart history but passed a nil store to the
seeder and the live tick, so the sqlite metrics store held almost nothing:
vmware/truenas/docker resources got one row per poll from the platform
ingestion paths and mock Proxmox guests got nothing at all (the unified sync
skips every resource in mock mode). Performance reports rendered
"Data Points: 0" for PVE guests and "Samples: 1" elsewhere, with no charts.
Re-enable store seeding behind an explicit PULSE_MOCK_SEED_METRICS_STORE
opt-in that scripts/hot-dev.sh and scripts/toggle-mock.sh export exactly
where they point PULSE_DATA_DIR at the isolated tmp/mock-data dir. Without
the opt-in (a production install flipping PULSE_MOCK_MODE on its real data
dir) the store stays untouched, which is what the old nil guard protected.
Replace the dormant dense seeding policy (every in-memory timestamp written
to both hourly and daily tiers, ~11M rows at current fixture scale) with
tier-correct backfill driven by the deterministic mock.SampleMetric runtime:
daily 30d at 4h spacing, hourly 7d at 2h, minute 24h at 15m, raw left to the
live tick (now also store-connected under the opt-in). Timestamps sit on the
spacing grid and a new Store.MaxTimestampsForTier coverage query fills only
the gap since the previous boot, so restarts neither duplicate rows nor
re-pay the seed: a fresh seed wrote 264,960 rows in 8.4s (71MB) and a
restart wrote only the gap plus fixture series whose mock IDs are not
boot-stable (k8s pod names, ceph FSID, a pre-existing generator defect).
Verified live in mock mode: 30-day PDF reports for a mock PVE VM
(checkout-web-01), a vSphere VM, the TrueNAS host, and a Docker container
now show 1260/1260/1260/540 data points with rendered time-series charts.
golangci-lint run ./... failed on ~190 pre-existing errcheck violations and
5 unformatted files, burying any new regression in noise. Fix all of them:
- Test files that hand-rolled mock-mode set/restore (vmware, truenas, and
friends) now use the canonical setMockModeForTest/testutil.SetMockMode
helper instead of drift copies that ignored SetEnabled errors.
- internal/mock and internal/monitoring tests get package-local
mustSetEnabled/mustSetMockEnabled/mustSetMonitorMockMode helpers that
fail the test on toggle errors.
- pkg/auth/sqlite_manager.go, pkg/metrics/store.go, pkg/server/server.go:
rollbacks in defers use the explicit-discard idiom, migration renames and
rollup commits log failures, the hosted reaper goroutine logs an error
exit, shutdown mock-disable logs failures.
- Remaining test sites check errors with t.Fatalf/t.Errorf or explicitly
discard best-effort calls (restore-chmods, handler-closure unmarshals)
per existing repo style.
- gofmt: internal/api/maintenance_verification.go, internal/ai/demo.go and
three findings test files.
Only dupl findings remain (44 pre-existing production-code duplication
pairs) — those need real refactors, not mechanical fixes.
Full test suites pass for every touched package.
Follow-up to 2b8ce06c0: that commit's contract update states reclaim skips
the WAL checkpoint entirely when the freelist is empty, but the prior commit
captured the pre-refinement store.go (the re-stage was lost to a .git/index.lock
race). Make the code match the contract: return early on a freelist_count read
error and when freelist == 0, so a steady-state no-op cycle does not run an
extra hourly wal_checkpoint(TRUNCATE).
runRetention reclaimed only PRAGMA incremental_vacuum(5000) (~20MB) and
only when that hourly cycle deleted rows. On instances where an hourly
retention pass frees more than 5000 pages, the freelist grows net-positive
every cycle and the sqlite file bloats unboundedly (5GB+ of free pages over
~60MB of live data) even though row retention works. This is the documented
'50+ resources -> 5GB+' symptom the 5000-page batch was meant to fix.
Reclaim every cycle (so a pre-existing backlog drains even in a no-delete
hour) and size each reclaim to the current freelist, capped at 50000 pages
(~200MB) so a large backlog drains over several cycles without holding the
write lock for minutes. Skip the checkpoint when the freelist is empty so
steady-state WAL cadence is unchanged.
Surfaced by the demo server (pulse-relay): metrics.db had grown to 5.9GB
(5GB freelist), filling its 25GB disk and breaking the nightly backup.
Adds TestStoreRetentionReclaimsFreePages to the perf-and-scalability proof
set and documents the reclaim invariant in the subsystem contract.
Tighten v5-to-v6 upgrade safety, release installability, provider MSP mode handling, AI cost accounting, metrics flushing, and frontend guardrails for the v6.0.0 GA candidate.
Extract alert config types, normalization, and identity helpers into internal/alerts/config while preserving the existing alerts package API through aliases and wrappers.
Move Manager callback lifecycle state into a same-package callbackBus, keeping public Set/Subscribe methods unchanged.
Harden metrics SQLite artifacts to owner-only permissions and cover permissive umask behavior.
Proof: go test -json ./internal/api -count=1; go test ./internal/alerts/... ./internal/monitoring ./internal/ai/... ./internal/websocket ./internal/config ./pkg/metrics; go test ./internal/alerts/... ./pkg/metrics
rcourtman