Reduce metrics SQLite write amplification

docs/release-control/v6/internal/records/known-rc-issue-closure-for-ga-metrics-write-amplification-2026-05-03.md

@@ -60,3 +60,49 @@ The metrics write-amplification path reported in `#1124` has a v6 runtime fix,
 operator-facing controls, documentation, and subsystem-contract coverage. The
 `known-rc-issue-closure-for-ga` gate remains satisfied for the current v6
 release candidate.
+
+## 2026-07-24 Physical-Write Revalidation
+
+The earlier proof established rollup controls and bounded retention, but it did
+not attribute physical writes by SQLite object or quantify WAL/checkpoint
+amplification. Issue `#1124` therefore remained open after a v6.1.1 operator
+reported continued writes. A deterministic 2,197-resource estate
+(Docker containers, Proxmox guests, agents, Kubernetes pods, storage targets,
+and physical disks) now persists 393,630 samples over 30 simulated 10-second
+polls and reports logical payload, per-object `dbstat`, page-cache, WAL,
+checkpoint, process-write, restart, integrity, and retention counters.
+
+The v6.1.1 and pre-fix `main` store implementations were byte-identical. Their
+four-index schema wrote 279,063 WAL frames (1,149,739,592 bytes) for
+21,214,200 logical payload bytes. The retained main database was 106,991,616
+bytes, of which the overlapping `idx_metrics_lookup` and
+`idx_metrics_unique` trees each occupied 24,014,848 bytes. This identifies the
+amplification source directly: each metric insert maintained the table,
+`sqlite_sequence`, and four indexes before checkpointing the dirty pages back
+to the main file.
+
+The revalidated schema makes the lookup index unique and orders its columns as
+the metric identity, eliminating the second overlapping tree without changing
+query order or durability settings. On the same workload it wrote 187,616 WAL
+frames (772,977,952 bytes), a 32.8% frame/byte reduction, and retained
+82,948,096 bytes. Linux process I/O accounting measured WAL-phase writes
+falling from 1,155,710,976 to 774,930,432 bytes and explicit checkpoint writes
+from 106,983,424 to 82,939,904 bytes (32.1% fewer total bytes). With the
+production 4,000-page auto-checkpoint enabled, a traced 157,452-sample run
+reduced total process write bytes from 392,445,952 to 257,208,320 and `fsync`
+calls from 42 to 32.
+
+The migration keeps the legacy unique index authoritative until deferred
+startup maintenance transactionally swaps the indexes. Tests cover legacy
+duplicates, close/restart, a process exit between index drop and replacement,
+backup/restore from both schema generations, writes racing the migration, and
+reads remaining available on the four-connection WAL pool. Query-plan checks
+still require indexed searches. Concurrent read/write p95 remained 0.88 ms
+against a 5 ms SLO, and the 500-node/10-reader dashboard p95 remained 2.37 ms
+against a 30 ms SLO. Retention removed all 393,630 expired rows and incremental
+vacuum reduced the 82,948,096-byte database to 36,864 bytes with zero freelist
+pages in 2.57 seconds.
+
+The durable store remains in WAL mode with `synchronous=NORMAL`; no
+authoritative history moved to volatile memory. The fix is on `main` for the
+next release and is not part of v6.1.1.

docs/release-control/v6/internal/status.json

@@ -9206,21 +9206,7 @@
       ]
     }
   ],
-  "work_claims": [
-    {
-      "id": "issue-1124-metrics-write-amplification-lane-l10",
-      "agent_id": "issue-1124-metrics-write-amplification",
-      "summary": "Measure and reduce metrics.db physical write amplification for issue #1124 without weakening durability or bounded runtime behavior.",
-      "target_id": "v6-product-lane-expansion",
-      "claimed_at": "2026-07-24T08:05:22Z",
-      "heartbeat_at": "2026-07-24T08:05:22Z",
-      "expires_at": "2026-07-24T16:05:22Z",
-      "work_item": {
-        "kind": "lane",
-        "id": "L10"
-      }
-    }
-  ],
+  "work_claims": [],
   "open_decisions": [],
   "source_of_truth_file": "docs/release-control/v6/internal/SOURCE_OF_TRUTH.md",
   "resolved_decisions": [

docs/release-control/v6/internal/subsystems/performance-and-scalability.md

@@ -1201,6 +1201,23 @@ WAL checkpoints more aggressive again, reopens duplicate-write failures,
 removes the bounded rollup-window checkpointing, or removes the metrics DB
 path/cadence controls must re-prove the metrics-store hot path with the owned
 store tests rather than assuming the earlier vacuum fixes are sufficient.
+The metrics identity and single-series lookup must also share one unique
+`idx_metrics_lookup(resource_type, resource_id, metric_type, tier, timestamp)`
+B-tree. Reintroducing a second full-width identity index is a write-amplification
+regression even when retained `metrics.db` size looks small: every sample would
+dirty both trees in the WAL and every checkpoint would copy both sets of pages
+back into the main file. Legacy databases may keep their existing unique index
+authoritative while the replacement index is built, but that O(rows) migration
+must run on deferred startup maintenance and swap indexes in one SQLite
+transaction so constructor latency, concurrent reads, uniqueness, and crash
+recovery remain intact. The deterministic issue-1124 profile must continue to
+attribute writes by logical payload, table/index `dbstat` pages, page-cache
+writes/spills, WAL frames, checkpoint frames, process write bytes where the
+host exposes them, and traced sync calls rather than treating final file size
+as a proxy. Its checked invariant persists 157,452 samples across 2,197 mixed
+provider resources and caps the no-auto-checkpoint workload at 40,000 WAL
+frames; the former four-index schema produces 50,516 frames while the
+consolidated schema produces 35,030.
 Retention must also return freed SQLite pages to the OS
 proportionally to the current freelist, bounded per cycle, and on every
 retention cycle rather than only when that cycle deleted rows: a fixed small