fix(mesh): persist PilotMetrics counters across central restart (#1078)

The 12 PilotMetrics counters (proxy_dials_failed, proxy_bridges_total,
proxy_idle_closes, the pilot-tunnel counters, and the peer-to-central
callback counters) live in a singleton holding scalar fields, so process
restart wipes them. Operators investigating rotation- or dial-failure
trends across a central restart lost the aggregate metric.

Persist the snapshot under a pilot_metrics_counters JSON row on the
existing system_state KV table. Hydrate via PilotMetrics.load() in
bootstrap/startup.ts before any service that increments runs; final-flush
in bootstrap/shutdown.ts before the SQLite handle closes. Between those
boundaries a buffered flush mirrors the audit-log pattern: every 1 s or
every 100 increments, whichever fires first. Hot increment path stays a
single property write plus a pending-counter check.

Defensive parse on the row: object check + numeric filter, so an operator
hand-edit or schema drift across releases degrades to zero state with a
warn rather than crashing boot. Schema additions back-fill missing keys
with zero so a release that adds a counter does not need a migration.

Closes F-R-4 from the mesh tracker.
This commit is contained in:
Anso
2026-05-17 01:55:17 -04:00
committed by GitHub
parent 9aaa8573c0
commit 6b728599c4
5 changed files with 369 additions and 23 deletions
+35
View File
@@ -605,6 +605,8 @@ export interface ScanSummary {
const AUDIT_LOG_FLUSH_INTERVAL_MS = 1_000;
const AUDIT_LOG_FLUSH_THRESHOLD = 100;
export const PILOT_METRICS_COUNTERS_KEY = 'pilot_metrics_counters';
export class DatabaseService {
private static instance: DatabaseService;
private db: Database.Database;
@@ -1775,6 +1777,39 @@ export class DatabaseService {
this.db.prepare('INSERT OR REPLACE INTO system_state (key, value) VALUES (?, ?)').run(key, value);
}
/**
* Persisted PilotMetrics counters. Returns the parsed JSON object (a
* numeric record keyed by counter name) or null when the row is missing
* or unparseable. Callers (PilotMetrics.load) handle per-field defaulting
* so a missing counter in the persisted blob does not break a new release
* that added the counter.
*
* This is the first JSON blob stored in `system_state`; mirror this
* parse/validate shape (object check + numeric filter) for any future
* JSON-shaped system_state row so an operator-edited row cannot crash
* boot.
*/
public getPilotMetricsCounters(): Record<string, number> | null {
const raw = this.getSystemState(PILOT_METRICS_COUNTERS_KEY);
if (raw === null) return null;
try {
const parsed = JSON.parse(raw) as unknown;
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) return null;
const out: Record<string, number> = {};
for (const [k, v] of Object.entries(parsed as Record<string, unknown>)) {
if (typeof v === 'number' && Number.isFinite(v)) out[k] = v;
}
return out;
} catch (err) {
console.warn('[DatabaseService] pilot_metrics_counters JSON parse failed:', (err as Error).message);
return null;
}
}
public setPilotMetricsCounters(counters: Record<string, number>): void {
this.setSystemState(PILOT_METRICS_COUNTERS_KEY, JSON.stringify(counters));
}
/**
* Run `fn` inside a single SQLite transaction. better-sqlite3 promotes a
* nested call to a SAVEPOINT, so callers can compose this with methods