feat: health-gated updates and rollback readiness (#1354)

* feat: classify stack deploy and update failures with suggested next actions

Failed deploy and update responses now carry a failure classification
(cause category, headline, and suggested next step) derived from the
compose error output. The recovery panel and chip render the
classification and include it in copied diagnostics, and gateway-style
failures surface as a node-unreachable cause.

* feat: add update and rollback readiness reports for stacks

Before a manual update, Sencho now shows an advisory readiness verdict
computed from the stored preflight result, open drift findings, live
container health, the pending image change, the rollback backup slot,
and node disk headroom. The Stack Dossier gains a rollback readiness
section that states what a rollback can restore and explicitly
discloses that volume and bind-mounted data are not covered. Toolbar
and sidebar updates now share one update path, and admins can create a
fleet snapshot from the readiness dialog before updating. Nodes that do
not advertise the capability keep the direct update flow.

* feat: observe stack health after updates with a post-deploy health gate

After a deploy or update succeeds, Sencho now watches the stack for a
configurable observation window and records a passed, failed, or
unknown verdict: containers must stay running, healthchecks must report
healthy, and restart loops or disappearing containers fail the gate.
The deploy panel shows the observation live and holds off auto-closing
until the verdict lands, a failed gate surfaces the existing recovery
actions including rollback, and the stack timeline records update
started and gate verdict events. Scheduled, webhook, bulk, and
git-source updates are gated the same way; rollbacks and installs are
deliberately not. The gate is observational only and can be tuned or
disabled per node under host alert settings.

* docs: document health-gated updates and rollback readiness

New operator page covering the update readiness dialog, the post-update
health gate and its settings, the rollback readiness disclosure, and
classified failures, with cross-links from the atomic deployments and
deploy progress pages. The API reference gains the readiness and
health-gate endpoints, the healthGateId success field, and the failure
classification schema on deploy and update error responses.

* feat: withhold the success verdict while the health gate observes

An update used to show a green Succeeded that a failed health gate then
contradicted moments later. The deploy modal now reports Verifying
health while the gate observes, shows success only when the gate
passes, and makes a failed or unknown gate the headline result; success
toasts soften to a verifying message while a gate runs. The mobile
recovery card groups its actions behind one bottom-right Take action
menu so it stays compact on a phone, with the classified cause still
visible on the card. A successful image update now also counts as the
last known-good marker in rollback readiness, and the docs gain
screenshots of the readiness dialog, gate states, dossier section, and
settings.

* fix: harden log format strings and the env existence path check

Log calls that interpolated the stack name into the console format
string now use constant format strings with placeholder arguments, and
envExists validates path containment inline at its filesystem access,
matching the established patterns used elsewhere in the same files.

* test: adapt deploy modal success specs to the post-deploy health gate

The deploy feedback modal now withholds its success verdict while the
health gate observes the new containers, showing "Verifying health"
until the gate passes. The two success-path E2E tests waited for
"Succeeded" within the gate's 90s default window and timed out.

Shorten the observation window to the 15s minimum for these tests via
the settings API, assert the verify-then-succeed sequence the modal
actually renders, and restore the default window afterward so the test
value does not leak into later runs.

* fix: serialize health gate polling and harden gate observation

Address race conditions in the post-update health gate found in review.

Backend: the gate poller used setInterval, so a Docker observe slower
than the 5s tick could overlap the next poll and corrupt the restart and
missing-container accounting, and a wedged socket could leave a poll
pending forever. Polling is now single-flight: each cycle self-schedules
the next only after it settles, and the observe is bounded by an 8s
timeout so a hung probe counts as a poll error and resolves the gate
unknown after three in a row.

Frontend: the gate poller could overlap requests, letting a slow earlier
"observing" response overwrite an already-applied terminal verdict. It is
now single-flight with a terminal latch, so a late response can never
roll the UI back from passed or failed.

Also reject a non-digit nodeId on the snapshot coverage route instead of
letting parseInt coerce it, document that turning off the deploy progress
panel opts out of the live gate UI while the gate still runs server-side,
and add gate-coverage tests for the webhook, git source, and auto-update
apply paths plus the new single-flight, observe-timeout, and recovery
cases.
This commit is contained in:
Anso
2026-06-11 00:26:26 -04:00
committed by GitHub
parent 739bbf990e
commit 38aabe7064
66 changed files with 5076 additions and 79 deletions
@@ -36,6 +36,7 @@ export const CAPABILITIES = [
'self-update',
'vulnerability-scanning',
'compose-doctor',
'update-guard',
] as const;
export type Capability = (typeof CAPABILITIES)[number];
+103
View File
@@ -106,6 +106,23 @@ export interface PreflightRunRow {
created_by: string | null;
}
/** One post-update health gate observation run. */
export interface HealthGateRunRow {
id: string;
node_id: number;
stack_name: string;
/** Named trigger_action because TRIGGER is reserved in SQLite. */
trigger_action: 'update' | 'deploy';
status: 'observing' | 'passed' | 'failed' | 'unknown';
reason: string | null;
window_seconds: number;
/** JSON array of per-container end states for display. */
containers_json: string;
started_at: number;
ended_at: number | null;
created_by: string | null;
}
/** One finding within a stored preflight run. Never carries an environment value. */
export interface PreflightFindingRow {
id: string;
@@ -914,6 +931,7 @@ export class DatabaseService {
);
CREATE INDEX IF NOT EXISTS idx_snapshot_files_snapshot ON fleet_snapshot_files(snapshot_id);
CREATE INDEX IF NOT EXISTS idx_snapshot_files_node_stack ON fleet_snapshot_files(node_id, stack_name);
CREATE TABLE IF NOT EXISTS audit_log (
id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -1272,6 +1290,22 @@ export class DatabaseService {
CREATE INDEX IF NOT EXISTS idx_preflight_findings_run
ON preflight_findings(run_id);
CREATE TABLE IF NOT EXISTS health_gate_runs (
id TEXT PRIMARY KEY,
node_id INTEGER NOT NULL,
stack_name TEXT NOT NULL,
trigger_action TEXT NOT NULL CHECK (trigger_action IN ('update','deploy')),
status TEXT NOT NULL CHECK (status IN ('observing','passed','failed','unknown')),
reason TEXT,
window_seconds INTEGER NOT NULL,
containers_json TEXT NOT NULL DEFAULT '[]',
started_at INTEGER NOT NULL,
ended_at INTEGER,
created_by TEXT
);
CREATE INDEX IF NOT EXISTS idx_health_gate_runs_node_stack
ON health_gate_runs(node_id, stack_name, started_at);
CREATE TABLE IF NOT EXISTS secrets (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL UNIQUE,
@@ -1400,6 +1434,8 @@ export class DatabaseService {
stmt.run('mesh_auto_recreate', '0');
stmt.run('prune_on_update', '1');
stmt.run('reclaim_hero', '1');
stmt.run('health_gate_enabled', '1');
stmt.run('health_gate_window_seconds', '90');
// Seed the default local node if none exists
const nodeCount = (this.db.prepare('SELECT COUNT(*) as count FROM nodes').get() as any)?.count || 0;
@@ -2301,6 +2337,61 @@ export class DatabaseService {
).all(runId) as PreflightFindingRow[];
}
// --- Health Gate Runs ---
public insertHealthGateRun(run: HealthGateRunRow): void {
this.db.prepare(
`INSERT INTO health_gate_runs
(id, node_id, stack_name, trigger_action, status, reason, window_seconds, containers_json, started_at, ended_at, created_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`
).run(
run.id, run.node_id, run.stack_name, run.trigger_action, run.status, run.reason,
run.window_seconds, run.containers_json, run.started_at, run.ended_at, run.created_by,
);
// Bounded history: keep only the 10 most recent runs per stack.
this.db.prepare(
`DELETE FROM health_gate_runs
WHERE node_id = ? AND stack_name = ?
AND id NOT IN (
SELECT id FROM health_gate_runs
WHERE node_id = ? AND stack_name = ?
ORDER BY started_at DESC, id DESC LIMIT 10
)`
).run(run.node_id, run.stack_name, run.node_id, run.stack_name);
}
public finalizeHealthGateRun(
id: string,
status: 'passed' | 'failed' | 'unknown',
reason: string | null,
endedAt: number,
containersJson: string,
): void {
this.db.prepare(
'UPDATE health_gate_runs SET status = ?, reason = ?, ended_at = ?, containers_json = ? WHERE id = ?'
).run(status, reason, endedAt, containersJson, id);
}
public getHealthGateRun(nodeId: number, stackName: string, id: string): HealthGateRunRow | undefined {
return this.db.prepare(
'SELECT * FROM health_gate_runs WHERE node_id = ? AND stack_name = ? AND id = ?'
).get(nodeId, stackName, id) as HealthGateRunRow | undefined;
}
public getLatestHealthGateRun(nodeId: number, stackName: string): HealthGateRunRow | undefined {
return this.db.prepare(
'SELECT * FROM health_gate_runs WHERE node_id = ? AND stack_name = ? ORDER BY started_at DESC, id DESC LIMIT 1'
).get(nodeId, stackName) as HealthGateRunRow | undefined;
}
/** Finalize runs left observing by a previous process (startup sweep). */
public markInterruptedHealthGateRuns(reason: string, endedAt: number): number {
const result = this.db.prepare(
"UPDATE health_gate_runs SET status = 'unknown', reason = ?, ended_at = ? WHERE status = 'observing'"
).run(reason, endedAt);
return result.changes;
}
// --- Notification History ---
private mapNotificationRow(row: any): NotificationHistory {
@@ -2641,6 +2732,7 @@ export class DatabaseService {
this.db.prepare('DELETE FROM stack_drift_findings WHERE node_id = ?').run(id);
this.db.prepare('DELETE FROM preflight_findings WHERE run_id IN (SELECT id FROM preflight_runs WHERE node_id = ?)').run(id);
this.db.prepare('DELETE FROM preflight_runs WHERE node_id = ?').run(id);
this.db.prepare('DELETE FROM health_gate_runs WHERE node_id = ?').run(id);
this.db.prepare('UPDATE blueprints SET pinned_node_id = NULL WHERE pinned_node_id = ?').run(id);
this.deleteRoleAssignmentsByResource('node', String(id));
this.db.prepare('DELETE FROM fleet_sync_status WHERE node_id = ?').run(id);
@@ -3267,6 +3359,17 @@ export class DatabaseService {
return rows.map(row => ({ ...row, content: crypto.decrypt(row.content) }));
}
/** Created-at of the most recent fleet snapshot covering a stack, or null. */
public getLatestSnapshotTimestampFor(nodeId: number, stackName: string): number | null {
const row = this.db.prepare(
`SELECT MAX(s.created_at) AS latest
FROM fleet_snapshots s
JOIN fleet_snapshot_files f ON f.snapshot_id = s.id
WHERE f.node_id = ? AND f.stack_name = ?`
).get(nodeId, stackName) as { latest: number | null } | undefined;
return row?.latest ?? null;
}
public deleteSnapshot(id: number): void {
this.db.prepare('DELETE FROM fleet_snapshots WHERE id = ?').run(id);
}
+52 -1
View File
@@ -361,8 +361,17 @@ export class FileSystemService {
async envExists(stackName: string): Promise<boolean> {
const stackDir = this.resolveStackDir(stackName);
// Canonical js/path-injection barrier inline with the access sink, the
// same pattern backupStackFiles/restoreStackFiles use: stackName is
// already validated by resolveStackDir above, but static analysis only
// credits the containment check when it sits at the sink itself.
const baseResolved = path.resolve(this.baseDir);
const target = path.resolve(stackDir, '.env');
if (!target.startsWith(baseResolved + path.sep)) {
return false;
}
try {
await fsPromises.access(path.join(stackDir, '.env'));
await fsPromises.access(target);
return true;
} catch {
return false;
@@ -920,6 +929,48 @@ export class FileSystemService {
};
}
/**
* Names-only summary of the backup slot's env coverage for rollback
* readiness: whether a backup exists, whether it contains a .env, and the
* variable names defined in it. Values never leave this method.
*/
async getBackupEnvSummary(stackName: string): Promise<{ exists: boolean; envPresent: boolean; keys: string[] }> {
if (!isValidStackName(stackName)) {
return { exists: false, envPresent: false, keys: [] };
}
// Canonical js/path-injection barrier inline with the read sink, mirroring
// backupStackFiles/restoreStackFiles.
const backupRoot = path.resolve(getBackupBaseDir());
const backupDir = path.resolve(backupRoot, String(this.nodeId), stackName);
if (!backupDir.startsWith(backupRoot + path.sep)) {
throw Object.assign(new Error('Path escapes backup directory'), { code: 'INVALID_PATH' });
}
try {
await fsPromises.access(backupDir);
} catch (e: unknown) {
// Only a missing slot may report "no backup"; an unreadable one (EACCES
// on a root-created dir) must propagate so callers degrade to unknown
// instead of falsely promising the next update will create one.
if ((e as NodeJS.ErrnoException)?.code !== 'ENOENT') throw e;
return { exists: false, envPresent: false, keys: [] };
}
try {
const content = await fsPromises.readFile(path.join(backupDir, '.env'), 'utf-8');
const keys: string[] = [];
for (const line of content.split(/\r?\n/)) {
const match = /^\s*([A-Za-z_][A-Za-z0-9_]*)=/.exec(line);
if (match) keys.push(match[1]);
}
return { exists: true, envPresent: true, keys };
} catch (e: unknown) {
// ENOENT means the backup genuinely has no env file. Anything else
// (EACCES, EISDIR) must propagate: reporting it as "no env in backup"
// would falsely claim a rollback cannot restore env changes.
if ((e as NodeJS.ErrnoException)?.code !== 'ENOENT') throw e;
return { exists: true, envPresent: false, keys: [] };
}
}
async getBackupInfo(stackName: string): Promise<{ exists: boolean; timestamp: number | null }> {
const backupDir = this.getBackupDir(stackName);
try {
+2
View File
@@ -8,6 +8,7 @@ import { CryptoService } from './CryptoService';
import { DatabaseService, type StackGitSource, type GitSourceAuthType } from './DatabaseService';
import { FileSystemService } from './FileSystemService';
import { ComposeService } from './ComposeService';
import { HealthGateService } from './HealthGateService';
import { NodeRegistry } from './NodeRegistry';
import { assertPolicyGateAllows, buildSystemPolicyGateOptions } from '../helpers/policyGate';
import { isDebugEnabled } from '../utils/debug';
@@ -1062,6 +1063,7 @@ export class GitSourceService {
}),
);
await ComposeService.getInstance().deployStack(stackName);
HealthGateService.getInstance().begin(nodeId, stackName, 'deploy', 'system:git-source');
console.log(`[GitSource] Applied and deployed ${stackName} at ${commitSha.slice(0, 7)}`);
return { applied: true, deployed: true };
} catch (e) {
+472
View File
@@ -0,0 +1,472 @@
import { randomUUID } from 'crypto';
import DockerController from './DockerController';
import { DatabaseService, type HealthGateRunRow } from './DatabaseService';
import { sanitizeForLog } from '../utils/safeLog';
import { getErrorMessage } from '../utils/errors';
import { withTimeout } from '../utils/withTimeout';
import type { HealthGateContainer, HealthGateReport } from './updateGuard/types';
const POLL_INTERVAL_MS = 5_000;
// Per-observe ceiling so a hung Docker socket cannot leave a poll pending
// forever. Above POLL_INTERVAL_MS so a slow-but-live probe is not cut short; a
// timeout counts as a poll error and three in a row resolve the gate unknown.
const OBSERVE_TIMEOUT_MS = 8_000;
// A stack whose containers never appear gives up after this long.
const EMPTY_GRACE_MS = 15_000;
const DEFAULT_WINDOW_SECONDS = 90;
const MIN_WINDOW_SECONDS = 15;
const MAX_WINDOW_SECONDS = 600;
// Backstop against runaway concurrency (a burst of webhook or scheduled
// updates). Gates past the cap finalize immediately as unknown.
const MAX_CONCURRENT_GATES = 25;
interface ObservedContainer {
id: string;
name: string;
startedAt: string | null;
/** Docker's raw RestartCount at observation time. */
restartCount: number;
/** Gate-maintained restart tally since the baseline; 0 on a fresh snapshot,
* set by poll()'s accounting pass, at most one increment per poll. */
restarts: number;
state: string;
health: string | null;
}
interface ActiveGate {
runId: string;
nodeId: number;
stackName: string;
windowSeconds: number;
startedAt: number;
/** Self-scheduling poll timer, armed only between settled poll cycles. */
timer: ReturnType<typeof setTimeout> | null;
/** Expected container set, keyed by name; null until the first non-empty poll. */
expected: Map<string, ObservedContainer> | null;
consecutivePollErrors: number;
/** Names missing on the previous poll (two consecutive misses fail the gate). */
missingLastPoll: Set<string>;
/** Names in `restarting` state on the previous poll. */
restartingLastPoll: Set<string>;
/**
* Set by finalize so a poll that was mid-await when this gate was superseded
* or stopped can never overwrite the terminal verdict with a stale one.
*/
finalized: boolean;
}
/**
* Post-update health gate: after a deploy/update succeeds, observe the
* stack's containers for a configurable window and record a passed / failed /
* unknown verdict plus an activity timeline event. Purely observational: it
* never restarts, heals, or rolls anything back. AutoHeal needs no special
* handling: the unhealthy or exited state that triggers it is seen by the
* gate's own polls and fails the gate, and repeated restarts trip the
* restart-loop check.
*
* begin() is the single shared post-success hook for every gated deploy and
* update path; excluded paths (rollback, installs, reconciler loops) simply
* never call it.
*/
export class HealthGateService {
private static instance: HealthGateService;
private readonly active = new Map<string, ActiveGate>();
private started = false;
public static getInstance(): HealthGateService {
if (!HealthGateService.instance) {
HealthGateService.instance = new HealthGateService();
}
return HealthGateService.instance;
}
/** Sweep runs left observing by a previous process, then accept begin() calls. */
public start(): void {
this.started = true;
try {
const swept = DatabaseService.getInstance().markInterruptedHealthGateRuns(
'Sencho restarted during observation', Date.now(),
);
if (swept > 0) {
console.log(`[HealthGate] Marked ${swept} interrupted observation(s) as unknown`);
}
} catch (error) {
console.error('[HealthGate] Startup sweep failed:', getErrorMessage(error, 'unknown'));
}
}
/** Clear every poll timer and finalize in-flight gates as unknown. */
public stop(): void {
this.started = false;
for (const gate of [...this.active.values()]) {
this.finalize(gate, 'unknown', 'shutdown during observation', []);
}
}
/**
* Begin observing a stack after a successful deploy/update. Returns the gate
* run id for response correlation, or null when gating is disabled, the
* service is not started, or recording fails internally. Inserts the row
* synchronously so the caller can include the id in its response;
* observation then runs on a timer. Never throws.
*
* Also records the `update_started` activity event for update triggers, so
* every gated update path gets the timeline marker even when the gate
* itself is disabled.
*/
public begin(
nodeId: number,
stackName: string,
trigger: 'update' | 'deploy',
actor: string | null,
): string | null {
// Refuses work outside the start()/stop() lifecycle so a late call during
// shutdown cannot leave a dangling poll timer.
if (!this.started) return null;
try {
const db = DatabaseService.getInstance();
const settings = this.readSettings();
if (trigger === 'update') {
this.recordActivity(nodeId, stackName, 'info', 'update_started', `${stackName} update started`, actor);
}
if (!settings.enabled) return null;
// A newer operation supersedes an in-flight gate for the same stack.
const key = `${nodeId}:${stackName}`;
const existing = this.active.get(key);
if (existing) {
this.finalize(existing, 'unknown', 'superseded by a newer update', []);
}
const runId = randomUUID();
const startedAt = Date.now();
const row: HealthGateRunRow = {
id: runId,
node_id: nodeId,
stack_name: stackName,
trigger_action: trigger,
status: 'observing',
reason: null,
window_seconds: settings.windowSeconds,
containers_json: '[]',
started_at: startedAt,
ended_at: null,
created_by: actor,
};
if (this.active.size >= MAX_CONCURRENT_GATES) {
db.insertHealthGateRun({ ...row, status: 'unknown', reason: 'too many concurrent observations', ended_at: startedAt });
return runId;
}
db.insertHealthGateRun(row);
const gate: ActiveGate = {
runId,
nodeId,
stackName,
windowSeconds: settings.windowSeconds,
startedAt,
timer: null,
expected: null,
consecutivePollErrors: 0,
missingLastPoll: new Set(),
restartingLastPoll: new Set(),
finalized: false,
};
this.active.set(key, gate);
this.scheduleNextPoll(gate);
return runId;
} catch (error) {
// The gate is an observer; its failure must never fail the operation.
console.error(
'[HealthGate] begin (%s) failed for %s on node %d:',
trigger, sanitizeForLog(stackName), nodeId, error,
);
return null;
}
}
/** A specific run by id, the latest run, or the never-run sentinel. */
public getReport(nodeId: number, stackName: string, gateId?: string): HealthGateReport {
const db = DatabaseService.getInstance();
const row = gateId
? db.getHealthGateRun(nodeId, stackName, gateId)
: db.getLatestHealthGateRun(nodeId, stackName);
if (!row) {
return {
stack: stackName, id: null, status: 'never-run', trigger: null, reason: null,
windowSeconds: null, startedAt: null, endedAt: null, containers: [],
};
}
let containers: HealthGateContainer[] = [];
try {
const parsed: unknown = JSON.parse(row.containers_json);
if (Array.isArray(parsed)) containers = parsed as HealthGateContainer[];
} catch {
// A corrupt blob only loses the per-container detail, never the verdict.
console.warn('[HealthGate] Unreadable containers_json for run %s', sanitizeForLog(row.id));
}
return {
stack: stackName,
id: row.id,
status: row.status,
trigger: row.trigger_action,
reason: row.reason,
windowSeconds: row.window_seconds,
startedAt: row.started_at,
endedAt: row.ended_at,
containers,
};
}
/**
* Orchestrate one poll cycle and arm the next. Polls are single-flight: the
* next timer is scheduled only after this cycle fully settles (the finally
* below), so a slow or timed-out observe can never overlap the following poll
* or corrupt the restart/missing accounting that assumes one poll at a time.
*/
private async poll(gate: ActiveGate): Promise<void> {
const key = `${gate.nodeId}:${gate.stackName}`;
// A late timer fire after supersede, stop, or finalize must do nothing.
if (gate.finalized || this.active.get(key) !== gate) return;
try {
await this.runPollCycle(gate, key);
} finally {
if (!gate.finalized && this.active.get(key) === gate) {
this.scheduleNextPoll(gate);
}
}
}
/** Arm the next poll. No-op once the gate is finalized. */
private scheduleNextPoll(gate: ActiveGate): void {
if (gate.finalized) return;
gate.timer = setTimeout(() => { void this.poll(gate); }, POLL_INTERVAL_MS);
}
private async runPollCycle(gate: ActiveGate, key: string): Promise<void> {
let observed: ObservedContainer[];
try {
observed = await withTimeout(
this.observeContainers(gate), OBSERVE_TIMEOUT_MS, 'health gate observe',
);
} catch (error) {
gate.consecutivePollErrors += 1;
console.warn(
'[HealthGate] poll error %d for %s:',
gate.consecutivePollErrors, sanitizeForLog(gate.stackName), getErrorMessage(error, 'unknown'),
);
if (gate.consecutivePollErrors >= 3) {
this.finalize(gate, 'unknown', 'Docker became unreachable during observation', []);
}
return;
}
// The await above can straddle a supersede or stop; never act on a gate
// that was finalized mid-flight.
if (gate.finalized || this.active.get(key) !== gate) return;
gate.consecutivePollErrors = 0;
const elapsedMs = Date.now() - gate.startedAt;
if (gate.expected === null) {
if (observed.length > 0) {
gate.expected = new Map(observed.map(c => [c.name, c]));
} else if (elapsedMs >= EMPTY_GRACE_MS) {
this.finalize(gate, 'unknown', 'no containers found to observe', []);
}
return;
}
const byName = new Map(observed.map(c => [c.name, c]));
// First pass: restart accounting for every expected container still
// present, so the summary below reflects the tallies the checks act on. A
// restart counts when the container was replaced (new id), relaunched
// (StartedAt moved), or Docker bumped its RestartCount; at most one
// restart is tallied per poll regardless of how many occurred in the gap.
for (const [name, baseline] of gate.expected) {
const current = byName.get(name);
if (!current) continue;
const restarted =
current.id !== baseline.id ||
current.restartCount > baseline.restartCount ||
(current.startedAt !== null && baseline.startedAt !== null && current.startedAt !== baseline.startedAt);
current.restarts = baseline.restarts + (restarted ? 1 : 0);
}
const summary = this.summarize(gate.expected, byName);
// Second pass: fail fast on a clearly bad state.
for (const [name, baseline] of gate.expected) {
const current = byName.get(name);
if (!current) {
if (gate.missingLastPoll.has(name)) {
this.finalize(gate, 'failed', `container ${name} disappeared during observation`, summary);
return;
}
gate.missingLastPoll.add(name);
continue;
}
gate.missingLastPoll.delete(name);
if (current.state === 'exited' && baseline.restarts === current.restarts) {
// An exit with no restart attempt is terminal for the window.
this.finalize(gate, 'failed', `container ${name} exited during observation`, summary);
return;
}
if (current.health === 'unhealthy') {
this.finalize(gate, 'failed', `container ${name} reported unhealthy`, summary);
return;
}
if (current.restarts >= 2) {
this.finalize(gate, 'failed', `container ${name} is restart looping`, summary);
return;
}
if (current.state === 'restarting') {
if (gate.restartingLastPoll.has(name)) {
this.finalize(gate, 'failed', `container ${name} is stuck restarting`, summary);
return;
}
gate.restartingLastPoll.add(name);
} else {
gate.restartingLastPoll.delete(name);
}
// Carry the running restart tally forward as the new baseline.
gate.expected.set(name, current);
}
if (elapsedMs < gate.windowSeconds * 1000) return;
// Window complete: pass requires everything running and healthy wherever a
// healthcheck exists. A health state still 'starting' is not a pass.
const stillStarting = observed.filter(c => c.health === 'starting');
if (stillStarting.length > 0) {
this.finalize(gate, 'unknown', 'a healthcheck was still starting when the observation window ended', summary);
return;
}
const notRunning = [...gate.expected.keys()].filter(name => byName.get(name)?.state !== 'running');
if (notRunning.length > 0) {
this.finalize(gate, 'failed', `not running at the end of the window: ${notRunning.join(', ')}`, summary);
return;
}
this.finalize(gate, 'passed', null, summary);
}
private async observeContainers(gate: ActiveGate): Promise<ObservedContainer[]> {
const docker = DockerController.getInstance(gate.nodeId).getDocker();
const listed = await docker.listContainers({
all: true,
filters: { label: [`com.docker.compose.project=${gate.stackName}`] },
});
const observed = await Promise.all(
listed.map(async (info): Promise<ObservedContainer | null> => {
try {
const inspect = await docker.getContainer(info.Id).inspect();
return {
id: info.Id,
name: info.Names?.[0]?.replace(/^\//, '') ?? info.Id.slice(0, 12),
startedAt: inspect.State?.StartedAt ?? null,
restartCount: typeof inspect.RestartCount === 'number' ? inspect.RestartCount : 0,
restarts: 0,
state: inspect.State?.Status ?? info.State ?? 'unknown',
health: inspect.State?.Health?.Status ?? null,
};
} catch (e: unknown) {
// Removed between list and inspect; the missing-container logic will
// see its absence on this or the next poll.
if ((e as { statusCode?: number })?.statusCode === 404) return null;
throw e;
}
}),
);
return observed.filter((c): c is ObservedContainer => c !== null);
}
private summarize(
expected: Map<string, ObservedContainer>,
current: Map<string, ObservedContainer>,
): HealthGateContainer[] {
return [...expected.values()].map(baseline => {
const now = current.get(baseline.name);
return {
name: baseline.name,
state: now?.state ?? 'missing',
health: now?.health ?? null,
restarts: now?.restarts ?? baseline.restarts,
};
});
}
private finalize(
gate: ActiveGate,
status: 'passed' | 'failed' | 'unknown',
reason: string | null,
containers: HealthGateContainer[],
): void {
if (gate.finalized) return;
gate.finalized = true;
if (gate.timer) clearTimeout(gate.timer);
const key = `${gate.nodeId}:${gate.stackName}`;
if (this.active.get(key) === gate) this.active.delete(key);
try {
DatabaseService.getInstance().finalizeHealthGateRun(
gate.runId, status, reason, Date.now(), JSON.stringify(containers),
);
} catch (error) {
// The verdict is lost from the DB (the startup sweep will later rewrite
// the row as unknown), so log everything needed to reconstruct it.
console.error(
'[HealthGate] Failed to persist verdict %s (%s) for run %s, stack %s:',
status, sanitizeForLog(reason ?? 'no reason'), gate.runId, sanitizeForLog(gate.stackName),
getErrorMessage(error, 'unknown'),
);
}
if (status === 'passed') {
this.recordActivity(gate.nodeId, gate.stackName, 'info', 'health_gate_passed',
`${gate.stackName} health gate passed after ${gate.windowSeconds}s`, 'system');
} else if (status === 'failed') {
this.recordActivity(gate.nodeId, gate.stackName, 'warning', 'health_gate_failed',
`${gate.stackName} health gate failed: ${reason ?? 'unknown reason'}`, 'system');
}
}
private recordActivity(
nodeId: number,
stackName: string,
level: 'info' | 'warning',
category: 'update_started' | 'health_gate_passed' | 'health_gate_failed',
message: string,
actor: string | null,
): void {
try {
DatabaseService.getInstance().addNotificationHistory(nodeId, {
level,
category,
message,
timestamp: Date.now(),
stack_name: stackName,
actor_username: actor,
});
} catch (error) {
console.warn('[HealthGate] Failed to record activity for %s:', sanitizeForLog(stackName), getErrorMessage(error, 'unknown'));
}
}
private readSettings(): { enabled: boolean; windowSeconds: number } {
try {
const settings = DatabaseService.getInstance().getGlobalSettings();
const windowRaw = parseInt(settings['health_gate_window_seconds'] ?? '', 10);
const windowSeconds = Number.isFinite(windowRaw)
? Math.min(MAX_WINDOW_SECONDS, Math.max(MIN_WINDOW_SECONDS, windowRaw))
: DEFAULT_WINDOW_SECONDS;
return { enabled: settings['health_gate_enabled'] !== '0', windowSeconds };
} catch (error) {
// Safe default: observing is non-destructive, so a settings read
// failure keeps the gate on with the default window.
console.warn('[HealthGate] Settings read failed; using defaults:', getErrorMessage(error, 'unknown'));
return { enabled: true, windowSeconds: DEFAULT_WINDOW_SECONDS };
}
}
}
@@ -27,6 +27,11 @@ export type NotificationCategory =
// from ALL_NOTIFICATION_CATEGORIES (the routable-category whitelist) below.
| 'drift_detected'
| 'drift_resolved'
// Update lifecycle markers from the post-update health gate. History-only
// for the same reason as the drift pair above.
| 'update_started'
| 'health_gate_passed'
| 'health_gate_failed'
| 'system';
export const ALL_NOTIFICATION_CATEGORIES: readonly NotificationCategory[] = [
+2
View File
@@ -6,6 +6,7 @@ import { PROXY_TIER_HEADER } from './license-headers';
import DockerController from './DockerController';
import { ComposeService } from './ComposeService';
import { FileSystemService } from './FileSystemService';
import { HealthGateService } from './HealthGateService';
import { ImageUpdateService } from './ImageUpdateService';
import type { ImageCheckResult } from './ImageUpdateService';
import { isDebugEnabled } from '../utils/debug';
@@ -861,6 +862,7 @@ export class SchedulerService {
const atomic = true;
await compose.updateStack(stackName, undefined, atomic);
db.clearStackUpdateStatus(nodeId, stackName);
HealthGateService.getInstance().begin(nodeId, stackName, 'update', 'system:scheduler');
this.safeDispatch(
'info',
+175
View File
@@ -0,0 +1,175 @@
import si from 'systeminformation';
import DockerController from './DockerController';
import { DatabaseService } from './DatabaseService';
import { FileSystemService } from './FileSystemService';
import { ComposeDoctorService } from './ComposeDoctorService';
import { UpdatePreviewService } from './UpdatePreviewService';
import { withTimeout } from '../utils/withTimeout';
import { getErrorMessage } from '../utils/errors';
import { sanitizeForLog } from '../utils/safeLog';
import {
aggregateRollbackOverall,
aggregateVerdict,
backupSlotSignal,
buildRollbackItems,
containersSignal,
diskSignal,
driftSignal,
healthchecksSignal,
preflightSignal,
updatePreviewSignal,
type Errored,
} from './updateGuard/readiness';
import type { ContainerProbe, RollbackReadinessReport, UpdateReadinessReport } from './updateGuard/types';
// Bound on the network-and-socket-backed inputs (container probe, update
// preview, disk stats) so a hung registry or Docker socket cannot stall the
// report past the dialog's own fetch timeout; the remaining inputs are local
// DB/file reads. A timed-out input degrades to its 'unknown' signal instead
// of failing the report.
const INPUT_TIMEOUT_MS = 3_000;
/**
* Computes update readiness and rollback readiness for a stack, on demand,
* from existing per-feature stores (preflight runs, drift findings, the atomic
* backup slot, the update preview, live Docker state). Derived data only;
* nothing here is persisted.
*/
export class UpdateGuardService {
private static instance: UpdateGuardService;
public static getInstance(): UpdateGuardService {
if (!UpdateGuardService.instance) {
UpdateGuardService.instance = new UpdateGuardService();
}
return UpdateGuardService.instance;
}
/**
* Probe the stack's containers via the compose project label, normalized for
* the pure scoring functions. Throws on Docker errors; callers map that to
* the 'error' sentinel.
*/
async probeContainers(nodeId: number, stackName: string): Promise<ContainerProbe[]> {
const docker = DockerController.getInstance(nodeId).getDocker();
const listed = await docker.listContainers({
all: true,
filters: { label: [`com.docker.compose.project=${stackName}`] },
});
const probes = await Promise.all(
listed.map(async (info): Promise<ContainerProbe | null> => {
const name = info.Names?.[0]?.replace(/^\//, '') ?? info.Id.slice(0, 12);
let inspect: Awaited<ReturnType<ReturnType<typeof docker.getContainer>['inspect']>>;
try {
inspect = await docker.getContainer(info.Id).inspect();
} catch (e: unknown) {
// A container removed between list and inspect (auto-heal or update
// churn) should not collapse the whole probe; skip just that one.
if ((e as { statusCode?: number })?.statusCode === 404) return null;
throw e;
}
const mounts = (inspect.Mounts ?? []).map(m =>
m.Type === 'volume' ? `volume ${m.Name ?? 'unnamed'}` : `${m.Type} ${m.Source ?? ''}`.trim(),
);
return {
name,
state: inspect.State?.Status ?? info.State ?? 'unknown',
health: inspect.State?.Health?.Status ?? null,
exitCode: typeof inspect.State?.ExitCode === 'number' ? inspect.State.ExitCode : null,
hasHealthcheck: !!inspect.Config?.Healthcheck?.Test?.length,
restartPolicy: inspect.HostConfig?.RestartPolicy?.Name || null,
mounts,
};
}),
);
return probes.filter((p): p is ContainerProbe => p !== null);
}
async computeUpdateReadiness(nodeId: number, stackName: string): Promise<UpdateReadinessReport> {
const db = DatabaseService.getInstance();
const now = Date.now();
const [preflight, drift, containers, preview, backup, disk] = await Promise.all([
this.collect('preflight', stackName, async () => ComposeDoctorService.getInstance().getLatest(nodeId, stackName)),
this.collect('drift', stackName, async () => db.getOpenDriftFindings(nodeId, stackName).length),
this.collect('containers', stackName, () =>
withTimeout(this.probeContainers(nodeId, stackName), INPUT_TIMEOUT_MS, 'readiness container probe')),
this.collect('update preview', stackName, () =>
withTimeout(UpdatePreviewService.getInstance().getPreview(nodeId, stackName), INPUT_TIMEOUT_MS, 'readiness update preview')),
this.collect('backup info', stackName, () => FileSystemService.getInstance(nodeId).getBackupInfo(stackName)),
this.collect('disk', stackName, () => this.readDiskUsage()),
]);
const settings = db.getGlobalSettings();
const limitPercent = parseInt(settings['host_disk_limit'] ?? '90', 10) || 90;
const signals = [
preflightSignal(preflight),
driftSignal(drift),
containersSignal(containers),
healthchecksSignal(containers),
updatePreviewSignal(preview === 'error' ? 'error' : preview.summary),
backupSlotSignal(backup, now),
diskSignal(typeof disk === 'number' ? { usePercent: disk, limitPercent } : 'error'),
];
return { stack: stackName, computedAt: now, verdict: aggregateVerdict(signals), signals };
}
async computeRollbackReadiness(nodeId: number, stackName: string): Promise<RollbackReadinessReport> {
const db = DatabaseService.getInstance();
const fsSvc = FileSystemService.getInstance(nodeId);
const now = Date.now();
const [backup, envSummary, stackHasEnv, preview, lastDeployAt, containers] = await Promise.all([
this.collect('backup info', stackName, () => fsSvc.getBackupInfo(stackName)),
this.collect('backup env summary', stackName, () => fsSvc.getBackupEnvSummary(stackName)),
this.collect('stack env presence', stackName, () => fsSvc.envExists(stackName)),
this.collect('update preview', stackName, () =>
withTimeout(UpdatePreviewService.getInstance().getPreview(nodeId, stackName), INPUT_TIMEOUT_MS, 'rollback readiness update preview')),
this.collect('activity history', stackName, async () => {
const events = db.getStackActivity(nodeId, stackName, { limit: 50 });
// A successful update is as good a known-good marker as a deploy.
return events.find(e => e.category === 'deploy_success' || e.category === 'image_update_applied')?.timestamp ?? null;
}),
this.collect('containers', stackName, () =>
withTimeout(this.probeContainers(nodeId, stackName), INPUT_TIMEOUT_MS, 'rollback readiness container probe')),
]);
const items = buildRollbackItems({
backup,
envSummary,
stackHasEnv,
rollbackTarget: preview === 'error' ? 'error' : { target: preview.rollback_target },
lastDeployAt,
containers,
}, now);
return { stack: stackName, computedAt: now, overall: aggregateRollbackOverall(items), items };
}
/** Host disk use percent for the main filesystem, or null when unavailable. */
private async readDiskUsage(): Promise<number | null> {
const fsSize = await withTimeout(si.fsSize(), INPUT_TIMEOUT_MS, 'readiness disk stats');
const mainDisk = fsSize.find(fs => fs.mount === '/' || fs.mount === 'C:') || fsSize[0];
if (typeof mainDisk?.use !== 'number') {
console.warn('[UpdateGuard] disk stats returned no usable mount; disk signal degrades to unknown');
return null;
}
return mainDisk.use;
}
/** Run one input collector; a failure degrades to the 'error' sentinel. */
private async collect<T>(label: string, stackName: string, fn: () => Promise<T>): Promise<T | Errored> {
try {
return await fn();
} catch (error) {
console.warn(
'[UpdateGuard] %s unavailable for %s:',
label, sanitizeForLog(stackName),
sanitizeForLog(getErrorMessage(error, 'unknown')),
);
return 'error';
}
}
}
+3
View File
@@ -3,6 +3,7 @@ import { ComposeService } from './ComposeService';
import { DatabaseService, type Webhook } from './DatabaseService';
import { FileSystemService } from './FileSystemService';
import { GitSourceService } from './GitSourceService';
import { HealthGateService } from './HealthGateService';
import { LicenseService } from './LicenseService';
import { PROXY_TIER_HEADER } from './license-headers';
import { NodeRegistry } from './NodeRegistry';
@@ -133,6 +134,7 @@ export class WebhookService {
buildSystemPolicyGateOptions('webhook', { auditPath: `/api/webhooks/${webhookId}/execute` }),
);
await compose.deployStack(stackName, undefined, atomic);
HealthGateService.getInstance().begin(nodeId, stackName, 'deploy', 'system:webhook');
break;
case 'restart':
await compose.runCommand(stackName, 'restart');
@@ -150,6 +152,7 @@ export class WebhookService {
buildSystemPolicyGateOptions('webhook', { auditPath: `/api/webhooks/${webhookId}/execute` }),
);
await compose.updateStack(stackName, undefined, atomic);
HealthGateService.getInstance().begin(nodeId, stackName, 'update', 'system:webhook');
break;
case 'git-pull':
return this.executeLocalGitPull(webhookId, stackName, action, triggerSource, startTime);
@@ -0,0 +1,125 @@
import type { FailureClassification } from './types';
interface ClassifierRule extends FailureClassification {
pattern: RegExp;
}
const DOCKER_UNREACHABLE: FailureClassification = {
reason: 'node_unreachable',
label: 'Docker unreachable',
suggestion: 'Check that Docker is running and reachable on this node, then retry.',
};
/**
* Maps the redacted error text a failed deploy/update throws (the accumulated
* compose stdout/stderr from ComposeService.execute, or a sentinel like
* CONTAINER_CRASHED) onto an operator-facing cause and next step.
*
* First match wins, so ordering is load-bearing:
* - the CONTAINER_CRASHED and stall sentinels are exact, so they go first;
* - env_missing precedes compose_render_failed because a render failure caused
* by a missing variable should classify as the actionable cause;
* - healthcheck_failed precedes dependency_unavailable because compose phrases
* an unhealthy dependency as "dependency failed to start: ... is unhealthy".
*
* The app-store install route has its own message-prettifier (utils/ErrorParser)
* with a different output shape; installs are out of scope here.
*/
const RULES: ClassifierRule[] = [
{
reason: 'container_exited',
label: 'Container exited after start',
suggestion: 'Check the container logs for the exit cause; roll back if the previous version was healthy.',
pattern: /CONTAINER_CRASHED/,
},
{
// The idle-stall backstop terminated the step; the real cause is unknown.
reason: 'unknown',
label: 'Operation stalled',
suggestion: 'The operation stopped producing output and was terminated. Check Docker activity on the node, then retry.',
pattern: /STACK_STALLED_OUTPUT/,
},
{
...DOCKER_UNREACHABLE,
pattern: /cannot connect to the docker daemon|docker daemon is not running|error during connect|docker daemon is unreachable/i,
},
{
reason: 'env_missing',
label: 'Missing environment variable',
suggestion: 'Define the missing variable in the stack environment file, then retry.',
pattern: /required variable\s+\S+ is missing|variable is not set|invalid interpolation format|env file .+ not found|couldn't find env file/i,
},
{
reason: 'image_pull_failed',
label: 'Image pull failed',
suggestion: 'Check the image name and tag, registry credentials, and registry rate limits, then retry.',
pattern: /pull access denied|manifest unknown|manifest for .+ not found|toomanyrequests|failed to resolve reference|no matching manifest|error pulling image|repository does not exist|unauthorized: authentication required/i,
},
{
reason: 'port_conflict',
label: 'Host port conflict',
suggestion: 'Free the conflicting host port or change the published port, then retry.',
pattern: /port is already allocated|bind: address already in use|ports are not available|failed to bind host port/i,
},
{
reason: 'bind_path_missing',
label: 'Bind mount path missing',
suggestion: 'Create the missing host path or correct the bind mount source, then retry.',
pattern: /bind source path does not exist|mounts denied|invalid mount config/i,
},
{
reason: 'permission_denied',
label: 'Permission denied',
suggestion: 'Check file and Docker socket permissions for the affected path, then retry.',
pattern: /permission denied|EACCES|operation not permitted/i,
},
{
reason: 'healthcheck_failed',
label: 'Healthcheck failed',
suggestion: 'Check the failing service logs and its healthcheck command; roll back if the previous version was healthy.',
pattern: /is unhealthy/i,
},
{
reason: 'dependency_unavailable',
label: 'Dependency unavailable',
suggestion: 'Start or create the missing dependency (service, external network, or volume) first, then retry.',
pattern: /dependency failed to start|depends on undefined service|declared as external, but could not be found/i,
},
{
reason: 'compose_render_failed',
label: 'Compose file invalid',
suggestion: 'Review the compose file syntax (Compose Doctor can pinpoint the issue), then retry.',
pattern: /yaml:|mapping values are not allowed|cannot unmarshal|additional propert|undefined volume|undefined network|invalid compose/i,
},
];
const UNKNOWN_FAILURE: FailureClassification = {
reason: 'unknown',
label: 'Unclassified failure',
suggestion: 'Open the deploy log and copy the troubleshooting details for the full error.',
};
/**
* Classify a failed deploy/update. Total: always returns a classification,
* falling back to `unknown`. `opts.dockerUnavailable` short-circuits to
* node_unreachable for errors the route already identified as a dead daemon
* (their message shape varies too much for patterns alone).
*
* Note: a ComposeRollbackError's message is already the underlying cause's
* message (its constructor copies it), so callers can pass
* getErrorMessage(error) for wrapped rollback failures unchanged.
*/
export function classifyFailure(
message: string,
opts?: { dockerUnavailable?: boolean },
): FailureClassification {
if (opts?.dockerUnavailable) {
return { ...DOCKER_UNREACHABLE };
}
for (const rule of RULES) {
if (rule.pattern.test(message)) {
return { reason: rule.reason, label: rule.label, suggestion: rule.suggestion };
}
}
return { ...UNKNOWN_FAILURE };
}
@@ -0,0 +1,277 @@
import type { PreflightStatus } from '../preflight/types';
import type { UpdatePreviewSummary } from '../UpdatePreviewService';
import type {
ContainerProbe,
ReadinessSignal,
ReadinessVerdict,
RollbackOverall,
RollbackReadinessItem,
SignalStatus,
} from './types';
/**
* Pure readiness scoring. UpdateGuardService gathers the inputs (each of which
* degrades independently to the 'error' sentinel) and these functions map them
* to signals and a verdict, so every grading rule is synchronously testable.
*/
/** Sentinel for an input whose collection failed. */
export type Errored = 'error';
const formatAge = (timestamp: number, now: number): string => {
const minutes = Math.max(0, Math.round((now - timestamp) / 60_000));
if (minutes < 60) return `${minutes}m ago`;
const hours = Math.round(minutes / 60);
if (hours < 48) return `${hours}h ago`;
return `${Math.round(hours / 24)}d ago`;
};
export function preflightSignal(
input: { status: PreflightStatus } | Errored,
): ReadinessSignal {
const base = { id: 'preflight' as const, title: 'Compose Doctor' };
if (input === 'error') {
return { ...base, status: 'unknown', affectsVerdict: false, detail: 'The stored preflight report could not be read.' };
}
switch (input.status) {
case 'never-run':
return { ...base, status: 'unknown', affectsVerdict: false, detail: 'Compose Doctor has not been run for this stack yet. Run it for a deeper pre-update check.' };
case 'blocker':
return { ...base, status: 'blocked', affectsVerdict: true, detail: 'The last preflight found a blocker. Resolve it before updating.' };
case 'unrenderable':
return { ...base, status: 'attention', affectsVerdict: true, detail: 'The compose file did not render in the last preflight; the update is likely to fail the same way.' };
case 'high':
return { ...base, status: 'attention', affectsVerdict: true, detail: 'The last preflight found high-risk findings. Review them before updating.' };
case 'warning':
return { ...base, status: 'warning', affectsVerdict: true, detail: 'The last preflight found warnings.' };
case 'pass':
case 'info':
return { ...base, status: 'ok', affectsVerdict: true, detail: 'The last preflight passed.' };
}
}
export function driftSignal(input: number | Errored): ReadinessSignal {
const base = { id: 'drift' as const, title: 'Drift' };
if (input === 'error') {
return { ...base, status: 'unknown', affectsVerdict: false, detail: 'Drift findings could not be read.' };
}
if (input > 0) {
const plural = input === 1 ? 'finding' : 'findings';
return {
...base,
status: 'warning',
affectsVerdict: true,
detail: `${input} open drift ${plural}: the running state has diverged from the compose file, so the rollback target may not match what is running.`,
};
}
return { ...base, status: 'ok', affectsVerdict: true, detail: 'No open drift findings.' };
}
export function containersSignal(input: ContainerProbe[] | Errored): ReadinessSignal {
const base = { id: 'containers' as const, title: 'Current containers' };
if (input === 'error') {
return { ...base, status: 'unknown', affectsVerdict: true, detail: 'Container state could not be read from Docker.' };
}
if (input.length === 0) {
return { ...base, status: 'warning', affectsVerdict: true, detail: 'The stack is not running; updating will start it.' };
}
const troubled = input.filter(
c => c.health === 'unhealthy' || c.state === 'restarting' || (c.state === 'exited' && (c.exitCode ?? 0) !== 0),
);
if (troubled.length > 0) {
const names = troubled.map(c => c.name).join(', ');
return {
...base,
status: 'attention',
affectsVerdict: true,
detail: `Already unhealthy before the update: ${names}. An update on top of a failing stack is hard to evaluate; consider fixing or stopping it first.`,
};
}
return { ...base, status: 'ok', affectsVerdict: true, detail: `${input.length} container${input.length === 1 ? '' : 's'} running normally.` };
}
export function healthchecksSignal(input: ContainerProbe[] | Errored): ReadinessSignal {
const base = { id: 'healthchecks' as const, title: 'Healthcheck coverage', status: 'ok' as SignalStatus, affectsVerdict: false };
if (input === 'error' || input.length === 0) {
return { ...base, detail: 'Coverage is unknown until the stack runs. Containers without healthchecks are verified by run state only after an update.' };
}
const withCheck = input.filter(c => c.hasHealthcheck).length;
const withoutRestart = input.filter(c => !c.restartPolicy || c.restartPolicy === 'no').length;
const parts = [
`${withCheck} of ${input.length} container${input.length === 1 ? '' : 's'} define a healthcheck; the rest are verified by run state only after an update.`,
];
if (withoutRestart > 0) {
parts.push(`${withoutRestart} ha${withoutRestart === 1 ? 's' : 've'} no restart policy.`);
}
return { ...base, detail: parts.join(' ') };
}
export function updatePreviewSignal(input: UpdatePreviewSummary | Errored): ReadinessSignal {
const base = { id: 'update_preview' as const, title: 'Pending update' };
if (input === 'error') {
return { ...base, status: 'unknown', affectsVerdict: false, detail: 'The update preview is unavailable.' };
}
if (input.blocked) {
return {
...base,
status: 'blocked',
affectsVerdict: true,
detail: input.blocked_reason ?? 'A scan policy blocks this update.',
};
}
if (input.has_update && input.semver_bump === 'major') {
const change = input.current_tag && input.next_tag ? ` (${input.current_tag} to ${input.next_tag})` : '';
return { ...base, status: 'attention', affectsVerdict: true, detail: `A major version bump is pending${change}. Review the upstream changelog for breaking changes.` };
}
if (input.has_update && input.semver_bump === 'unknown') {
return { ...base, status: 'warning', affectsVerdict: true, detail: 'An image update is pending but the version change could not be classified.' };
}
if (input.has_update) {
const kind = input.update_kind === 'digest' ? 'a same-tag image refresh' : `a ${input.semver_bump} update`;
return { ...base, status: 'ok', affectsVerdict: true, detail: `Pending: ${kind}.` };
}
return { ...base, status: 'ok', affectsVerdict: true, detail: 'No pending image update detected; the update re-pulls and recreates with current tags.' };
}
export function backupSlotSignal(
input: { exists: boolean; timestamp: number | null } | Errored,
now: number,
): ReadinessSignal {
const base = { id: 'backup_slot' as const, title: 'Rollback backup' };
if (input === 'error') {
return { ...base, status: 'unknown', affectsVerdict: false, detail: 'The backup slot could not be read.' };
}
if (!input.exists) {
return { ...base, status: 'warning', affectsVerdict: true, detail: 'No rollback backup exists yet; one is created automatically when the update starts.' };
}
const age = input.timestamp ? ` (from ${formatAge(input.timestamp, now)})` : '';
return { ...base, status: 'ok', affectsVerdict: true, detail: `A compose and env file backup exists${age} and is refreshed when the update starts.` };
}
export function diskSignal(
input: { usePercent: number; limitPercent: number } | null | Errored,
): ReadinessSignal {
const base = { id: 'disk' as const, title: 'Node disk' };
if (input === 'error' || input === null) {
return { ...base, status: 'unknown', affectsVerdict: false, detail: 'Disk usage could not be read.' };
}
const use = Math.round(input.usePercent);
if (input.usePercent >= input.limitPercent) {
return { ...base, status: 'attention', affectsVerdict: true, detail: `Disk usage is at ${use}%, at or above the ${input.limitPercent}% alert threshold. Image pulls may fail; free space first.` };
}
if (input.usePercent >= input.limitPercent - 5) {
return { ...base, status: 'warning', affectsVerdict: true, detail: `Disk usage is at ${use}%, close to the ${input.limitPercent}% alert threshold.` };
}
return { ...base, status: 'ok', affectsVerdict: true, detail: `Disk usage is at ${use}%.` };
}
/**
* Severity precedence: blocked > attention (review required) > verdict-affecting
* unknown > warning > ready. Informational unknowns never affect the verdict.
*/
export function aggregateVerdict(signals: ReadinessSignal[]): ReadinessVerdict {
const affecting = signals.filter(s => s.affectsVerdict);
if (affecting.some(s => s.status === 'blocked')) return 'blocked';
if (affecting.some(s => s.status === 'attention')) return 'review_required';
if (affecting.some(s => s.status === 'unknown')) return 'unknown';
if (affecting.some(s => s.status === 'warning')) return 'ready_with_warnings';
return 'ready';
}
// ── Rollback readiness ───────────────────────────────────────────────────────
export interface RollbackInputs {
backup: { exists: boolean; timestamp: number | null } | Errored;
envSummary: { exists: boolean; envPresent: boolean; keys: string[] } | Errored;
/** Whether the stack currently has an env file (distinguishes "no env to cover"). */
stackHasEnv: boolean | Errored;
/**
* UpdatePreview.rollback_target wrapped in an object so the Errored sentinel
* cannot be absorbed into the string domain (an image literally named
* "error" must not read as a failed preview).
*/
rollbackTarget: { target: string | null } | Errored;
/** Timestamp of the most recent deploy_success activity event, if any. */
lastDeployAt: number | null | Errored;
containers: ContainerProbe[] | Errored;
}
export function buildRollbackItems(inputs: RollbackInputs, now: number): RollbackReadinessItem[] {
const items: RollbackReadinessItem[] = [];
const backupExists = inputs.backup !== 'error' && inputs.backup.exists;
if (inputs.backup === 'error') {
items.push({ id: 'compose_source', state: 'unknown', label: 'Previous compose file', detail: 'The backup slot could not be read.' });
} else if (backupExists) {
const age = inputs.backup.timestamp ? ` from ${formatAge(inputs.backup.timestamp, now)}` : '';
items.push({ id: 'compose_source', state: 'ready', label: 'Previous compose file', detail: `A backup${age} is available to restore.` });
} else {
items.push({ id: 'compose_source', state: 'missing', label: 'Previous compose file', detail: 'No backup exists yet. One is created automatically by the next update or deploy.' });
}
if (inputs.envSummary === 'error') {
items.push({ id: 'env_keys', state: 'unknown', label: 'Previous env file', detail: 'The backed-up env file could not be read.' });
} else if (inputs.envSummary.envPresent) {
const n = inputs.envSummary.keys.length;
items.push({ id: 'env_keys', state: 'ready', label: 'Previous env file', detail: `${n} variable name${n === 1 ? '' : 's'} captured in the backup (values are restored with the file, never shown here).` });
} else if (inputs.stackHasEnv === true && backupExists) {
items.push({ id: 'env_keys', state: 'missing', label: 'Previous env file', detail: 'The stack has an env file but the backup does not contain one; a rollback would not restore env changes.' });
} else if (!backupExists) {
items.push({ id: 'env_keys', state: 'missing', label: 'Previous env file', detail: 'No backup exists yet.' });
} else {
items.push({ id: 'env_keys', state: 'ready', label: 'Previous env file', detail: 'The stack uses no env file, so there is nothing to restore.' });
}
if (inputs.rollbackTarget === 'error') {
items.push({ id: 'previous_images', state: 'unknown', label: 'Previous image tag', detail: 'The update preview is unavailable.' });
} else if (inputs.rollbackTarget.target) {
items.push({ id: 'previous_images', state: 'ready', label: 'Previous image tag', detail: `Known rollback target: ${inputs.rollbackTarget.target}. If the compose file uses a moving tag, restoring files alone does not revert the image; pin this tag to be exact.` });
} else {
items.push({ id: 'previous_images', state: 'unknown', label: 'Previous image tag', detail: 'The previous image tag could not be determined. A rollback restores compose and env files; a moving tag may keep the newer image.' });
}
if (inputs.lastDeployAt === 'error') {
items.push({ id: 'last_deploy', state: 'unknown', label: 'Last successful deploy', detail: 'The activity history could not be read.' });
} else if (inputs.lastDeployAt) {
items.push({ id: 'last_deploy', state: 'ready', label: 'Last successful deploy', detail: `Recorded ${formatAge(inputs.lastDeployAt, now)}; the backup reflects a configuration that deployed successfully.` });
} else {
items.push({ id: 'last_deploy', state: 'missing', label: 'Last successful deploy', detail: 'No successful deploy is recorded in the recent activity history.' });
}
if (inputs.containers === 'error') {
items.push({ id: 'healthchecks', state: 'unknown', label: 'Healthchecks', detail: 'Container state could not be read from Docker.' });
} else if (inputs.containers.some(c => c.hasHealthcheck)) {
items.push({ id: 'healthchecks', state: 'ready', label: 'Healthchecks', detail: 'At least one service defines a healthcheck, so a rollback can be verified beyond run state.' });
} else {
items.push({ id: 'healthchecks', state: 'missing', label: 'Healthchecks', detail: 'No service defines a healthcheck; rollback verification relies on run state only.' });
}
const mounts = inputs.containers === 'error'
? []
: [...new Set(inputs.containers.flatMap(c => c.mounts))];
const mountDetail = mounts.length > 0 ? ` This stack mounts: ${mounts.join(', ')}.` : '';
items.push({
id: 'volume_data',
state: 'not_covered',
label: 'Application data',
detail: `Named volumes and bind-mounted data are not included in file backups. Rolling back restores compose and env files only; application data keeps its current state.${mountDetail}`,
});
return items;
}
/**
* compose_source gates not_ready; ready additionally requires env coverage and
* a known previous image tag. volume_data, healthchecks, and last_deploy are
* disclosures and never gate the overall state.
*/
export function aggregateRollbackOverall(items: RollbackReadinessItem[]): RollbackOverall {
const byId = new Map(items.map(i => [i.id, i.state]));
if (byId.get('compose_source') !== 'ready') {
return byId.get('compose_source') === 'unknown' ? 'partial' : 'not_ready';
}
if (byId.get('env_keys') === 'ready' && byId.get('previous_images') === 'ready') {
return 'ready';
}
return 'partial';
}
+115
View File
@@ -0,0 +1,115 @@
/** Overall verdict of an update readiness check. */
export type ReadinessVerdict = 'ready' | 'ready_with_warnings' | 'review_required' | 'blocked' | 'unknown';
/** Graded status of a single readiness signal. */
export type SignalStatus = 'ok' | 'warning' | 'attention' | 'blocked' | 'unknown';
/** One input to the readiness verdict (preflight, drift, containers, ...). */
export interface ReadinessSignal {
id: 'preflight' | 'drift' | 'containers' | 'healthchecks' | 'update_preview' | 'backup_slot' | 'disk';
status: SignalStatus;
/** Short headline ("Compose Doctor", "Running containers"). */
title: string;
/** What was observed and why it matters. Never carries an env value. */
detail: string;
/**
* False for informational unknowns (preflight never run, preview
* unavailable) that should not drag the overall verdict to `unknown`.
*/
affectsVerdict: boolean;
}
/** Computed on demand; not persisted (the inputs keep their own history). */
export interface UpdateReadinessReport {
stack: string;
computedAt: number;
verdict: ReadinessVerdict;
signals: ReadinessSignal[];
}
/** State of one rollback readiness item. */
export type RollbackItemState = 'ready' | 'missing' | 'unknown' | 'not_covered';
export interface RollbackReadinessItem {
id: 'compose_source' | 'env_keys' | 'previous_images' | 'last_deploy' | 'healthchecks' | 'volume_data';
state: RollbackItemState;
label: string;
/** Names only for env coverage; values never appear here. */
detail: string;
}
export type RollbackOverall = 'ready' | 'partial' | 'not_ready';
export interface RollbackReadinessReport {
stack: string;
computedAt: number;
overall: RollbackOverall;
items: RollbackReadinessItem[];
}
/** Normalized per-container probe used by readiness and the health gate. */
export interface ContainerProbe {
name: string;
/** Docker container state (running, exited, restarting, ...), or 'unknown'. */
state: string;
/** Docker health status (healthy | unhealthy | starting) or null without a healthcheck. */
health: string | null;
exitCode: number | null;
hasHealthcheck: boolean;
/** RestartPolicy.Name, or null/'no' when none is set. */
restartPolicy: string | null;
/** Mount descriptors ("volume data", "bind /srv/app"), for coverage disclosure. */
mounts: string[];
}
/** Lifecycle of a post-update health gate observation. */
export type HealthGateStatus = 'observing' | 'passed' | 'failed' | 'unknown';
/** Per-container end state captured when a gate run finalizes. */
export interface HealthGateContainer {
name: string;
/** Docker container state, or 'missing' when it vanished mid-observation. */
state: string;
health: string | null;
restarts: number;
}
/** Payload of GET /:stackName/health-gate ('never-run' when no run exists). */
export interface HealthGateReport {
stack: string;
id: string | null;
status: HealthGateStatus | 'never-run';
trigger: 'update' | 'deploy' | null;
reason: string | null;
windowSeconds: number | null;
startedAt: number | null;
endedAt: number | null;
containers: HealthGateContainer[];
}
/** Categories a failed stack deploy or update can be classified into. */
export type FailureReason =
| 'image_pull_failed'
| 'compose_render_failed'
| 'env_missing'
| 'port_conflict'
| 'bind_path_missing'
| 'permission_denied'
| 'container_exited'
| 'healthcheck_failed'
| 'dependency_unavailable'
| 'node_unreachable'
| 'unknown';
/**
* Operator-facing classification of a failed deploy/update, attached to the
* route's error response so the recovery surfaces can show a cause and a next
* step instead of only the raw compose output.
*/
export interface FailureClassification {
reason: FailureReason;
/** Short display headline ("Host port conflict"). */
label: string;
/** Suggested next action, one sentence. */
suggestion: string;
}