feat(recovery): add safe-mode recovery surface and emergency CLI (#1286)

* feat(recovery): add safe-mode recovery surface and emergency CLI

Add a read-only Recovery tab under Settings (admin-only) backed by a new
GET /api/diagnostics endpoint reporting app version, database integrity,
encryption-key status, Docker reachability, account and SSO counts, and
non-secret configuration. The endpoint loads without Docker or live metrics
so it stays available when the dashboard does not, requires a genuine admin
session, and builds its config block from a non-secret allowlist so no
credentials are ever exposed.

Expand the emergency command-line toolkit beyond the two-factor reset with
seven host-level commands: reset-password, create-emergency-admin,
clear-sessions, disable-sso, diagnostics, validate-db, and backup-data. Each
prints its result, exits with a meaningful status code, and writes an audit
entry where it changes state.

Document the toolkit in a new operator guide and link it from the recovery
and two-factor pages.

* feat(recovery): download the emergency command reference as a text file

The recovery commands are needed exactly when the dashboard is unreachable,
so reading them only in-app is a chicken-and-egg problem. Add a Download
button to the command-line section that saves the full
`docker compose exec sencho ...` reference as a text file, letting operators
keep it on hand before they need it. Reuses a shared download helper with the
existing diagnostics export.

* fix(recovery): harden diagnostics, backup, and emergency-admin against edge cases

Address findings from an independent review of the recovery toolkit:

- DiagnosticsService now degrades instead of throwing when a queried table is
  missing or corrupt: each read falls back and is folded into database.ok, so a
  broken database reports "problem detected" rather than failing the whole
  endpoint or showing a misleading healthy state with zeroed counts.
- backup-data refuses a destination that resolves to the live database, which
  would otherwise report success while producing no separate copy.
- create-emergency-admin now applies the same username rule as the user-
  management route, extracted to a shared helper so both stay in sync.

Adds tests for a missing read table, a malformed emergency-admin username, and
the backup same-target rejection.
This commit is contained in:
Anso
2026-06-02 16:11:24 -04:00
committed by GitHub
parent 06b25262cc
commit c6d1631afe
27 changed files with 1339 additions and 11 deletions
+16
View File
@@ -2680,12 +2680,28 @@ export class DatabaseService {
this.db.prepare('UPDATE users SET token_version = token_version + 1, updated_at = ? WHERE id = ?').run(Date.now(), userId);
}
/**
* Invalidate every active session by bumping every user's token_version in
* one statement. Returns the number of users affected. Used by the
* emergency `clear-sessions` CLI when a stolen cookie or a wedged login
* state needs a clean sign-out of every user on this node.
*/
public bumpAllTokenVersions(): number {
const result = this.db.prepare('UPDATE users SET token_version = token_version + 1, updated_at = ?').run(Date.now());
return result.changes;
}
// --- User MFA ---
public getUserMfa(userId: number): UserMfa | undefined {
return this.db.prepare('SELECT * FROM user_mfa WHERE user_id = ?').get(userId) as UserMfa | undefined;
}
/** Count of users with a completed (enabled) two-factor enrolment. */
public getMfaEnrolledCount(): number {
return (this.db.prepare('SELECT COUNT(*) as count FROM user_mfa WHERE enabled = 1').get() as { count: number })?.count || 0;
}
/**
* Create or merge a user_mfa row. Any field left undefined on the update
* object is preserved. Boolean flags are normalized to 0/1.
+174
View File
@@ -0,0 +1,174 @@
/**
* Read-only recovery diagnostics shared by the admin Recovery settings tab
* (GET /api/diagnostics) and the `diagnostics` / `validate-db` emergency CLI
* commands. It answers the first questions an operator asks when the dashboard
* is misbehaving: is the database intact, is the encryption key present, is
* Docker reachable, and is at least one admin able to sign in.
*
* The report carries no secrets. The `config` block is built from an allowlist
* of non-sensitive global settings, so tokens, password hashes, OIDC client
* secrets, and cloud-backup credentials can never leak into an exported bundle.
*/
import fs from 'fs';
import path from 'path';
import { DatabaseService } from './DatabaseService';
import { CryptoService } from './CryptoService';
import { getSenchoVersion } from './CapabilityRegistry';
// Tables the app cannot function without. A missing one means the schema did
// not initialize and the database should be treated as broken.
const CORE_TABLES = ['users', 'global_settings', 'sso_config', 'audit_log'] as const;
// Allowlist of global_settings keys safe to surface. Anything not listed here
// (auth_jwt_secret, auth_password_hash, cloud_backup_secret_key, OIDC/LDAP
// secrets, etc.) is omitted by construction rather than filtered out.
const SAFE_SETTING_KEYS = [
'host_cpu_limit',
'host_ram_limit',
'host_disk_limit',
'host_alert_suppression_mins',
'docker_janitor_gb',
'global_crash',
'developer_mode',
'template_registry_url',
'metrics_retention_hours',
'log_retention_days',
'audit_retention_days',
'mesh_auto_recreate',
'scan_history_per_image_limit',
'cloud_backup_provider',
] as const;
export interface DiagnosticsReport {
version: string | null;
database: {
ok: boolean;
integrity: string;
path: string;
missingTables: string[];
};
encryptionKey: { present: boolean; valid: boolean };
docker: { reachable: boolean; error?: string };
auth: {
adminCount: number;
userCount: number;
mfaEnrolledCount: number;
ssoProviders: Array<{ provider: string; enabled: boolean }>;
};
config: Record<string, string>;
}
export interface CollectDiagnosticsOptions {
/**
* Optional Docker reachability probe. The HTTP route passes a bounded
* `docker.ping()`; the CLI omits it because it runs without a Docker
* connection and reports `reachable: false`.
*/
checkDocker?: () => Promise<boolean>;
}
function dataDir(): string {
return process.env.DATA_DIR || path.join(process.cwd(), 'data');
}
/**
* Confirm the encryption key is on disk and actually usable: present, a 32-byte
* hex value, and able to round-trip through CryptoService. A present-but-corrupt
* key reads as `{ present: true, valid: false }` so the operator knows a restore
* is needed rather than a fresh key generation.
*/
function checkEncryptionKey(): { present: boolean; valid: boolean } {
const keyPath = path.join(dataDir(), 'encryption.key');
if (!fs.existsSync(keyPath)) return { present: false, valid: false };
try {
const raw = fs.readFileSync(keyPath, 'utf-8').trim();
if (Buffer.from(raw, 'hex').length !== 32) return { present: true, valid: false };
const crypto = CryptoService.getInstance();
const probe = crypto.encrypt('diagnostics-probe');
return { present: true, valid: crypto.decrypt(probe) === 'diagnostics-probe' };
} catch (err) {
// The UI verdict is binary, but log the cause so an unreadable key file
// (permissions) can be told apart from a corrupt one when debugging.
console.warn(`[diagnostics] encryption key check failed: ${(err as Error).message}`);
return { present: true, valid: false };
}
}
export async function collectDiagnostics(opts: CollectDiagnosticsOptions = {}): Promise<DiagnosticsReport> {
const db = DatabaseService.getInstance();
const handle = db.getDb();
// A read that may throw if its table is missing or corrupt degrades to
// `fallback` instead of failing the whole report, and records that a read
// failed so `database.ok` reflects it. This keeps the surface usable on the
// broken database it exists to diagnose, without reporting a degraded `0`
// (e.g. adminCount) as if the database were healthy.
let readFailed = false;
const safe = <T>(read: () => T, fallback: T, label: string): T => {
try {
return read();
} catch (err) {
readFailed = true;
console.warn(`[diagnostics] ${label} failed: ${String((err as Error)?.message ?? err)}`);
return fallback;
}
};
let integrity: string;
let missingTables: string[];
let integrityOk = false;
try {
integrity = String(handle.pragma('integrity_check', { simple: true }));
const present = new Set(
(handle.prepare("SELECT name FROM sqlite_master WHERE type='table'").all() as Array<{ name: string }>)
.map(row => row.name),
);
missingTables = CORE_TABLES.filter(table => !present.has(table));
integrityOk = integrity === 'ok';
} catch (err) {
integrity = `error: ${(err as Error).message}`;
missingTables = [];
}
const config = safe(() => {
const settings = db.getGlobalSettings();
const out: Record<string, string> = {};
for (const key of SAFE_SETTING_KEYS) {
if (settings[key] !== undefined) out[key] = settings[key];
}
return out;
}, {}, 'global_settings read');
const auth = {
adminCount: safe(() => db.getAdminCount(), 0, 'getAdminCount'),
userCount: safe(() => db.getUserCount(), 0, 'getUserCount'),
mfaEnrolledCount: safe(() => db.getMfaEnrolledCount(), 0, 'getMfaEnrolledCount'),
ssoProviders: safe(
() => db.getSSOConfigs().map(c => ({ provider: c.provider, enabled: c.enabled === 1 })),
[] as Array<{ provider: string; enabled: boolean }>,
'getSSOConfigs',
),
};
// Reads run before this so a present-but-unreadable table (which the
// integrity check may still call "ok") also marks the database not-ok.
const dbOk = integrityOk && missingTables.length === 0 && !readFailed;
let docker: DiagnosticsReport['docker'] = { reachable: false };
if (opts.checkDocker) {
try {
docker = { reachable: await opts.checkDocker() };
} catch (err) {
docker = { reachable: false, error: (err as Error).message };
}
}
return {
version: getSenchoVersion(),
database: { ok: dbOk, integrity, path: handle.name, missingTables },
encryptionKey: checkEncryptionKey(),
docker,
auth,
config,
};
}