feat: auto-heal policies for unhealthy containers (#671)

* feat(db): add auto_heal_policies and auto_heal_history schema and CRUD

Adds two new SQLite tables (auto_heal_policies, auto_heal_history) to
DatabaseService.initSchema() and exposes CRUD methods: getAutoHealPolicies,
getAutoHealPolicy, addAutoHealPolicy, updateAutoHealPolicy,
deleteAutoHealPolicy, recordAutoHealHistory, getAutoHealHistory,
incrementConsecutiveFailures, resetConsecutiveFailures, setPolicyEnabled.
Also adds AutoHealPolicy and AutoHealHistoryEntry TypeScript interfaces.

* feat(events): track health-status duration and expose state accessors

- Add healthStatus and unhealthySince fields to InternalContainerState
- onHealthStatus now records unhealthySince timestamp on first transition
  to unhealthy, and clears it when the container recovers or restarts
- onStart resets both fields so a restarted container begins from 'starting'
- Add listContainerStates() and getContainerState() public accessors for
  use by the upcoming AutoHealService evaluator

* fix(auto-heal): key allowlist in updateAutoHealPolicy, cascade delete, extract ContainerHealthSnapshot

* feat: add AutoHealService evaluator singleton

Polls every 30 s, matches containers to enabled policies via Compose
labels, and restarts containers that have been unhealthy beyond the
configured threshold. Enforces cooldown, per-hour rate cap, and
recent-user-action suppression; auto-disables policies after repeated
consecutive failures. Also adds DockerEventManager.getService() accessor
required by the evaluator.

* fix(auto-heal): prune stale restartTimestamps, guard undefined policy id

- Prune restartTimestamps entries for containers no longer running after
  each container list fetch, preventing unbounded map growth from dead
  container IDs.
- Guard against policies with undefined id at the start of the per-policy
  loop; warn and skip rather than proceed with a non-null assertion.
- Extract handleAutoDisable private helper to bring executeHeal under 30
  lines and isolate the auto-disable side-effect sequence.
- Move ContainerInfo type to module scope.

* feat: add auto-heal API routes and wire AutoHealService lifecycle

Registers five REST endpoints under /api/auto-heal/policies (list, create,
patch, delete, history) with requirePaid + requireAdmin guards and Zod
validation. Wires AutoHealService.start()/stop() into the server startup
and graceful-shutdown blocks alongside MonitorService.

* test: add AutoHealService and DatabaseService auto-heal unit tests

- 15 unit tests for AutoHealService.shouldHeal covering all decision branches
  (healthy state, duration threshold, user-action suppression, cooldown,
  rate limiting, and correct skipReason values)
- 13 integration tests for DatabaseService auto-heal CRUD: policy round-trip,
  stack-name filter, partial update, cascade delete, history ordering/limit,
  consecutive failure counters, and setPolicyEnabled toggle

* fix: log AutoHealService shutdown errors consistently

* fix(api): requireAdmin-first guard order and try/catch on auto-heal routes

* feat(ui): add StackAutoHealSheet component

* feat(ui): add Auto-Heal context menu item to EditorLayout

* fix(ui): StackAutoHealSheet label, token, a11y, and useEffect fixes

- Rename 'All services in stack' to 'All services' in combobox options and placeholder
- Replace text-green-600 with text-success design token in actionColorClass
- Add htmlFor/id pairs to all four numeric form inputs for accessibility
- Inline fetch logic into useEffect, removing stale closure risk and eslint-disable comment
- Remove now-unused fetchPolicies and fetchServices standalone functions
- Update 'Auto-disable after' label to 'Auto-disable after (failures)' for clarity
- Add toast.error in policy fetch failure path; services fetch silently skips as before

* docs: add auto-heal-policies feature documentation

* test(e2e): add auto-heal policies CRUD spec

* fix(docs): correct auto-heal-policies nav position in docs.json

backend/src/index.ts

@@ -23,6 +23,7 @@ import { HostTerminalService } from './services/HostTerminalService';
 import { DatabaseService, Node, AuthProvider, ScheduledTask, UserRole, ResourceType } from './services/DatabaseService';
 import { NotificationService } from './services/NotificationService';
 import { MonitorService } from './services/MonitorService';
+import { AutoHealService } from './services/AutoHealService';
 import { DockerEventManager } from './services/DockerEventManager';
 import { ImageUpdateService } from './services/ImageUpdateService';
 import { templateService } from './services/TemplateService';
@@ -5697,6 +5698,16 @@ const AlertCreateSchema = z.object({
   cooldown_mins: z.coerce.number().int().min(0).max(10080),
 });
 
+const AutoHealPolicyCreateSchema = z.object({
+  stack_name: z.string().min(1).max(255),
+  service_name: z.string().min(1).max(255).nullable().optional(),
+  unhealthy_duration_mins: z.coerce.number().int().min(1).max(1440),
+  cooldown_mins: z.coerce.number().int().min(1).max(1440).default(5),
+  max_restarts_per_hour: z.coerce.number().int().min(1).max(60).default(3),
+  auto_disable_after_failures: z.coerce.number().int().min(1).max(100).default(5),
+});
+const AutoHealPolicyUpdateSchema = AutoHealPolicyCreateSchema.partial().omit({ stack_name: true });
+
 app.post('/api/alerts', authMiddleware, async (req: Request, res: Response) => {
   if (!requireAdmin(req, res)) return;
   const parsed = AlertCreateSchema.safeParse(req.body);
@@ -5724,6 +5735,100 @@ app.delete('/api/alerts/:id', authMiddleware, async (req: Request, res: Response
   }
 });
 
+// ─── Auto-Heal Policies ───────────────────────────────────────────────────────
+
+app.get('/api/auto-heal/policies', authMiddleware, (req: Request, res: Response): void => {
+  if (!requirePaid(req, res)) return;
+  const stackName = typeof req.query.stackName === 'string' ? req.query.stackName : undefined;
+  try {
+    res.json(DatabaseService.getInstance().getAutoHealPolicies(stackName));
+  } catch (err) {
+    console.error('[AutoHeal] Failed to list policies:', err instanceof Error ? err.message : err);
+    res.status(500).json({ error: 'Internal server error' });
+  }
+});
+
+app.post('/api/auto-heal/policies', authMiddleware, (req: Request, res: Response): void => {
+  if (!requireAdmin(req, res)) return;
+  if (!requirePaid(req, res)) return;
+  const parsed = AutoHealPolicyCreateSchema.safeParse(req.body);
+  if (!parsed.success) {
+    res.status(400).json({ error: parsed.error.issues[0]?.message ?? 'Invalid input' });
+    return;
+  }
+  const { stack_name, service_name, unhealthy_duration_mins, cooldown_mins, max_restarts_per_hour, auto_disable_after_failures } = parsed.data;
+  const now = Date.now();
+  try {
+    const policy = DatabaseService.getInstance().addAutoHealPolicy({
+      stack_name,
+      service_name: service_name ?? null,
+      unhealthy_duration_mins,
+      cooldown_mins,
+      max_restarts_per_hour,
+      auto_disable_after_failures,
+      enabled: 1,
+      consecutive_failures: 0,
+      last_fired_at: 0,
+      created_at: now,
+      updated_at: now,
+    });
+    res.status(201).json(policy);
+  } catch (err) {
+    console.error('[AutoHeal] Failed to create policy:', err instanceof Error ? err.message : err);
+    res.status(500).json({ error: 'Internal server error' });
+  }
+});
+
+app.patch('/api/auto-heal/policies/:id', authMiddleware, (req: Request, res: Response): void => {
+  if (!requireAdmin(req, res)) return;
+  if (!requirePaid(req, res)) return;
+  const id = parseInt(req.params.id as string, 10);
+  if (isNaN(id)) { res.status(400).json({ error: 'Invalid id' }); return; }
+  const parsed = AutoHealPolicyUpdateSchema.safeParse(req.body);
+  if (!parsed.success) {
+    res.status(400).json({ error: parsed.error.issues[0]?.message ?? 'Invalid input' });
+    return;
+  }
+  try {
+    const db = DatabaseService.getInstance();
+    if (!db.getAutoHealPolicy(id)) { res.status(404).json({ error: 'Policy not found' }); return; }
+    db.updateAutoHealPolicy(id, parsed.data);
+    res.json(db.getAutoHealPolicy(id));
+  } catch (err) {
+    console.error('[AutoHeal] Failed to update policy:', err instanceof Error ? err.message : err);
+    res.status(500).json({ error: 'Internal server error' });
+  }
+});
+
+app.delete('/api/auto-heal/policies/:id', authMiddleware, (req: Request, res: Response): void => {
+  if (!requireAdmin(req, res)) return;
+  if (!requirePaid(req, res)) return;
+  const id = parseInt(req.params.id as string, 10);
+  if (isNaN(id)) { res.status(400).json({ error: 'Invalid id' }); return; }
+  try {
+    const db = DatabaseService.getInstance();
+    if (!db.getAutoHealPolicy(id)) { res.status(404).json({ error: 'Policy not found' }); return; }
+    db.deleteAutoHealPolicy(id);
+    res.json({ success: true });
+  } catch (err) {
+    console.error('[AutoHeal] Failed to delete policy:', err instanceof Error ? err.message : err);
+    res.status(500).json({ error: 'Internal server error' });
+  }
+});
+
+app.get('/api/auto-heal/policies/:id/history', authMiddleware, (req: Request, res: Response): void => {
+  if (!requirePaid(req, res)) return;
+  const id = parseInt(req.params.id as string, 10);
+  if (isNaN(id)) { res.status(400).json({ error: 'Invalid id' }); return; }
+  const limit = Math.min(parseInt(String(req.query.limit ?? '50'), 10) || 50, 100);
+  try {
+    res.json(DatabaseService.getInstance().getAutoHealHistory(id, limit));
+  } catch (err) {
+    console.error('[AutoHeal] Failed to fetch history:', err instanceof Error ? err.message : err);
+    res.status(500).json({ error: 'Internal server error' });
+  }
+});
+
 app.get('/api/notifications', authMiddleware, async (req: Request, res: Response) => {
   try {
     const history = DatabaseService.getInstance().getNotificationHistory();
@@ -8535,6 +8640,7 @@ async function startServer() {
 
   // Start Background Watchdog
   MonitorService.getInstance().start();
+  AutoHealService.getInstance().start();
 
   // Start Docker Event Stream (causal crash/OOM/health detection per local node)
   await DockerEventManager.getInstance().start();
@@ -8609,6 +8715,7 @@ const gracefulShutdown = (signal: string) => {
     try { MonitorService.getInstance().stop(); } catch (e) {
       console.warn('[Shutdown] MonitorService cleanup failed:', (e as Error).message);
     }
+    try { AutoHealService.getInstance().stop(); } catch (e) { console.warn('[Shutdown] AutoHealService cleanup failed:', (e as Error).message); }
     try { DockerEventManager.getInstance().stop(); } catch (e) {
       console.warn('[Shutdown] DockerEventManager cleanup failed:', (e as Error).message);
     }