11 Commits

Author	SHA1	Message	Date
Anso	63213c0960	feat: add service-scoped Compose update and restore (#1648) ... * feat: add service-scoped Compose update and restore Allow updating or rebuilding one declared Compose service on multi-service stacks without recreating siblings, with recovery snapshots, health-gate observation, and prune holds for rollback images. Full-stack update paths and single-service UX stay unchanged. * fix: sanitize service-scoped update log messages for CodeQL * fix: address service-scoped update audit findings B-01 through B-07 * fix: complete service-scoped update audit metadata and surfaces * test: wrap Updates readiness tests for deploy-feedback context * fix: keep service recovery reachable without Deploy Progress Make failed service-gate recovery discoverable when Deploy Progress is disabled or dismissed, suppress stale image-scan notification side effects, normalize ComposeService line endings, and add focused regression coverage. * fix: resurface ContainersHealth density and expand on multi-service stacks Service grouping hid the summary strip and Compact/Detailed/Expand controls that still applied to multi-container stacks.	2026-07-19 02:42:29 -04:00
Anso	f794702171	feat(security): action-posture Security dashboard with exploit intel and triage (#1424) ... * feat(security): reframe masthead as action posture, not worst-CVE severity Derive the Security masthead from an action posture (Action needed / Monitoring / Secure / Unknown) instead of raw scanner severity, and label the raw Critical/High counts as scanner detections. "Secure" now means nothing is actionable right now, never a claim that no vulnerabilities exist; Unknown covers a missing scanner or a node with no completed scan. Phase-1 bootstrap: "actionable" is approximated from the overview facts that already exist (fixable findings, secrets, misconfigs); a later phase moves the bucketing to the backend. * feat(security): derive overview action posture from triaged facts Add deriveSecurityPosture as the single bucketing function and extend /security/overview with posture facts (fixableCriticalHigh, dangerousCompose, accepted, rawCritical/rawHigh, plus knownExploited/publiclyExposed placeholders that later phases populate) and the derived posture verb. Suppression- and acknowledgement-aware counts come from one bounded read-time pass over the latest-scan Critical/High findings, grouped per image so the existing read-time filters apply unchanged. The pass is capped and flags posturePartial, so a large node degrades gracefully instead of scanning every detail row. The masthead now prefers the backend posture and keeps the local bootstrap only as a fallback for older remote nodes reached through the proxy. * feat(security): capture Trivy finding enrichment (status, CVSS, vendor, purl, layer) parseTrivyOutput now keeps the per-finding fields Trivy already returns and we previously discarded: Status (fixed / will_not_fix / end_of_life / ...), CVSS (score + vector, preferring the NVD source then falling back), vendor severity, package URL, package path, and layer digest. Persisted on vulnerability_details via additive nullable columns (guarded ALTER), bound null when absent, and carried through the cached-scan reconstruction path. These fields separate scary from exploitable and feed the action posture and the per-finding evidence tags. Field paths verified against Trivy's documented image-scan JSON; covered by parse and insert/read round-trip tests. * feat(security): add CVE exploit-intel service (CISA KEV + FIRST EPSS) Add CveIntelService, a daily background cache of CISA KEV membership and FIRST EPSS scores stored in a new cve_intel table and joined to findings at read time by CVE id (never frozen onto scan rows, so a CVE entering KEV later lights up on scans already stored). EPSS is fetched only for CVE ids present in stored findings, batched; both feeds are best-effort and keep the last cache on failure, so the Security page degrades gracefully offline. Wired into startup/shutdown like the other background services. The overview now counts known-exploited Critical/High findings, and KEV membership escalates posture to Action needed even when no fix is available. A per-instance "Exploit intelligence" toggle on the scanner setup surface lets air-gapped or firewalled hosts disable the outbound fetch; the daily tick keeps running but skips the fetch body when it is off. * feat(security): show per-finding evidence tags (KEV, EPSS, vendor status, CVSS) The vulnerabilities endpoint joins read-time exploit intel (KEV membership and EPSS score) onto each finding by CVE id, and the scan sheet renders evidence tags beside each CVE: known-exploited, EPSS probability, vendor will-not-fix / end-of-life, and the CVSS score. Severity becomes one signal among several so an operator can tell scary from exploitable, with no invented composite score. * feat(security): evolve CVE suppressions into triage decisions Layer a triage status and optional OpenVEX justification onto CVE suppressions. Statuses: needs review / affected / not affected / accepted risk / fixed / false positive / ignored. Dismissing states (not affected, accepted, fixed, false positive, ignored) stop a finding from driving the action posture; needs review and affected stay actionable and are surfaced as counts. Existing rows default to "accepted" (the prior suppress behavior), so nothing changes for them. The overview now reports needsReview / notAffected / accepted as distinct facts derived from the triage status. The decision replicates across the fleet (snapshot + replicated-insert carry status + justification) so a replica's posture matches the control node. The inline suppress dialog gains a triage decision selector; the read-time filter surfaces the status and justification on every finding. * feat(security): export fleet triage decisions as OpenVEX (Admiral) Add an OpenVEX exporter that turns the instance's CVE triage decisions into a standard VEX document (not_affected / fixed / affected / under_investigation, with justifications), and a GET /security/vex/export endpoint to download it. Authoring fleet VEX is a governance capability, so it is gated to Admiral (paid) plus admin, mirroring the SARIF export gate; the Suppressions panel shows an Export VEX action only on Admiral. * docs(security): document action posture, evidence tags, exploit intel, and triage Update the Security page and CVE suppressions docs for the action-posture masthead (scanner detections vs product posture), per-finding evidence tags (KEV / EPSS / CVSS / vendor status), the exploit-intelligence toggle (CISA KEV + FIRST EPSS) on scanner setup, triage decisions layered on suppressions, and OpenVEX export of fleet triage decisions. * test(security): match intel hosts exactly in CveIntelService test Route the fetch stub and its call assertions by exact hostname (www.cisa.gov / api.first.org) instead of a domain substring check. Resolves the js/incomplete-url-substring-sanitization code-scanning alerts on the test's URL routing; behavior is unchanged.	2026-06-23 17:42:11 -04:00
Anso	38aabe7064	feat: health-gated updates and rollback readiness (#1354) ... * feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.	2026-06-11 00:26:26 -04:00
Anso	6b728599c4	fix(mesh): persist PilotMetrics counters across central restart (#1078) ... The 12 PilotMetrics counters (proxy_dials_failed, proxy_bridges_total, proxy_idle_closes, the pilot-tunnel counters, and the peer-to-central callback counters) live in a singleton holding scalar fields, so process restart wipes them. Operators investigating rotation- or dial-failure trends across a central restart lost the aggregate metric. Persist the snapshot under a pilot_metrics_counters JSON row on the existing system_state KV table. Hydrate via PilotMetrics.load() in bootstrap/startup.ts before any service that increments runs; final-flush in bootstrap/shutdown.ts before the SQLite handle closes. Between those boundaries a buffered flush mirrors the audit-log pattern: every 1 s or every 100 increments, whichever fires first. Hot increment path stays a single property write plus a pending-counter check. Defensive parse on the row: object check + numeric filter, so an operator hand-edit or schema drift across releases degrades to zero state with a warn rather than crashing boot. Schema additions back-fill missing keys with zero so a release that adds a counter does not need a migration. Closes F-R-4 from the mesh tracker.	2026-05-17 01:55:17 -04:00
Anso	33b15d6cba	feat(fleet-sync): retry failed pushes and backfill on add-node (#970) ... A control instance now retries fleet-sync pushes that hit a transient failure and backfills the security state on a freshly registered remote without waiting for the next policy edit. New service: - FleetSyncRetryService (singleton, start/stop) wakes 30s after boot and ticks every 5min. For each fleet resource, queries getFailedSyncTargets within a 24h window and re-pushes via FleetSyncService.pushResourceToNode through the same per-node mutex, so a normal fanout in flight serializes naturally with a retry. - After STALE_THRESHOLD_MS (1h) of continuous failure for a previously-working node, dispatches a single warning notification per cooldown window. Brand-new nodes that have never succeeded do not alert via this path; misconfigured remotes are caught by the test-connection affordance at registration time. - Wired into bootstrap startup/shutdown next to AutoHealService. Public surface: - FleetSyncService.pushResourceToNode(node, resource): targeted push to one node that re-uses the per-node mutex. Used by the retry service and any future targeted-resync flow. - routes/nodes.ts POST /api/nodes fires pushResourceAsync for both resources after a remote-proxy node row commits. Tuning constants centralized in fleetSyncConstants.ts: - RETRY_MAX_AGE_MS = 24h - STALE_THRESHOLD_MS = 1h Tests: - 8 vitest cases covering replica skip, retry dispatch, missing-node skip, alert-once-per-cooldown across the threshold window, no-alert for recent failures, no-alert for brand-new never-succeeded nodes, no-alert when the retry itself succeeds, start/stop idempotency. - Full backend suite: 1781 pass / 5 skipped.	2026-05-07 13:16:24 -04:00
Anso	e5b1c7b22b	refactor(backend): collapse entitlement provider abstraction back to LicenseService (#889) ... Removes backend/src/entitlements/ (registry, loadProvider, CommunityEntitlementProvider, types, headers, normalize) and the two abstraction-only tests. Relocates headers/normalize/types to services/license-*.ts. Swaps 22 consumer call sites from getEntitlementProvider() to LicenseService.getInstance(). Drops the Dockerfile install step plus PRO_PACKAGE_VERSION build-arg and github_token BuildKit secret in docker-publish.yml. Removes the now stale no-restricted-imports rule in backend/eslint.config.mjs. Net: 37 files changed, ~700 lines removed, no behavior change. Local dev no longer requires GitHub Packages auth to start the backend. Rationale and revisit conditions in docs/internal/adrs/2026-05-02-collapse-entitlement-provider.md.	2026-05-02 23:45:44 -04:00
Anso	3324616e59	refactor(backend): extract EntitlementProvider abstraction (Phase 1) (#878) ... * refactor(backend): extract EntitlementProvider abstraction (Phase 1) Phase 1 of the open-core hybrid extraction described in docs/internal/adrs/2026-05-02-open-core-hybrid-strategy.md. Introduces the abstraction without moving any code out of the public repo; Phase 2 will actually move services/LicenseService.ts to a private @studio-saelix/sencho-pro package. The new backend/src/entitlements/ module contains: - types.ts. The EntitlementProvider interface plus all tier/license types (LicenseTier, LicenseVariant, LicenseInfo, SeatLimits, ActivationResult, etc.). The interface mirrors the existing LicenseService public surface so the migration was mechanical. - registry.ts. Module-scope holder for the active provider with setEntitlementProvider, getEntitlementProvider, and a test-only reset helper. getEntitlementProvider throws if called before bootstrap registers a provider; the throw is intentional fail-fast on a bootstrap-order bug rather than a silent degradation. - CommunityEntitlementProvider.ts. Phase 2 fallback that returns community tier and rejects activate(). NOT instantiated in production today; a smoke test keeps it covered against bitrot. - loadProvider.ts. Async resolver. Phase 1 returns LicenseService.getInstance() directly. The async signature matches what Phase 2 needs (dynamic import of @studio-saelix/sencho-pro with a "module not found" vs "construction threw" narrowing); the call site does not change between phases. - headers.ts. PROXY_TIER_HEADER and PROXY_VARIANT_HEADER constants. These are part of the wire contract between Sencho instances and belong in the public core regardless of which entitlement provider is bound. - normalize.ts. isLicenseTier, isLicenseVariant, normalizeTier, normalizeVariant. Domain knowledge about Sencho's tier model (legacy name maps from pre-0.38.1 versions), not LemonSqueezy internals. Phase 2 keeps these in the public core. services/LicenseService.ts now imports its types from entitlements/types and adds an "implements EntitlementProvider" clause. Re-exports the types for back-compat with ~20 type-only consumers; a follow-up PR will sweep those imports to entitlements/ types directly before Phase 2 deletes the file. bootstrap/startup.ts awaits loadEntitlementProvider, registers the result, then calls initialize. shutdown.ts calls getEntitlementProvider().destroy() instead of the LicenseService singleton. middleware/tierGates.ts, the chokepoint for ~154 tier-check call sites, now reads through getEntitlementProvider. Sixteen other production files (routes/{fleet,imageUpdates,license,permissions, scheduledTasks,security,stacks,templates,users,webhooks}, services/{BlueprintService,CloudBackupService,SchedulerService, SSOService}, proxy/remoteNodeProxy, websocket/{hostConsole, remoteForwarder}, middleware/auth) had their LicenseService.getInstance calls and utility-export imports redirected to the entitlements module. The only remaining LicenseService.getInstance in production code is in entitlements/loadProvider.ts itself, which is the intentional Phase-1 binding site. Test infrastructure: setupTestDb registers LicenseService.getInstance() as the active provider so existing test files using the helper need no changes. The mocking pattern many tests use, vi.spyOn(LicenseService.getInstance(), 'getTier'), keeps working because LicenseService.getInstance() and getEntitlementProvider() return the same singleton in Phase 1. scheduler-service.test.ts is the only test that does not use setupTestDb but exercises tier-gating; it now mocks entitlements/registry alongside its existing LicenseService mock. Adds a smoke test for CommunityEntitlementProvider so the Phase 2 fallback class stays covered. Adds an architecture doc at docs/internal/architecture/entitlement-provider.md covering the runtime registry, bootstrap order invariants, and the Phase 1 vs Phase 2 binding table. Test results: 89/89 backend test files pass, 1657 passing tests, 5 pre-existing skips. The pre-existing database-metrics > handles 1000+ metrics stress test continues to flake under parallel load and pass when re-run solo, same flake observed in PRs #862, #863. * chore(backend): drop unused entitlement type imports from LicenseService Phase 1 of the EntitlementProvider extraction left five type imports (ActivationResult, BillingPortalError, BillingPortalResult, DeactivationResult, ValidationResult) unreferenced after the runtime methods that produced them began inferring their result shapes via the EntitlementProvider interface contract. ESLint's no-unused-vars rule flagged them as errors and failed the lint step in CI.	2026-05-02 05:07:00 -04:00
Anso	685d5d729e	feat(blueprints): backend foundation for fleet-wide compose templates (#860) ... * feat(blueprints): add backend foundation for fleet-wide compose templates Introduces the Blueprint Model: a docker-compose.yml plus a node selector (labels or explicit IDs) that Sencho reconciles across the fleet. Backend foundation only; the frontend tab and documentation follow. Schema (DatabaseService): - node_labels table for fleet-level orchestration tagging - blueprints table with compose content, selector, drift_mode, classification - blueprint_deployments table for per-node materialized state - New idempotent migrate methods following the existing pattern Services: - BlueprintAnalyzer: pure compose-YAML classifier (stateless / stateful / unknown) with 17 covered cases including named volumes, bind mounts, external volumes, and tmpfs - NodeLabelService: label CRUD plus selector matching helper (any/all/ids) - BlueprintService: local + remote deploy/withdraw orchestration, marker file management, name-conflict guard, per-(blueprint,node) lock - BlueprintReconciler: 60-second loop with three-mode drift policy (observe/suggest/enforce), state-aware guards, and Enforce-downgrade for volume-destroying drift Routes (gated requirePaid + requireAdmin on mutations): - /api/blueprints (CRUD + apply + withdraw + accept + preview + analyze) - /api/node-labels (CRUD + listAll + listDistinct) Notifications: four new categories registered in NotificationService for deploy/failure/drift events. Bootstrap: reconciler start/stop wired in startup and shutdown. Tests: 45 new Vitest cases covering selector matching, classifier rules, state-aware guards, drift-mode branching, and marker parsing. Full backend suite (1625 tests) passes; tsc clean. * fix(lint): replace bare Function type in blueprint reconciler tests Replace 8 occurrences of `as unknown as { computeDecision: Function }` with a properly typed `ReconcilerWithCompute` alias that mirrors the real method signature. Export `ReconcileDecision` from BlueprintReconciler so the test can reference it. Resolves @typescript-eslint/no-unsafe-function-type errors that were failing the Backend (Lint) CI step.	2026-05-01 18:57:44 -04:00
Anso	7663f4cd8b	feat(fleet): sencho mesh in traffic and routing tab (#858) ... * feat(fleet): sencho mesh in traffic and routing tab Lights up Sencho Mesh: cross-node container forwarding rendered as if the container next to you were on localhost. Builds on the dormant TCP frame plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar package) and exposes the Admiral-only orchestrator surface. Backend - New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column via DatabaseService.migrateMeshTables. - MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with cascading override regeneration, request-based resolver from sidecar control WS, cross-node TCP forwarding via PilotTunnelManager (same-node fast path included), in-memory 1000-event activity ring buffer with durable mirror to audit_log for state-change events, per-node and per-route diagnostics, and the Test upstream probe. - MeshComposeOverride: pure YAML generator that injects extra_hosts using host-gateway. The user's docker-compose.yml is never mutated; overrides live under DATA_DIR/mesh/overrides. - ComposeService deploy/update splice the override file when the stack is opted in; non-mesh stacks behave identically to today. - Pilot agent resolveMeshTarget consults the local mesh_stacks table (defense in depth) and resolves Compose containers via Dockerode. - /api/mesh router with 13 Admiral-gated endpoints covering status, enable/disable, stack opt-in/out, alias listing, per-route diagnostic, Test upstream probe, per-node diagnostic, sidecar restart, activity log paginated and SSE. - meshControl WS slot at /api/mesh/control validates the mesh_sidecar JWT minted by MeshService; dispatched as upgrade slot 2 (canonical order preserved). Frontend - New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state typography from the audit. - RoutingTab masthead with mesh activity drawer, per-node card grid with TogglePill, alias rows with five-state pill taxonomy (healthy / degraded / unreachable / tunnel-down / not-authorized), inline Test buttons. - Four sheets: opt-in picker with port-collision inline error, per-route detail with diagnostic + filtered activity, per-node diagnostics with active streams + resolver cache + restart action, fleet-wide activity log with filters. - meshRouteState helper centralizes pill-state mapping; pure-function tests cover all five states. Docs - User docs at /docs/features/sencho-mesh.mdx covering opt-in, troubleshooting, security model (4 guarantees + 4 explicit non-guarantees), and V1 limitations. - Internal architecture and runbook pages. - websocket-dispatch internal doc updated with the new slot. * fix(mesh): validate stack name before path use; fix test DB lifecycle Two surgical fixes against the prior PR. Path-injection (CodeQL js/path-injection): MeshService.optInStack, optOutStack, ensureStackOverride, and removeStackOverride now validate stackName via isValidStackName from utils/validation, reject malicious names at the API boundary, and additionally check isPathWithinBase on the resolved override file path for defense in depth. The dataflow from req.params.stackName to fs.writeFile no longer reaches an unsanitized path expression. Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb / cleanupTestDb, which deletes the temp dir while DatabaseService still holds an open SQLite handle. On Linux CI this raises SQLITE_READONLY_DBMOVED on the next prepare() because the inode has been unlinked. Switched to file-scoped beforeAll/afterAll matching agents-routes.test.ts, with a per-test beforeEach that truncates mesh_stacks plus non-default nodes and resets the MeshService singleton in-memory state. Adds a new test case asserting the path-traversal rejection. * fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho writes its canonical compose file as `compose.yaml`, so any stack created via the UI failed to deploy with `open ...docker-compose.yml: no such file or directory`. When no mesh override applies, drop the explicit `-f` so docker compose's built-in discovery resolves the actual filename. When an override exists, look up the real base filename via FileSystemService.getComposeFilename() and pass both files explicitly. Also hoist the MeshService import to module top now that the dependency is known to be acyclic, and revert the matching unit-test assertion.	2026-05-01 01:50:53 -04:00
Anso	5cf4323511	perf(backend): batch audit_log inserts into a buffered transaction (#817) ... Every mutating /api/* request runs an individual INSERT into audit_log which serializes against other writers under burst load (SQLite's single-writer model). Buffer the writes in DatabaseService and flush them in a single transaction either every second or once the buffer reaches 100 entries, whichever comes first. Read paths (getAuditLogs, getAuditLogsInRange, cleanupOldAuditLogs) drain the buffer first so callers always see a consistent view, which keeps the existing test pattern of insert-then-read working. Graceful shutdown flushes before db.close() so no entries are lost on clean exit. The 1s flush timer is unref'd so the buffer cannot keep the process alive on its own. The CLI resetMfa script flushes explicitly before returning since it exits before the timer fires.	2026-04-28 00:59:04 -04:00
Anso	e9fce15010	refactor(backend): extract bootstrap into startup/shutdown modules (phase 5) (#745) ... Move the startup and shutdown lifecycles out of index.ts: - bootstrap/startup.ts exports startServer(server) - migration check, service initialization, background watchdogs, HTTP listen, pilot-agent loopback bind. - bootstrap/shutdown.ts exports installShutdownHandlers(server) - SIGTERM/SIGINT handlers, in-order service stop chain, 10s force-exit guard, SQLite close. Restructure MfaService to add an instance + lifecycle so the replay purge timer no longer lives as a module-scope setInterval in index.ts. MfaService keeps all existing static methods (generateSecret, verifyTotp, currentWindow, generateBackupCodes, hashBackupCodes, verifyBackupCode, formatBackupCodeForDisplay, normalizeBackupCode, buildOtpauthUri) so every existing caller stays unchanged. The new start() / stop() pair is idempotent and calls .unref() so test shutdown is not blocked. bootstrap/startup calls MfaService.getInstance().start(). bootstrap/shutdown calls MfaService.getInstance().stop(). index.ts drops from 305 to 147 lines and now contains only the Express app composition: createApp, route mounts, remote proxy, createServer, attachUpgrade, static/SPA fallback, errorHandler, installShutdownHandlers, and the require.main guard that boots the server when run directly. Behavior is byte-for-byte identical: shutdown service order, log strings, force-exit timer, pilot-agent loopback logic, and the MFA purge cadence and debug logging all preserved verbatim.	2026-04-23 23:44:00 -04:00