mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 00:47:52 +00:00
716daf77d083fc2270366d7ef1be0a59424a83d2
106 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
716daf77d0 |
feat(updates): auto-prune dangling images after updates (#1316)
* feat(updates): auto-prune dangling images after updates Each update pulls a fresh image and recreates containers, leaving the replaced image behind as a dangling layer that previously had to be pruned by hand. A new "Prune dangling images after updates" toggle under Settings > System > Docker hygiene reclaims these automatically. The setting is on by default and opt-out. When enabled, a successful stack update (manual or scheduled) and a Sencho self-update each remove the dangling image layers they orphaned. Only untagged layers are touched; tagged images, volumes, and data are never removed. The toggle requires an admin account and is per node: each instance honors its own value, so a remote node self-update applies that node's own preference. A prune failure never affects the update result: on the stack path it is caught and logged after the update has already succeeded, and on the self-update path the helper-shell prune runs only after a clean recreate and cannot change the exit code or the recorded update error. * security(self-update): shell-quote label-derived values in helper command Address review feedback on the prune-on-update change: - The self-update helper command interpolated the compose service name and config-file paths (both read from Docker Compose labels) straight into a shell string. Shell-quote them via shQuote so a label carrying shell metacharacters stays inert data and cannot break the exit-code capture, error-file write, or prune guard. - Correct the settings copy and docs: the prune is a standard dangling-image prune, so it reclaims every untagged layer on the node, not only the one the current update orphaned. Tagged images, volumes, and data remain untouched. - Add tests: shell-metacharacter neutralization and prune-output suppression in the self-update command, and an atomic-update case asserting a prune failure does not trigger a rollback. * fix(updates): omit the reclaim figure when the daemon reports zero bytes End-to-end testing on a Docker daemon backed by the containerd image store showed the post-update prune removing a dangling image while the prune API returned SpaceReclaimed=0, so the stream printed "reclaimed 0.0 MB" even though an image was removed. Show the reclaimed figure only when the daemon reports a non-zero value; otherwise the line reads "=== Pruned dangling images ===". The overlay2 store still reports real figures and shows them. Add a test covering both branches. |
||
|
Anso
|
865d792874 |
feat(pricing): collapse to two tiers (#1309)
* feat(pricing): collapse to two tiers (Community + Admiral) Collapse Sencho's pricing from three tiers (Community / Skipper / Admiral) to two: a generous free Community tier and a single paid Admiral tier. The Skipper tier is removed. Now free in Community: auto-heal, auto-update, scheduled operations, webhooks, notification routing, Fleet Actions and bulk operations, SSO preset providers (Google / GitHub / Okta), unlimited users with admin and viewer roles, and deploy safety (atomic deploys, auto-rollback, and one-click rollback). Admiral (paid) is focused on running and governing a fleet: blueprints, Fleet Secrets, deploy enforcement, vulnerability report export, audit log, host console, private registries, mesh networking, node cordon, managed cloud backup, LDAP / Active Directory SSO, and the advanced RBAC roles (deployer, node-admin, auditor) with per-resource scoped assignments. Internally the license variant distinction is removed so tier is binary (community / paid). License validation still verifies the Lemon Squeezy store and product before granting paid status. Docs and the contributor guide are updated to the two-tier model. * docs(pricing): correct licensing page to two-tier pricing and tidy stale tier wording The licensing docs page kept the old Admiral pricing plus a Founder Lifetime column and an Enterprise paragraph after the two-tier collapse. Update it to $12/month or $99/year, drop the lifetime and Enterprise content, and link to the pricing page for current pricing. Also fix stale "Skipper" wording in CLA.md, SUPPORT.md, one test title, and three test comments. Historical CHANGELOG entries and the retired-Skipper license-guard test are intentionally left as-is. * docs: align licensing and SSO pages with the two-tier model Correct the SSO overview so the Google, GitHub, and Okta presets read as available on every tier, matching the provider table; only LDAP and Active Directory require Sencho Admiral. Remove the lifetime-plan references from the licensing, settings, and troubleshooting pages so they reflect subscription-only Admiral pricing. * fix(rbac): omit scoped permissions from /me on the Community tier Scoped role assignments only take effect on the paid tier, but GET /api/permissions/me returned them unconditionally, so a downgraded instance with leftover assignments rendered per-resource affordances the API then rejected with 403. The endpoint now mirrors the permission middleware and includes scoped permissions only on the paid tier. Adds a regression test covering the downgrade case. * docs: use custom-pricing wording on the contact page The two-tier model has no Enterprise tier; reword the contact page's enterprise pricing/deals to custom pricing/deals so it does not imply a tier that no longer exists. |
||
|
Anso
|
c6d1631afe |
feat(recovery): add safe-mode recovery surface and emergency CLI (#1286)
* feat(recovery): add safe-mode recovery surface and emergency CLI Add a read-only Recovery tab under Settings (admin-only) backed by a new GET /api/diagnostics endpoint reporting app version, database integrity, encryption-key status, Docker reachability, account and SSO counts, and non-secret configuration. The endpoint loads without Docker or live metrics so it stays available when the dashboard does not, requires a genuine admin session, and builds its config block from a non-secret allowlist so no credentials are ever exposed. Expand the emergency command-line toolkit beyond the two-factor reset with seven host-level commands: reset-password, create-emergency-admin, clear-sessions, disable-sso, diagnostics, validate-db, and backup-data. Each prints its result, exits with a meaningful status code, and writes an audit entry where it changes state. Document the toolkit in a new operator guide and link it from the recovery and two-factor pages. * feat(recovery): download the emergency command reference as a text file The recovery commands are needed exactly when the dashboard is unreachable, so reading them only in-app is a chicken-and-egg problem. Add a Download button to the command-line section that saves the full `docker compose exec sencho ...` reference as a text file, letting operators keep it on hand before they need it. Reuses a shared download helper with the existing diagnostics export. * fix(recovery): harden diagnostics, backup, and emergency-admin against edge cases Address findings from an independent review of the recovery toolkit: - DiagnosticsService now degrades instead of throwing when a queried table is missing or corrupt: each read falls back and is folded into database.ok, so a broken database reports "problem detected" rather than failing the whole endpoint or showing a misleading healthy state with zeroed counts. - backup-data refuses a destination that resolves to the live database, which would otherwise report success while producing no separate copy. - create-emergency-admin now applies the same username rule as the user- management route, extracted to a shared helper so both stay in sync. Adds tests for a missing read table, a malformed emergency-admin username, and the backup same-target rejection. |
||
|
Anso
|
c11a550b6a |
fix(fleet-snapshots): gate reads on admin role and encrypt content at rest (#1273)
* fix(fleet-snapshots): gate reads on admin role and encrypt content at rest Fleet snapshots capture every node's compose.yaml and .env, so the data is as sensitive as the live stacks. This hardens access and reliability across the snapshot pipeline. - Restrict snapshot reads to administrators. GET /api/fleet/snapshots and /:id now require the admin role, matching create, restore, and delete; the Fleet "Snapshots" tab and its panel render only for admins. Previously any authenticated user could enumerate snapshots and read every node's .env. - Encrypt snapshot file contents at rest with the instance key. Restore and cloud-archive paths decrypt on read, so cloud archives stay portable and a database copy no longer exposes stack secrets in plaintext. Rows written before this change still read back as plaintext. - Surface partial captures. A stack whose compose file cannot be read or fetched, or a file over the 1 MB capture cap, is recorded as a warning and shown on the snapshot instead of being silently dropped, so a snapshot is never mistaken for complete. Remote .env read errors are now distinguished from a genuinely absent .env. Adds route-authz, capture-warning, and encryption round-trip tests; updates the Fleet-Wide Backups feature docs. * fix(fleet-snapshots): gate cloud snapshot reads on admin role The cloud snapshot read routes were guarded by provider/license only, not by role, while their write counterparts (upload, delete) already required admin and the Cloud Backup settings surface is admin-only. Because a downloaded archive contains plaintext compose and .env files, a non-admin could list and download cloud snapshots and read every node's secrets, the same exposure the local snapshot reads were just closed against. - Require admin on GET /api/cloud-backup/snapshots, /status/:id, and /object/:keyB64/download, matching the local snapshot reads and the admin-only Cloud Backup settings section. - When capturing a remote node, treat a 200 response carrying X-Env-Exists: false as a stack with no .env (matching the local ENOENT path) instead of storing an empty .env that restore would later write back. Adds non-admin authorization tests for the cloud read routes and a remote absent-.env capture test. |
||
|
Anso
|
085267b466 |
feat(security): make CVE suppressions optionally honored by deploy-block policies (#1269)
* feat(security): make CVE suppressions optionally honored by deploy-block policies
Block-on-deploy policies evaluate the raw scan result, so a CVE an admin
has accepted in CVE Suppressions still blocks the deploy. Add an opt-in,
per-instance toggle ("Honor suppressions in deploy blocks", Settings ->
Security) that, when on, re-derives each image's severity from the
suppression-filtered findings before comparing to the policy threshold. A
deploy that proceeds only because suppressions dropped it below the gate is
recorded in the audit log. Default off, so the strict raw-scan behavior is
unchanged unless an operator enables it.
The setting governs the instance that runs the deploy and is not
fleet-replicated. The gate fails safe: a suppression-read error or an
empty detail set falls back to raw scan severity rather than dropping it.
Also surface a previously swallowed error in the CVE suppressions and
misconfig acknowledgement settings panels so a failed list load shows a
toast instead of an empty list.
* fix(security): gate on raw severity when preflight detail rows are truncated
The suppression-aware deploy gate re-derived image severity from the stored
vulnerability_details rows, assuming any non-empty set was complete. A cached
pre-deploy scan keeps the full aggregate counts but copies only a bounded slice
of detail rows, so recomputing from that slice could drop an unsuppressed
blocking CVE below the threshold and let a deploy through.
Guard the recompute: when the loaded detail rows do not match the scan's total
finding count, gate on the raw scan severity (never drops severity). Suppression
awareness still applies for scans whose details are stored in full, which is the
common case.
|
||
|
Anso
|
b61388c675 |
fix(rbac): enforce admin seat cap on promotion and harden last-admin and audit paths (#1266)
Promoting a user to admin now respects the per-tier admin seat limit the same way user creation does, closing a path that let an operator exceed the cap by creating an account and then editing its role to admin. The last-admin guard for demote and delete now runs the admin-count re-check and the write in a single transaction, so two concurrent admin changes can no longer race the admin count to zero and lock everyone out. Admin two-factor resets are now recorded once with their own audit summary instead of being mislabeled as a user creation by the audit middleware. |
||
|
Anso
|
7e65a2ae19 |
fix(mfa): enforce single-use backup codes under concurrent verification (#1262)
* fix(mfa): enforce single-use backup codes under concurrent verification Backup-code consumption read the stored hash set, awaited bcrypt.compare, then wrote the shrunk set back. Two concurrent /login/mfa requests carrying the same code could both read the same set, both match, and both persist, so a single backup code yielded two authenticated sessions. Move consumption into a synchronous transaction: verifyBackupCode now returns the matched hash without mutating state, and a new consumeBackupCodeHash re-reads and shrinks the set atomically, returning whether the hash was still present. Login gates success on that result, so exactly one concurrent request wins. Reshape the verify result into a discriminated union so a match always carries its hash. Add coverage: a deterministic consume test (same hash twice, distinct hashes, absent hash), a concurrent same-code login race, backup-code exhaustion, and a disable-via-backup-code path. * test(mfa): make the concurrent backup-code test deterministic The race test relied on incidental scheduling to interleave the two requests, so it could false-green if they happened to serialize. Add a barrier that holds both requests just after verification (once both have read the same stored set) until both arrive, then releases them into the atomic consume. This forces the race every run, so the test fails against a non-atomic consume and passes only when exactly one request wins. A timeout releases the barrier if only one request arrives, so a setup fault fails loudly instead of hanging. |
||
|
Anso
|
5e66b54153 |
fix(audit-log): neutralize CSV export injection, clamp pagination, bound anomaly history (#1259)
* fix(audit-log): neutralize CSV export injection, clamp pagination, bound anomaly history Harden the Admiral audit log without changing its tier or hub-only gating. - CSV export now defuses formula injection: any field that a spreadsheet would evaluate as a formula (leading = + - @, or a trigger behind leading whitespace, or a leading tab/CR) is prefixed with a single quote before RFC 4180 quoting. Audit summaries embed user-controlled resource names, so this closes a path where a crafted name could execute on export open. - Clamp page and limit to positive bounds on the list endpoint so a negative limit can no longer reach SQLite as "unlimited" and dump the whole table. - Bound the anomaly and stats history reads to a capped slice of recent rows so the analysis paths stay within fixed memory and latency on large histories instead of scanning the full retention window per request. - Surface failed audit list and stats fetches through the standard error toast instead of leaving the view silently stale. - Add a developer-mode-gated diagnostic log to the stats endpoint for parity with the list and export handlers. Covered by new unit and HTTP-integration tests (CSV neutralization through the real export route, pagination clamps, bounded history, stats endpoint, anomaly annotation) and verified end to end in the browser. * fix(audit-log): compute signal-rail stats with exact SQL aggregates Address review feedback on the earlier history-cap change. The cap was correct for the anomaly baseline but made the stats tiles (events, actors, failure rate, hourly series) silently undercount on a hub with more than the cap's worth of rows in the window, since they were derived from the capped row slice. - Add DatabaseService.getAuditStatsInputs: exact counts via SQL COUNT / COUNT(DISTINCT) / GROUP BY hour, and new-ip detection over the small DISTINCT (user, ip) pair sets. No row cap, so the tiles stay exact at any window size while memory stays bounded. - Reduce computeAuditStats to a pure formatter over those aggregates. - Keep the bounded history read only for the list endpoint's anomaly annotation, where a recent-activity baseline is an acceptable heuristic. - Skip the redundant load-failure toast when a fetch fails with a handled 401, so an expired session does not stack toasts on top of logout. Adds exactness tests for the aggregate counts, distinct-actor handling, and new-ip detection, and strengthens the pagination-clamp tests. * fix(audit-log): exclude future-dated rows and make the new-ip sample deterministic Two small parity fixes on the stats aggregates: upper-bound every current window by `now` so a future-dated row (clock skew or a fixture) cannot inflate the live counts, and order the new-ip pair scan so the sample actor shown in the tile detail is stable. Adds a test asserting a future row is excluded. |
||
|
Anso
|
265fece988 |
fix(notifications): prevent self-container stack routing (#1242)
* fix(notifications): prevent self-container stack routing * fix(stack-files): stabilize download metrics in CI |
||
|
Anso
|
adcd04b01a |
refactor(auto-update): retire per-stack gate, drive auto-update from schedules only (#1233)
* refactor(auto-update): retire per-stack gate, drive auto-update from schedules only
The per-stack Auto-update toggle in the stack sidebar context menu wrote a
gate row to `stack_auto_update_settings`, but actual updates only ran when a
`scheduled_tasks` row with `action='update'` fired. On a fresh install the
toggle was inert: detection ran every 6h, nothing was applied.
The same context menu already exposes `Schedule task`, which opens
ScheduledOperationsView pre-filled for the stack where the user can pick
`Auto-update Stack` and any cron. Keeping the toggle alongside that flow
duplicated the same action and turned the gate table into a parallel store
of "is a covering schedule active" derivable from `scheduled_tasks` itself.
Drop the gate model entirely:
- Backend: remove the `stack_auto_update_settings` table and its four
accessors, the three routes under /api/stacks/*/auto-update, the per-stack
skip in /api/auto-update/execute and SchedulerService.executeUpdate's
fleet branch, and the clearStackAutoUpdateSetting call on stack delete.
Dashboard `autoUpdate` count derives from scheduled_tasks (action='update'
rows pinned to the node, total/enabled split).
- Frontend: drop the Auto-update entry from the sidebar context menu and its
optimistic toggle plumbing. Drop autoUpdateSettings state, the
/stacks/auto-update-settings fetch, and the auto-update-settings-changed
WebSocket branch. Slim useSidebarActivitySummary (just nextRunAt; no
enabled/total counts). AutoUpdateReadinessView's per-card autoUpdateEnabled
now means "a covering enabled action='update' schedule exists" (per-stack
row or fleet row on this node, earliest next_run_at wins, per-stack row
wins on ties), with the gate-fetch removed.
- New: scheduledTasksRouter broadcasts scope: 'scheduled-tasks' on POST,
PUT, PATCH /toggle, and DELETE so useConfigurationStatus and
useNextAutoUpdateRun refetch under the 250ms debounce instead of waiting
for the 60s poll. The broadcast is wrapped so a broken subscriber socket
cannot turn a successful mutation into a 500.
- Docs: rewrite the "Per-stack control" section of auto-update-policies.mdx
to describe the schedule-based model; update the matching troubleshooting
entry. The misleading fleet-update help text in ScheduledOperationsView
is corrected to reflect that every stack on the node is covered.
Tier parity: the surviving auto-update path (Schedule task -> Auto-update
Stack / All Stacks) is gated `requirePaid + requireAdmin` backend and
`isPaid + isAdmin` frontend, matching the gate the deleted routes carried.
The pre-commit grep returns no tier-related diff outside this PR's scope.
No data migration is provided: greenfield rules apply, and the leftover
table on already-shipped instances is harmless because no code reads or
writes it after this PR.
* docs: sweep remaining references to the per-stack auto-update toggle
The previous commit retired the per-stack Auto-update gate in favor of
configuring auto-update purely through scheduled tasks. This commit
removes the now-stale mentions of that toggle across the operator docs:
- docs/features/sidebar.mdx: drop the Auto-update entry from the Inspect
group description, the matching screenshot alt-text, and the Skipper
Note that listed it. Schedule task now carries the cross-link to
Auto-Update Policies.
- docs/features/stack-management.mdx: drop the Auto-update list item;
refresh the Schedule task entry to mention the Auto-update Stack action.
- docs/features/dashboard.mdx: rename the Configuration Status row from
"Auto-update stacks" to "Auto-update schedules" with the new value
shape, and rewrite the troubleshooting accordion to describe the
scheduled-tasks invalidation path.
- docs/features/scheduled-operations.mdx: rewrite the Auto-update All
Stacks row and helper text to reflect that every stack on the node is
covered (no per-stack opt-out from this surface anymore).
- docs/features/multi-node.mdx: rewrite the Updates column definition to
derive the Auto/Off flag from enabled Auto-update Stack / Auto-update
All Stacks schedules instead of the removed per-stack policy.
The auto-update-policies.mdx rewrite in the previous commit already
covered the main reference page. The sidebar-context-menu.png screenshot
will be refreshed on release once the new menu is live in production;
the alt text is updated in this commit so it accurately describes the
shipping state.
No website edits needed: the Auto-Update Policies feature card description
("Schedule automatic image pulls and redeployments per stack on your own
cadence") and the feature matrix labels ("Auto-update stack schedule",
"Auto-update all stacks schedule") remain accurate under the new model.
* fix(stacks): drop orphaned requireAdmin import after auto-update route removal
CI's backend lint step flagged this PR's earlier deletion of the three
/api/stacks/*/auto-update routes: those handlers were the only callers of
`requireAdmin` inside routes/stacks.ts, leaving the named import on line 15
unreferenced. `requirePaid` and `effectiveTier` from the same line are still
in use elsewhere in the file and stay.
tsc --noEmit does not flag unused named imports; ESLint's no-unused-vars
does. Local backend lint reproduces and now reports 0 errors against the
existing 334-warning baseline.
|
||
|
Anso
|
42e8d3a78c |
feat(security): per-image scroll + retention cap in scan history (#1231)
* feat(security): per-image scroll + retention cap in scan history Long scan histories for hot images used to monopolise the Scan history sheet: a single image with dozens of scans pushed every other image off screen, and the underlying vulnerability_scans table grew without bound. Each image group's table now renders inside its own ScrollArea capped at max-h-64 (~6 rows visible) so a busy image scrolls independently while the list of images stays navigable. A new global setting scan_history_per_image_limit (default 50, min 5, max 1000) backs both a window-function query that caps the response per image_ref and a prune step that runs on the existing MonitorService cleanup tick. The response now carries cappedImageRefs + perImageLimit so the UI can render a "Capped at N · older scans pruned" hint on groups sitting at the ceiling without a second settings round-trip. Single-image deep-dive (imageRef query param) bypasses the cap so a user clicking into one image can still see its full history. The prune uses self-contained subqueries to avoid SQLITE_MAX_VARIABLE_NUMBER issues on first-run installs with large backlogs, and explicitly deletes child rows from vulnerability_details, secret_findings, and misconfig_findings inside a transaction since FK cascade is not enabled at the connection level. Settings → Developer → Data retention gains a "Scan history per image" field. * fix(security): skip searchDraft debounce on mount to stop page-reset race The searchDraft debounce useEffect fires once on initial mount with the unchanged value and, 300ms later, unconditionally calls setPage(0). When a user (or a test) paginates inside that 300ms window, the pending debounce silently undoes the page advance. CI surfaced this as a flaky 3rd fetch in the "advances offset when the user pages forward" test once the per-image cap work added enough state-update overhead to push the click past the 300ms threshold on the slower Linux jsdom run. Track searchDraft with a ref and exit the effect when the value has not actually changed, so the debounce only runs in response to real user typing. |
||
|
Anso
|
2d56ea958a |
fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228)
* fix(stack-activity): per-stack history integrity, attribution, sanitization Address the Stack Activity audit findings (PR 1 of 2): - Per-stack history integrity: drop the per-insert 100-row prune in addNotificationHistory that evicted quieter stacks' history whenever another stack got chatty. Periodic cleanupOldNotifications now caps per (node, stack) at 500 rows and per-node unattached system events at 1000 rows, on top of the existing 30-day retention. Signature takes an options bag and returns a per-stage summary so MonitorService can log what actually ran each cycle. - Actor attribution: thread req.user?.username through every notifyActionFailure call site and add synthetic actors at service emit sites (system:autoheal, system:scheduler, system:image-update, system:docker-events, system:blueprint, system:monitor, system:policy). The timeline renders system actors as "via <Label>" so an autoheal redeploy is no longer indistinguishable from a user redeploy. - Message sanitization: new sanitizeNotificationMessage at NotificationService.dispatchAlert strips KEY=VALUE pairs whose key ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and truncates to 1000 chars. Applied to the stored history and to every downstream Discord/Slack/webhook channel. The ImageUpdateService recovery-path direct DB write also runs through the sanitizer. - Composite pagination cursor: getStackActivity now accepts a (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only form silently dropped events when a single compose up emitted many events sharing one millisecond. Route rejects beforeId without before. - Frontend hardening: distinct error state with retry button (initial fetch failure no longer renders as the genuine empty state), strict positive-integer parsing on cursor params, overrequest-by-1 pagination so the last page does not leave a dead "Load more" click, runtime guard on liveEvents merge that validates the level union, per-minute day-bucket recompute so an open panel does not stay on "Today" past midnight. No tier, role, or capability gate touched. Route permission gate remains stack:read on the named stack. * fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir External review surfaced two leak paths in the message sanitizer: - The sensitive-key regex was uppercase-only. Compose env names are conventionally uppercase but lowercase forms (db_password, jwt_secret, github_token) are valid and do leak through the same Docker and compose-parse error paths. Make the regex case-insensitive and tighten it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word, while still leaving BYPASS, COMPASS, and similar non-secret keys alone. - The compose-dir path collapse only read process.env.COMPOSE_DIR, but the real resolution chain is node.compose_dir (per-node DB override) -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom compose_dir could still leak absolute paths into stored history and downstream channels. Route both the dispatchAlert call and the ImageUpdateService recovery-path direct write through NodeRegistry.getInstance().getComposeDir(localNodeId) so the collapse covers every resolution outcome. Tests now assert lowercase keys are redacted and that BYPASS-style non-secrets stay intact in both cases. notification-routing mock extended to stub the new getComposeDir call. * chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal Close three small follow-ups on the per-stack activity timeline: - A11y: each day-group gets role="list" and each event row gets role="listitem" so screen readers traverse the timeline as a list instead of a wall of text. The day-group container also carries an aria-label naming the bucket. - Visibility-aware day-bucket tick: the 60s setInterval that re-derives Today/Yesterday/Earlier now short-circuits when document.hidden, so a backgrounded panel does not re-render every minute for no visible effect. - Live-disconnect signal: useNotifications dispatches a sencho:notifications-connection custom event on WebSocket open and close. The timeline listens and, when explicitly disconnected, shows a one-line "Live updates offline; reconnecting…" hint above the list. The sidebar ticker already surfaces fleet-wide connection state; this adds an in-context cue for users who are focused on a single stack. Stack-name case normalization was considered and rejected: stack names are case-permissive per the isValidStackName validator, and lowercasing on read or write would silently rename or hide a user's "MyApp" stack. * ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex ESLint no-useless-escape errored on \- inside the character class [a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the end of the class so it's an unambiguous literal and the escape is no longer required. Behavior is identical; sanitizer tests still pass. * revert(stack-activity): drop unvalidated E2E spec from this PR The spec was committed without ever running against a real Docker daemon, then failed in CI when it ran for the first time: deploy returned 200 but no notification appeared on the activity endpoint within the polling window, suggesting either a deploy-notification race or a node-id resolution mismatch in the CI environment. Backend unit tests (route + composite cursor + sanitizer) and frontend component tests cover the same logic. The E2E spec will land in a dedicated follow-up once it has been authored against a working CI environment. |
||
|
Anso
|
d727a55a5f |
feat(stacks): surface post-deploy scan attempt status (#1198)
triggerPostDeployScan was fire-and-forget. When Trivy was missing on a
node, when the registry refused the digest lookup, or when a single
image scan threw, the failure went to console.error and the user
never learned. Open the security tab later, see stale data, no
indicator that the scan even tried.
Backend:
- New stack_scan_attempts table (node_id, stack_name, status,
attempted_at, error_message). One row per stack; latest attempt
overwrites the previous one.
- DatabaseService gains recordStackScanAttempt /
getStackScanAttempt / clearStackScanAttempts. Status is one of
'ok' | 'partial' | 'failed' | 'skipped'.
- triggerPostDeployScan in helpers/policyGate.ts now records every
exit path: 'skipped' when Trivy is unavailable or no images to
scan; 'failed' when container enumeration or all images fail;
'partial' when some images scan and others fail; 'ok' on full
success.
- New GET /api/stacks/:name/scan-status returns { status,
attemptedAt, errorMessage } or { status: null } when never tried.
- DELETE /:stackName cleanup chain now clears the row alongside
the existing update-status / auto-update cleanups.
Frontend:
- StackAnatomyPanel fetches /scan-status on stackName change.
- Renders a small warning strip below the update banner when
status !== 'ok' (failed / partial / skipped). Hidden when status
is 'ok' or unknown (never attempted). Title attribute carries
the full error message for hover inspection.
Cross-feature note: the audit doc flagged this as M-6 with a
coordination note for the pending Security feature audit. The
schema kept intentionally narrow (one row per stack, simple
status enum) so the Security audit can extend it (richer history,
per-image-row breakdown, etc.) without a destructive migration.
Resolves M-6 from the stack-management audit.
|
||
|
Anso
|
aa3d99a594 |
fix(mesh): re-evaluate data plane every 10s and add opt-in auto-recreate (#1184)
* fix(mesh): re-evaluate data plane every 10s and add opt-in auto-recreate MeshService.dataPlaneStatus was written exactly once at boot in setupMeshNetwork() and never re-evaluated. After the operator removed sencho_mesh at runtime (or it was recreated externally, or Sencho was disconnected from it), /api/health and the dashboard banner kept returning the stale boot-time discriminator until the next process restart. Adds a 10s revalidator that inspects the current Docker truth in one network-inspect call and transitions dataPlaneStatus to reflect it. Short-circuits in not_started / not_in_docker / subnet_invalid (states that cannot change within this process) and in concurrent ticks. Transitions are idempotent on stable state, so the timer can tick indefinitely on a healthy mesh without log noise. New 'not_found' reason value for the network-was-removed-at-runtime case. Existing reasons (subnet_mismatch, subnet_overlap, attach_failed) also surface from the revalidator when their underlying conditions arise post-boot. transitionDataPlane keeps message and subnet fields fresh across consecutive observations even when reason is unchanged, so /api/health never reports stale numbers (e.g. two consecutive subnet_mismatch observations against different external subnets). Adds an opt-in mesh_auto_recreate global setting (default off). When on, the revalidator additionally calls attemptInPlaceRecreate() after surfacing not_found. The helper hard-prefers the boot-chosen subnet (this.meshSubnet) and never iterates candidates, because changing the subnet here would invalidate every existing extra_hosts override on disk. A real conflict on the original subnet is reported as subnet_overlap and preserved during the 60s recreate throttle window so the operator-actionable reason is not flapped back to not_found between attempts. Self-attachment is checked via Name match (operator --hostname X matches container Name /X) or full container-ID prefix for hex HOSTNAMEs >= 12 chars (Docker default short ID). Non-hex HOSTNAMEs cannot collide with container IDs at all so a Name miss is conclusive; short hex HOSTNAMEs preserve the prior status as 'unknown' rather than risking a false-positive prefix match. Frontend surfaces: - types/mesh.ts: 'not_found' added to MeshDataPlaneReason. - MeshDataPlaneBanner: 'not_found' headline copy. - Settings > System > Mesh data plane: TogglePill bound to mesh_auto_recreate, default off, helper text explains the tradeoff. Backend coverage in backend/src/__tests__/mesh-data-plane-revalidate.test.ts (25 cases): short-circuits, idempotent stable-state, recovery from subnet_mismatch / subnet_overlap, transition to not_found / subnet_mismatch / attach_failed, transient-Docker anti-flap, re-entrancy guard, name match path, ID-prefix path, short hex hostname ambiguity, non-hex hostname certainty, transition message refresh on observation drift, auto-recreate off (default), auto-recreate success with senchoIp preservation, auto-recreate overlap classification with no subnet drift, throttle window preserves classified reason, throttle release. Lifecycle test covers timer wiring in start()/stop(). Existing mesh-setup-error-classification suite (27 cases) still green. Resolves: F-4 in the v1.0 audit tracker. * fix(mesh): address Codex review of PR #1184 Three findings from the independent review: BLOCKER: attemptInPlaceRecreate() called recordSetupFailure() on create / attach failures, which clears this.senchoIp. The next revalidator tick's attachment check is guarded on senchoIp, so with it null the check is skipped and the snapshot path can silently flip the status back to ok against a network where Sencho is in fact not attached. Also: a later successful recreate would call ensureSelfAttached() with senchoIp null, which short-circuits, so the network gets recreated without binding Sencho. Replaced the recordSetupFailure() calls in attemptInPlaceRecreate with a new recordRecreateFailure() that uses transitionDataPlane and preserves senchoIp. Added two tests: create-fails-then-succeeds (verifies senchoIp survives the failure and the later retry binds Sencho correctly) and create-succeeds-attach-fails-then-next-tick (verifies the snapshot path surfaces attach_failed on the next tick instead of falsely reporting ok). SHOULD-FIX 1: single-key POST /api/settings wrote String(value) without re-validating against the per-key schema, so an allowlisted enum-shaped key like mesh_auto_recreate could persist arbitrary strings ('banana', 'true') that the bulk PATCH would later refuse. Routed the single-key path through SettingsPatchSchema.safeParse so both write paths validate identically. Added regression tests for an invalid mesh_auto_recreate value, a valid mesh_auto_recreate write, and an out-of-range numeric value. SHOULD-FIX 2: the new Mesh data plane subsection lived inside a section the registry exposes to non-admins, who would see the toggle and only learn it was admin-only after the save 403'd. Gated the subsection on `isAdmin` from useAuth so non-admins do not see the control. The other system controls keep their existing visibility pattern (read-only for non-admins). 71/71 backend tests green (revalidate + mesh-setup + settings-routes). 276/276 frontend tests green. tsc clean on backend + frontend. |
||
|
Anso
|
fcff8e9047 |
fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) (#1175)
* fix(monitor): collapse repeated host-metric alerts into per-window summary (F-11) A host metric over threshold previously dispatched one notification every 5 minutes for the duration of the breach, producing 7+ identical messages in 35 minutes and spamming Discord/Slack routes. Replace the hardcoded 5-minute cooldown for CPU/RAM/disk with a per-metric suppression window (default 60 min, configurable via host_alert_suppression_mins). The first breach fires immediately; subsequent cycles within the window are silently counted; the next dispatch after the window elapses carries a summary suffix listing how many cycles were suppressed and when the breach first crossed threshold. Recovery clears the counter so re-breach fires fresh. The pattern mirrors PolicyEnforcement.notifyTrivyMissingOnce: module-scope Map, in-memory only, in-cycle dedup, with a test-reset helper. The existing system_state row keeps post-restart re-fires bounded. Janitor and per-stack alert rules are unchanged; they already have adequate cadence and per-rule cooldown respectively. * fix(ci): restore backend and frontend checks * fix(e2e): remove create button timing race * fix(e2e): harden create double-click test * fix(monitor): clear persisted F-11 timestamp on recovery + clamp suppression window Independent audit on the previous commit surfaced two issues. 1. clearHostMetricSuppression early-returned on missing in-memory state, leaving a stale system_state.last_host_*_alert_ts row alive after a process restart. Scenario: breach fires + persists timestamp, process restarts, metric recovers before another evaluate cycle re-seeds the in-memory Map, recovery cleanup early-returns. Next re-breach inside the original window hits the restart-survivability branch and is silently suppressed instead of firing fresh. Fix: read persisted state in clearHostMetricSuppression and reset to '0' independently of in-memory presence. The read-before-write also skips redundant writes when the row is already cleared. 2. host_alert_suppression_mins is validated by zod on the bulk PATCH path but the single-key POST /api/settings path accepts allowlisted keys without re-validation. A 999999999-minute value would silence host alerts for centuries. Add MAX_HOST_ALERT_SUPPRESSION_MIN = 1440 mirroring the zod max, and clamp via Math.min in evaluateGlobalSettings. Two new vitest cases (restart-then-recovery-then-rebreach; the 1440 clamp) confirmed failing before the fix, passing after. The existing "metric drop" case updated to use a mock-backed persistence pattern consistent with the new restart-scenario tests. 73/73 monitor-service tests green; full backend suite 2507/2510 (same pre-existing Windows EBUSY flake on filesystem-backup.test.ts as baseline). |
||
|
Anso
|
e05099f2a1 |
fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.
Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
`sticky_error_got` columns to `fleet_sync_status` via an idempotent
migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
flag on a clean push. getFailedSyncTargets SQL adds
`AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
sticky is set (covers event-driven pushResourceAsync calls). On a 409
with code CONTROL_IDENTITY_MISMATCH, records the failure once and
pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
the peer with `{override:true}` using the stored Bearer node_proxy
token. On peer 200, clears every sticky row for the node so the next
push re-anchors and resumes replication. Distinct 502 / 504 responses
for peer-rejected / peer-unreachable so the UI can show a useful toast.
Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
(30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
SummaryRow per remote node card. In sync / degraded / paused with
a tooltip; no action buttons (the action lives in NodeManager).
Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
mismatch, short-circuit on subsequent pushes, null fingerprints,
and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
happy path, peer 401 -> 502, peer unreachable -> 504, local-node
rejection, unknown node id, and community-tier 403.
Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.
Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.
Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
|
||
|
Anso
|
f6e42535c8 |
fix(mesh): route peer→central traffic over the existing forward WS (#1094)
* fix(mesh): route peer→central traffic over the existing forward WS The reverse mesh callback path (`/api/mesh/proxy-tunnel-from-peer`) needed SENCHO_PRIMARY_URL on central plus a publicly reachable origin from the peer's perspective. In a typical homelab where central sits behind NAT, peer→central dispatch silently failed at the dialer's short-circuit and the headline "call any service on any node by hostname" worked one way only. The forward WS at `/api/mesh/proxy-tunnel` is already bidirectional end to end. Make the bridge a persistent control-plane primitive: dial every mesh-enabled proxy peer at startup, reconcile every 60 s, never idle-close. Peer→central traffic multiplexes over the same WS via `tcp_open_reverse`. Removed: - `meshProxyTunnelFromPeer.ts` WS handler and dispatch - `MeshCentralRegistry`, `PeerToCentralMeshSessionDialer` - `mesh_handshake` first-frame state machine in `meshProxyTunnel.ts` - `maybeSendBootstrap`, `buildHandshakeFrame` in the dialer - `mesh_proxy_callback_bootstrap` capability and `maybeWarnUnsetPrimaryUrl` - `mesh_centrals` table (drop migration; greenfield, no users) - `PilotTunnelManager.replaceOrRegisterProxyBridge` (dead after handler removal) - twelve associated unit/integration tests plus the peer-recovery branch in `MeshService.openCrossNode` Added: - `MeshService.proactiveBridgeFanout` selects every mesh-enabled proxy peer (no longer gated on `mesh_stacks` rows) - `startBridgeReconcileLoop` runs the fanout every 60 s (override via `SENCHO_MESH_RECONCILE_INTERVAL_MS`) - `MeshProxyTunnelDialer` default idle TTL is now `0` and exposes `isDialing(nodeId)` for the status surface - `MeshNodeStatus.reverseCallbackStatus` discriminator (`connected | connecting | unavailable | not_applicable`) surfaced via `/api/mesh/status` and rendered as a pill in the Routing tab - `openCrossNode` error message distinguishes "no proxy target" from "waiting for central to dial the reverse bridge" - New tests: `mesh-service-proxy-tunnel-reconcile`, `mesh-status-reverse-callback`, `mesh-proxy-tunnel-dialer-no-idle-close` SENCHO_PRIMARY_URL is no longer required for any mesh function. * fix(mesh): rewrite proxy-tunnel reconcile test contents The previous commit renamed the file but the rewritten test bodies stayed unstaged on top of the rename. This commit lands the actual rewrite: the fanout assertion now requires every mesh-enabled proxy peer to be dialed, not just those with `mesh_stacks` rows, and adds a reconcile-tick repeated-call test. |
||
|
Anso
|
50b89db3b8 |
chore(mesh): drop mesh_stacks table on pilot-mode DBs (#1085)
Per the C-3 design, mesh state lives on central. Pilots learn aliases via the D-1 override push and hold them in `pilotAliasOverlay`. The local `mesh_stacks` table on pilots has been dead since C-3 shipped. This is a greenfield cleanup (Directive 20): - `migrateMeshTables()` runs `DROP TABLE IF EXISTS mesh_stacks` when `SENCHO_MODE === 'pilot'` and skips the CREATE. Central-mode behavior is unchanged. - The four mesh CRUD methods (`listMeshStacks`, `isMeshStackEnabled`, `insertMeshStack`, `deleteMeshStack`) short-circuit on the same check. Reads early-return empty/false; writes early-return; `insertMeshStack` also warns so an accidental pilot-side write is loud. - New `isPilotMode()` helper mirrors the existing one in `bootstrap/startup.ts`. Layering rules forbid the shared import; a third call-site would justify extraction to `helpers/`. Operator-visible behavior is unchanged on either side. On central, the table and its rows are untouched. On pilots, the table is removed (it held no rows in steady state) and any caller of the CRUD methods continues to see empty list / false / no-op without touching SQLite. Test plan - `npx tsc --noEmit`: clean. - New `database-service-pilot-mesh-stacks.test.ts`: 5/5 green (table absence + four short-circuit assertions). - `mesh-service.test.ts`, `mesh-diagnostic-local.test.ts`, and `mesh-service-proactive-bootstrap-fanout.test.ts`: 58/58 green. - Full backend suite: 2280/2281; the single failure is a pre-existing Windows EBUSY flake in `filesystem-backup.test.ts` that reproduces on a clean tree. - Code reviewed; findings applied. |
||
|
Anso
|
6b728599c4 |
fix(mesh): persist PilotMetrics counters across central restart (#1078)
The 12 PilotMetrics counters (proxy_dials_failed, proxy_bridges_total, proxy_idle_closes, the pilot-tunnel counters, and the peer-to-central callback counters) live in a singleton holding scalar fields, so process restart wipes them. Operators investigating rotation- or dial-failure trends across a central restart lost the aggregate metric. Persist the snapshot under a pilot_metrics_counters JSON row on the existing system_state KV table. Hydrate via PilotMetrics.load() in bootstrap/startup.ts before any service that increments runs; final-flush in bootstrap/shutdown.ts before the SQLite handle closes. Between those boundaries a buffered flush mirrors the audit-log pattern: every 1 s or every 100 increments, whichever fires first. Hot increment path stays a single property write plus a pending-counter check. Defensive parse on the row: object check + numeric filter, so an operator hand-edit or schema drift across releases degrades to zero state with a warn rather than crashing boot. Schema additions back-fill missing keys with zero so a release that adds a counter does not need a migration. Closes F-R-4 from the mesh tracker. |
||
|
Anso
|
cf618dd866 |
feat(mesh): symmetric WS dial for proxy-mode mesh peers (#1066)
* chore(mesh): foundation for symmetric callback dial Adds the data-plane scaffolding that the symmetric callback dial fix builds on: - mesh_centrals table for peer-side bootstrap material - MeshCentralRegistry service (upsert/getActive/clear/markUsed/markRejected) - PilotTunnelManager kind discriminator and replaceOrRegisterProxyBridge - mesh_proxy_callback_bootstrap capability registration - MeshProxyTunnelDialer reason-tagged proxy-bridge-down events from a single tearDownBridge emission point - Reactive redial scheduler that skips idle and auth_failed reasons * feat(mesh): add reverse-direction activity log entries (closes R1-B) acceptReverseLocal now emits route.resolve.ok with direction=reverse on connect ack and route.resolve.fail with direction=reverse plus reason=container_not_found / connect_error pre-connect. Post-connect close/error stays silent. Reuses existing event types via the new details.direction discriminator so frontend filters are unaffected. * feat(mesh): add peer-to-central callback dial path (closes R1-A2) Closes the architectural gap where proxy-mode mesh peers could not re-establish their tunnel to central after any non-idle bridge teardown (idle close, network blip, central restart, peer reboot). Central remains the hub for the data plane; the change is purely about WS initiation. Symmetric WS initiation, asymmetric protocol roles. Central retains PilotTunnelBridge ownership; peer retains TcpStreamSwitchboard + reverseDialer ownership. Central bootstraps callback credentials over the first authenticated central-initiated mesh tunnel via a one-shot mesh_handshake JSON frame; peer persists the material in a new mesh_centrals SQLite table and dials central's new /api/mesh/proxy-tunnel-from-peer endpoint when local cross-node traffic needs a bridge and none is live. Mesh_tunnel JWT (HS256, signed with auth_jwt_secret) carries scope, audience, issuer (central instance id), peer api_token fingerprint, kid. Validation on inbound peer dial: algorithm pin, signature, scope, audience, instance, time bounds, node existence and mode, fingerprint match. Failures return HTTP 401 with a machine-readable reason; peer routes the response per a clear-vs-keep cache matrix. Triggers proactive bootstrap on mesh-enable and api_token rotation; central startup fans out to mesh-enabled proxy-mode nodes with mesh_stacks rows (throttled, fire-and-forget). Reactive redial on non-idle bridge loss. Capability-gated handshake send (mesh_proxy_callback_bootstrap) makes the upgrade path safe against older peers in mixed-version fleets. Adds peer-side /api/system/pilot-tunnels centralCallback diag block, bounded counter metrics for bootstrap and dial events. SENCHO_PRIMARY_URL preflight warning when unset on a central with mesh-enabled proxy nodes. Tested with unit suites for the validation chain, registry, manager, and both dialers; integration tests for bootstrap E2E (asserts protocol-role invariant), api_token rotation, instance id change, version skew, and pilot-mode regression. * fix(mesh): green CI on the symmetric callback branch Two independent CI failures, both surgical: 1. Backend tests (11 fails): four mesh test files called setupTestDb in beforeEach. setupTestDb does not reset the DatabaseService singleton, so the per-test afterEach rm of the previous tmpdir left the singleton connection pointing at a deleted file. The next beforeEach's line-55 write threw SQLITE_READONLY_DBMOVED on Linux. Windows file-lock semantics hid this locally. Hoist setupTestDb / cleanupTestDb to file-scope beforeAll / afterAll; per-test state resets stay in beforeEach. Matches the convention in the eight mesh test files that already pass. 2. CodeQL (4 high alerts): js/insufficient-password-hash flagged sha256(api_token) at four sites. The api_token is a 256-bit opaque bearer (sen_sk_-prefixed), not a human password; sha256 is the correct fingerprint primitive for binding the mesh_tunnel JWT to a specific token. Add the two production files plus the two test files that mint the fingerprint to the existing path-scoped query-filter for that rule. * fix(mesh): drop unused afterEach import and revert dead codeql config ESLint flagged afterEach as unused in mesh-central-registry.test.ts:1 after the previous commit hoisted setup/teardown to file-scope beforeAll/afterAll. Remove from the vitest import line. Revert the codeql-config.yml additions from the previous commit. The paths: sub-key under query-filters > exclude is not a documented CodeQL feature and silently no-ops. The four js/insufficient-password-hash alerts on api_token fingerprinting are tracked as dismissed false positives in the GitHub Security tab rather than via dead config. |
||
|
Anso
|
e7a3b544c0 |
fix: harden auto-heal policies (#1042)
* fix: harden auto-heal policies * fix: resolve auto-heal lint failure |
||
|
Anso
|
c31d48b933 |
fix: harden git source webhooks (#1033)
* fix: harden git source webhooks * fix: make path validation visible to CodeQL static analysis Add explicit isValidStackName guard in getEnvContent, isValidGitSourcePath pre-validation in readRepoFile, and URL hostname check in remoteStackRequest to satisfy CodeQL taint-tracking so the pipeline passes. * fix: use path.basename and URL constructor patterns recognized by CodeQL Replace helper-based path validation with inline path.basename and path.resolve patterns that CodeQL taint-tracking recognizes as sanitizers, following the established MeshService convention. Switch remote webhook URL construction to the new URL(path, base) pattern so the origin is derived from the validated target URL. * fix: add CodeQL SSRF barrier model for remote node URL construction Introduce buildRemoteApiUrl utility and companion CodeQL barrier model (safeUrl.model.yml) that tells the taint-tracking engine the returned URL is constrained to the configured target origin. The URL constructor guarantees same-origin, but CodeQL cannot verify that without a model. * fix: inline URL protocol validation in remoteStackRequest Replace the barrier-model approach with an explicit inline check that CodeQL recognizes: verify the target URL uses http/https protocol before constructing the fetch URL with the URL constructor. * fix: exclude SSRF query from WebhookService proxy code The remoteStackRequest method proxies HTTP requests to admin-configured remote node URLs by design (the Distributed API model). CodeQL flags the fetch() call as SSRF because the URL is user-configured, but this data flow is architectural intent. Exclude js/server-side-request-forgery from this file. * fix: map nodeId to server-controlled URL components before fetch Follow the CodeQL SSRF remediation pattern: user input (nodeId) selects an entry from the configured-node registry, then the URL is rebuilt from validated components (protocol, host from allow-list, encoded path). Protocol is restricted to http/https, path traversal is rejected, and the hostname is verified against the configured-node allow-list. * fix: remove unnecessary escape in endpoint validation regex |
||
|
Anso
|
3b650523c1 |
Audit-hardening pass for secret and misconfiguration scanning (#977)
* fix(security): dedupe concurrent compose-stack scans
Track stack scans in scanningImages keyed stack:<nodeId>:<stackName>.
The /scan/stack route returns 409 when an in-flight scan exists, and
the service-side check is the real correctness barrier (the route
pre-check is a fast-path optimization that mirrors scanImage). The
dedup key release lives in a try/finally so failed scans free the
slot for retry.
Why: scanComposeStack had no equivalent of scanImage's scanningImages
guard, so two simultaneous calls for the same stack would both run
trivy config, both insert a vulnerability_scans row, and double-
process the result.
* feat(security): acknowledge misconfig findings
Adds a parallel acknowledgement system for Trivy misconfig findings
that mirrors cve_suppressions: a new misconfig_acknowledgements table,
read-time enrichment via the new misconfig-ack-filter utility, REST
CRUD endpoints, fleet-sync replication from control to replicas, a
Settings panel, and an Acknowledge button on the Misconfigs tab.
Schema and behavior parity with cve_suppressions:
- UNIQUE(rule_id, COALESCE(stack_pattern, '')) so fleet-wide acks
collide as expected
- blockIfReplica on every write
- Audit-log entries name the scope (rule_id, stack_pattern) but
never the reason text
- replicated_from_control flag controls UI delete affordance and
drives clearReplicatedRows on demote/reanchor
- Validators reused: validateStackPatternForRedos for glob safety,
sanitizeForLog for log fragments
SARIF export emits an external/accepted suppression entry per
acknowledged misconfig, matching the CVE pattern.
Per-row Acknowledge dialog prefills stack_pattern with the scan's
stack_context so the default scope is "rule + this stack only" and an
operator must broaden explicitly.
Tests: misconfig-ack-filter (15) and misconfig-ack-routes (23)
including the duplicate-409 case for both pinned and fleet-wide acks.
* fix(security): reap orphaned trivy tmp dirs at startup
When the buildEnv path writes a per-scan DOCKER_CONFIG dir under
os.tmpdir() and the process crashes before the finally block runs,
the dir leaks. Mirrors GitSourceService.sweepStaleTempDirs:
exported sweepStaleTrivyTempDirs is fire-and-forget at boot,
removes prefix-matching dirs older than 1 hour, swallows
permission/race failures, logs a single line if any were reaped.
* perf(security): emit per-batch summary for scanAllNodeImages
Adds one diag() line at the end of scanAllNodeImages summarising
unique image count, scanned, skipped, failed, violation count, and
elapsed time. Per-image diag inside scanImage stays useful for
debugging individual scans; the summary gives operators a single
fleet-level checkpoint when developer_mode is on.
* perf(security): cap SARIF export at 5000 findings per type
Replace the unbounded fetchAllPages walk on /scans/:id/sarif with a
hard limit of 5000 findings per type. When any type trips the cap,
emit run-level properties.truncated=true plus row_limit and per-type
totals so downstream tooling can flag the export as partial.
Console-warns for ops visibility.
A scan with 50k vulns previously streamed every row into memory
before serialising; the cap bounds memory and serialisation time at
the cost of completeness on pathological scans.
* docs(env): document TRIVY_BIN host-binary override
The env var is honored by TrivyService.detectTrivy as a fallback when
no managed install is present, but it was undocumented in
.env.example. Adds the var with a comment explaining precedence
(managed > TRIVY_BIN > PATH).
* test(security): cover scanComposeStack failure modes
Two new cases drive the existing try/catch through real failure
paths:
- Malformed Trivy stdout: row flips to status='failed' with the
parser error preserved on `error`.
- execFile rejection: row flips to status='failed' with a string
error message.
Pairs with the existing dedup tests so the failure path now also
verifies the scan row state, not just the thrown exception.
* test(e2e): security scanner + misconfig acknowledgement flow
Seven Playwright tests covering the scanner UI and the new
acknowledgement system end-to-end:
- Trivy availability gate (skips suite when binary absent so CI
without Trivy can opt out via E2E_SKIP_TRIVY=1)
- Stack config scan completes and records misconfig findings
- Concurrent stack scan returns 409 from the dedup gate
- Misconfig ack POST creates and lists on Settings
- Duplicate (rule_id, stack_pattern) returns 409
- Malformed rule_id (shell metacharacters) returns 400
- Misconfigs tab renders against a real stack scan
Tests drive the API for behaviour assertions and the UI only for
shell-rendering checks; the visual snapshot suite owns screenshots.
* docs(features): add misconfig acknowledgement workflow and SARIF cap
Refreshes vulnerability-scanning.mdx with:
- Misconfig acknowledgements section covering the per-row dialog,
Settings panel, scope/matching rules, and SARIF emission
- Tier table row for the new feature
- SARIF section note on the 5000 row-per-type cap and the
properties.truncated marker for partial exports
- Troubleshooting entries: SARIF cap, hidden Acknowledge button,
findings resurfacing after delete, Trivy DB phone-home, and
409 on concurrent compose-stack scans
* fix(ci): clear backend lint and CodeQL alerts
- Remove the dead fetchAllPages helper in routes/security.ts. It lost
its callers when the SARIF endpoint switched to direct paged reads
for the truncation cap. ESLint flagged it as unused.
- Switch the trivy-tmp-cleanup test helper to fs.mkdtempSync. Building
paths under os.tmpdir() with predictable names tripped CodeQL's
js/insecure-temporary-file rule (high severity), which warns about
symlink-pre-creation attacks even in test code. mkdtempSync appends
a process-random suffix and creates the dir atomically; the
sencho-trivy- prefix is preserved so the production sweep still
matches the test fixtures.
|
||
|
Anso
|
a284732a95 |
feat(fleet-sync): hide other replicas' identity-scoped policies on a replica (#973)
GET /api/security/policies on a replica now returns only the policies
that apply to THIS replica. Replicated rows with a node_identity
targeting a sibling replica are filtered out so an operator cannot
enumerate the names and rules of policies meant for another node in
the fleet. Defense in depth: the security panel is admin-only, but a
backend filter is bypass-proof and matches Sencho's privacy posture.
Internal evaluators (getMatchingPolicy, evaluateScanAgainstPolicies)
keep using the unfiltered list because they already enforce identity
matching at evaluation time.
CVE suppressions are fleet-wide on every replica (no node_identity
column) so no analogous filter is needed.
Public surface:
- DatabaseService.getScanPoliciesForUi(role, selfIdentity): the
filtered variant, called from securityRouter.get('/policies').
Tests:
- 3 new vitest cases: control sees full set; replica hides
other-replica scoped rows; replica always includes locally created
rows.
- Full backend suite: 1795 pass / 5 skipped.
|
||
|
Anso
|
4007709590 |
fix(fleet-sync): hygiene pass on receiver behavior and cleanup (#972)
A bundle of small file-local fixes to the receiver path and node-deletion flow. Changes: - F4 receiver audit log: applyIncomingSync now writes a system audit entry on every applied push so mirrored security-rule changes show up in the replica's audit panel with a clear control-side origin. - F7 pilot-agent skip: pushResource explicitly excludes pilot-agent nodes (they have no api_url for HTTP push) and warns once per node id so the operator sees they will not receive replicated policies. - B4 identity-drift notification: when targetIdentity differs from the cached fleet_self_identity, dispatch a warning so the operator can audit any identity-scoped policies that may need re-targeting. - B6 stack_pattern ReDoS guard: reject patterns with 4+ consecutive wildcards or more than 8 wildcards total. Both control-side validators (POST/PUT scan policies) and the receiver-side row validator share the helper. - B9 deleteNode cascade: clear fleet_sync_status rows for the node inside the existing transaction so the sync-status panel does not render ghost entries after a node is removed. - S6 last_error redaction: formatError strips Bearer tokens and JWT-shaped values from error messages and caps at 500 chars before storing in fleet_sync_status.last_error or logging. Tests: - 8 new vitest cases covering audit-log entry, identity-drift alert, pilot-agent warn-once, formatError redaction (Bearer + JWT), ReDoS validator rejection, and a backtracking-time smoke test. - New database-fleet-sync-cascade.test.ts: deleteNode removes fleet_sync_status rows for the deleted node and leaves siblings untouched. - Full backend suite: 1792 pass / 5 skipped. |
||
|
Anso
|
f8c75aa6cd |
fix(fleet-sync): clear stale policy_evaluation on replica sync swap (#971)
Replicated scan policies always insert with fresh ids on the replica side. Any vulnerability_scans.policy_evaluation row referencing the prior set was instantly stale the moment the swap completed, so the replica's UI would surface violations from a policy that no longer exists. replaceReplicatedScanPolicies now calls clearOrphanPolicyEvaluations() inside the same transaction as the delete + inserts, so the cleanup commits atomically with the row swap. Local-only policies and their cached evaluations remain untouched. CVE suppressions do not have an analogous cache (suppressions are applied at read time, never persisted on scan rows), so replaceReplicatedCveSuppressions does not need a sibling call. Tests: - database-replicated-policies.test.ts: stale policy_evaluation cleared after a swap; local-policy evaluation preserved across swaps. - Full backend suite: 1783 pass / 5 skipped. |
||
|
Anso
|
7dde257e1f |
feat(fleet-sync): replica self-demote endpoint and role UX (#969)
A replica admin can now demote the instance back to a standalone
control without raw SQLite access. The Settings → Security UI surfaces
a confirm-gated button when the role is replica; the role probe also
surfaces a soft banner when it cannot determine fleet role rather
than silently defaulting to control.
Backend:
- POST /api/fleet/role/demote (admin, requires `{confirm: true}`):
flips fleet_role to 'control', clears fleet_self_identity,
fleet_control_identity, and both received_pushed_at:* watermarks,
drops every replicated_from_control row from scan_policies and
cve_suppressions, nulls out any orphaned policy_evaluation cache.
Returns 409 ALREADY_CONTROL when invoked on a control.
- DatabaseService gains `clearOrphanPolicyEvaluations()` and
`clearReplicatedRows()` helpers. Reanchor consolidates onto
clearReplicatedRows so it shares the same code path.
- `FleetSyncService.demote()` returns boolean for the route to
translate into 200 or 409.
Frontend:
- SecuritySection probes /fleet/role and now records explicit success
vs failure rather than silently treating an error as control. A
soft banner appears when probe fails.
- Replica banner gains a "Demote to control" button and a destructive
ConfirmModal explaining the wipe.
Tests:
- 4 new route-level vitest cases (401, 400 without confirm,
end-to-end demote with replica setup, 409 ALREADY_CONTROL with
explicit precondition).
- Service unit test asserts the consolidated clearReplicatedRows path.
- Full backend suite: 1773 pass / 5 skipped. Frontend: 185 pass.
|
||
|
Anso
|
27660f622b |
fix(fleet-sync): version the wire protocol and serialize per-node pushes (#967)
Hardens the scan-policy and CVE-suppression replication channel as the foundation of a multi-PR fleet sync hardening track. No new endpoints, no new tables, no schema changes; receivers still tolerate legacy payloads (absent pushedAt and controlIdentity) for rollout safety. Wire protocol: - Sender stamps every push with a strictly-increasing pushedAt and a placeholder controlIdentity. Receiver rejects strictly-older pushedAt with 409 STALE_SYNC_PUSH so the next write retries. - pushedAt comparison plus row replacement plus watermark write run in a single SQLite transaction; a partial-write window cannot leave the watermark behind the row state. Concurrency and limits: - Per-node mutex on the sender so concurrent control writes serialize per remote and never apply older state on top of newer. - Sender-side row cap at MAX_SYNC_ROWS=5000 with a 6-hour throttled truncation alert so flapping configs cannot flood the operator. - Route-level body limit raised to 5MB on POST /api/fleet/sync/:resource only; the global 100KB cap is unchanged. Oversize bodies return a structured 413 SYNC_PAYLOAD_TOO_LARGE. Determinism and hygiene: - getMatchingPolicy gains an id-ASC tiebreaker so two replicas resolve the same winner when policies tie on scope class. - Inline comment documents why getMatchingPolicy filters node_id at SQL yet still relies on JS identity matching for replicated rows. - Comment on the receive endpoint documents why no requirePaid is enforced (control's tier authorizes; replica trusts the bearer). - STALE_SYNC_PUSH 409s no longer record a node failure; they are expected protocol outcomes, not health issues. Public surface additions: - DatabaseService.transaction(fn): generic SAVEPOINT-friendly wrapper. - DatabaseService.getLocalScanPolicies / getLocalCveSuppressions: SQL filter on replicated_from_control = 0. - StaleSyncPushError: typed sentinel the route translates to 409. - fleetSyncConstants: shared MAX_SYNC_ROWS, body limit, state-key and error-code maps so the wire protocol has one source of truth. Tests: - 24 new vitest cases across fleet-sync-service, fleet-sync-routes, and database-matching-policy covering monotonic pushedAt, per-node serialization, row truncation and throttle, stale-push suppression, receiver back-compat, oversize-body 413, deterministic matching. - Full backend suite: 1757 pass / 5 skipped. |
||
|
Anso
|
0f0b22c51a |
feat(fleet): Fleet Secrets tab with env-var bundles (v1 MVP) (#965)
* feat(fleet): add Fleet Secrets tab with versioned env-var bundles (Skipper+) Centralized, encrypted-at-rest secret bundles that can be pushed to labeled nodes' stacks. Each save bumps a monotonic version; each push records a per-node-per-version row in `secret_pushes` plus an entry in `audit_log`. Conflict detection shows added/changed/unchanged/removed (informational) diffs before write. Overlay merge preserves keys missing from the bundle. - Adds `secrets`, `secret_versions`, `secret_pushes` tables. - New `SecretsService` reuses CryptoService for AES-256-GCM, NodeLabelService for selectors, and direct fetch + Bearer for outbound calls to remote nodes. - New `secretsRouter` with 9 endpoints under `/api/secrets`, gated by `requirePaid`. Mounted after the auth gate. - Audit summary patterns added for the new routes. - New Fleet › Secrets tab with bundle list, editor sheet (key=value rows, versions tab), and push wizard (selector, target stack, env file picker, per-node diff preview, results pills). - Documentation: docs/features/fleet-secrets.mdx + docs.json nav entry. - 26 Vitest cases cover parser, encryption, versioning, push aggregation, tier gating. * fix(fleet): use const for rawValue in env parser ESLint prefer-const flagged the let declaration as a CI-blocking error; the variable is never reassigned. |
||
|
Anso
|
52b46753af |
feat(fleet): add Federation tab with cordon and pin policy (Admiral) (#964)
Ships the v1 MVP for the Federation tab as placement control, not
placement automation:
- Cordon a node: marks the node unschedulable so the BlueprintReconciler
skips it for new placements only. Existing deployments continue to
drift-check and redeploy on revision changes; cordon never triggers
withdraw or eviction. Toggle on the NodeCard kebab (Admiral, admin
role); Cordoned pill renders for all tiers.
- Pin a blueprint to a node: stores blueprints.pinned_node_id, replacing
the desired set with the pinned node regardless of selector. Pin
overrides cordon by design. Action lives only in the Federation tab;
BlueprintDetail and the deployment table show read-only Pinned
indicators.
Backend: idempotent migrations add nodes.cordoned/cordoned_at/cordoned_reason
and blueprints.pinned_node_id. New routes POST /api/nodes/:id/cordon,
POST /api/nodes/:id/uncordon, PUT /api/blueprints/:id/pin, all gated by
requireAdmiral plus requireAdmin. Audit summaries added so the existing
auditLog middleware records every operator action. deleteNode clears
dangling pins.
Reconciler: pin override evaluated before selector match; cordon filter
applied only to the new-placement branch (deploy/stateReview without an
existing deployment). 11 new Vitest cases cover cordon filter, pin
override, pin-overrides-cordon, missing pin target, pin shrinks
desired set (stateless withdraw + stateful evict_blocked), and pin
clearing on node delete.
Frontend: new FederationTab.tsx with cordoned-nodes summary and
pin-policy table. Federation moved out of the experimental flag into
{isAdmiral && (...)} + AdmiralGate, mirroring the Routing tab pattern.
Secrets stays under experimental.
Tests pass: backend tsc, full Vitest suite (1704 passed), frontend
tsc -b, ESLint (0 errors). Manual verification via the local dev
instance confirmed the tab is hidden at Community, the kebab and pill
render at Admiral, and cordon and pin endpoints round-trip end to end.
Refs cut-line-1.0.md Federation v1 MVP.
|
||
|
Anso
|
775fab7d64 |
feat(dashboard): replace duplicate Recent Activity card with Fleet Heartbeat / Stack Restart Map (#932)
* feat: open security basics, manual fleet ops, and basic fleet management to Community
Realign tier guards to the user-stated philosophy: Community covers
deploy/monitor at scale plus security basics, Skipper adds automation
and advanced fleet management, Admiral keeps enterprise control.
Community now includes:
- Trivy install / uninstall / update from the Settings Hub (admin role)
- CVE suppressions CRUD (admin role; replicates fleet-wide)
- Manual image scan with vuln, secret, and misconfig results
- Stack-config scan, scan comparison
- Manual fleet snapshots: create, list, view, restore, delete
- Per-node Sencho self-update (Check Updates + per-node Update)
- Fleet Overview search, sort, filters, node-card expand, auto-refresh
Stays paid:
- Scan policies with block_on_deploy enforcement (Skipper+)
- SBOM (SPDX, CycloneDX), SARIF export (Skipper+)
- Bulk Update All across the fleet (Skipper+)
- Scheduled snapshot create (now Skipper, was Admiral)
- Trivy auto-update toggle, fleet-wide policy push (Admiral)
The Settings -> Security tab is unhidden by setting the registry tier to
null. The SecuritySection no longer early-returns a PaidGate; the policy
list, Add Policy button, and policy dialogs are wrapped in {isPaid && }.
The Fleet view drops isPaid gates on the Snapshots tab, Check Updates
button, per-node update handlers, OverviewToolbar grid controls, the
NodeCard expand affordance, and the auto-refresh notice. The
NodeUpdatesSheet receives a canBulkUpdate prop and gates the Update All
button on it. useFleetUpdateStatus and useFleetPolling drop their isPaid
guards so polling runs for Community; useFleetOverview drops the isPaid
wrap on the filter and sort path.
Backend route guards are flipped per the matrix above. The scheduler
tick and requireScheduledTaskTier add 'snapshot' to the Skipper+ branch.
Backend test assertions are inverted for the now-Community endpoints
and a positive Skipper-snapshot-task test is added.
Documentation across features/, api-reference/, and operations/ is
updated to reflect the new tier mapping.
* feat: add node last-contact tracking, fleet latency, and stack-restart summary
- DatabaseService: add last_successful_contact column to nodes table via
idempotent migration; expose updateNodeLastContact() and getStackRestartSummary()
methods; include the column in NODE_COLUMNS so getNodes/getNode return it
- fleet.ts: record latency_ms and last_successful_contact on each remote
node overview fetch; pilot-agent nodes surface pilot_last_seen instead;
pass db singleton into fetchRemoteNodeOverview to avoid redundant getInstance calls
- dashboard.ts: replace /recent-activity with /stack-restarts endpoint that
groups notification_history events by stack and category (crash/autoheal/manual)
over a configurable window (default 7 days, max 30)
* refactor(dashboard): remove redundant per-route authMiddleware
All routes under /api/ are covered by the global auth gate in app.ts.
The inline authMiddleware arguments on /configuration and /stack-restarts
were redundant with that gate and inconsistent with every other route in
the file. Remove them and drop the now-unused import.
* refactor(backend): consolidate Date.now(), move SQL aggregation, normalize node row mapping
- Capture a single completedAt timestamp in fetchRemoteNodeOverview to
eliminate two separate Date.now() calls and ensure latency_ms and
last_successful_contact are derived from the same instant
- Inline the redundant contactedAt variable; use completedAt directly
- Move stack-restart aggregation from JS into SQL (GROUP BY stack_name
with CASE/SUM counts), replacing the Map loop in the route handler
- Export StackRestartSummary interface from DatabaseService and remove
the duplicate local definition in dashboard.ts; handler now returns
the query result directly
- Add last_successful_contact normalization in decryptNodeRow, mirroring
the existing pilot_last_seen pattern
- Add authGate reliance comment above dashboardRouter route handlers
* feat(dashboard): replace Recent Activity card with context-aware Fleet Heartbeat / Stack Restart Map
- Multi-node installs (≥1 remote node): shows Fleet Heartbeat — real-time
reachability, latency, and container count per registered node
- Local-only installs: shows Stack Restart Map — 7-day restart frequency
per stack grouped by crash / auto-heal / manual category
- Conditional wrapper (DashboardActivityCard) switches states automatically
when the node list changes, with no page reload required
- Deletes RecentActivity card and hook (duplicated data already in Recent Alerts)
- Extracts formatRelativeTime to frontend/src/lib/utils.ts for reuse
* fix(dashboard): add pilot_last_seen to FleetNodeOverview and use it in getLastSeenLabel
* fix(fleet): expose mode and pilot_last_seen in overview, consolidate formatRelativeTime, drop em dash
- Add `mode` and `pilot_last_seen` (in seconds) to the FleetNodeOverview
interface and to both the pilot-agent and HTTP-proxy return paths in
fetchRemoteNodeOverview so the frontend getLastSeenLabel pilot branch
can fire correctly
- Remove the private formatRelativeTime from RecentAlerts.tsx and use
the shared implementation from lib/utils, converting the millisecond
timestamp at the call site
- Replace the em dash in getLatencyLabel with 'n/a' per project rules
|
||
|
Anso
|
2e8ca76103 | fix(backend): batch-insert stress test metrics to avoid per-insert fsync timeout (#911) | ||
|
Anso
|
685d5d729e |
feat(blueprints): backend foundation for fleet-wide compose templates (#860)
* feat(blueprints): add backend foundation for fleet-wide compose templates
Introduces the Blueprint Model: a docker-compose.yml plus a node selector
(labels or explicit IDs) that Sencho reconciles across the fleet. Backend
foundation only; the frontend tab and documentation follow.
Schema (DatabaseService):
- node_labels table for fleet-level orchestration tagging
- blueprints table with compose content, selector, drift_mode, classification
- blueprint_deployments table for per-node materialized state
- New idempotent migrate methods following the existing pattern
Services:
- BlueprintAnalyzer: pure compose-YAML classifier (stateless / stateful /
unknown) with 17 covered cases including named volumes, bind mounts,
external volumes, and tmpfs
- NodeLabelService: label CRUD plus selector matching helper (any/all/ids)
- BlueprintService: local + remote deploy/withdraw orchestration, marker
file management, name-conflict guard, per-(blueprint,node) lock
- BlueprintReconciler: 60-second loop with three-mode drift policy
(observe/suggest/enforce), state-aware guards, and Enforce-downgrade for
volume-destroying drift
Routes (gated requirePaid + requireAdmin on mutations):
- /api/blueprints (CRUD + apply + withdraw + accept + preview + analyze)
- /api/node-labels (CRUD + listAll + listDistinct)
Notifications: four new categories registered in NotificationService for
deploy/failure/drift events.
Bootstrap: reconciler start/stop wired in startup and shutdown.
Tests: 45 new Vitest cases covering selector matching, classifier rules,
state-aware guards, drift-mode branching, and marker parsing. Full backend
suite (1625 tests) passes; tsc clean.
* fix(lint): replace bare Function type in blueprint reconciler tests
Replace 8 occurrences of `as unknown as { computeDecision: Function }`
with a properly typed `ReconcilerWithCompute` alias that mirrors the
real method signature. Export `ReconcileDecision` from
BlueprintReconciler so the test can reference it.
Resolves @typescript-eslint/no-unsafe-function-type errors that were
failing the Backend (Lint) CI step.
|
||
|
Anso
|
7663f4cd8b |
feat(fleet): sencho mesh in traffic and routing tab (#858)
* feat(fleet): sencho mesh in traffic and routing tab Lights up Sencho Mesh: cross-node container forwarding rendered as if the container next to you were on localhost. Builds on the dormant TCP frame plumbing from the prior PR (pilot tunnel TCP frames + sencho-mesh sidecar package) and exposes the Admiral-only orchestrator surface. Backend - New mesh_stacks table (per-node opt-ins) + nodes.mesh_enabled column via DatabaseService.migrateMeshTables. - MeshService singleton: sidecar lifecycle via Dockerode, opt-in/out with cascading override regeneration, request-based resolver from sidecar control WS, cross-node TCP forwarding via PilotTunnelManager (same-node fast path included), in-memory 1000-event activity ring buffer with durable mirror to audit_log for state-change events, per-node and per-route diagnostics, and the Test upstream probe. - MeshComposeOverride: pure YAML generator that injects extra_hosts using host-gateway. The user's docker-compose.yml is never mutated; overrides live under DATA_DIR/mesh/overrides. - ComposeService deploy/update splice the override file when the stack is opted in; non-mesh stacks behave identically to today. - Pilot agent resolveMeshTarget consults the local mesh_stacks table (defense in depth) and resolves Compose containers via Dockerode. - /api/mesh router with 13 Admiral-gated endpoints covering status, enable/disable, stack opt-in/out, alias listing, per-route diagnostic, Test upstream probe, per-node diagnostic, sidecar restart, activity log paginated and SSE. - meshControl WS slot at /api/mesh/control validates the mesh_sidecar JWT minted by MeshService; dispatched as upgrade slot 2 (canonical order preserved). Frontend - New Traffic Routing tab in FleetView, gated by isAdmiral and wrapped in AdmiralGate. Tab uses the cyan brand glyph and italic-serif state typography from the audit. - RoutingTab masthead with mesh activity drawer, per-node card grid with TogglePill, alias rows with five-state pill taxonomy (healthy / degraded / unreachable / tunnel-down / not-authorized), inline Test buttons. - Four sheets: opt-in picker with port-collision inline error, per-route detail with diagnostic + filtered activity, per-node diagnostics with active streams + resolver cache + restart action, fleet-wide activity log with filters. - meshRouteState helper centralizes pill-state mapping; pure-function tests cover all five states. Docs - User docs at /docs/features/sencho-mesh.mdx covering opt-in, troubleshooting, security model (4 guarantees + 4 explicit non-guarantees), and V1 limitations. - Internal architecture and runbook pages. - websocket-dispatch internal doc updated with the new slot. * fix(mesh): validate stack name before path use; fix test DB lifecycle Two surgical fixes against the prior PR. Path-injection (CodeQL js/path-injection): MeshService.optInStack, optOutStack, ensureStackOverride, and removeStackOverride now validate stackName via isValidStackName from utils/validation, reject malicious names at the API boundary, and additionally check isPathWithinBase on the resolved override file path for defense in depth. The dataflow from req.params.stackName to fs.writeFile no longer reaches an unsanitized path expression. Test DB lifecycle: mesh-service.test.ts used per-test setupTestDb / cleanupTestDb, which deletes the temp dir while DatabaseService still holds an open SQLite handle. On Linux CI this raises SQLITE_READONLY_DBMOVED on the next prepare() because the inode has been unlinked. Switched to file-scoped beforeAll/afterAll matching agents-routes.test.ts, with a per-test beforeEach that truncates mesh_stacks plus non-default nodes and resets the MeshService singleton in-memory state. Adds a new test case asserting the path-traversal rejection. * fix(compose): use discovered compose filename instead of hardcoded docker-compose.yml composeArgs() hardcoded `-f docker-compose.yml` for every deploy. Sencho writes its canonical compose file as `compose.yaml`, so any stack created via the UI failed to deploy with `open ...docker-compose.yml: no such file or directory`. When no mesh override applies, drop the explicit `-f` so docker compose's built-in discovery resolves the actual filename. When an override exists, look up the real base filename via FileSystemService.getComposeFilename() and pass both files explicitly. Also hoist the MeshService import to module top now that the dependency is known to be acyclic, and revert the matching unit-test assertion. |
||
|
Anso
|
3e01daf76f |
feat(stack): per-stack activity timeline with actor attribution (#852)
* feat(stack): per-stack activity timeline with actor attribution Adds an Activity tab to the Stack Anatomy panel showing a timestamped event log for each stack: deploys, restarts, starts, stops, and image updates, attributed to the user who triggered them or 'system' for automated actions. Backend: - Extends notification_history with actor_username column (idempotent migration) and a partial composite index on (node_id, stack_name, timestamp DESC) for efficient per-stack lookups. - NotificationService.dispatchAlert() accepts an optional actor that is written to the new column. - Success-side dispatchAlert calls added after deploy, bulkContainerOp (start/stop/restart), and update handlers in routes/stacks.ts so user-initiated operations are recorded, not just failures. - New GET /api/stacks/:stackName/activity?limit&before endpoint with stack:read permission gate and cursor-based pagination. Frontend: - StackAnatomyPanel grows an Anatomy / Activity tab pair using the existing Tabs primitive. - StackActivityTimeline fetches the initial 50 events, paginates on demand, and prepends live events arriving over the existing WS notifications stream without duplicates. - NotificationPanel bell dropdown suppresses user-initiated success events (start/stop/restart/deploy/update triggered by a real user), keeping the tray focused on alerts and system events. * docs(stack): add stack activity timeline feature page and internal arch docs * fix(test): add actor_username to notification-routing history assertions dispatchAlert now passes actor_username to addNotificationHistory after the activity timeline PR added the column. Update the two exact-match assertions that were failing because the expected object shape was missing this field. |
||
|
Anso
|
5cf4323511 |
perf(backend): batch audit_log inserts into a buffered transaction (#817)
Every mutating /api/* request runs an individual INSERT into audit_log which serializes against other writers under burst load (SQLite's single-writer model). Buffer the writes in DatabaseService and flush them in a single transaction either every second or once the buffer reaches 100 entries, whichever comes first. Read paths (getAuditLogs, getAuditLogsInRange, cleanupOldAuditLogs) drain the buffer first so callers always see a consistent view, which keeps the existing test pattern of insert-then-read working. Graceful shutdown flushes before db.close() so no entries are lost on clean exit. The 1s flush timer is unref'd so the buffer cannot keep the process alive on its own. The CLI resetMfa script flushes explicitly before returning since it exits before the timer fires. |
||
|
Anso
|
836e384d17 |
perf(backend): cache global_settings reads in DatabaseService (#814)
getGlobalSettings() runs a SELECT * on every call and is hit from 22 files, including the auth middleware (every authenticated request), the WebSocket upgrade handler (every connection), and the debug-mode gate (every diagnostic log line). Cache the result inside the service on first read and invalidate on updateGlobalSetting(). The cached snapshot is Object.freeze'd and the public return type is now Readonly<Record<string, string>> so accidental mutations are caught at compile time. The settings GET handler that delete'd private keys now takes a defensive shallow copy first. The 5-second TTL cache in utils/debug.ts is now redundant and removed; the service-level cache is strictly fresher (invalidates on write rather than going stale for up to 5s). |
||
|
Anso
|
abee078741 |
feat(scheduler): add auto_backup, auto_stop, auto_down, auto_start and delete_after_run one-shot mode (#777)
* feat(scheduler): add auto_backup, auto_stop, auto_down, auto_start actions and delete_after_run one-shot mode Extends the scheduler with four new stack-targeted actions: - auto_backup: backs up stack compose files and .env using the existing FileSystemService.backupStackFiles primitive - auto_stop: runs compose stop (containers preserved) - auto_down: runs compose down (containers removed) - auto_start: runs compose up -d via deployStack (universal start for both stopped and down stacks) Adds delete_after_run boolean column to scheduled_tasks. When enabled, the task self-deletes after its first successful execution; failures keep the task so the user can debug and retry. All four new actions gate at Admiral tier, consistent with restart/snapshot/prune. Migration is idempotent (maybeAddCol). * docs(scheduler): update scheduled-operations doc with new lifecycle actions and delete-after-run Adds the four new actions (Backup Stack Files, Stop Stack, Take Stack Down, Start Stack) to the action table. Documents the delete-after-run one-shot mode with its success-only deletion semantics. Adds the Stack Lifecycle Scheduling section explaining stop-vs-down semantics and the local-execution boundary. Adds three troubleshooting entries: auto-start on a missing compose folder, auto-backup single-slot overwrite by design, and one-shot task disappearing after successful run. Updates the timeline description from four to five lanes. Refreshes screenshots to show the new dialog layout with the Lifecycle lane visible. |
||
|
Anso
|
e0034132b4 |
feat(notifications): match routing rules by labels and categories (#776)
- Add label_ids and categories columns to notification_routes via idempotent migration - Matcher logic always evaluates routes (AND semantics across all non-empty matchers) - getStackLabelIds skips DB call when no enabled route uses label filtering - Extract ALL_NOTIFICATION_CATEGORIES array from NotificationService as single source of truth - Derive VALID_CATEGORIES set from the array in the route handler - Extract validateLabelIds and validateCategories helpers to remove POST/PUT duplication - Extract tryAddColumn as a private DatabaseService class method (removes 5 local re-declarations) - Extract CATEGORY_LABELS to frontend/src/lib/notificationCategories.ts (shared by NotificationPanel and NotificationRoutingSection) - Frontend form adds label and category multiselects with AND-filter hint - Route cards show label and category badges; empty-matcher routes show 'Matches all alerts' - Add tests for category-only, label-only, and combined AND-semantics routing |
||
|
Anso
|
fcbdd59ec2 |
fix(notifications): scope routing rules to nodes via node_id column (#775)
Adds a nullable node_id column to notification_routes (null = any node, integer = fire only when the alert originates from that specific node). This fixes a multi-node fleet defect where a route scoped to "my-app" would fire on every node that hosts a stack with that name. Backend changes: - DatabaseService: idempotent migration adds node_id INTEGER NULL and a composite index on (node_id, enabled, priority); the two statements are in separate try-catch blocks so the index is always created even when the column was added in an earlier run - NotificationService: route matcher now pre-filters by node_id before checking stack_patterns (== null matches any node) - notifications route: POST/PUT accept optional node_id, validated to be null or the local node's ID; NodeRegistry guards against cross-node misroutes Frontend changes: - NotificationRoutingSection: node scope Select field uses useNodes() from NodeContext (no extra API call) to populate the local node option - Route cards show a node badge when node_id is set Tests: 3 new tests covering node-match, node-mismatch, and null-scope; all 75 files (1413 tests) passing. |
||
|
Anso
|
44dba59cab |
feat(notifications): add structured category enum to dispatcher and history (#774)
Introduce a NotificationCategory string-literal union (11 values) and thread it through dispatchAlert as a required second argument. All callers (DockerEventService, AutoHealService, ImageUpdateService, MonitorService, PolicyEnforcement, policyGate, SchedulerService, imageUpdates route) pass an explicit category at every call site, giving TypeScript compile-time enforcement that no new emit site can be added without choosing a category. DatabaseService gains an idempotent migration that adds a nullable category TEXT column to notification_history; existing rows keep category=NULL (displayed as Uncategorized in the UI). The getNotificationHistory method accepts an optional category filter that is forwarded from the GET /api/notifications/history route via a ?category= query param. NotificationPanel gains a category Select dropdown so users can filter history by category. The frontend types mirror the backend union so API responses are type-safe end-to-end. All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert signature and passing. |
||
|
Anso
|
af9cb0aa63 |
feat(auto-update): per-stack auto-update enable/disable toggle (#771)
* feat(auto-update): add per-stack auto-update enable/disable toggle Paid users (Skipper and Admiral) can now opt individual stacks out of scheduled auto-updates from the stack context menu without disabling the global feature. - Add stack_auto_update_settings table (node_id, stack_name) with default enabled=true; four typed DatabaseService accessors with parameterized queries. - Add GET /stacks/auto-update-settings, GET /stacks/:name/auto-update, and PUT /stacks/:name/auto-update (requirePaid + requireAdmin). PUT broadcasts state-invalidate with action auto-update-settings-changed so all open tabs refresh immediately. - Stack DELETE clears the auto-update setting row alongside stack_update_status. - autoUpdateRouter /execute skips disabled stacks before any registry call; skip is recorded in the results array. Manual Update actions are not affected. - Add Auto-update: Enabled/Disabled toggle in the stack inspect group (paid tiers only, hidden for Community, consistent with Auto-Heal). Toggle uses optimistic update with revert-on-error toast. - AutoUpdateReadinessView shows an Auto: Off pill and disables the Apply now button for stacks with auto-updates off. Detection still runs so the readiness card remains visible. - Add 21 backend Vitest tests covering DB round-trips, endpoint auth and tier gates, execute skip for both wildcard and named targets. Add 3 frontend hook tests for toggle visibility and callback behavior. * docs(auto-update): document per-stack auto-update control Add a Per-stack control section to the auto-update readiness page explaining how to disable and re-enable auto-updates for individual stacks, what disabling means (scheduled apply skipped; detection still runs; manual update unaffected), and a troubleshooting entry for scheduled runs not applying to a specific stack. |
||
|
Anso
|
24c0a2833b |
fix(security): clear cached policy evaluations when a scan policy is deleted (#758)
Vulnerability scans cache their policy verdict as a JSON blob in vulnerability_scans.policy_evaluation. Deleting a scan policy used to remove only the policies row and leave those blobs intact, so the scheduler kept emitting violations and stacks remained marked as blocked against a policy that no longer existed. deleteScanPolicy now nulls out policy_evaluation on every scan whose JSON references the deleted policy id, then deletes the policy row, in one transaction. |
||
|
Anso
|
ca5a930c68 |
refactor(backend): extract authMiddleware and introduce createApp factory (phase 2) (#732)
Phase 2 of the index.ts refactor. Pulls the auth middleware and session cookie issuers into their own module, and introduces the app.ts factory that owns the first nine steps of the canonical middleware pipeline. New modules: - middleware/auth.ts: authMiddleware, issueSessionCookie, issueMfaPendingCookie, clearMfaPendingCookie - app.ts: createApp() factory installing trust proxy, helmet, cors, compression, cookieParser, rate limiters, conditionalJsonParser, and nodeContextMiddleware. A header comment documents all 16 canonical middleware steps and where each currently lives. Changes: - middleware/authGate.ts: createAuthGate factory removed; authGate now imports authMiddleware directly (the factory existed only to avoid a circular import while authMiddleware lived in index.ts). - services/DatabaseService.ts: added API_TOKEN_SCOPE_TO_ROLE map so the auth middleware no longer inlines a stringly-typed record. - index.ts drops ~260 lines; auth routes, authGate, auditLog, apiTokenScope, remaining routes, static serving, and the error handler continue to be registered there until their respective phases. - vitest.config.ts bumps testTimeout to 30s and hookTimeout to 45s so fork-pool workers have enough headroom to ts-node-transform the growing module graph under CPU contention (64 workers each import the full Express stack in beforeAll). Code review fixes: use getErrorMessage() util in the auth catch block instead of inline cast; promote the scope-to-role map to a typed module-level constant. |
||
|
Anso
|
661b9c638b |
feat(security): enforce scan policies as a pre-deploy gate (#719)
Policies with block_on_deploy=1 now scan every stack image before docker compose up runs and reject the deploy with HTTP 409 on violation. The UI opens a dialog listing offending images; admins can override per deploy with ?ignorePolicy=true, and every bypass is recorded in the audit log with the originating route, actor, policy, and image list. When Trivy is not installed on the target node the gate fails open with a warning notification, so teams are never locked out by tooling state. Post-deploy and scheduled scans still evaluate matching policies and dispatch warnings on violations to surface drift on long-running stacks. Public API additions: policy and suppression CRUD under /api/security, plus the documented 409 block-response shape on all deploy paths. |
||
|
Anso
|
08f57c7141 |
feat(settings): surface security, notifications, and app store on remote nodes (#716)
Flip Security (Trivy), Notifications (agents + history), and App Store from global-and-hidden-on-remote to node-scoped so operators can manage them when a remote node is selected in the node picker. The primary instance proxies the calls to each remote, which resolves the correct per-instance binary state, agent config, and template registry. Backend: key `agents` and `notification_history` by `node_id` with idempotent column-add migrations and a `(node_id, type)` unique index on agents, matching the Labels pattern. Thread `req.nodeId` through the /api/agents and /api/notifications routes. Internal NotificationService and ImageUpdateService writes resolve the middleware default via `NodeRegistry.getDefaultNodeId()` so monitor-emitted rows share a bucket with user-facing ones (avoids split-brain where the UI sees test notifications but not internal alerts). Frontend: split Security on remote to render only the scanner card and hide scan policies and CVE suppressions (those remain control-plane-only). Drop the misleading "Always Local" badge on Developer since retention windows govern backend jobs, not UI state. Flip the App Store registry to node-scoped. Docs: add a "What Settings apply per node" table to multi-node, clarify remote alert setup in alerts-notifications, and note Trivy's per-host install in vulnerability-scanning. |
||
|
Anso
|
b95442c8f7 |
feat(ui): redesign global logs as cockpit surface (#699)
Rework the Logs view into the cockpit language: a PageMasthead strip with cyan rail, pulsing live dot, italic state word, and tracked-mono kicker; a SignalRail of EVENTS/MIN (with rolling 60s sparkline), ERRORS, WARNINGS, and CONTAINERS tiles; a segmented filter strip; a day-banded feed with severity dots and whole-row tint on ERROR/WARN; and a floating glass chip strip for pause/resume, clear, and download. Decouple the live log stream from the Developer Mode toggle so SSE runs by default for everyone, with polling as an invisible fallback when EventSource is unavailable. Drop the Standard Log Polling Rate setting and the global_logs_refresh key it persisted; Developer Mode still gates Real-Time Metrics and Debug Diagnostics and nothing else. Introduces the shared PageMasthead and SignalRail primitives for reuse across other cockpit pages. |
||
|
Anso
|
ed2a16af79 |
feat(notifications): deep-link bell rows to source stack and container logs (#692)
Add stack_name and container_name columns to notification_history so bell rows can act as jump points. Producers (AutoHeal, Docker events) pass the container context through dispatchAlert; the panel renders routable rows as buttons that load the target stack and, when a container name is present, open its logs modal. Non-structural notifications stay as passive display rows. |
||
|
Anso
|
591dc75d1e |
feat(audit-log): signal rail, day-banded stream, anomaly detection (#682)
Add a Stream view to the Audit Log that leads with a four-tile signal rail (events, actors, failure rate with inline sparkline, peak hour) and presents the feed grouped by day with severity dots, relative times, and inline anomaly callouts. The existing Table view is preserved behind a toggle for power users. Anomaly flags are computed at read time against strictly prior history and returned on demand via ?with_anomalies=1: - unusual_hour: hour outside the actor's central 7-day window - new_ip: IP unseen for this actor in the last 30 days - first_seen_actor: no prior history in the 30-day window New /audit-log/stats endpoint returns the signal-rail aggregates over 24h/7d/30d windows; stats are derived from a single 30-day scan. |
||
|
Anso
|
ec7620675e |
feat(app-store): editorial hero, category rail, security scan signal per tile (#679)
* feat(app-store): editorial hero, category rail, security scan signal per tile Rework the App Store view into an editorial layout with a 180px category rail, a featured-template hero, and compact tiles that surface star counts and vulnerability scan status at a glance. The deploy sheet splits into Essentials (one-click deploy) and Advanced (full port/volume/env control) tabs. Backend enriches /api/templates with scan_status, scan_cve_count, and a featured flag computed from the highest-star template with recorded stars. Scan lookups use a single batched SQL query to avoid N+1 round-trips. * fix(app-store): split firstSentence helper into its own module The firstSentence helper lived alongside the TemplateLogo component, which violates react-refresh/only-export-components — a file that exports a component must not also export non-component values, or Fast Refresh cannot establish an HMR boundary. Move the helper into appstore/util.ts and update the two import sites. |