9 Commits

Author	SHA1	Message	Date
Anso	38aabe7064	feat: health-gated updates and rollback readiness (#1354) ... * feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.	2026-06-11 00:26:26 -04:00
Anso	d369b03a38	feat: detect stalled stack updates and add in-app recovery actions (#1347) ... * feat: detect stalled stack updates and add in-app recovery actions Add a backend idle-output backstop that stops a deploy/update compose step that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a hung image pull surfaces a fast failure instead of spinning indefinitely. Surface failed, timed-out, and stalled operations with recovery actions on the stack page: a desktop chip plus popover menu and an inline mobile card offering retry, restart, roll back (when a backup exists), refresh state, and copy diagnostics, all gated by deploy permission. The streaming deploy/update progress modal is now on by default and warns when output goes quiet. Container state is refreshed after a failed or stalled operation, and the UI never sits in an indefinite spinner. * fix: harden rollback against policy-blocked file mutation and refine recovery Address review findings on the stalled-update recovery work: - The rollback route restored backup files before running the policy gate, so a policy-blocked rollback could leave the on-disk config rolled back while the deployed containers were unchanged. Snapshot the current files first and revert them when the gate blocks; if that revert itself fails, escalate it on the persistent alert feed since the 409 is already sent. - Refresh container state after a successful manual rollback (rollback redeploys), without mis-recording a refetch failure as a rollback failure. - Suppress the stalled-output warning once live progress is unavailable. * test: mock snapshotStackFiles in the atomic-deploy rollback route tests The rollback route now snapshots stack files before restoring a backup, so its FileSystemService mock needs snapshotStackFiles. Without it the mocked call threw and the route returned 500, failing the success-path rollback assertions.	2026-06-10 10:12:24 -04:00
Anso	865d792874	feat(pricing): collapse to two tiers (#1309) ... * feat(pricing): collapse to two tiers (Community + Admiral) Collapse Sencho's pricing from three tiers (Community / Skipper / Admiral) to two: a generous free Community tier and a single paid Admiral tier. The Skipper tier is removed. Now free in Community: auto-heal, auto-update, scheduled operations, webhooks, notification routing, Fleet Actions and bulk operations, SSO preset providers (Google / GitHub / Okta), unlimited users with admin and viewer roles, and deploy safety (atomic deploys, auto-rollback, and one-click rollback). Admiral (paid) is focused on running and governing a fleet: blueprints, Fleet Secrets, deploy enforcement, vulnerability report export, audit log, host console, private registries, mesh networking, node cordon, managed cloud backup, LDAP / Active Directory SSO, and the advanced RBAC roles (deployer, node-admin, auditor) with per-resource scoped assignments. Internally the license variant distinction is removed so tier is binary (community / paid). License validation still verifies the Lemon Squeezy store and product before granting paid status. Docs and the contributor guide are updated to the two-tier model. * docs(pricing): correct licensing page to two-tier pricing and tidy stale tier wording The licensing docs page kept the old Admiral pricing plus a Founder Lifetime column and an Enterprise paragraph after the two-tier collapse. Update it to $12/month or $99/year, drop the lifetime and Enterprise content, and link to the pricing page for current pricing. Also fix stale "Skipper" wording in CLA.md, SUPPORT.md, one test title, and three test comments. Historical CHANGELOG entries and the retired-Skipper license-guard test are intentionally left as-is. * docs: align licensing and SSO pages with the two-tier model Correct the SSO overview so the Google, GitHub, and Okta presets read as available on every tier, matching the provider table; only LDAP and Active Directory require Sencho Admiral. Remove the lifetime-plan references from the licensing, settings, and troubleshooting pages so they reflect subscription-only Admiral pricing. * fix(rbac): omit scoped permissions from /me on the Community tier Scoped role assignments only take effect on the paid tier, but GET /api/permissions/me returned them unconditionally, so a downgraded instance with leftover assignments rendered per-resource affordances the API then rejected with 403. The endpoint now mirrors the permission middleware and includes scoped permissions only on the paid tier. Adds a regression test covering the downgrade case. * docs: use custom-pricing wording on the contact page The two-tier model has no Enterprise tier; reword the contact page's enterprise pricing/deals to custom pricing/deals so it does not imply a tier that no longer exists.	2026-06-04 17:45:53 -04:00
Anso	2d56ea958a	fix(stack-activity): per-stack history integrity, attribution, sanitization (#1228) ... * fix(stack-activity): per-stack history integrity, attribution, sanitization Address the Stack Activity audit findings (PR 1 of 2): - Per-stack history integrity: drop the per-insert 100-row prune in addNotificationHistory that evicted quieter stacks' history whenever another stack got chatty. Periodic cleanupOldNotifications now caps per (node, stack) at 500 rows and per-node unattached system events at 1000 rows, on top of the existing 30-day retention. Signature takes an options bag and returns a per-stage summary so MonitorService can log what actually ran each cycle. - Actor attribution: thread req.user?.username through every notifyActionFailure call site and add synthetic actors at service emit sites (system:autoheal, system:scheduler, system:image-update, system:docker-events, system:blueprint, system:monitor, system:policy). The timeline renders system actors as "via <Label>" so an autoheal redeploy is no longer indistinguishable from a user redeploy. - Message sanitization: new sanitizeNotificationMessage at NotificationService.dispatchAlert strips KEY=VALUE pairs whose key ends in TOKEN/KEY/PASSWORD/SECRET/CREDENTIALS/AUTH, scrubs HTTP basic auth in URLs and Bearer tokens, collapses COMPOSE_DIR paths, and truncates to 1000 chars. Applied to the stored history and to every downstream Discord/Slack/webhook channel. The ImageUpdateService recovery-path direct DB write also runs through the sanitizer. - Composite pagination cursor: getStackActivity now accepts a (timestamp, id) cursor (?before=&beforeId=). The legacy timestamp-only form silently dropped events when a single compose up emitted many events sharing one millisecond. Route rejects beforeId without before. - Frontend hardening: distinct error state with retry button (initial fetch failure no longer renders as the genuine empty state), strict positive-integer parsing on cursor params, overrequest-by-1 pagination so the last page does not leave a dead "Load more" click, runtime guard on liveEvents merge that validates the level union, per-minute day-bucket recompute so an open panel does not stay on "Today" past midnight. No tier, role, or capability gate touched. Route permission gate remains stack:read on the named stack. * fix(stack-activity): sanitizer covers lowercase env vars and per-node compose dir External review surfaced two leak paths in the message sanitizer: - The sensitive-key regex was uppercase-only. Compose env names are conventionally uppercase but lowercase forms (db_password, jwt_secret, github_token) are valid and do leak through the same Docker and compose-parse error paths. Make the regex case-insensitive and tighten it to also catch bare TOKEN= / KEY= / PASSWORD= without a prefix word, while still leaving BYPASS, COMPASS, and similar non-secret keys alone. - The compose-dir path collapse only read process.env.COMPOSE_DIR, but the real resolution chain is node.compose_dir (per-node DB override) -> process.env.COMPOSE_DIR -> /app/compose. A node with a custom compose_dir could still leak absolute paths into stored history and downstream channels. Route both the dispatchAlert call and the ImageUpdateService recovery-path direct write through NodeRegistry.getInstance().getComposeDir(localNodeId) so the collapse covers every resolution outcome. Tests now assert lowercase keys are redacted and that BYPASS-style non-secrets stay intact in both cases. notification-routing mock extended to stub the new getComposeDir call. * chore(stack-activity): a11y roles, visibility-aware tick, live-disconnect signal Close three small follow-ups on the per-stack activity timeline: - A11y: each day-group gets role="list" and each event row gets role="listitem" so screen readers traverse the timeline as a list instead of a wall of text. The day-group container also carries an aria-label naming the bucket. - Visibility-aware day-bucket tick: the 60s setInterval that re-derives Today/Yesterday/Earlier now short-circuits when document.hidden, so a backgrounded panel does not re-render every minute for no visible effect. - Live-disconnect signal: useNotifications dispatches a sencho:notifications-connection custom event on WebSocket open and close. The timeline listens and, when explicitly disconnected, shows a one-line "Live updates offline; reconnecting…" hint above the list. The sidebar ticker already surfaces fleet-wide connection state; this adds an in-context cue for users who are focused on a single stack. Stack-name case normalization was considered and rejected: stack names are case-permissive per the isValidStackName validator, and lowercasing on read or write would silently rename or hide a user's "MyApp" stack. * ci(stack-activity): drop unnecessary escape in URL_BASIC_AUTH regex ESLint no-useless-escape errored on \- inside the character class [a-zA-Z0-9+.\-] at notificationMessage.ts:14. Move the dash to the end of the class so it's an unambiguous literal and the escape is no longer required. Behavior is identical; sanitizer tests still pass. * revert(stack-activity): drop unvalidated E2E spec from this PR The spec was committed without ever running against a real Docker daemon, then failed in CI when it ran for the first time: deploy returned 200 but no notification appeared on the activity endpoint within the polling window, suggesting either a deploy-notification race or a node-id resolution mismatch in the CI environment. Backend unit tests (route + composite cursor + sanitizer) and frontend component tests cover the same logic. The E2E spec will land in a dedicated follow-up once it has been authored against a working CI environment.	2026-05-25 21:09:00 -04:00
Anso	523ba5854c	fix(stacks): return 404 for nonexistent stacks on deploy/down/update (F-7) (#1108) ... POST /api/stacks/:name/{deploy,down,update} previously returned HTTP 500 with body {"error":"spawn docker ENOENT"} when invoked against a stack whose compose directory was missing. The status code was wrong (the named resource did not exist, so 404 is the right answer) and the message misled operators into thinking the docker CLI was unavailable. Add a small requireStackExists(nodeId, stackName, res) helper in routes/stacks.ts that validates the stack name and confirms a compose file is present via FileSystemService.hasComposeFile before any of the three handlers spawn docker compose. The helper is called immediately after requirePermission and before runPolicyGate so unauthorized callers still get 403 first and the policy gate never runs against a phantom stack. In ComposeService.execute(), narrow the child.on('error') handler so the genuine docker-binary-missing case (ENOENT on the spawn itself) rejects with "Docker CLI unavailable on this node" instead of the raw "spawn docker ENOENT". This is defense in depth for the rare case the pre-check cannot cover, and it fixes the misleading-message half of the bug as well. Cover the new contract with stack-actions-missing-stack.test.ts (four cases: deploy/down/update return 404, invalid name returns 400). Mock ComposeService as a tripwire so a future code path that bypasses the guard would fail loudly. Fix stacks-failure-notifications.test.ts by adding hasComposeFile to its FileSystemService partial mock so the existing happy-path-error-handling cases continue to flow into ComposeService.	2026-05-19 00:13:57 -04:00
Anso	5461bc316b	fix: harden stack management operations (#1046)	2026-05-14 10:21:18 -04:00
Anso	328a98439d	fix: harden vulnerability scan scheduling (#1035)	2026-05-13 09:40:45 -04:00
Anso	74ae2ce0c6	fix: harden atomic deployment rollback (#1029) ... * fix: harden atomic deployment rollback * fix: update Docker toolchain to Go 1.26.3 * fix: repair Dockerfile tr argument split across lines * fix: bump protobufjs to clear npm audit high-severity advisories * fix: sanitize error objects in console.error to prevent log injection	2026-05-12 15:58:30 -04:00
Anso	dd9d33813b	feat(deploy-logs): opt-in deploy progress modal with structured log rows (#779) ... * feat(notifications): dispatch deploy_failure alert on stack action errors * feat(terminal): add onReady and onMessage callback props * feat(deploy-logs): add DeployLogContext with runWithLog API * feat(deploy-logs): add DeployLogPanel bottom drawer with resize and minimize * feat(deploy-logs): wire DeployLogContext to App and EditorLayout action runners * test(deploy-logs): add E2E test for deploy log panel open, failure, and minimize * docs(deploy-logs): add user-facing and internal architecture docs * feat(deploy-logs): redesign as opt-in modal with structured log rows Replace the full-width bottom drawer (DeployLogPanel) with a centered modal that streams structured log output for deploy, stop, restart, update, install, and Git apply operations. The modal is disabled by default; users opt in from Settings -> Appearance. Core changes: - New DeployFeedbackContext with runWithLog() API: if opt-in is off, silently bypasses the UI so all call sites degrade to the existing toast behavior without code changes. - composeLogParser.ts: pure parser that strips ANSI escapes and classifies compose output into stage badges (PULL, BUILD, CREATE, START, STOP, DOWN, WARN, ERR, LOG). 15 unit tests. - StructuredLogRow.tsx: memoized row with timestamp, stage badge, and message. Error rows get a rose left rail; warn rows get a tinted bg. - DeployFeedbackModal: Dialog-based, max-w-640px/max-h-70vh, elapsed timer, auto-close 4s on success (hover cancels), persistent on failure. Raw xterm output collapsible in footer. - DeployFeedbackPill: minimized state anchored top-right, survives navigation, click restores modal. - Wires App Store install (action: install), Git apply (action: deploy), and Git pull (action: update) in addition to the existing EditorLayout actions. - Fixes Terminal.tsx WS URL in generic mode (was connecting to root path not proxied by Vite; now uses /ws). - Settings: adds "Show deploy progress modal" checkbox to Appearance. - Docs: renames deploy-logs.mdx to deploy-progress.mdx; updates internal architecture doc. * fix(deploy-logs): connect Terminal in generic mode and move pill to bottom-center Terminal was passed stackName which routes it to the stack logs WS (container stdout). In that mode onReady is never called, so the deployStarted gate never resolves and the compose command never runs. Remove stackName so Terminal uses generic WS mode, which calls onReady on open and streams compose output. Also reposition the minimized pill from top-right to bottom-center (fixed bottom-6 left-1/2 -translate-x-1/2) per UX feedback. * docs(deploy-logs): update pill position to bottom center * test(deploy-logs): rewrite E2E spec for deploy feedback modal The old spec targeted the removed bottom-drawer DeployLogPanel and used the wrong field name when calling POST /api/stacks (sent 'name' but the endpoint reads 'stackName'), causing every test to fail with a 400 before any UI assertions ran. Fixes: - POST /api/stacks body now uses 'stackName' matching the API contract - All locators updated to target the new DeployFeedbackModal and DeployFeedbackPill components (data-testid attributes added) - Added enableDeployFeedback helper to opt-in via localStorage before each test that expects the modal (feature is off by default) - Added opt-in OFF test to confirm the modal is suppressed when disabled - Minimize/expand test now asserts the pill appears and contains the stack name before clicking to restore the modal * test(deploy-logs): fix compose file write endpoint in E2E helper createStackViaApi was calling PUT /api/stacks/:name/files/docker-compose.yml which does not exist. The correct endpoint is PUT /api/stacks/:name with { content } in the body. * test(deploy-logs): use addInitScript to persist opt-in across reloads The opt-in flag was set via page.evaluate before setupDeployStack, which calls page.reload() and loginAs (a second navigation). Although localStorage should persist across same-origin reloads, the React tree was reading 'false' on remount in CI. Switching to addInitScript guarantees the localStorage value is set before any page script on every navigation, so useDeployFeedbackEnabled's useState initializer always sees the right value when React mounts. * test(deploy-logs): verify localStorage and re-dispatch event before deploy Adds syncDeployFeedbackState() called right before each deploy click in the ON tests. It both verifies localStorage is set (failing the test loudly with a clear message if not) and re-dispatches the SENCHO_SETTINGS_CHANGED event to defeat any stale React state after navigation. If the modal still does not appear with the assertion green, the issue is downstream of localStorage and we have a clear signal. * test(deploy-logs): wait for React re-render after dispatching opt-in event After syncDeployFeedbackState dispatches SENCHO_SETTINGS_CHANGED, React schedules the state update but does not flush it synchronously. The click that follows can fire against the stale closure where isEnabled is still false, so runWithLog takes its early-return path and the modal never opens. A 200ms wait is enough to let React commit the new state before the next interaction. * test(deploy-logs): wait for stack file fetch before clicking deploy deployStack() in EditorLayout returns early at 'if (!selectedFile)' without calling runWithLog. selectedFile is set inside loadFile() after GET /api/stacks/:name resolves. The previous setup clicked the stack in the sidebar and immediately asked the test to click Deploy, racing the fetch. CI backend logs confirmed no deploy POST ever fired for the ON tests, while the OFF test passed only because it asserts non-existence. Now setup awaits both the stack click and the file response together, then verifies the action bar's deploy button is visible before returning. * test(deploy-logs): wait for network idle and capture browser logs Adds a networkidle wait plus a 500ms settle after the stack click so React commits selectedFile and any follow-up env/container/backup fetches drain before the deploy click. Also mirrors browser console errors and pageerrors into the Playwright output so the next failure ships with the React stack trace instead of just a 'modal not visible' message. * test(deploy-logs): temporary debug logging in runWithLog Adds a console.log at the entry of runWithLog so we can see in CI logs whether it is being called and what isEnabled value the closure has. Also widens the test's console capture to include these debug lines. This is diagnostic only and will be removed once the root cause of the modal-not-opening-in-CI failure is identified. * test(deploy-logs): debug log at deployStack entry to trace click path Adds console.log at the first line of deployStack handler so we can confirm in CI whether the click is reaching it at all and what selectedFile/isStackBusy resolve to. Combined with the existing runWithLog debug logs, this isolates whether the modal failure is in deployStack guarding out, runWithLog early-returning, or something else entirely. * test(deploy-logs): drop filter, log every browser console msg The previous filter only emitted error/warning plus the deploy-feedback substring. The deploy-feedback debug logs never appeared, so we don't yet know whether the log itself is firing. Remove the filter so the full console stream shows up in CI. * test(deploy-logs): app-level console log to verify capture pipeline If even an unconditional log at App component render time does not appear in CI browser logs, then the console capture listener is broken or the dispatched logs are being filtered upstream of Playwright. This isolates whether the issue is in the production code or the test harness. * test(deploy-logs): use testid locator for stack action button Replaces the regex-based getByRole locator (/Deploy|Start/i) with getByTestId('stack-deploy-button'). The regex matched something other than the actual deploy button: backend logs proved no deploy POST ever fired, and instrumentation confirmed neither deployStack nor runWithLog ran on click despite the test claiming success. Adds data-testid='stack-deploy-button' to both the Restart and Start button branches in EditorLayout's action bar so the same locator works whether the stack is running or not. Also drops the temporary debug console.log entries in deployStack, runWithLog, and App, and restores the test's console listener filter to only emit error and warning messages. * test(deploy-logs): park cursor in corner so auto-close countdown fires After clicking the deploy button, the cursor lands inside the centered modal. The modal pauses its 4s auto-close countdown on hover, so the HAPPY test was waiting for a close that never happened. page.mouse.move to (0,0) parks the cursor outside the modal before the success banner appears, letting the countdown complete. * test(deploy-logs): drop redundant loginAs after page.reload page.reload preserves auth cookies, so the page lands back on the dashboard without needing a fresh login. The loginAs call after reload was racing on isLoginPage(): a transient login-page state during page load made loginAs commit to filling #username, then the dashboard committed and #username never came back. Playwright's auto-wait then hung the fill until the test's 120s timeout, which also dragged later stacks.spec tests down with collateral timeouts. waitForStacksLoaded is enough to confirm we're on the dashboard with the sidebar populated before clicking the new stack. * test(e2e): make loginAs race-safe when login page is a false positive isLoginPage() reports the page as a login screen if the Login button locator reports visible at the moment of the check. Under CI load (more real container deploys from the deploy-log-panel suite), the auth context can render the login form for one paint, then redirect to the dashboard. The original code committed to filling #username and hung until the test timeout when the field was no longer there. Now the login branch waits up to 2s for #username to actually appear before filling. If it never appears, we fall through to the dashboard check instead of hanging.	2026-04-26 00:43:54 -04:00