Commit Graph

5 Commits

Author SHA1 Message Date
Anso 63213c0960 feat: add service-scoped Compose update and restore (#1648)
* feat: add service-scoped Compose update and restore

Allow updating or rebuilding one declared Compose service on multi-service
stacks without recreating siblings, with recovery snapshots, health-gate
observation, and prune holds for rollback images. Full-stack update paths
and single-service UX stay unchanged.

* fix: sanitize service-scoped update log messages for CodeQL

* fix: address service-scoped update audit findings B-01 through B-07

* fix: complete service-scoped update audit metadata and surfaces

* test: wrap Updates readiness tests for deploy-feedback context

* fix: keep service recovery reachable without Deploy Progress

Make failed service-gate recovery discoverable when Deploy Progress is
disabled or dismissed, suppress stale image-scan notification side
effects, normalize ComposeService line endings, and add focused
regression coverage.

* fix: resurface ContainersHealth density and expand on multi-service stacks

Service grouping hid the summary strip and Compact/Detailed/Expand controls that still applied to multi-container stacks.
2026-07-19 02:42:29 -04:00
Anso 48cebf9501 fix: bind deploy progress, request, and health gate to the captured node (#1357)
* fix: bind deploy progress, request, and health gate to the captured node

A deploy/update/install/git-apply re-read the active node from localStorage
independently at three points: the progress WebSocket at mount, the POST at call
time, and the health-gate poll. If the active node changed between the click and
any of those, the operation, its live output, and its health verdict could
target different nodes, and the socket and POST splitting across nodes broke
output streaming.

Capture the operation's node once when it starts and thread it through every
leg. A new nodeId option on apiFetch overrides the active-node read, the
progress terminal takes a nodeId prop for its socket URL, the health gate polls
on the captured node, and a failed gate records its recovery entry only on the
node it ran on. The surface, the request, and the gate now always agree.

* fix: scope failed-gate recovery to the file list's node and harden node targeting

Addresses review findings on the captured-node binding:

- Track the node the stack file list was fetched for (filesNodeId) and record a
  failed gate's recovery entry only when it matches the gate's node. This closes
  a race where switching back to the gate's node could match a same-named stack
  from the previous node's still-loaded list before the new list lands, keying
  the record to the wrong file and blocking the correct one. refreshStacks now
  carries a sequence token so an out-of-order resolution cannot leave files and
  filesNodeId inconsistent.
- Make an explicit apiFetch nodeId authoritative over a caller-supplied
  x-node-id header.
- Add the missing stack-logs nodeId cases (null, and active-node fallback) to the
  terminal tests.
2026-06-11 14:36:20 -04:00
Anso e20f1fe415 feat: add an inline deploy-progress style for the stack detail (#1355)
* feat: add an inline deploy-progress style for the stack detail

Deploy progress gains a presentation choice under Settings > Appearance >
Display: Modal (the default centered overlay) or Inline. In Inline style a
compact status band on the stack detail shows the running operation, its
elapsed time, the live phase, the latest output line, and the post-update
health gate result. A "View output" button opens the full log modal on
demand, a dismiss control clears the band, and the band auto-clears a few
seconds after a clean completion.

The live progress socket is lifted to an always-mounted owner so the band
streams without the modal; the default Modal style is unchanged. Operations
carry their node so a band never bleeds onto a same-named stack on another
node.

The stack detail's redundant "CONTAINERS" section heading is removed; the
band reserves that vertical space.

* fix: keep inline deploy progress reachable off the stack detail

Review of the inline presentation found a gap: a failed operation, an App
Store install, or navigating away leaves the inline session with no visible
surface, since the band only renders on the operation's own stack detail.
Restore the minimized pill as the inline fallback, shown only when the band
is not covering the session, so there is always a click-through to the log
without ever overlapping the band. Closing the modal for a failed op now
ends the session (the band has stepped aside) instead of only hiding it.

Also document the unsupported mid-operation style switch, and refresh the
deploy-progress, settings, appearance, and app-store docs for the renamed
"Deploy progress" setting and the Modal/Inline choice.
2026-06-11 10:33:57 -04:00
Anso 38aabe7064 feat: health-gated updates and rollback readiness (#1354)
* feat: classify stack deploy and update failures with suggested next actions

Failed deploy and update responses now carry a failure classification
(cause category, headline, and suggested next step) derived from the
compose error output. The recovery panel and chip render the
classification and include it in copied diagnostics, and gateway-style
failures surface as a node-unreachable cause.

* feat: add update and rollback readiness reports for stacks

Before a manual update, Sencho now shows an advisory readiness verdict
computed from the stored preflight result, open drift findings, live
container health, the pending image change, the rollback backup slot,
and node disk headroom. The Stack Dossier gains a rollback readiness
section that states what a rollback can restore and explicitly
discloses that volume and bind-mounted data are not covered. Toolbar
and sidebar updates now share one update path, and admins can create a
fleet snapshot from the readiness dialog before updating. Nodes that do
not advertise the capability keep the direct update flow.

* feat: observe stack health after updates with a post-deploy health gate

After a deploy or update succeeds, Sencho now watches the stack for a
configurable observation window and records a passed, failed, or
unknown verdict: containers must stay running, healthchecks must report
healthy, and restart loops or disappearing containers fail the gate.
The deploy panel shows the observation live and holds off auto-closing
until the verdict lands, a failed gate surfaces the existing recovery
actions including rollback, and the stack timeline records update
started and gate verdict events. Scheduled, webhook, bulk, and
git-source updates are gated the same way; rollbacks and installs are
deliberately not. The gate is observational only and can be tuned or
disabled per node under host alert settings.

* docs: document health-gated updates and rollback readiness

New operator page covering the update readiness dialog, the post-update
health gate and its settings, the rollback readiness disclosure, and
classified failures, with cross-links from the atomic deployments and
deploy progress pages. The API reference gains the readiness and
health-gate endpoints, the healthGateId success field, and the failure
classification schema on deploy and update error responses.

* feat: withhold the success verdict while the health gate observes

An update used to show a green Succeeded that a failed health gate then
contradicted moments later. The deploy modal now reports Verifying
health while the gate observes, shows success only when the gate
passes, and makes a failed or unknown gate the headline result; success
toasts soften to a verifying message while a gate runs. The mobile
recovery card groups its actions behind one bottom-right Take action
menu so it stays compact on a phone, with the classified cause still
visible on the card. A successful image update now also counts as the
last known-good marker in rollback readiness, and the docs gain
screenshots of the readiness dialog, gate states, dossier section, and
settings.

* fix: harden log format strings and the env existence path check

Log calls that interpolated the stack name into the console format
string now use constant format strings with placeholder arguments, and
envExists validates path containment inline at its filesystem access,
matching the established patterns used elsewhere in the same files.

* test: adapt deploy modal success specs to the post-deploy health gate

The deploy feedback modal now withholds its success verdict while the
health gate observes the new containers, showing "Verifying health"
until the gate passes. The two success-path E2E tests waited for
"Succeeded" within the gate's 90s default window and timed out.

Shorten the observation window to the 15s minimum for these tests via
the settings API, assert the verify-then-succeed sequence the modal
actually renders, and restore the default window afterward so the test
value does not leak into later runs.

* fix: serialize health gate polling and harden gate observation

Address race conditions in the post-update health gate found in review.

Backend: the gate poller used setInterval, so a Docker observe slower
than the 5s tick could overlap the next poll and corrupt the restart and
missing-container accounting, and a wedged socket could leave a poll
pending forever. Polling is now single-flight: each cycle self-schedules
the next only after it settles, and the observe is bounded by an 8s
timeout so a hung probe counts as a poll error and resolves the gate
unknown after three in a row.

Frontend: the gate poller could overlap requests, letting a slow earlier
"observing" response overwrite an already-applied terminal verdict. It is
now single-flight with a terminal latch, so a late response can never
roll the UI back from passed or failed.

Also reject a non-digit nodeId on the snapshot coverage route instead of
letting parseInt coerce it, document that turning off the deploy progress
panel opts out of the live gate UI while the gate still runs server-side,
and add gate-coverage tests for the webhook, git source, and auto-update
apply paths plus the new single-flight, observe-timeout, and recovery
cases.
2026-06-11 00:26:26 -04:00
Anso d369b03a38 feat: detect stalled stack updates and add in-app recovery actions (#1347)
* feat: detect stalled stack updates and add in-app recovery actions

Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.

Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.

* fix: harden rollback against policy-blocked file mutation and refine recovery

Address review findings on the stalled-update recovery work:

- The rollback route restored backup files before running the policy gate, so
  a policy-blocked rollback could leave the on-disk config rolled back while the
  deployed containers were unchanged. Snapshot the current files first and
  revert them when the gate blocks; if that revert itself fails, escalate it on
  the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
  redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.

* test: mock snapshotStackFiles in the atomic-deploy rollback route tests

The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.
2026-06-10 10:12:24 -04:00