* fix: bind deploy progress, request, and health gate to the captured node
A deploy/update/install/git-apply re-read the active node from localStorage
independently at three points: the progress WebSocket at mount, the POST at call
time, and the health-gate poll. If the active node changed between the click and
any of those, the operation, its live output, and its health verdict could
target different nodes, and the socket and POST splitting across nodes broke
output streaming.
Capture the operation's node once when it starts and thread it through every
leg. A new nodeId option on apiFetch overrides the active-node read, the
progress terminal takes a nodeId prop for its socket URL, the health gate polls
on the captured node, and a failed gate records its recovery entry only on the
node it ran on. The surface, the request, and the gate now always agree.
* fix: scope failed-gate recovery to the file list's node and harden node targeting
Addresses review findings on the captured-node binding:
- Track the node the stack file list was fetched for (filesNodeId) and record a
failed gate's recovery entry only when it matches the gate's node. This closes
a race where switching back to the gate's node could match a same-named stack
from the previous node's still-loaded list before the new list lands, keying
the record to the wrong file and blocking the correct one. refreshStacks now
carries a sequence token so an out-of-order resolution cannot leave files and
filesNodeId inconsistent.
- Make an explicit apiFetch nodeId authoritative over a caller-supplied
x-node-id header.
- Add the missing stack-logs nodeId cases (null, and active-node fallback) to the
terminal tests.
* feat: detect stalled stack updates and add in-app recovery actions
Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.
Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.
* fix: harden rollback against policy-blocked file mutation and refine recovery
Address review findings on the stalled-update recovery work:
- The rollback route restored backup files before running the policy gate, so
a policy-blocked rollback could leave the on-disk config rolled back while the
deployed containers were unchanged. Snapshot the current files first and
revert them when the gate blocks; if that revert itself fails, escalate it on
the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.
* test: mock snapshotStackFiles in the atomic-deploy rollback route tests
The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.
* refactor(auto-update): retire per-stack gate, drive auto-update from schedules only
The per-stack Auto-update toggle in the stack sidebar context menu wrote a
gate row to `stack_auto_update_settings`, but actual updates only ran when a
`scheduled_tasks` row with `action='update'` fired. On a fresh install the
toggle was inert: detection ran every 6h, nothing was applied.
The same context menu already exposes `Schedule task`, which opens
ScheduledOperationsView pre-filled for the stack where the user can pick
`Auto-update Stack` and any cron. Keeping the toggle alongside that flow
duplicated the same action and turned the gate table into a parallel store
of "is a covering schedule active" derivable from `scheduled_tasks` itself.
Drop the gate model entirely:
- Backend: remove the `stack_auto_update_settings` table and its four
accessors, the three routes under /api/stacks/*/auto-update, the per-stack
skip in /api/auto-update/execute and SchedulerService.executeUpdate's
fleet branch, and the clearStackAutoUpdateSetting call on stack delete.
Dashboard `autoUpdate` count derives from scheduled_tasks (action='update'
rows pinned to the node, total/enabled split).
- Frontend: drop the Auto-update entry from the sidebar context menu and its
optimistic toggle plumbing. Drop autoUpdateSettings state, the
/stacks/auto-update-settings fetch, and the auto-update-settings-changed
WebSocket branch. Slim useSidebarActivitySummary (just nextRunAt; no
enabled/total counts). AutoUpdateReadinessView's per-card autoUpdateEnabled
now means "a covering enabled action='update' schedule exists" (per-stack
row or fleet row on this node, earliest next_run_at wins, per-stack row
wins on ties), with the gate-fetch removed.
- New: scheduledTasksRouter broadcasts scope: 'scheduled-tasks' on POST,
PUT, PATCH /toggle, and DELETE so useConfigurationStatus and
useNextAutoUpdateRun refetch under the 250ms debounce instead of waiting
for the 60s poll. The broadcast is wrapped so a broken subscriber socket
cannot turn a successful mutation into a 500.
- Docs: rewrite the "Per-stack control" section of auto-update-policies.mdx
to describe the schedule-based model; update the matching troubleshooting
entry. The misleading fleet-update help text in ScheduledOperationsView
is corrected to reflect that every stack on the node is covered.
Tier parity: the surviving auto-update path (Schedule task -> Auto-update
Stack / All Stacks) is gated `requirePaid + requireAdmin` backend and
`isPaid + isAdmin` frontend, matching the gate the deleted routes carried.
The pre-commit grep returns no tier-related diff outside this PR's scope.
No data migration is provided: greenfield rules apply, and the leftover
table on already-shipped instances is harmless because no code reads or
writes it after this PR.
* docs: sweep remaining references to the per-stack auto-update toggle
The previous commit retired the per-stack Auto-update gate in favor of
configuring auto-update purely through scheduled tasks. This commit
removes the now-stale mentions of that toggle across the operator docs:
- docs/features/sidebar.mdx: drop the Auto-update entry from the Inspect
group description, the matching screenshot alt-text, and the Skipper
Note that listed it. Schedule task now carries the cross-link to
Auto-Update Policies.
- docs/features/stack-management.mdx: drop the Auto-update list item;
refresh the Schedule task entry to mention the Auto-update Stack action.
- docs/features/dashboard.mdx: rename the Configuration Status row from
"Auto-update stacks" to "Auto-update schedules" with the new value
shape, and rewrite the troubleshooting accordion to describe the
scheduled-tasks invalidation path.
- docs/features/scheduled-operations.mdx: rewrite the Auto-update All
Stacks row and helper text to reflect that every stack on the node is
covered (no per-stack opt-out from this surface anymore).
- docs/features/multi-node.mdx: rewrite the Updates column definition to
derive the Auto/Off flag from enabled Auto-update Stack / Auto-update
All Stacks schedules instead of the removed per-stack policy.
The auto-update-policies.mdx rewrite in the previous commit already
covered the main reference page. The sidebar-context-menu.png screenshot
will be refreshed on release once the new menu is live in production;
the alt text is updated in this commit so it accurately describes the
shipping state.
No website edits needed: the Auto-Update Policies feature card description
("Schedule automatic image pulls and redeployments per stack on your own
cadence") and the feature matrix labels ("Auto-update stack schedule",
"Auto-update all stacks schedule") remain accurate under the new model.
* fix(stacks): drop orphaned requireAdmin import after auto-update route removal
CI's backend lint step flagged this PR's earlier deletion of the three
/api/stacks/*/auto-update routes: those handlers were the only callers of
`requireAdmin` inside routes/stacks.ts, leaving the named import on line 15
unreferenced. `requirePaid` and `effectiveTier` from the same line are still
in use elsewhere in the file and stay.
tsc --noEmit does not flag unused named imports; ESLint's no-unused-vars
does. Local backend lint reproduces and now reports 0 errors against the
existing 334-warning baseline.
Per-node fetch failures in useCrossNodeStackSearch were silently
swallowed into []. The user could not tell the difference between
"this node has no matching stack" and "this node timed out or 502'd".
Adds a third return value to the hook: failedNodes: FailedNode[]
where FailedNode is { nodeId, nodeName, reason }. The reason captures
the HTTP status text (e.g. "list returned HTTP 502") or the
underlying Error message ("connect ECONNREFUSED 192.168.x.x:1852")
without leaking node URLs. AbortError from the effect cleanup path
is intentionally excluded - it's expected when the user keeps typing.
Threads the new field through useStackListState and EditorLayout into
StackList. StackList renders a warning chip below the "Other nodes"
header when the array is non-empty:
! N nodes unreachable > (expand)
Click expands the chip to a vertical list of "node: reason" lines.
Hover surfaces the full reasons in the title attribute too, so a
user inspecting at a glance can read them without clicking. The chip
is suppressed entirely when the search yields no failures (no zero
state). Color is the existing warning token, not error - the failure
is recoverable (node may come back) and not a system-wide problem.
GlobalCommandPalette also calls useCrossNodeStackSearch but ignores
the third return value; its destructuring is additive-safe.
Resolves M-2 from the stack-management audit.
After applying a stack update the sidebar's "update available" dot stayed
visible and the stack's status indicator was stuck on the optimistic value
until the page was manually refreshed. Two root causes:
1. Image-updates state refresh was a fire-and-forget call in some paths and
entirely missing from the bulk-update, auto-update, and state-invalidate
WebSocket-handler paths.
2. stackActionsRef.current was resynced only at render time, so the post-
update refreshStacks(true) running in the action's finally block read a
stale "busy" map and preserved the optimistic mask via prev[file] ?? status.
Backend now broadcasts a state-invalidate event with scope='image-updates'
and action='stack-updated' after every successful update (single-stack route
and auto-update loop). The frontend useNotifications hook routes this to a
new onImageUpdatesChange callback wired to fetchImageUpdates in EditorLayout,
so every connected client refreshes the dot through the same code path.
Bulk update also calls fetchImageUpdates directly for fast local feedback,
and setStackAction/clearStackAction now keep stackActionsRef synchronously
in sync with state so the busy-stack check inside refreshStacks observes
the cleared map immediately.
Adds 3 unit tests covering the new WS branch (positive, scope-mismatch
negative, auto-update-settings-changed negative).
Moves the stack-list state cluster (14 useState calls, plus refs, memos,
effects, and 7 absorbed sub-hooks) into a new hook at
EditorLayout/hooks/useStackListState.ts, following the same pattern as
useEditorViewState (B4-5).
EditorLayout useState count: 42 -> 28. useEffect count unchanged.
Also fixes a stale-closure bug in isStackBusy: previously read from
stackActions state, making isBusy stale inside buildMenuCtx whenever
a stack action fired between dep-array updates. Now reads from
stackActionsRef.current so it is always current without needing to be
listed as a dep. Wrapping it in useCallback makes the function
reference stable.
Anso