* feat: detect stalled stack updates and add in-app recovery actions
Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.
Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.
* fix: harden rollback against policy-blocked file mutation and refine recovery
Address review findings on the stalled-update recovery work:
- The rollback route restored backup files before running the policy gate, so
a policy-blocked rollback could leave the on-disk config rolled back while the
deployed containers were unchanged. Snapshot the current files first and
revert them when the gate blocks; if that revert itself fails, escalate it on
the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.
* test: mock snapshotStackFiles in the atomic-deploy rollback route tests
The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.
* fix(atomic-deploy): harden rollback locking, restore fidelity, and tier gating
Hardens the Atomic Deployments feature found during a full audit:
- Rollback now holds the per-stack lifecycle lock (deploy/update already do),
so a rollback can no longer race a concurrent deploy on the same compose
files. Adds a 'rollback' lifecycle action and releases the lock in finally.
- restoreStackFiles is now a faithful revert: it removes managed compose/.env
files added after the backup before copying, so a rollback no longer leaves a
hybrid of old and new configuration. Scope is the protected file set only;
user data is untouched. Aborts (rather than reporting success) if a stale
managed file cannot be removed.
- The scheduled image-update path derives the atomic flag from the licence tier
instead of hardcoding it on, keeping the paid capability explicit at the call
site (the scheduler is already paid-gated; this prevents silent drift).
- The backup-metadata read (GET /stacks/:name/backup) now requires a paid
licence, matching the rollback flow that is the only caller.
- Manual rollback dispatches a success/failure notification, alongside the
existing audit-log entry.
Adds route integration tests (lock acquisition/release, tier 403, notifications,
no-backup 404), filesystem tests for the faithful restore (orphan removal,
variant switch, abort path, non-managed files preserved), a community-tier
scheduler test, and a developer-mode logging matrix. Documents the restore
semantics and reconciles the scheduled-update wording in the feature guide.
* fix(atomic-deploy): assert restore target stays within the compose dir before unlink
The orphan-removal step in restoreStackFiles joins the stack directory with a
managed filename and unlinks it. The stack directory is already validated and
contained by resolveStackDir (allowlist stack name + within-base assertion), but
the containment guard was not reapplied to the joined target at the delete sink,
so static analysis flagged the path as derived from user input. Reassert
containment on the final path before unlinking, matching the barrier the other
write/read helpers in this service already apply. No behavior change for valid
stacks; defense-in-depth at the sink.
* fix(atomic-deploy): inline the path-containment barrier at the restore unlink sink
The wrapped within-base assertion was not recognized as a sanitizer by the
static path-injection analysis, which still traced the stack name to the unlink
sink. Replace it with the inline path.resolve + startsWith containment check the
other write helpers in this service already use (the recognized barrier), kept
in the same scope as the sink. Behavior is unchanged for valid stack names.
* fix(atomic-deploy): clear stale managed files from the backup slot before writing
The backup directory is reused across runs and was only ever added to, never
cleared. A managed file removed from the stack since the last backup (e.g. a
deleted .env or a switched compose variant) lingered in the slot, so a later
rollback restored a file that did not exist immediately before the failed run,
contradicting the faithful-revert guarantee. Clear the protected file set from
the slot before copying the current files, with the same inline containment
barrier the restore path uses. A clear failure is logged, not fatal, since it
only risks a stale future rollback and should not block a valid deploy.
The Skipper/Admiral atomic deploy/update path used to create
.sencho-backup/ inside the user's stack folder, which silently failed
with EACCES whenever a container had chowned the bind mount (swag,
tautulli, linuxserver/* images, etc). That broke auto-rollback and the
manual rollback endpoint for those stacks. Stack backups now live under
<DATA_DIR>/backups/<stackName>/ next to sencho.db, which is always
writable by the Sencho user.
While stress-testing the same scenario, MonitorService also flooded the
error log with "Error parsing stats for container ... 404 no such
container" because per-container stats polls (30s tick) raced with
docker compose recreating containers. The 404 case is now skipped
silently; non-404 stats failures still log at error level.
Anso