Restore verification iterated the backup directory listing, so a file
recorded in the .checksums manifest but absent from the backup slot was
never checked. The orphan removal then deleted the live file and the copy
restored nothing, reporting success while leaving the stack unrecoverable.
Walk the manifest instead of the directory listing: a recorded file that
the slot no longer holds now aborts the restore before any file is touched,
alongside the existing corrupt-content check. Also fail backup creation when
a managed file exists but cannot be read (non-ENOENT), instead of silently
omitting it and producing an incomplete backup with the same failure mode.
Give every scheduled action an explicit, predictable target model
(Action then Node then Stack then Options then Schedule):
- System Prune now exposes a Node picker and requires a node, so it can
no longer run silently on the default node.
- Vulnerability Scan and System Prune list local nodes only; both run on
the hub-local Docker daemon and reject remote nodes on the backend.
- Restart Stack service discovery loads services from the selected node
via fetchForNode instead of the active or local node.
- Fleet Snapshot shows a read-only "Scope: Entire fleet" summary.
Backend gains a shared local-node guard and prune node validation on
create and update, plus an executor-level remote-node guard, so the
frontend and backend validation now agree for every action.
* fix(deploy): verify atomic-deploy backup integrity before restore
Atomic deploy and the Rollback action restore a stack from a backup of
its compose file and .env. A backup truncated or corrupted at write time
(out of disk, interrupted copy) was copied back silently, overwriting a
working stack with bad content.
The backup now writes a .checksums manifest holding a SHA-256 of each
backed-up file, and a restore re-hashes every file and compares it before
touching the stack. A mismatch aborts the restore with a clear error and
leaves the live files unchanged. Backups without a manifest, and files
with no recorded checksum, are restored unverified so a rollback is never
blocked by missing integrity data.
* fix(deploy): guard backup source reads with an inline path barrier
The integrity change reads each managed file from the stack directory before
hashing it. Static analysis flags those reads because the source path derives
from the user-provided stack name and the containment check lived in a helper
it does not trace. Re-establish containment inline at each read sink, resolving
against the compose base and confirming the path stays within it, mirroring
snapshotStackFiles. Behavior is unchanged for valid stacks; the guard only
rejects a path that escapes the compose directory, which validation already
prevents.
* test(deploy): assert compose stays put when the backup .env is corrupt
Strengthen the .env-corruption test so it also mutates the live compose.yaml
and asserts it is left untouched, proving the integrity abort halts before any
file is copied back rather than relying on the backup happening to match. Also
note on the test hash oracle that it matches production for UTF-8 text fixtures.
* feat: detect stalled stack updates and add in-app recovery actions
Add a backend idle-output backstop that stops a deploy/update compose step
that has gone silent (SENCHO_COMPOSE_STALL_TIMEOUT_MS, default 10m), so a
hung image pull surfaces a fast failure instead of spinning indefinitely.
Surface failed, timed-out, and stalled operations with recovery actions on
the stack page: a desktop chip plus popover menu and an inline mobile card
offering retry, restart, roll back (when a backup exists), refresh state,
and copy diagnostics, all gated by deploy permission. The streaming
deploy/update progress modal is now on by default and warns when output
goes quiet. Container state is refreshed after a failed or stalled
operation, and the UI never sits in an indefinite spinner.
* fix: harden rollback against policy-blocked file mutation and refine recovery
Address review findings on the stalled-update recovery work:
- The rollback route restored backup files before running the policy gate, so
a policy-blocked rollback could leave the on-disk config rolled back while the
deployed containers were unchanged. Snapshot the current files first and
revert them when the gate blocks; if that revert itself fails, escalate it on
the persistent alert feed since the 409 is already sent.
- Refresh container state after a successful manual rollback (rollback
redeploys), without mis-recording a refetch failure as a rollback failure.
- Suppress the stalled-output warning once live progress is unavailable.
* test: mock snapshotStackFiles in the atomic-deploy rollback route tests
The rollback route now snapshots stack files before restoring a backup, so its
FileSystemService mock needs snapshotStackFiles. Without it the mocked call
threw and the route returned 500, failing the success-path rollback assertions.
* fix(atomic-deploy): harden rollback locking, restore fidelity, and tier gating
Hardens the Atomic Deployments feature found during a full audit:
- Rollback now holds the per-stack lifecycle lock (deploy/update already do),
so a rollback can no longer race a concurrent deploy on the same compose
files. Adds a 'rollback' lifecycle action and releases the lock in finally.
- restoreStackFiles is now a faithful revert: it removes managed compose/.env
files added after the backup before copying, so a rollback no longer leaves a
hybrid of old and new configuration. Scope is the protected file set only;
user data is untouched. Aborts (rather than reporting success) if a stale
managed file cannot be removed.
- The scheduled image-update path derives the atomic flag from the licence tier
instead of hardcoding it on, keeping the paid capability explicit at the call
site (the scheduler is already paid-gated; this prevents silent drift).
- The backup-metadata read (GET /stacks/:name/backup) now requires a paid
licence, matching the rollback flow that is the only caller.
- Manual rollback dispatches a success/failure notification, alongside the
existing audit-log entry.
Adds route integration tests (lock acquisition/release, tier 403, notifications,
no-backup 404), filesystem tests for the faithful restore (orphan removal,
variant switch, abort path, non-managed files preserved), a community-tier
scheduler test, and a developer-mode logging matrix. Documents the restore
semantics and reconciles the scheduled-update wording in the feature guide.
* fix(atomic-deploy): assert restore target stays within the compose dir before unlink
The orphan-removal step in restoreStackFiles joins the stack directory with a
managed filename and unlinks it. The stack directory is already validated and
contained by resolveStackDir (allowlist stack name + within-base assertion), but
the containment guard was not reapplied to the joined target at the delete sink,
so static analysis flagged the path as derived from user input. Reassert
containment on the final path before unlinking, matching the barrier the other
write/read helpers in this service already apply. No behavior change for valid
stacks; defense-in-depth at the sink.
* fix(atomic-deploy): inline the path-containment barrier at the restore unlink sink
The wrapped within-base assertion was not recognized as a sanitizer by the
static path-injection analysis, which still traced the stack name to the unlink
sink. Replace it with the inline path.resolve + startsWith containment check the
other write helpers in this service already use (the recognized barrier), kept
in the same scope as the sink. Behavior is unchanged for valid stack names.
* fix(atomic-deploy): clear stale managed files from the backup slot before writing
The backup directory is reused across runs and was only ever added to, never
cleared. A managed file removed from the stack since the last backup (e.g. a
deleted .env or a switched compose variant) lingered in the slot, so a later
rollback restored a file that did not exist immediately before the failed run,
contradicting the faithful-revert guarantee. Clear the protected file set from
the slot before copying the current files, with the same inline containment
barrier the restore path uses. A clear failure is logged, not fatal, since it
only risks a stale future rollback and should not block a valid deploy.
The Skipper/Admiral atomic deploy/update path used to create
.sencho-backup/ inside the user's stack folder, which silently failed
with EACCES whenever a container had chowned the bind mount (swag,
tautulli, linuxserver/* images, etc). That broke auto-rollback and the
manual rollback endpoint for those stacks. Stack backups now live under
<DATA_DIR>/backups/<stackName>/ next to sencho.db, which is always
writable by the Sencho user.
While stress-testing the same scenario, MonitorService also flooded the
error log with "Error parsing stats for container ... 404 no such
container" because per-container stats polls (30s tick) raced with
docker compose recreating containers. The 404 case is now skipped
silently; non-404 stats failures still log at error level.
Anso