mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 20:00:08 +00:00
Anso
45844b92ca
* fix(atomic-deploy): harden rollback locking, restore fidelity, and tier gating Hardens the Atomic Deployments feature found during a full audit: - Rollback now holds the per-stack lifecycle lock (deploy/update already do), so a rollback can no longer race a concurrent deploy on the same compose files. Adds a 'rollback' lifecycle action and releases the lock in finally. - restoreStackFiles is now a faithful revert: it removes managed compose/.env files added after the backup before copying, so a rollback no longer leaves a hybrid of old and new configuration. Scope is the protected file set only; user data is untouched. Aborts (rather than reporting success) if a stale managed file cannot be removed. - The scheduled image-update path derives the atomic flag from the licence tier instead of hardcoding it on, keeping the paid capability explicit at the call site (the scheduler is already paid-gated; this prevents silent drift). - The backup-metadata read (GET /stacks/:name/backup) now requires a paid licence, matching the rollback flow that is the only caller. - Manual rollback dispatches a success/failure notification, alongside the existing audit-log entry. Adds route integration tests (lock acquisition/release, tier 403, notifications, no-backup 404), filesystem tests for the faithful restore (orphan removal, variant switch, abort path, non-managed files preserved), a community-tier scheduler test, and a developer-mode logging matrix. Documents the restore semantics and reconciles the scheduled-update wording in the feature guide. * fix(atomic-deploy): assert restore target stays within the compose dir before unlink The orphan-removal step in restoreStackFiles joins the stack directory with a managed filename and unlinks it. The stack directory is already validated and contained by resolveStackDir (allowlist stack name + within-base assertion), but the containment guard was not reapplied to the joined target at the delete sink, so static analysis flagged the path as derived from user input. Reassert containment on the final path before unlinking, matching the barrier the other write/read helpers in this service already apply. No behavior change for valid stacks; defense-in-depth at the sink. * fix(atomic-deploy): inline the path-containment barrier at the restore unlink sink The wrapped within-base assertion was not recognized as a sanitizer by the static path-injection analysis, which still traced the stack name to the unlink sink. Replace it with the inline path.resolve + startsWith containment check the other write helpers in this service already use (the recognized barrier), kept in the same scope as the sink. Behavior is unchanged for valid stack names. * fix(atomic-deploy): clear stale managed files from the backup slot before writing The backup directory is reused across runs and was only ever added to, never cleared. A managed file removed from the stack since the last backup (e.g. a deleted .env or a switched compose variant) lingered in the slot, so a later rollback restored a file that did not exist immediately before the failed run, contradicting the faithful-revert guarantee. Clear the protected file set from the slot before copying the current files, with the same inline containment barrier the restore path uses. A clear failure is logged, not fatal, since it only risks a stale future rollback and should not block a valid deploy.
84 lines
8.0 KiB
Plaintext
84 lines
8.0 KiB
Plaintext
---
|
|
title: Atomic Deployments
|
|
description: Wrap every deploy and update in a backup, a 3-second health probe, and an automatic rollback when a container crashes.
|
|
---
|
|
|
|
Sencho wraps every protected deploy in a four-step safety net: it copies the current `compose.yaml` and `.env` to a writable backup directory, runs the requested compose action, waits 3 seconds for the new containers to settle, then checks them for a non-zero exit code. If any container has crashed, Sencho restores the backed-up files and re-deploys automatically.
|
|
|
|
The same backup also powers the **Rollback** action in the stack editor, so you can roll a stack back to its last good configuration on demand.
|
|
|
|
<Note>
|
|
Atomic Deployments require a Sencho **Skipper** or **Admiral** license. Community Edition runs the same compose actions without a backup or automatic rollback.
|
|
</Note>
|
|
|
|
## How it works
|
|
|
|
1. **Backup.** Before the action runs, Sencho copies `compose.yaml` (or `compose.yml` / `docker-compose.yaml` / `docker-compose.yml`) and `.env`, if present, into the backup directory. The deploy progress modal streams `=== Backup created for atomic deployment ===` once the copy completes, before any `docker compose` output.
|
|
2. **Run the action.** Sencho executes the requested compose action: `up -d` for a deploy, or a pull-then-`up -d` recreate for an update.
|
|
3. **Health probe.** Sencho waits 3 seconds, then lists every container with the `com.docker.compose.project=<stack>` label and checks each one for a non-zero exit code. Any container that has exited with a non-zero status counts as a crash.
|
|
4. **Auto-rollback on failure.** When a crash is detected, Sencho streams `=== Deployment failed - rolling back to previous version ===`, restores the backed-up files, and re-runs `docker compose up -d` with the restored configuration. On success it streams `=== Rolled back successfully ===`. The original deploy error is preserved and reported as the deploy result, so a failed-then-rolled-back deploy still registers as a failure in the deploy progress modal.
|
|
|
|
If the rollback itself fails (for example, the re-deploy step cannot pull a previously available image, or the file restore is blocked by filesystem permissions), Sencho streams `=== Rollback failed - manual intervention may be required ===`. The backup files remain at `<DATA_DIR>/backups/<stack>/` so you can copy them back manually.
|
|
|
|
## Which operations are protected
|
|
|
|
Atomic deployments wrap:
|
|
|
|
- **Deploy** and **Update** from the stack editor's action bar.
|
|
- **App Store** installs of a new stack.
|
|
- **Webhook** triggers for deploy and pull actions.
|
|
- **Image auto-updates** triggered by an auto-update policy.
|
|
|
|
A scheduled image-update task uses the same atomic wrapper as a manual update, so a recurring update still takes a backup and rolls back automatically when a container crashes. Scheduled lifecycle actions (start, stop, restart) change no stack configuration and run `docker compose` directly without a backup.
|
|
|
|
## Manual rollback
|
|
|
|
The stack editor's action bar carries a **More actions** overflow menu (the three-dot icon next to **Update**). Open it on a Skipper or Admiral instance and you'll see **Rollback** at the top, with the timestamp of the most recent backup rendered beneath the label. Selecting it restores the backed-up files and re-runs `docker compose up -d` non-atomically, to avoid nesting a rollback inside another atomic wrapper and overwriting the good backup with the broken state from the just-failed deploy.
|
|
|
|
<Frame>
|
|
<img src="/images/atomic-deployments/rollback-menu.png" alt="Stack editor action bar with the More actions overflow menu open, showing the Rollback entry at the top with the backup timestamp rendered beneath the label, followed by Scan config and Delete entries" />
|
|
</Frame>
|
|
|
|
The menu entry is hidden when no backup exists for the stack, for example on a freshly created stack that has never been deployed atomically, and on a Community Edition instance. The endpoint additionally requires the `stack:deploy` permission, so a user without it will see the menu entry but receive a permission error if they invoke it.
|
|
|
|
## Where backups are stored
|
|
|
|
Backups live under `<DATA_DIR>/backups/<stack>/`, in the same writable volume Sencho uses for its database and other persisted state. They are intentionally kept outside the user's compose folder, so the operation works even when a container has chowned its bind-mounted stack directory to root.
|
|
|
|
Each backup is a flat copy of the compose file Sencho found, plus `.env` if it exists, plus a `.timestamp` marker that records when the backup was taken. There is one backup slot per stack: every protected deploy or update overwrites the previous backup, so the **Rollback** menu always reverts to the configuration that was on disk immediately before the most recent run.
|
|
|
|
A restore is a faithful revert, not an overlay. Sencho replaces the compose file and `.env` with the backed-up copies and removes any compose variant or `.env` that was added after the backup was taken, so the stack returns to exactly the file set it had before the run. For example, if a deploy switched the stack from `compose.yaml` to `docker-compose.yml` or introduced a new `.env`, a rollback undoes both. Files Sencho does not manage are left untouched.
|
|
|
|
## Community Edition behavior
|
|
|
|
On Community Edition, Sencho runs the same `docker compose` commands without the atomic wrapper. There is no backup, no health probe, and no automatic rollback, and the **Rollback** menu entry is hidden. On a Skipper or Admiral license, atomic deployments are active immediately for every protected action; no configuration is required.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="The Rollback option is not in the More actions menu">
|
|
Sencho hides the entry whenever a rollback is not possible. The most common reasons are:
|
|
|
|
- The stack has never been deployed atomically, so no backup file exists yet. Run **Deploy** or **Update** once and the entry will appear.
|
|
- The instance is on Community Edition. Atomic Deployments require Skipper or Admiral.
|
|
|
|
A user without the `stack:deploy` permission will still see the menu entry; the rejection comes from the backend with a permission error after they click. Ask an admin to grant `stack:deploy` through **Settings · Roles & Access** if that happens.
|
|
</Accordion>
|
|
<Accordion title="The deploy succeeded but a service crashed seconds later">
|
|
The health probe is a 3-second window after `docker compose up -d` returns. Crashes that happen after that window are out of scope for atomic rollback by design, because Sencho cannot tell whether a late exit is a real failure or a normal restart. For ongoing health, use **Auto-Heal Policies** to restart unhealthy containers automatically and **Alert Rules** to page you when a container exits unexpectedly.
|
|
</Accordion>
|
|
<Accordion title="The deploy progress modal showed 'Rollback failed - manual intervention may be required'">
|
|
This message means the auto-rollback attempted to restore the backup and re-deploy, but the restore step or the re-deploy itself errored out. The backup files are still at `<DATA_DIR>/backups/<stack>/`. To recover:
|
|
|
|
1. Copy `compose.yaml` (or the variant Sencho backed up) and `.env` from `<DATA_DIR>/backups/<stack>/` back into the stack directory.
|
|
2. Open the stack in the editor and click **Deploy** to re-run with the restored configuration.
|
|
|
|
The most common causes are filesystem permissions on the stack directory and a missing image in a private registry that the original deploy could not pull.
|
|
</Accordion>
|
|
<Accordion title="Rollback ran but the stack still has the broken configuration">
|
|
Sencho keeps a single backup per stack. If you ran two atomic deploys back to back, the second deploy overwrote the first backup with the broken configuration before it failed. **Rollback** then restores that broken configuration, because as far as Sencho is concerned it is the most recent known state.
|
|
|
|
To recover, edit the compose file or `.env` directly in the editor, fix the bad change, and click **Deploy**. The next protected deploy will write a fresh backup of the now-good configuration.
|
|
</Accordion>
|
|
</AccordionGroup>
|