mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-15 21:23:17 +00:00
feat(recovery): complete authored-project atomic rollback generations (#1819)
* feat(recovery): capture complete authored Compose project for atomic rollback Replace the root-compose-only backup slot with staged recovery generations that record the managed inventory, exact Compose invocation, and prior image identity, and wire the same engine through deploy, update, manual rollback, and Git apply. * fix(recovery): satisfy CodeQL path barriers and update-guard mock Inline resolve+startsWith checks at generation/inventory fs sinks and stub getCurrentStackUpdateRecovery in UpdateGuardService tests. * fix(recovery): drop unused FileSystemService import in generation store test * fix(recovery): harden authored-project rollback for upgrade and restore safety Preserve legacy UUID backup rows, restore Git deploy state with files, make multi-file restore recoverable, evaluate policy on the restored target, and fail closed when Git capture cannot cover an apply. * fix(recovery): unblock Git apply unit tests and CodeQL pre-restore TOCTOU Mock recovery capture in git-source-service tests after fail-closed apply capture, and re-resolve live paths immediately before pre-restore snapshot reads. * fix(recovery): fall back to authored inventory when Git manifesto is missing First Git apply captures before promote, so a missing managed-project manifesto must not block rollback capture when the live stack already has authored files. * fix(recovery): make authored-project rollback atomic across Git state Restore the managed-project manifesto with files, keep nullable Git identity on first-apply captures, persist Git side-state in restore intents for startup reconcile, compensate legacy materialize failures, and refuse directory collisions before mutation. * fix(recovery): satisfy CodeQL path and TOCTOU barriers on manifesto restore Add inline resolve barriers for manifesto read/clear sinks and remove the access-then-read race when restoring a generation manifesto snapshot. * fix(recovery): close third-audit rollback generation blockers Fail closed on incomplete Git inventory fallbacks, execute captured Compose invocation during recovery, refuse startup and mutations while restore intents remain unresolved, propagate legacy stale-delete failures, and add Docker-level exact prior-image coverage plus regression tests. * fix(recovery): mark acquired before handoff in prior-image Docker test Match the production updateStack CAS sequence so the exact prior-image integration test does not fail handoff from the captured phase. * fix(recovery): close fourth-audit rollback safety blockers Evaluate policy against held images, use index-based pre-restore snapshots, hold the shared stack lock across Git apply, replay Mesh and empty captured invocations exactly, restore POSIX modes with fail-closed sensitive permissions, keep case-sensitive paths, and link Git auto-deploy health gates. Add regression coverage for these cases. * test(recovery): fix mocks for health-gate link and authored compose args Add linkGateOrRetain to the Git apply recovery mock, and mock authoredComposeArgs so the case-collision inventory test is not masked by a missing getComposeDir stub. * fix(recovery): close fifth-audit rollback safety blockers Share git_apply locking for webhook auto-apply, fail closed on malformed recovery service records, refuse mixed-image capture, and require exact probe counts with hold-tag eligibility checks. * fix(recovery): close sixth-audit rollback safety blockers Preserve the legacy backup slot during generation capture, encrypt sensitive pre-restore snapshots, revert files on a failed health probe without committing Git, fail closed when an absent-file revert would delete a directory, skip Compose one-offs, route manual and scheduled backup through the current generation, and persist runtime image platform identity. * fix(recovery): close seventh-audit rollback safety blockers Fleet snapshot restore and restore-all now capture a recovery generation under the stack lock before any authored file write, including on remote nodes. * fix(recovery): keep pre-deploy generations during health-gate observe Link deploy recovery generations to the observing gate so backup cannot replace them mid-observe. Distinguish missing hold tags from probe failures, refuse generation release when services metadata is corrupt, classify mixed-replica and coverage refusals, and toast the backend rollback message. * fix(recovery): wrap webhook deploy case for eslint const bindings in an unbraced switch case trip no-case-declarations. Match the pull case block.
This commit is contained in:
Anso
@@ -1,21 +1,21 @@
|
||||
---
|
||||
title: Atomic Deployments
|
||||
sidebarTitle: Atomic deploys
|
||||
description: Wrap every deploy and update in a backup, a 3-second health probe, and an automatic rollback when a container crashes.
|
||||
description: Wrap every deploy and update in a managed authored-project backup, prior-image holds, a 3-second health probe, and automatic rollback when a container crashes.
|
||||
---
|
||||
|
||||
Sencho wraps every protected deploy in a four-step safety net: it backs up the current compose file, `.env`, and any configured project env files, runs the compose action, waits 3 seconds for containers to settle, then checks for a non-zero exit code. If any container crashed, Sencho restores the backup and re-deploys automatically.
|
||||
Sencho wraps every protected deploy in a four-step safety net: it backs up the managed authored project (compose inventory, invocation metadata, and Git revision state when the stack is Git-linked), captures a recovery generation with prior image identity (opaque holds), runs the compose action, waits 3 seconds for containers to settle, then checks for a non-zero exit code. If any container crashed after handoff, Sencho restores from that generation and re-deploys automatically.
|
||||
|
||||
The same backup also powers the **Rollback** action in the stack editor, so you can roll a stack back to its last good configuration on demand. To see in advance whether that rollback would actually help, and to watch container health for longer than the 3-second probe, see [Health-Gated Updates](/features/health-gated-updates).
|
||||
The same recovery generation powers the **Rollback** action in the stack editor, so you can roll a stack back to its last good configuration on demand. To see in advance whether that rollback would actually help, and to watch container health for longer than the 3-second probe, see [Health-Gated Updates](/features/health-gated-updates).
|
||||
|
||||
## How it works
|
||||
|
||||
1. **Backup.** Before the action runs, Sencho copies `compose.yaml` (or `compose.yml` / `docker-compose.yaml` / `docker-compose.yml`), `.env` if present, and any project env files configured for the stack (for example, `stack.env` or `.env.production`) into the backup directory. The deploy progress modal streams `=== Backup created for atomic deployment ===` once the copy completes, before any `docker compose` output.
|
||||
1. **Backup.** Before the action runs, Sencho captures the managed authored project into a recovery generation: ordered compose files, overrides, discoverable `include:` / `extends` inputs, and env, label, config, and secret files when they can be enumerated. For Git-linked stacks it also records the managed-project manifesto and Git revision identity so a later restore can converge with the next pull. It records prior image identity as opaque holds. The deploy progress modal streams `=== Capturing rollback generation for atomic deploy ===` (or the matching update line) once capture completes, before any `docker compose` output.
|
||||
2. **Run the action.** Sencho executes the requested compose action: `up -d` for a deploy, or a pull-then-`up -d` recreate for an update.
|
||||
3. **Health probe.** Sencho waits 3 seconds, then lists every container with the `com.docker.compose.project=<stack>` label and checks each one for a non-zero exit code. Any container that has exited with a non-zero status counts as a crash.
|
||||
4. **Auto-rollback on failure.** When a crash is detected, Sencho streams `=== Deployment failed - restoring previous compose and env files ===`, restores the backed-up files, and re-runs `docker compose up -d` with the restored configuration. On success it streams `=== Restored previous compose and env files ===`. The restore reverts the compose and `.env` configuration. An image on a moving tag (such as `latest`) is not reverted, because the local tag still resolves to the newly pulled digest. The original deploy error is preserved as the deploy result, so a failed-then-rolled-back deploy still registers as a failure.
|
||||
3. **Health probe.** Sencho waits 3 seconds, then lists every container with the `com.docker.compose.project=<stack>` label and checks each one for a non-zero exit code. Any container that has exited with a non-zero status counts as a crash. Recovery capture and the post-rollback compensation probe skip Compose one-off (`docker compose run`) containers so they are not counted as service replicas.
|
||||
4. **Auto-rollback on failure.** When a crash is detected after handoff, Sencho streams `=== Deployment failed - restoring previous runtime from recovery generation ===` (or the matching update line), restores the captured authored inventory, and re-runs `docker compose up -d` with the restored configuration. Where the generation holds a prior image ID, restore retargets that exact image for moving tags and for supported local builds. Named volumes and bind-mounted application data are not restored. The original deploy error is preserved as the deploy result, so a failed-then-rolled-back deploy still registers as a failure.
|
||||
|
||||
If the rollback itself fails (for example, the re-deploy step cannot pull a previously available image, or the file restore is blocked by filesystem permissions), Sencho streams `=== Rollback failed. Manual intervention may be required ===`. The backup files remain at `<DATA_DIR>/backups/<nodeId>/<stack>/` so you can copy them back manually.
|
||||
If the rollback itself fails (for example, a held image is missing, or the file restore is blocked by filesystem permissions), Sencho streams `=== Rollback failed. Manual intervention may be required ===`. The recovery generation remains available so you can retry or recover manually.
|
||||
|
||||
## Which operations are protected
|
||||
|
||||
@@ -26,69 +26,65 @@ Atomic deployments wrap:
|
||||
- **Webhook** triggers for deploy and pull actions.
|
||||
- **Image auto-updates** triggered by an auto-update policy.
|
||||
|
||||
A scheduled image-update task uses the same atomic wrapper as a manual update, so a recurring update still takes a backup and rolls back automatically when a container crashes. Scheduled lifecycle actions (start, stop, restart) change no stack configuration and run `docker compose` directly without a backup.
|
||||
A scheduled image-update task uses the same atomic wrapper as a manual update, so a recurring update still takes a backup and rolls back automatically when a container crashes. Scheduled lifecycle actions (start, stop, restart) change no stack configuration and run `docker compose` directly without a backup. On-demand **Backup** and scheduled stack backup capture a current recovery generation of the managed authored project. Backup refuses while a health gate is still observing after a deploy or update, so the pre-operation generation stays available for rollback.
|
||||
|
||||
[Blueprint](/features/blueprint-model) deploys are a separate, fleet-wide desired-state path and do not use this atomic wrapper: a failed Blueprint apply is not backed up or rolled back automatically.
|
||||
|
||||
## Manual rollback
|
||||
|
||||
The stack editor's action bar has a **More actions** overflow menu (the three-dot icon next to **Update**). **Rollback** sits at the top, with the most recent backup's timestamp beneath the label. Selecting it restores the backed-up files and re-runs `docker compose up -d` non-atomically, so the rollback does not nest inside another atomic wrapper and overwrite the good backup with the just-failed state.
|
||||
The stack editor's action bar has a **More actions** overflow menu (the three-dot icon next to **Update**). **Rollback** sits at the top, with the most recent recovery generation's timestamp beneath the label. Selecting it prefers the current recovery generation: it restores the captured authored inventory (and prior image IDs where held) and re-runs `docker compose up -d` non-atomically, so the rollback does not nest inside another atomic wrapper and overwrite the good generation with the just-failed state.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/atomic-deployments/rollback-menu.png" alt="Stack editor header for the plex stack, with the More actions overflow menu open. Rollback sits at the top with the backup timestamp beneath the label, followed by Scan config, a Mute submenu for the stack's notifications, and Delete." />
|
||||
</Frame>
|
||||
|
||||
The menu entry is hidden when no backup exists for the stack, for example on a freshly created stack that has never been deployed. It is also hidden for users who lack the `stack:deploy` permission; the backend enforces that check as the authoritative guard. The **Mute** entry next to it controls notification suppression for the stack and is unrelated to atomic deployments; see [Alerts & Notifications](/features/alerts-notifications).
|
||||
The menu entry is hidden when no recovery generation (or fallback backup) exists for the stack, for example on a freshly created stack that has never been deployed. It is also hidden for users who lack the `stack:deploy` permission; the backend enforces that check as the authoritative guard. The **Mute** entry next to it controls notification suppression for the stack and is unrelated to atomic deployments; see [Alerts & Notifications](/features/alerts-notifications).
|
||||
|
||||
After a failed deploy or update, the stack page also surfaces a **Roll back** button in the recovery panel alongside Retry, Restart, and Refresh. This is the same rollback action, triggered in response to a failure rather than invoked on demand. See [Deploy Progress](/features/deploy-progress#recovery-actions) for the full recovery actions reference.
|
||||
|
||||
## Where backups are stored
|
||||
|
||||
Backups live under `<DATA_DIR>/backups/<nodeId>/<stack>/`, in the same writable volume Sencho uses for its database and other persisted state. They are intentionally kept outside the user's compose folder, so the operation works even when a container has chowned its bind-mounted stack directory to root.
|
||||
Recovery generations and file backups live under Sencho's writable data volume (outside the user's compose folder), so the operation works even when a container has chowned its bind-mounted stack directory to root. Generations are also listed in **Resources → Rollback**; see [Health-Gated Updates](/features/health-gated-updates#automatic-rollback-images).
|
||||
|
||||
Each backup is a flat copy of the compose file Sencho found, plus `.env` and any configured project env files if they exist, plus two markers: a `.timestamp` recording when the backup was taken and a `.checksums` integrity manifest holding a SHA-256 for each backed-up file. There is one backup slot per stack: every protected deploy or update overwrites the previous backup, so the **Rollback** menu always reverts to the configuration that was on disk immediately before the most recent run.
|
||||
Each capture stores the managed authored inventory for the stack: ordered compose files, overrides, discoverable include/extends inputs, and env/label/config/secret inputs when discoverable, plus integrity metadata. Opaque image holds record prior image identity when containers were inspectable. A stack keeps a current generation for manual rollback; superseded generations follow the retention settings on that page.
|
||||
|
||||
A restore is a faithful revert, not an overlay. Sencho replaces the compose file and `.env` with the backed-up copies and removes any compose variant or `.env` that was added after the backup was taken, so the stack returns to exactly the file set it had before the run. For example, if a deploy switched the stack from `compose.yaml` to `docker-compose.yml` or introduced a new `.env`, a rollback undoes both. Files Sencho does not manage are left untouched.
|
||||
A restore reverts the managed inventory recorded in the generation, not an overlay of unmanaged files. Sencho restores the captured authored set (and, for Git-linked stacks, the matching managed-project manifesto and revision identity) and, where held, retargets exact prior image IDs. Files Sencho does not manage are left untouched. Named volumes and bind-mounted application data are never part of the restore.
|
||||
|
||||
Before a restore overwrites anything, Sencho re-hashes each backed-up file and compares it against the `.checksums` manifest. If a file no longer matches (for example, a backup truncated by an out-of-disk write), Sencho aborts the restore with a clear error and leaves the stack exactly as it was, rather than copying the corrupt content back over a working configuration.
|
||||
Before a restore overwrites anything, Sencho verifies generation integrity. If content no longer matches (for example, a truncated write), Sencho aborts the restore with a clear error and leaves the stack exactly as it was, rather than copying corrupt content back over a working configuration.
|
||||
|
||||
## Rollback readiness
|
||||
|
||||
The **Stack Dossier** includes a **Rollback readiness** panel: a pre-flight read on whether rolling back will actually fix the problem. It carries an overall verdict (**Ready**, **Partial**, or **Not ready**) and evaluates five signals: whether a previous compose file exists and how old it is, whether a previous `.env` was captured, whether the previous image tag is pinned or moving (moving tags are not reverted), the age of the last successful deploy, and whether healthchecks are defined to verify recovery. A sixth, always-shown note covers application data: named volumes and bind-mounted data (database rows, uploaded files, anything outside the compose and env files) are outside the scope of any revert.
|
||||
The **Stack Dossier** includes a **Rollback readiness** panel: a pre-flight read on whether rolling back will actually fix the problem. It carries an overall verdict (**Ready**, **Partial**, or **Not ready**) and reports managed-input coverage, whether a current recovery generation exists, rollback policy eligibility, plus the existing disclosures for compose/env coverage, previous image identity, last successful deploy, and healthchecks. Application data is always marked not covered: named volumes and bind-mounted data (database rows, uploaded files, anything outside the managed authored files) stay outside the scope of any revert.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/atomic-deployments/rollback-readiness.png" alt="Rollback readiness panel in the Stack Dossier for the plex stack, showing a Partial verdict badge and five signal rows: Previous compose file (ready), Previous env file (ready), Previous image tag (a moving tag warning naming the rollback target), Last successful deploy (ready), and Healthchecks (missing)." />
|
||||
</Frame>
|
||||
|
||||
Check this panel in the dossier before rolling back a stack that has been running for a while. A rollback reverts only the compose and env files; if the problem is in a volume or in a database migration that already ran, rolling back the compose file alone will not help. See [Stack Dossier](/features/stack-dossier) for the full readout.
|
||||
Check this panel in the dossier before rolling back a stack that has been running for a while. A rollback restores the managed authored inventory and held prior images where available; if the problem is in a volume or in a database migration that already ran, rolling back files alone will not help. See [Stack Dossier](/features/stack-dossier) for the full readout.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="The Rollback option is not in the More actions menu">
|
||||
Sencho hides the entry whenever a rollback is not possible. The most common reason is that the stack has never been deployed, so no backup file exists yet. Run **Deploy** or **Update** once and the entry will appear.
|
||||
Sencho hides the entry whenever a rollback is not possible. The most common reason is that the stack has never been deployed, so no recovery generation exists yet. Run **Deploy**, **Update**, or **Backup** once and the entry will appear.
|
||||
|
||||
The entry is also hidden for users who lack the `stack:deploy` permission. Ask an admin to grant `stack:deploy` through **Settings · Access** if the entry does not appear.
|
||||
</Accordion>
|
||||
<Accordion title="The deploy succeeded but a service crashed seconds later">
|
||||
The health probe is a 3-second window after `docker compose up -d` returns. Crashes after that window are out of scope for atomic rollback, because Sencho cannot tell a late exit apart from a normal restart. The [health gate](/features/health-gated-updates) covers exactly this period: it observes the stack for a configurable window after the update, records a verdict on the stack timeline, and offers a manual rollback when containers do not stay healthy. For ongoing health beyond that, use [Auto-Heal Policies](/features/auto-heal-policies) to restart unhealthy containers automatically and [per-stack alert rules](/features/alerts-notifications#per-stack-alert-rules) to page you when a container exits unexpectedly.
|
||||
The health probe is a 3-second window after `docker compose up -d` returns. Crashes after that window are out of scope for atomic rollback, because Sencho cannot tell a late exit apart from a normal restart. The [health gate](/features/health-gated-updates) covers exactly this period: it observes the stack for a configurable window after the update, records a verdict on the stack timeline, and offers a manual rollback when containers do not stay healthy. The gate does not auto-compensate. For ongoing health beyond that, use [Auto-Heal Policies](/features/auto-heal-policies) to restart unhealthy containers automatically and [per-stack alert rules](/features/alerts-notifications#per-stack-alert-rules) to page you when a container exits unexpectedly.
|
||||
</Accordion>
|
||||
<Accordion title="The deploy progress modal showed 'Rollback failed. Manual intervention may be required'">
|
||||
This message means the auto-rollback attempted to restore the backup and re-deploy, but the restore step or the re-deploy itself errored out. The backup files are still at `<DATA_DIR>/backups/<nodeId>/<stack>/`. To recover:
|
||||
This message means the auto-rollback attempted to restore from the recovery generation and re-deploy, but the restore step or the re-deploy itself errored out. The generation remains listed under **Resources → Rollback**. To recover:
|
||||
|
||||
1. Copy `compose.yaml` (or the variant Sencho backed up), `.env`, and any project env files from `<DATA_DIR>/backups/<nodeId>/<stack>/` back into the stack directory.
|
||||
2. Open the stack in the editor and click **Deploy** to re-run with the restored configuration.
|
||||
1. Prefer **Rollback** from the stack editor (it uses the current recovery generation when one exists).
|
||||
2. If that still fails, open the stack in the editor, restore the authored files you need, and click **Deploy**.
|
||||
|
||||
The most common causes are filesystem permissions on the stack directory and a missing image in a private registry that the original deploy could not pull.
|
||||
The most common causes are filesystem permissions on the stack directory and a missing held image that the restore expected to retarget.
|
||||
</Accordion>
|
||||
<Accordion title="Rollback failed with a 'backup is corrupt (integrity check failed)' error">
|
||||
The backup slot holds a file whose contents no longer match the checksum recorded when the backup was taken, usually because the disk filled up or the write was interrupted while the backup was being written. Sencho refuses to copy that file back, so your live stack is left untouched rather than overwritten with corrupt content.
|
||||
The recovery generation holds content whose integrity no longer matches what was recorded at capture, usually because the disk filled up or the write was interrupted. Sencho refuses to copy that content back, so your live stack is left untouched rather than overwritten with corrupt content.
|
||||
|
||||
Edit the compose file or `.env` directly in the editor to the configuration you want, then click **Deploy**. The next protected deploy writes a fresh, verified backup, and **Rollback** works again from that point.
|
||||
Edit the authored files directly in the editor to the configuration you want, then click **Deploy**. The next protected deploy writes a fresh, verified generation, and **Rollback** works again from that point.
|
||||
</Accordion>
|
||||
<Accordion title="Rollback ran but the stack still has the broken configuration">
|
||||
Sencho keeps a single backup per stack. If you ran two atomic deploys back to back, the second deploy overwrote the first backup with the broken configuration before it failed. **Rollback** then restores that broken configuration, because as far as Sencho is concerned it is the most recent known state.
|
||||
Manual rollback prefers the current recovery generation. If a later protected run already handed off a generation that captured the broken authored state, **Rollback** restores that state, because it is the generation Sencho treats as current.
|
||||
|
||||
To recover, edit the compose file or `.env` directly in the editor, fix the bad change, and click **Deploy**. The next protected deploy will write a fresh backup of the now-good configuration.
|
||||
To recover, edit the authored files directly in the editor, fix the bad change, and click **Deploy**. The next protected deploy captures a fresh generation of the good configuration.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
@@ -89,12 +89,10 @@ By default, restoring a stack only writes back its files and leaves the stack's
|
||||
<img src="/images/fleet-backups/restore-dialog.png" alt="Restore plex on Local confirmation dialog with the warning that it overwrites the current compose files with the snapshot version, a Redeploy stack after restore checkbox, and Cancel and Restore buttons" />
|
||||
</Frame>
|
||||
|
||||
Sencho writes the snapshot's files back to the target node:
|
||||
- **Local nodes** have files written directly, with the current files backed up first (creating a rollback point via the atomic deployment system)
|
||||
- **Remote nodes** receive files via the Distributed API proxy
|
||||
Sencho writes the snapshot's files back to the target node. Restoring over an existing stack captures a recovery generation of the current authored project before any snapshot file is written. If that capture fails, the restore stops and the live files are left unchanged. A stack that does not yet exist is created from the snapshot files. Local and remote restores run that capture-then-write sequence on the target node. If a later file write fails after capture succeeded, live files may already have changed; the captured generation remains so you can use Rollback to return to the pre-restore project.
|
||||
|
||||
<Warning>
|
||||
Restoring overwrites the current compose and environment files on the target node. If atomic deployments are enabled, the current files are backed up before restoration.
|
||||
Restoring overwrites the current compose and environment files on the target node. For an existing stack, Sencho captures a recovery generation first so you can use Rollback if the restore stops after that capture.
|
||||
</Warning>
|
||||
|
||||
### Restoring an entire snapshot
|
||||
|
||||
@@ -248,5 +248,5 @@ Pulls, applies, and create-from-git operations on the same stack are serialized
|
||||
- **Unsupported inputs are refused, not guessed.** Inputs that cannot be safely reproduced fail the pull with an actionable message: URL includes, Git LFS pointers, submodule contents, symbolic links, build contexts that exceed the size bounds, and include or extends declarations that point outside the repository or use dynamic `\${VAR}` paths (their contents cannot be enumerated). Nothing is applied until the declaration is fixed. Absolute host paths, host bind mounts, external resources, and dynamic `\${VAR}` data paths are never claimed as covered: they resolve at deploy time from the environment or the node, and the manifest records them as unmanaged.
|
||||
- **Materialization bounds.** The materialized project is bounded by file count, total bytes, per-file size, path depth, and build-context size, each adjustable with a `GITSOURCE_*` variable (see configuration). Crossing a bound refuses the pull with the counts so far rather than producing a partial project.
|
||||
- **Detach and export.** Removing a Git source renders the effective compose model into a single `compose.yaml`, keeps the remaining materialized files, removes auto-discovered override files so the exported model is final, and removes Git tracking. Resolved environment values are baked into the exported file. If removal is interrupted before it completes, Sencho restores the original files automatically.
|
||||
- **Rollback scope.** Rollback of a Git-managed stack restores compose files and `.env`. Other materialized inputs are not reverted by rollback; re-apply the previous revision from Git to restore them.
|
||||
- **Rollback scope.** Rollback of a Git-managed stack restores the managed authored inventory captured in the recovery generation (the same contract as deploy and update: ordered compose files, overrides, include/extends, and discoverable env/label/config/secret inputs, plus held prior image IDs where available). Named volumes and bind-mounted application data are not restored. Git apply captures a generation before promote. If promote fails mid-flight, Sencho restores the prior files. If apply-with-deploy fails after promote, the applied files stay on disk and the generation remains available for manual rollback (partial success); Sencho does not auto-compensate the deploy failure after apply.
|
||||
- **Some read-only views read the primary file.** The dependency graph, drift snapshot, and networking inspector summarize the primary compose file, so a service declared only in an override may not appear in those views. Deploy, update, image-update checks, and mesh attachment use the full merged set.
|
||||
|
||||
@@ -71,7 +71,7 @@ The live verifying and recovery view is part of the deploy progress panel. If yo
|
||||
<img src="/images/health-gated-updates/modal-gate-failed.png" alt="Deploy progress modal with the Health gate failed headline and the banner identifying the unhealthy container, with the hint that rollback options are available on the stack" />
|
||||
</Frame>
|
||||
|
||||
When the gate fails, the stack page surfaces the same [recovery actions](/features/deploy-progress#recovery-actions) as a failed update: retry, restart, roll back when a backup exists, refresh the container state, or copy diagnostics. Rolling back is always your call; the gate never rolls anything back on its own.
|
||||
When the gate fails, the stack page surfaces the same [recovery actions](/features/deploy-progress#recovery-actions) as a failed update: retry, restart, roll back when a recovery generation exists, refresh the container state, or copy diagnostics. Rolling back is always your call; the gate never rolls anything back on its own.
|
||||
|
||||
The gate also runs for updates you did not click: scheduled image updates, webhook-triggered deploys and pulls, bulk updates, and Git source applies all record gate verdicts on the stack timeline. Rollbacks, App Store installs, and Sencho's own automation loops are deliberately not gated.
|
||||
|
||||
@@ -90,24 +90,24 @@ Open **Settings > Infrastructure > Stacks > Deploy Guardrails** on the node you
|
||||
|
||||
The Stack Dossier carries a **Rollback readiness** section that answers one question honestly: if this update goes wrong, what can a rollback actually restore? It reports an overall state of Ready, Partial, or Not ready, built from:
|
||||
|
||||
- **Previous compose file**: whether a backup slot exists and how old it is.
|
||||
- **Previous env file**: whether the backup contains the stack's env file. Sencho lists how many variable names are covered; the values themselves are restored with the file and never displayed.
|
||||
- **Previous image tag**: the known rollback target from the update preview. When the compose file uses a moving tag, restoring files alone does not revert the image, so the report names the exact tag you would pin to be precise.
|
||||
- **Last successful deploy**: whether the backup reflects a configuration that actually deployed successfully.
|
||||
- **Recovery generation**: whether a current generation exists for exact restore. Atomic deploy, full-stack update, Git apply, and on-demand or scheduled stack backup capture one.
|
||||
- **Managed inputs**: whether the managed authored inventory (ordered compose files, overrides, include/extends, and discoverable env/label/config/secret inputs) is covered exactly.
|
||||
- **Previous compose file** / **Previous env file**: compose and env coverage disclosures (driven by the current generation when one exists).
|
||||
- **Previous image tag**: the known rollback target. When a generation holds prior image IDs, restore can retarget moving tags and supported local builds to those exact images; when no prior ID was captured, restoring files alone may leave a moving tag on a newer digest.
|
||||
- **Last successful deploy**: whether recent activity shows a configuration that deployed successfully.
|
||||
- **Healthchecks**: whether a rollback can be verified beyond run state.
|
||||
- **Application data**: always reported as not covered. Named volumes and bind-mounted data are not included in file backups; a rollback restores compose and env files only, and your application data keeps its current state. This row exists so the limit is stated where you decide, not discovered during an incident.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/health-gated-updates/dossier-rollback-readiness.png" alt="Rollback readiness section in the Stack Dossier showing the overall state chip and the six rows: Previous compose file, Previous env file, Previous image tag, Last successful deploy, Healthchecks, and the Application data row marked not covered" />
|
||||
</Frame>
|
||||
- **Rollback eligibility**: whether policy allows restore for the current generation (integrity and held-image posture).
|
||||
- **Application data**: always reported as not covered. Named volumes and bind-mounted data are not included; a rollback restores the managed authored inventory (and held images where available), and your application data keeps its current state. This row exists so the limit is stated where you decide, not discovered during an incident.
|
||||
|
||||
## Automatic rollback images
|
||||
|
||||
Before a full-stack update runs, Sencho captures the running image of every service as an opaque, uniquely named copy so it can automatically restore the prior state if the update or its health gate fails. These copies exist in Docker as `sencho-rb/<generation>/<service>:hold`, but they are Sencho-internal recovery state, not part of your image inventory: they are kept out of **Resources → Images** and listed instead in **Resources → Rollback**. If a captured image still carries its original registry tag alongside the hold tag (a compose file pinned to an immutable tag, for example), it stays visible in the Images tab too, badged **Rollback protected** instead of the usual unused label, since it is held on purpose rather than left behind by accident.
|
||||
Before an atomic deploy, a full-stack update, or a Git apply that will promote (including apply-with-deploy), Sencho captures the running image of every service as an opaque, uniquely named copy so it can restore the prior runtime when a failure after handoff requires it. These copies exist in Docker as `sencho-rb/<generation>/<service>:hold`, but they are Sencho-internal recovery state, not part of your image inventory: they are kept out of **Resources → Images** and listed instead in **Resources → Rollback**. If a captured image still carries its original registry tag alongside the hold tag (a compose file pinned to an immutable tag, for example), it stays visible in the Images tab too, badged **Rollback protected** instead of the usual unused label, since it is held on purpose rather than left behind by accident.
|
||||
|
||||
Each capture is one **rollback generation**. The generation currently backing a stack's live deployment is retained for as long as it is current; once a newer update supersedes it, it is retained for a configurable window before Sencho cleans it up automatically. A stack updated repeatedly in a short span can have more than one superseded generation in that window at once.
|
||||
Each capture is one **rollback generation**. The generation currently backing a stack's live deployment is retained for as long as it is current; once a newer protected run supersedes it, it is retained for a configurable window before Sencho cleans it up automatically. A stack updated repeatedly in a short span can have more than one superseded generation in that window at once.
|
||||
|
||||
**Resources → Rollback** lists every generation on the node: the stack it belongs to, a short generation id, whether it is the current protection or a superseded one awaiting cleanup, and roughly when it clears. Search by stack name or generation id, filter to Current or Superseded, and click a column header to sort. An admin can release a generation's protection early from that list, including the current one, which immediately frees its image but means Sencho cannot automatically roll that stack back until its next successful full-stack update; the confirmation dialog says so before you proceed. A generation that is mid-recovery or still being observed by a health gate cannot be released until that finishes.
|
||||
The health gate remains observational: it records a verdict and surfaces recovery actions, but it never rolls anything back on its own. Automatic restore after handoff belongs to the atomic deploy/update path when a crash is detected; manual **Rollback** prefers the current recovery generation. On-demand and scheduled stack backup will not replace that generation while a health gate is still observing after a deploy or update.
|
||||
|
||||
**Resources → Rollback** lists every generation on the node: the stack it belongs to, a short generation id, whether it is the current protection or a superseded one awaiting cleanup, and roughly when it clears. Search by stack name or generation id, filter to Current or Superseded, and click a column header to sort. An admin can release a generation's protection early from that list, including the current one, which immediately frees its image but means Sencho cannot automatically roll that stack back until its next successful atomic deploy, full-stack update, Git apply, or stack backup capture; the confirmation dialog says so before you proceed. A generation that is mid-recovery or still being observed by a health gate cannot be released until that finishes.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/health-gated-updates/rollback-generations.png" alt="Resources Rollback tab showing search, All Current and Superseded filter buttons, an info icon for help, and sortable Stack, Generation, State, and Retention column headers above the generations table" />
|
||||
@@ -122,7 +122,7 @@ Two settings under **Settings > Infrastructure > Stacks > Deploy Guardrails** co
|
||||
|
||||
## Classified failures
|
||||
|
||||
When a deploy or update fails, Sencho classifies the failure from the compose output and shows the cause with a suggested next step in the recovery panel: an image pull failure, a missing environment variable, a host port conflict, a missing bind-mount path, a permission problem, a crashed container, a failed healthcheck, an unavailable dependency, an unreachable node or Docker daemon, or an invalid compose file. The classification also lands in **Copy details**, so a bug report carries the cause, not just the raw output.
|
||||
When a deploy or update fails, Sencho classifies the failure from the compose output and shows the cause with a suggested next step in the recovery panel: an image pull failure, a missing environment variable, a host port conflict, a missing bind-mount path, a permission problem, a crashed container, a failed healthcheck, an unavailable dependency, an unreachable node or Docker daemon, an invalid compose file, mixed replica images that cannot be captured exactly, or a project whose include/extends paths cannot be captured for exact rollback. The classification also lands in **Copy details**, so a bug report carries the cause, not just the raw output.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -140,15 +140,15 @@ When a deploy or update fails, Sencho classifies the failure from the compose ou
|
||||
The modal holds its auto-close while the gate observes so the verdict is not lost. You can close it at any time; the observation continues server-side and the verdict lands on the stack timeline. If the gate result repeatedly cannot be retrieved, the modal gives up with an unknown verdict instead of waiting forever.
|
||||
</Accordion>
|
||||
<Accordion title="Rollback readiness says my data is not covered">
|
||||
That is by design and true for every stack: file backups cover compose and env files, never named volumes or bind-mounted data. For point-in-time copies of stack files across the fleet, use [fleet snapshots](/features/fleet-backups); for application data, use a backup tool appropriate to the workload (database dumps, volume backups) before risky updates.
|
||||
That is by design and true for every stack: recovery generations cover the managed authored inventory and held prior images, never named volumes or bind-mounted data. For point-in-time copies of stack files across the fleet, use [fleet snapshots](/features/fleet-backups); for application data, use a backup tool appropriate to the workload (database dumps, volume backups) before risky updates.
|
||||
</Accordion>
|
||||
<Accordion title="Updates from the sidebar menu now show a dialog first">
|
||||
The sidebar's per-stack **Update** action runs the same path as the editor toolbar, so it shows the same readiness dialog and deploy progress. One click on **Update now** proceeds. On nodes that do not advertise the capability, updates run directly without the dialog.
|
||||
</Accordion>
|
||||
<Accordion title="Why do I see sencho-rb/... images in docker images on the host">
|
||||
Those are automatic rollback images: an opaque copy of a service's prior image, held so Sencho can restore it if a full-stack update fails. They are not leftovers. Sencho keeps them out of **Resources → Images** and **Security** on purpose (they are recovery state, not image inventory or scan targets) and lists them in **Resources → Rollback** instead, showing which stack and generation each one belongs to and how soon it clears on its own. If one still carries a normal registry tag too, it also stays visible in the Images tab with a **Rollback protected** badge, and Security continues to scan that registry tag.
|
||||
Those are automatic rollback images: an opaque copy of a service's prior image, held so Sencho can restore it if an atomic deploy, full-stack update, or Git apply-with-deploy path needs the prior runtime after handoff. They are not leftovers. Sencho keeps them out of **Resources → Images** and **Security** on purpose (they are recovery state, not image inventory or scan targets) and lists them in **Resources → Rollback** instead, showing which stack and generation each one belongs to and how soon it clears on its own. If one still carries a normal registry tag too, it also stays visible in the Images tab with a **Rollback protected** badge, and Security continues to scan that registry tag.
|
||||
</Accordion>
|
||||
<Accordion title="Deleting a rollback-protected image fails">
|
||||
That failure is intentional: the image is protected by an active or recently superseded rollback generation. Open **Resources → Rollback**, find the matching generation, and use **Release** there if you are sure you do not need it. Releasing the current generation means Sencho cannot automatically roll that stack back until its next successful full-stack update.
|
||||
That failure is intentional: the image is protected by an active or recently superseded rollback generation. Open **Resources → Rollback**, find the matching generation, and use **Release** there if you are sure you do not need it. Releasing the current generation means Sencho cannot automatically roll that stack back until its next successful atomic deploy, full-stack update, or Git apply capture.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
@@ -108,7 +108,7 @@ Events from the Drift tab (drift detected, drift resolved) appear in the stack's
|
||||
|
||||
### Rollback readiness
|
||||
|
||||
At the bottom of the Dossier tab, on nodes that advertise the `update-guard` capability, a **Rollback readiness** section reports an overall **Ready**, **Partial**, or **Not ready** state for the stack, built from six signals: whether a compose backup exists, whether that backup covers the stack's env file, whether the image tag is pinned or moving, whether the last deploy succeeded, whether any service defines a healthcheck, and an explicit note that application data in named volumes and bind mounts is never restored by a rollback. It shares the same capability-plus-fetch-failure behavior described above for the Networking and Storage export summaries. See [Health-Gated Updates](/features/health-gated-updates#rollback-readiness) for what each signal means and how to act on it.
|
||||
At the bottom of the Dossier tab, on nodes that advertise the `update-guard` capability, a **Rollback readiness** section reports an overall **Ready**, **Partial**, or **Not ready** state for the stack. It covers managed-input coverage, whether a current recovery generation exists, rollback policy eligibility, plus the existing disclosures for compose/env coverage, previous image identity, last successful deploy, and healthchecks. Application data in named volumes and bind mounts remains explicitly not covered. It shares the same capability-plus-fetch-failure behavior described above for the Networking and Storage export summaries. See [Health-Gated Updates](/features/health-gated-updates#rollback-readiness) for what each signal means and how to act on it.
|
||||
|
||||
## Permissions and per-node scoping
|
||||
|
||||
|
||||
@@ -300,7 +300,7 @@ If an image update is available, or the stack declares one or more services with
|
||||
|
||||
For actionable registry updates, the detail line also reads `patch · safe to apply` (green), `minor · review recommended` (amber), or `major · breaking changes possible` (rose). Build-only stacks show **Rebuild available** instead of a version bump, with a **Rebuild & Update** button. A stack that mixes registry images and local builds still shows a single banner, with the rebuild note folded into the same detail line. Major bumps use the rose styling and are worth reviewing before applying.
|
||||
|
||||
Rebuilds can take longer than a registry pull and depend on the local Dockerfile context, network access, and base-image availability. Atomic rollback restores compose and env files only; previously built image layers are not rolled back automatically.
|
||||
Rebuilds can take longer than a registry pull and depend on the local Dockerfile context, network access, and base-image availability. When a recovery generation captured the prior image ID, supported local-build recovery restores that exact image identity. Without a held prior ID, restore falls back to the authored files alone and does not reconstruct previously built layers.
|
||||
|
||||
A separate amber **scan** banner appears here if the most recent post-deploy vulnerability scan failed or was skipped for one or more images; deploys are never blocked by a scan failure. See [Vulnerability Scanning](/features/vulnerability-scanning).
|
||||
|
||||
@@ -383,7 +383,7 @@ The stack header groups actions by frequency of use. The most common action is t
|
||||
| Secondary | **Stop** | `docker compose stop` | Stops containers without removing them. State is preserved. |
|
||||
| Secondary | **Take down** | `docker compose down` | Removes containers and compose-created networks. The stack definition stays on disk so you can deploy again later. Optional volume removal is available in the confirmation dialog. |
|
||||
| Secondary | **Update** | `docker compose pull` + `up -d` (or build-aware rebuild when services declare `build:`) | Pulls registry images and recreates containers. When one or more services use `build:`, Update rebuilds those images from source (`compose build --pull`), pulls any remaining registry images, then recreates containers. |
|
||||
| Overflow | **Rollback** | Restores backup | Reverts compose and env files to the pre-deploy snapshot and redeploys. Only shown when a backup exists. |
|
||||
| Overflow | **Rollback** | Restores recovery generation | Prefers the current recovery generation: restores the managed authored inventory and held prior image IDs where available, then redeploys. Only shown when a generation or fallback backup exists. |
|
||||
| Overflow | **Scan config** | Trivy config scan | Scans the compose file for misconfigurations (admin role). |
|
||||
| Overflow | **Monitor** | Opens Monitor sheet | Opens the stack **Monitor** sheet on the Alerts tab (alert rules and Auto-heal). Same sheet as sidebar **Alerts** / **Auto-Heal**. |
|
||||
| Overflow | **Mute** | Creates a mute rule | Quick presets to mute notifications, deploy-success noise, or monitor alerts for this stack, plus a link to manage its mute rules in full. See [Alerts and Notifications](/features/alerts-notifications). |
|
||||
|
||||
Reference in New Issue
Block a user