mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-13 12:17:34 +00:00
Anso
f5178889eb
* feat(recovery): capture complete authored Compose project for atomic rollback Replace the root-compose-only backup slot with staged recovery generations that record the managed inventory, exact Compose invocation, and prior image identity, and wire the same engine through deploy, update, manual rollback, and Git apply. * fix(recovery): satisfy CodeQL path barriers and update-guard mock Inline resolve+startsWith checks at generation/inventory fs sinks and stub getCurrentStackUpdateRecovery in UpdateGuardService tests. * fix(recovery): drop unused FileSystemService import in generation store test * fix(recovery): harden authored-project rollback for upgrade and restore safety Preserve legacy UUID backup rows, restore Git deploy state with files, make multi-file restore recoverable, evaluate policy on the restored target, and fail closed when Git capture cannot cover an apply. * fix(recovery): unblock Git apply unit tests and CodeQL pre-restore TOCTOU Mock recovery capture in git-source-service tests after fail-closed apply capture, and re-resolve live paths immediately before pre-restore snapshot reads. * fix(recovery): fall back to authored inventory when Git manifesto is missing First Git apply captures before promote, so a missing managed-project manifesto must not block rollback capture when the live stack already has authored files. * fix(recovery): make authored-project rollback atomic across Git state Restore the managed-project manifesto with files, keep nullable Git identity on first-apply captures, persist Git side-state in restore intents for startup reconcile, compensate legacy materialize failures, and refuse directory collisions before mutation. * fix(recovery): satisfy CodeQL path and TOCTOU barriers on manifesto restore Add inline resolve barriers for manifesto read/clear sinks and remove the access-then-read race when restoring a generation manifesto snapshot. * fix(recovery): close third-audit rollback generation blockers Fail closed on incomplete Git inventory fallbacks, execute captured Compose invocation during recovery, refuse startup and mutations while restore intents remain unresolved, propagate legacy stale-delete failures, and add Docker-level exact prior-image coverage plus regression tests. * fix(recovery): mark acquired before handoff in prior-image Docker test Match the production updateStack CAS sequence so the exact prior-image integration test does not fail handoff from the captured phase. * fix(recovery): close fourth-audit rollback safety blockers Evaluate policy against held images, use index-based pre-restore snapshots, hold the shared stack lock across Git apply, replay Mesh and empty captured invocations exactly, restore POSIX modes with fail-closed sensitive permissions, keep case-sensitive paths, and link Git auto-deploy health gates. Add regression coverage for these cases. * test(recovery): fix mocks for health-gate link and authored compose args Add linkGateOrRetain to the Git apply recovery mock, and mock authoredComposeArgs so the case-collision inventory test is not masked by a missing getComposeDir stub. * fix(recovery): close fifth-audit rollback safety blockers Share git_apply locking for webhook auto-apply, fail closed on malformed recovery service records, refuse mixed-image capture, and require exact probe counts with hold-tag eligibility checks. * fix(recovery): close sixth-audit rollback safety blockers Preserve the legacy backup slot during generation capture, encrypt sensitive pre-restore snapshots, revert files on a failed health probe without committing Git, fail closed when an absent-file revert would delete a directory, skip Compose one-offs, route manual and scheduled backup through the current generation, and persist runtime image platform identity. * fix(recovery): close seventh-audit rollback safety blockers Fleet snapshot restore and restore-all now capture a recovery generation under the stack lock before any authored file write, including on remote nodes. * fix(recovery): keep pre-deploy generations during health-gate observe Link deploy recovery generations to the observing gate so backup cannot replace them mid-observe. Distinguish missing hold tags from probe failures, refuse generation release when services metadata is corrupt, classify mixed-replica and coverage refusals, and toast the backend rollback message. * fix(recovery): wrap webhook deploy case for eslint const bindings in an unbraced switch case trip no-case-declarations. Match the pull case block.
91 lines
12 KiB
Plaintext
91 lines
12 KiB
Plaintext
---
|
|
title: Atomic Deployments
|
|
sidebarTitle: Atomic deploys
|
|
description: Wrap every deploy and update in a managed authored-project backup, prior-image holds, a 3-second health probe, and automatic rollback when a container crashes.
|
|
---
|
|
|
|
Sencho wraps every protected deploy in a four-step safety net: it backs up the managed authored project (compose inventory, invocation metadata, and Git revision state when the stack is Git-linked), captures a recovery generation with prior image identity (opaque holds), runs the compose action, waits 3 seconds for containers to settle, then checks for a non-zero exit code. If any container crashed after handoff, Sencho restores from that generation and re-deploys automatically.
|
|
|
|
The same recovery generation powers the **Rollback** action in the stack editor, so you can roll a stack back to its last good configuration on demand. To see in advance whether that rollback would actually help, and to watch container health for longer than the 3-second probe, see [Health-Gated Updates](/features/health-gated-updates).
|
|
|
|
## How it works
|
|
|
|
1. **Backup.** Before the action runs, Sencho captures the managed authored project into a recovery generation: ordered compose files, overrides, discoverable `include:` / `extends` inputs, and env, label, config, and secret files when they can be enumerated. For Git-linked stacks it also records the managed-project manifesto and Git revision identity so a later restore can converge with the next pull. It records prior image identity as opaque holds. The deploy progress modal streams `=== Capturing rollback generation for atomic deploy ===` (or the matching update line) once capture completes, before any `docker compose` output.
|
|
2. **Run the action.** Sencho executes the requested compose action: `up -d` for a deploy, or a pull-then-`up -d` recreate for an update.
|
|
3. **Health probe.** Sencho waits 3 seconds, then lists every container with the `com.docker.compose.project=<stack>` label and checks each one for a non-zero exit code. Any container that has exited with a non-zero status counts as a crash. Recovery capture and the post-rollback compensation probe skip Compose one-off (`docker compose run`) containers so they are not counted as service replicas.
|
|
4. **Auto-rollback on failure.** When a crash is detected after handoff, Sencho streams `=== Deployment failed - restoring previous runtime from recovery generation ===` (or the matching update line), restores the captured authored inventory, and re-runs `docker compose up -d` with the restored configuration. Where the generation holds a prior image ID, restore retargets that exact image for moving tags and for supported local builds. Named volumes and bind-mounted application data are not restored. The original deploy error is preserved as the deploy result, so a failed-then-rolled-back deploy still registers as a failure.
|
|
|
|
If the rollback itself fails (for example, a held image is missing, or the file restore is blocked by filesystem permissions), Sencho streams `=== Rollback failed. Manual intervention may be required ===`. The recovery generation remains available so you can retry or recover manually.
|
|
|
|
## Which operations are protected
|
|
|
|
Atomic deployments wrap:
|
|
|
|
- **Deploy** and **Update** from the stack editor's action bar.
|
|
- **App Store** installs of a new stack.
|
|
- **Webhook** triggers for deploy and pull actions.
|
|
- **Image auto-updates** triggered by an auto-update policy.
|
|
|
|
A scheduled image-update task uses the same atomic wrapper as a manual update, so a recurring update still takes a backup and rolls back automatically when a container crashes. Scheduled lifecycle actions (start, stop, restart) change no stack configuration and run `docker compose` directly without a backup. On-demand **Backup** and scheduled stack backup capture a current recovery generation of the managed authored project. Backup refuses while a health gate is still observing after a deploy or update, so the pre-operation generation stays available for rollback.
|
|
|
|
[Blueprint](/features/blueprint-model) deploys are a separate, fleet-wide desired-state path and do not use this atomic wrapper: a failed Blueprint apply is not backed up or rolled back automatically.
|
|
|
|
## Manual rollback
|
|
|
|
The stack editor's action bar has a **More actions** overflow menu (the three-dot icon next to **Update**). **Rollback** sits at the top, with the most recent recovery generation's timestamp beneath the label. Selecting it prefers the current recovery generation: it restores the captured authored inventory (and prior image IDs where held) and re-runs `docker compose up -d` non-atomically, so the rollback does not nest inside another atomic wrapper and overwrite the good generation with the just-failed state.
|
|
|
|
<Frame>
|
|
<img src="/images/atomic-deployments/rollback-menu.png" alt="Stack editor header for the plex stack, with the More actions overflow menu open. Rollback sits at the top with the backup timestamp beneath the label, followed by Scan config, a Mute submenu for the stack's notifications, and Delete." />
|
|
</Frame>
|
|
|
|
The menu entry is hidden when no recovery generation (or fallback backup) exists for the stack, for example on a freshly created stack that has never been deployed. It is also hidden for users who lack the `stack:deploy` permission; the backend enforces that check as the authoritative guard. The **Mute** entry next to it controls notification suppression for the stack and is unrelated to atomic deployments; see [Alerts & Notifications](/features/alerts-notifications).
|
|
|
|
After a failed deploy or update, the stack page also surfaces a **Roll back** button in the recovery panel alongside Retry, Restart, and Refresh. This is the same rollback action, triggered in response to a failure rather than invoked on demand. See [Deploy Progress](/features/deploy-progress#recovery-actions) for the full recovery actions reference.
|
|
|
|
## Where backups are stored
|
|
|
|
Recovery generations and file backups live under Sencho's writable data volume (outside the user's compose folder), so the operation works even when a container has chowned its bind-mounted stack directory to root. Generations are also listed in **Resources → Rollback**; see [Health-Gated Updates](/features/health-gated-updates#automatic-rollback-images).
|
|
|
|
Each capture stores the managed authored inventory for the stack: ordered compose files, overrides, discoverable include/extends inputs, and env/label/config/secret inputs when discoverable, plus integrity metadata. Opaque image holds record prior image identity when containers were inspectable. A stack keeps a current generation for manual rollback; superseded generations follow the retention settings on that page.
|
|
|
|
A restore reverts the managed inventory recorded in the generation, not an overlay of unmanaged files. Sencho restores the captured authored set (and, for Git-linked stacks, the matching managed-project manifesto and revision identity) and, where held, retargets exact prior image IDs. Files Sencho does not manage are left untouched. Named volumes and bind-mounted application data are never part of the restore.
|
|
|
|
Before a restore overwrites anything, Sencho verifies generation integrity. If content no longer matches (for example, a truncated write), Sencho aborts the restore with a clear error and leaves the stack exactly as it was, rather than copying corrupt content back over a working configuration.
|
|
|
|
## Rollback readiness
|
|
|
|
The **Stack Dossier** includes a **Rollback readiness** panel: a pre-flight read on whether rolling back will actually fix the problem. It carries an overall verdict (**Ready**, **Partial**, or **Not ready**) and reports managed-input coverage, whether a current recovery generation exists, rollback policy eligibility, plus the existing disclosures for compose/env coverage, previous image identity, last successful deploy, and healthchecks. Application data is always marked not covered: named volumes and bind-mounted data (database rows, uploaded files, anything outside the managed authored files) stay outside the scope of any revert.
|
|
|
|
Check this panel in the dossier before rolling back a stack that has been running for a while. A rollback restores the managed authored inventory and held prior images where available; if the problem is in a volume or in a database migration that already ran, rolling back files alone will not help. See [Stack Dossier](/features/stack-dossier) for the full readout.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="The Rollback option is not in the More actions menu">
|
|
Sencho hides the entry whenever a rollback is not possible. The most common reason is that the stack has never been deployed, so no recovery generation exists yet. Run **Deploy**, **Update**, or **Backup** once and the entry will appear.
|
|
|
|
The entry is also hidden for users who lack the `stack:deploy` permission. Ask an admin to grant `stack:deploy` through **Settings · Access** if the entry does not appear.
|
|
</Accordion>
|
|
<Accordion title="The deploy succeeded but a service crashed seconds later">
|
|
The health probe is a 3-second window after `docker compose up -d` returns. Crashes after that window are out of scope for atomic rollback, because Sencho cannot tell a late exit apart from a normal restart. The [health gate](/features/health-gated-updates) covers exactly this period: it observes the stack for a configurable window after the update, records a verdict on the stack timeline, and offers a manual rollback when containers do not stay healthy. The gate does not auto-compensate. For ongoing health beyond that, use [Auto-Heal Policies](/features/auto-heal-policies) to restart unhealthy containers automatically and [per-stack alert rules](/features/alerts-notifications#per-stack-alert-rules) to page you when a container exits unexpectedly.
|
|
</Accordion>
|
|
<Accordion title="The deploy progress modal showed 'Rollback failed. Manual intervention may be required'">
|
|
This message means the auto-rollback attempted to restore from the recovery generation and re-deploy, but the restore step or the re-deploy itself errored out. The generation remains listed under **Resources → Rollback**. To recover:
|
|
|
|
1. Prefer **Rollback** from the stack editor (it uses the current recovery generation when one exists).
|
|
2. If that still fails, open the stack in the editor, restore the authored files you need, and click **Deploy**.
|
|
|
|
The most common causes are filesystem permissions on the stack directory and a missing held image that the restore expected to retarget.
|
|
</Accordion>
|
|
<Accordion title="Rollback failed with a 'backup is corrupt (integrity check failed)' error">
|
|
The recovery generation holds content whose integrity no longer matches what was recorded at capture, usually because the disk filled up or the write was interrupted. Sencho refuses to copy that content back, so your live stack is left untouched rather than overwritten with corrupt content.
|
|
|
|
Edit the authored files directly in the editor to the configuration you want, then click **Deploy**. The next protected deploy writes a fresh, verified generation, and **Rollback** works again from that point.
|
|
</Accordion>
|
|
<Accordion title="Rollback ran but the stack still has the broken configuration">
|
|
Manual rollback prefers the current recovery generation. If a later protected run already handed off a generation that captured the broken authored state, **Rollback** restores that state, because it is the generation Sencho treats as current.
|
|
|
|
To recover, edit the authored files directly in the editor, fix the bad change, and click **Deploy**. The next protected deploy captures a fresh generation of the good configuration.
|
|
</Accordion>
|
|
</AccordionGroup>
|