mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-07-26 11:49:16 +00:00
Anso
38aabe7064
* feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.
73 lines
7.8 KiB
Plaintext
73 lines
7.8 KiB
Plaintext
---
|
|
title: Atomic Deployments
|
|
description: Wrap every deploy and update in a backup, a 3-second health probe, and an automatic rollback when a container crashes.
|
|
---
|
|
|
|
Sencho wraps every protected deploy in a four-step safety net: it copies the current `compose.yaml` and `.env` to a writable backup directory, runs the requested compose action, waits 3 seconds for the new containers to settle, then checks them for a non-zero exit code. If any container has crashed, Sencho restores the backed-up files and re-deploys automatically.
|
|
|
|
The same backup also powers the **Rollback** action in the stack editor, so you can roll a stack back to its last good configuration on demand. To see in advance whether that rollback would actually help, and to watch container health for longer than the 3-second probe, see [Health-Gated Updates](/features/health-gated-updates).
|
|
|
|
## How it works
|
|
|
|
1. **Backup.** Before the action runs, Sencho copies `compose.yaml` (or `compose.yml` / `docker-compose.yaml` / `docker-compose.yml`) and `.env`, if present, into the backup directory. The deploy progress modal streams `=== Backup created for atomic deployment ===` once the copy completes, before any `docker compose` output.
|
|
2. **Run the action.** Sencho executes the requested compose action: `up -d` for a deploy, or a pull-then-`up -d` recreate for an update.
|
|
3. **Health probe.** Sencho waits 3 seconds, then lists every container with the `com.docker.compose.project=<stack>` label and checks each one for a non-zero exit code. Any container that has exited with a non-zero status counts as a crash.
|
|
4. **Auto-rollback on failure.** When a crash is detected, Sencho streams `=== Deployment failed - rolling back to previous version ===`, restores the backed-up files, and re-runs `docker compose up -d` with the restored configuration. On success it streams `=== Rolled back successfully ===`. The original deploy error is preserved and reported as the deploy result, so a failed-then-rolled-back deploy still registers as a failure in the deploy progress modal.
|
|
|
|
If the rollback itself fails (for example, the re-deploy step cannot pull a previously available image, or the file restore is blocked by filesystem permissions), Sencho streams `=== Rollback failed - manual intervention may be required ===`. The backup files remain at `<DATA_DIR>/backups/<stack>/` so you can copy them back manually.
|
|
|
|
## Which operations are protected
|
|
|
|
Atomic deployments wrap:
|
|
|
|
- **Deploy** and **Update** from the stack editor's action bar.
|
|
- **App Store** installs of a new stack.
|
|
- **Webhook** triggers for deploy and pull actions.
|
|
- **Image auto-updates** triggered by an auto-update policy.
|
|
|
|
A scheduled image-update task uses the same atomic wrapper as a manual update, so a recurring update still takes a backup and rolls back automatically when a container crashes. Scheduled lifecycle actions (start, stop, restart) change no stack configuration and run `docker compose` directly without a backup.
|
|
|
|
## Manual rollback
|
|
|
|
The stack editor's action bar carries a **More actions** overflow menu (the three-dot icon next to **Update**). Open it and you'll see **Rollback** at the top, with the timestamp of the most recent backup rendered beneath the label. Selecting it restores the backed-up files and re-runs `docker compose up -d` non-atomically, to avoid nesting a rollback inside another atomic wrapper and overwriting the good backup with the broken state from the just-failed deploy.
|
|
|
|
<Frame>
|
|
<img src="/images/atomic-deployments/rollback-menu.png" alt="Stack editor action bar with the More actions overflow menu open, showing the Rollback entry at the top with the backup timestamp rendered beneath the label, followed by Scan config and Delete entries" />
|
|
</Frame>
|
|
|
|
The menu entry is hidden when no backup exists for the stack, for example on a freshly created stack that has never been deployed. The endpoint additionally requires the `stack:deploy` permission, so a user without it will see the menu entry but receive a permission error if they invoke it.
|
|
|
|
## Where backups are stored
|
|
|
|
Backups live under `<DATA_DIR>/backups/<stack>/`, in the same writable volume Sencho uses for its database and other persisted state. They are intentionally kept outside the user's compose folder, so the operation works even when a container has chowned its bind-mounted stack directory to root.
|
|
|
|
Each backup is a flat copy of the compose file Sencho found, plus `.env` if it exists, plus a `.timestamp` marker that records when the backup was taken. There is one backup slot per stack: every protected deploy or update overwrites the previous backup, so the **Rollback** menu always reverts to the configuration that was on disk immediately before the most recent run.
|
|
|
|
A restore is a faithful revert, not an overlay. Sencho replaces the compose file and `.env` with the backed-up copies and removes any compose variant or `.env` that was added after the backup was taken, so the stack returns to exactly the file set it had before the run. For example, if a deploy switched the stack from `compose.yaml` to `docker-compose.yml` or introduced a new `.env`, a rollback undoes both. Files Sencho does not manage are left untouched.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="The Rollback option is not in the More actions menu">
|
|
Sencho hides the entry whenever a rollback is not possible. The most common reason is that the stack has never been deployed, so no backup file exists yet. Run **Deploy** or **Update** once and the entry will appear.
|
|
|
|
A user without the `stack:deploy` permission will still see the menu entry; the rejection comes from the backend with a permission error after they click. Ask an admin to grant `stack:deploy` through **Settings · Roles & Access** if that happens.
|
|
</Accordion>
|
|
<Accordion title="The deploy succeeded but a service crashed seconds later">
|
|
The health probe is a 3-second window after `docker compose up -d` returns. Crashes that happen after that window are out of scope for atomic rollback by design, because Sencho cannot tell whether a late exit is a real failure or a normal restart. The [health gate](/features/health-gated-updates) covers exactly this period: it observes the stack for a configurable window after the update, records a verdict on the stack timeline, and offers a manual rollback when containers do not stay healthy. For ongoing health beyond that, use **Auto-Heal Policies** to restart unhealthy containers automatically and **Alert Rules** to page you when a container exits unexpectedly.
|
|
</Accordion>
|
|
<Accordion title="The deploy progress modal showed 'Rollback failed - manual intervention may be required'">
|
|
This message means the auto-rollback attempted to restore the backup and re-deploy, but the restore step or the re-deploy itself errored out. The backup files are still at `<DATA_DIR>/backups/<stack>/`. To recover:
|
|
|
|
1. Copy `compose.yaml` (or the variant Sencho backed up) and `.env` from `<DATA_DIR>/backups/<stack>/` back into the stack directory.
|
|
2. Open the stack in the editor and click **Deploy** to re-run with the restored configuration.
|
|
|
|
The most common causes are filesystem permissions on the stack directory and a missing image in a private registry that the original deploy could not pull.
|
|
</Accordion>
|
|
<Accordion title="Rollback ran but the stack still has the broken configuration">
|
|
Sencho keeps a single backup per stack. If you ran two atomic deploys back to back, the second deploy overwrote the first backup with the broken configuration before it failed. **Rollback** then restores that broken configuration, because as far as Sencho is concerned it is the most recent known state.
|
|
|
|
To recover, edit the compose file or `.env` directly in the editor, fix the bad change, and click **Deploy**. The next protected deploy will write a fresh backup of the now-good configuration.
|
|
</Accordion>
|
|
</AccordionGroup>
|