mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-07 01:14:14 +00:00
feat(recovery): make rollback-recovery image lifecycle visible and controllable (#1753)
* feat(recovery): make rollback-recovery image lifecycle visible and controllable GitHub discussion #1751 asked why Sencho creates sencho-rb/<id>/<service>:hold images during automatic updates and how to clean them up. That surfaced a real safety bug alongside the missing visibility: the manual single-image delete route did not consult the held-image predicate every other deletion path already honors, so a user could delete a rollback-protected image straight through the Images tab and silently break automatic recovery for that update. A short/truncated id also bypassed the predicate's full-id lookup. Fixes: - POST /images/delete now resolves the submitted id to its canonical form and checks the unified held-image predicate before deleting, returning 409 IMAGE_HELD_FOR_ROLLBACK for a protected image. - The Images tab no longer mislabels a protected image as plain "Unused"; a fully-synthetic hold image is kept out of the generic inventory entirely and surfaced instead in a new Resources -> Rollback tab, with an additive "Rollback protected" badge for images that still carry a normal tag too. New capability: - Two settings (Deploy Guardrails): superseded-generation retention (days, replaces a hardcoded 7) and a cap on retained generations per stack. - A new Resources -> Rollback tab lists every generation (stack, short id, state, retention) with an admin-gated manual release action, including releasing the current generation with an explicit warning that automatic rollback becomes unavailable until the next successful update. Release is a single atomic, server-revalidated transition so a stale UI read can never release a row that has since become ineligible. Also consolidated three near-duplicate implementations of the held-image predicate (two of which relied on a require() of a sibling .ts file that silently failed to resolve under the test runner and was never actually exercised by a real test before this change) into one shared module. Known follow-up, not fixed here: an orphaned sencho-rb tag whose recovery row no longer exists (DB restore, node re-add) is invisible in both the Images and Rollback tabs with no UI path to reclaim it. * fix(audit): add summary mapping for rollback generation release * fix(security): sanitize prune target in log sinks and cover release RBAC Closes two open js/log-injection findings on the system prune route by applying the same inline sanitizeForLog barrier the rest of the file already uses. The prune target is validated against an enum by parsePruneTargets before reaching these sinks, so the findings were false positives, but the barrier is cheap and removes the standing alerts on a file this change already touches. Also wraps the generation id in the release log line for consistency with the stack name beside it. Adds coverage for gaps a QA pass identified: - Release endpoint refuses a viewer and a deployer (Admin-only), leaving the generation and its artifacts untouched. - Viewer can still read the generations list, matching the sibling Resources routes. - The predicate the prune routes build reports full-stack rollback holds, not just service-scoped ones, and re-reads per call so a hold taken between plan and delete still gates the delete. - After releasing the current generation, no rollback point is claimed for the stack through any consumer of the current-generation lookup.
This commit is contained in:
@@ -101,6 +101,21 @@ The Stack Dossier carries a **Rollback readiness** section that answers one ques
|
||||
<img src="/images/health-gated-updates/dossier-rollback-readiness.png" alt="Rollback readiness section in the Stack Dossier showing the overall state chip and the six rows: Previous compose file, Previous env file, Previous image tag, Last successful deploy, Healthchecks, and the Application data row marked not covered" />
|
||||
</Frame>
|
||||
|
||||
## Automatic rollback images
|
||||
|
||||
Before a full-stack update runs, Sencho captures the running image of every service as an opaque, uniquely named copy so it can automatically restore the prior state if the update or its health gate fails. These copies exist in Docker as `sencho-rb/<generation>/<service>:hold`, but they are Sencho-internal recovery state, not part of your image inventory: they are kept out of **Resources → Images** and listed instead in **Resources → Rollback**. If a captured image still carries its original registry tag alongside the hold tag (a compose file pinned to an immutable tag, for example), it stays visible in the Images tab too, badged **Rollback protected** instead of the usual unused label, since it is held on purpose rather than left behind by accident.
|
||||
|
||||
Each capture is one **rollback generation**. The generation currently backing a stack's live deployment is retained for as long as it is current; once a newer update supersedes it, it is retained for a configurable window before Sencho cleans it up automatically. A stack updated repeatedly in a short span can have more than one superseded generation in that window at once.
|
||||
|
||||
**Resources → Rollback** lists every generation on the node: the stack it belongs to, a short generation id, whether it is the current protection or a superseded one awaiting cleanup, and roughly when it clears. An admin can release a generation's protection early from that list, including the current one, which immediately frees its image but means Sencho cannot automatically roll that stack back until its next successful full-stack update; the confirmation dialog says so before you proceed. A generation that is mid-recovery or still being observed by a health gate cannot be released until that finishes.
|
||||
|
||||
Because these images are deliberately held, deleting one directly (by id, including through the API) is refused. Release the generation from **Resources → Rollback** instead, or leave it to clear on its own.
|
||||
|
||||
Two settings under **Settings > Infrastructure > Stacks > Deploy Guardrails** control the automatic cleanup:
|
||||
|
||||
- **Superseded rollback retention** sets how many days a superseded generation is kept before its image is cleaned up. The current generation is unaffected by this window; it stays protected until it is superseded or manually released. Default 7 days.
|
||||
- **Maximum retained rollback generations per stack** caps how many generations a stack keeps at once, current generation included, so the oldest superseded generations beyond the cap are cleaned up ahead of the retention window. 0 (default) leaves the count unlimited and relies on the retention window alone.
|
||||
|
||||
## Classified failures
|
||||
|
||||
When a deploy or update fails, Sencho classifies the failure from the compose output and shows the cause with a suggested next step in the recovery panel: an image pull failure, a missing environment variable, a host port conflict, a missing bind-mount path, a permission problem, a crashed container, a failed healthcheck, an unavailable dependency, an unreachable node or Docker daemon, or an invalid compose file. The classification also lands in **Copy details**, so a bug report carries the cause, not just the raw output.
|
||||
@@ -126,4 +141,10 @@ When a deploy or update fails, Sencho classifies the failure from the compose ou
|
||||
<Accordion title="Updates from the sidebar menu now show a dialog first">
|
||||
The sidebar's per-stack **Update** action runs the same path as the editor toolbar, so it shows the same readiness dialog and deploy progress. One click on **Update now** proceeds. On nodes that do not advertise the capability, updates run directly without the dialog.
|
||||
</Accordion>
|
||||
<Accordion title="Why do I see sencho-rb/... images in docker images on the host">
|
||||
Those are automatic rollback images: an opaque copy of a service's prior image, held so Sencho can restore it if a full-stack update fails. They are not leftovers. Sencho keeps them out of **Resources → Images** on purpose (they are recovery state, not image inventory) and lists them in **Resources → Rollback** instead, showing which stack and generation each one belongs to and how soon it clears on its own. If one still carries a normal registry tag too, it also stays visible in the Images tab with a **Rollback protected** badge.
|
||||
</Accordion>
|
||||
<Accordion title="Deleting a rollback-protected image fails">
|
||||
That failure is intentional: the image is protected by an active or recently superseded rollback generation. Open **Resources → Rollback**, find the matching generation, and use **Release** there if you are sure you do not need it. Releasing the current generation means Sencho cannot automatically roll that stack back until its next successful full-stack update.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
Reference in New Issue
Block a user