sencho/docs/features/health-gated-updates.mdx

---
title: Health-Gated Updates
sidebarTitle: Health gates
description: Check update readiness before applying, watch container health after the update lands, and know exactly what a rollback can restore.
---

Updating a stack is the highest-anxiety operation in a homelab: a pull-and-recreate can break containers, networking, env configuration, or storage assumptions, and `docker compose` alone gives you no answer to "is it safe to update this right now?". Sencho makes updates deliberate and reversible in three layers: a readiness check before the update, a health gate after it, and an always-honest rollback readiness report on the stack.

None of these layers blocks you. Readiness is advisory, the health gate is purely observational, and rollback is always an explicit action you confirm. The only thing that hard-blocks an update is a [deploy enforcement policy](/features/deploy-enforcement).

## Update readiness

When you trigger **Update** from the stack editor toolbar or the sidebar menu, Sencho first opens a readiness dialog with a single verdict and the signals behind it:

| Verdict | Meaning |
|---------|---------|
| Ready | Nothing stands out; the update can proceed. |
| Ready with warnings | Proceed, but read the warnings first. |
| Review required | Something needs a look: a high-risk preflight finding, an already-unhealthy container, a pending major version bump, or low disk. |
| Blocked | A blocker was found, such as a Compose Doctor blocker or a scan policy that will stop the update. |
| Unknown | Readiness could not be verified, for example when the node or Docker is unreachable. |

The verdict is computed from signals Sencho already tracks, so the check is fast and never re-runs anything heavy:

- **Compose Doctor**: the stored result of the last [preflight run](/features/compose-doctor). A blocker finding makes the verdict Blocked; high-risk findings ask for review. If Compose Doctor has never run, the dialog says so without dragging the verdict down.
- **Drift**: open [drift findings](/features/stack-drift) warn you that the running state has diverged, so the rollback target may not match what is running.
- **Current containers**: a container that is already unhealthy, restarting, or crashed before the update makes the result hard to evaluate; the dialog asks you to look first.
- **Healthcheck coverage**: how many services define healthchecks, which determines how thoroughly the post-update health gate can verify the result.
- **Pending update**: the [update preview](/features/auto-update-policies) classifies the pending change. A major version bump asks for review; patch updates and same-tag refreshes pass quietly.
- **Local build services**: any service built from a local Dockerfile instead of pulled from a registry. The update rebuilds these from source, which depends on the local Dockerfile context and can take longer than a plain pull; this signal never blocks the verdict.
- **Rollback backup**: whether a backup slot exists and how old it is. A missing backup is only a note, because the update itself creates a fresh one when it starts.
- **Node disk**: disk usage near or above the node's alert threshold warns you before a large pull fills the disk.

Admins also see whether a [fleet snapshot](/features/fleet-backups) covers this stack and can tick **Create a fleet snapshot before updating** to capture one as part of proceeding. If the snapshot fails, the update does not start.

Every verdict keeps the **Update now** button enabled. The dialog informs the decision; it does not make it for you.

On stacks with more than one Compose service, you can also open readiness for a single service from that service's header. The dialog still checks stack-wide guardrails (Compose Doctor, disk, backup, Docker), but container health, pending change, and recovery signals focus on the selected service. Sibling and dependency issues stay advisory. The toolbar **Update** action and the Updates view's stack **Apply** always update the full stack.

When the selected service is build-backed and has no registry update, the proceed button reads **Rebuild** instead of **Update**. A per-service update badge appears only when a registry update is confirmed for that service.

<Frame>
  <img src="/images/health-gated-updates/readiness-dialog.png" alt="Update readiness dialog with a review required verdict, showing signal rows for Compose Doctor, Drift, Current containers, Healthcheck coverage, Pending update, Local build services, Rollback backup and Node disk, the fleet snapshot row, and the Cancel and Update now buttons" />
</Frame>

## The post-update health gate

The 3-second crash probe from [atomic deployments](/features/atomic-deployments) catches instant failures. The health gate is the longer observational layer behind it: after a deploy or update succeeds, Sencho watches the stack's containers for an observation window (90 seconds by default) and records a verdict.

During the window, the gate checks every container that came up with the stack:

- all expected containers stay **running** (or finish cleanly as a one-shot; see below),
- Docker healthchecks report **healthy** on long-running containers wherever a service defines one (residual health on a completed one-shot is ignored),
- no container **crashes** (non-zero exit), **disappears**, or enters a **restart loop**.

A one-shot or init service that finishes with exit code `0` and explicitly declares `restart: "no"` (or `deploy.restart_policy.condition: none`) counts as a successful completion, not a gate failure. Omitting `restart:` does not qualify: Compose defaults to no-restart, and Docker reports the same inspect value for both intentional jobs and bare long-running services. Residual Docker health state on a completed one-shot (`starting` or `unhealthy` from an inherited healthcheck) does not fail or leave the gate unknown. A clean exit under `unless-stopped`, `always`, or `on-failure`, or with no declared restart, still fails the gate. An exit whose code cannot be read also fails the gate.

A clear failure ends the observation immediately. Passing requires the full window, and a healthcheck still in its start period on a long-running container when the window ends records the verdict as unknown rather than claiming success.

After a **service-scoped** update or restore, the gate still watches the whole project, but it attributes failures: a problem on the updated service is a primary failure; a previously healthy sibling that regresses during the window is a collateral failure. Siblings that were already unhealthy before the update do not fail the gate on their own.

The deploy progress modal treats the gate verdict as the real result: while the gate observes, the status reads **Verifying health** rather than claiming success, and the auto-close waits. A passed gate shows the success state; a failed or unknown gate becomes the headline result with its reason. Closing the modal never stops the observation; the gate runs server-side and its result appears on the stack timeline either way, as **health gate passed** or **health gate failed** events alongside the **update started** marker.

The live verifying and recovery view is part of the deploy progress panel. If you have turned that panel off, the in-browser gate view does not appear, but the gate still runs on the node and records its verdict on the stack timeline.

<Frame>
  <img src="/images/health-gated-updates/modal-verifying.png" alt="Deploy progress modal after an update succeeded, showing the Verifying health status and the health gate banner observing containers" />
</Frame>

<Frame>
  <img src="/images/health-gated-updates/modal-gate-failed.png" alt="Deploy progress modal with the Health gate failed headline and the banner identifying the unhealthy container, with the hint that rollback options are available on the stack" />
</Frame>

When the gate fails, the stack page surfaces the same [recovery actions](/features/deploy-progress#recovery-actions) as a failed update: retry, restart, roll back when a recovery generation exists, refresh the container state, or copy diagnostics. Rolling back is always your call; the gate never rolls anything back on its own.

The gate also runs for updates you did not click: scheduled image updates, webhook-triggered deploys and pulls, bulk updates, and Git source applies all record gate verdicts on the stack timeline. Rollbacks, App Store installs, and Sencho's own automation loops are deliberately not gated.

### Tuning or disabling the gate

Open **Settings > Infrastructure > Stacks > Deploy Guardrails** on the node you want to configure:

- **Observe health after updates** turns the gate on or off for that node. On by default; turning it off only stops the observation and its timeline events, never the update itself.
- **Observation window** sets how long containers are watched, from 15 to 600 seconds. The default is 90 seconds; raise it for stacks that take a while to settle, since a long-running healthcheck still starting at the end of the window records an unknown verdict instead of a pass.

<Frame>
  <img src="/images/health-gated-updates/settings.png" alt="Stacks settings page showing the Deploy Guardrails subsection, with the Observe health after updates toggle set to On and the Observation window field set to 90 seconds" />
</Frame>

## Rollback readiness

The Stack Dossier carries a **Rollback readiness** section that answers one question honestly: if this update goes wrong, what can a rollback actually restore? It reports an overall state of Ready, Partial, or Not ready, built from:

- **Recovery generation**: whether a current generation exists for exact restore. Atomic deploy, full-stack update, Git apply, and on-demand or scheduled stack backup capture one.
- **Managed inputs**: whether the managed authored inventory (ordered compose files, overrides, include/extends, and discoverable env/label/config/secret inputs) is covered exactly.
- **Previous compose file** / **Previous env file**: compose and env coverage disclosures (driven by the current generation when one exists).
- **Previous image tag**: the known rollback target. When a generation holds prior image IDs, restore can retarget moving tags and supported local builds to those exact images; when no prior ID was captured, restoring files alone may leave a moving tag on a newer digest.
- **Last successful deploy**: whether recent activity shows a configuration that deployed successfully.
- **Healthchecks**: whether a rollback can be verified beyond run state.
- **Rollback eligibility**: whether policy allows restore for the current generation (integrity and held-image posture).
- **Application data**: always reported as not covered. Named volumes and bind-mounted data are not included; a rollback restores the managed authored inventory (and held images where available), and your application data keeps its current state. This row exists so the limit is stated where you decide, not discovered during an incident.

## Automatic rollback images

Before an atomic deploy, a full-stack update, or a Git apply that will promote (including apply-with-deploy), Sencho captures the running image of every service as an opaque, uniquely named copy so it can restore the prior runtime when a failure after handoff requires it. These copies exist in Docker as `sencho-rb/<generation>/<service>:hold`, but they are Sencho-internal recovery state, not part of your image inventory: they are kept out of **Resources → Images** and listed instead in **Resources → Rollback**. If a captured image still carries its original registry tag alongside the hold tag (a compose file pinned to an immutable tag, for example), it stays visible in the Images tab too, badged **Rollback protected** instead of the usual unused label, since it is held on purpose rather than left behind by accident.

Each capture is one **rollback generation**. The generation currently backing a stack's live deployment is retained for as long as it is current; once a newer protected run supersedes it, it is retained for a configurable window before Sencho cleans it up automatically. A stack updated repeatedly in a short span can have more than one superseded generation in that window at once.

The health gate remains observational: it records a verdict and surfaces recovery actions, but it never rolls anything back on its own. Automatic restore after handoff belongs to the atomic deploy/update path when a crash is detected; manual **Rollback** prefers the current recovery generation. On-demand and scheduled stack backup will not replace that generation while a health gate is still observing after a deploy or update.

**Resources → Rollback** lists every generation on the node: the stack it belongs to, a short generation id, whether it is the current protection or a superseded one awaiting cleanup, and roughly when it clears. Search by stack name or generation id, filter to Current or Superseded, and click a column header to sort. An admin can release a generation's protection early from that list, including the current one, which immediately frees its image but means Sencho cannot automatically roll that stack back until its next successful atomic deploy, full-stack update, Git apply, or stack backup capture; the confirmation dialog says so before you proceed. A generation that is mid-recovery or still being observed by a health gate cannot be released until that finishes.

<Frame>
  <img src="/images/health-gated-updates/rollback-generations.png" alt="Resources Rollback tab showing search, All Current and Superseded filter buttons, an info icon for help, and sortable Stack, Generation, State, and Retention column headers above the generations table" />
</Frame>

Because these images are deliberately held, deleting one directly (by id, including through the API) is refused. Release the generation from **Resources → Rollback** instead, or leave it to clear on its own.

Two settings under **Settings > Infrastructure > Stacks > Deploy Guardrails** control the automatic cleanup:

- **Superseded rollback retention** sets how many days a superseded generation is kept before its image is cleaned up. The current generation is unaffected by this window; it stays protected until it is superseded or manually released. Default 7 days.
- **Maximum retained rollback generations per stack** caps how many generations a stack keeps at once, current generation included, so the oldest superseded generations beyond the cap are cleaned up ahead of the retention window. 0 (default) leaves the count unlimited and relies on the retention window alone.

## Classified failures

When a deploy or update fails, Sencho classifies the failure from the compose output and shows the cause with a suggested next step in the recovery panel: an image pull failure, a missing environment variable, a host port conflict, a missing bind-mount path, a permission problem, a crashed container, a failed healthcheck, an unavailable dependency, an unreachable node or Docker daemon, an invalid compose file, mixed replica images that cannot be captured exactly, or a project whose include/extends paths cannot be captured for exact rollback. The classification also lands in **Copy details**, so a bug report carries the cause, not just the raw output.

## Troubleshooting

<AccordionGroup>
  <Accordion title="The readiness dialog shows Unknown">
    Unknown means one of the verdict-affecting signals could not be verified, most often because Docker or the node did not answer in time. The update path is never blocked by an unknown verdict; check the node's connection if it persists, and proceed when you are confident in the stack's state.
  </Accordion>
  <Accordion title="The health gate failed but my app seems fine">
    The gate fails on the first clear problem it observes: a non-zero exit, an unexpected clean exit of a service that is not an explicit one-shot, an unhealthy healthcheck on a long-running container, a restart loop, or a container disappearing mid-window. A one-shot that exits `0` with explicit `restart: "no"` (or `deploy.restart_policy.condition: none`) does not fail the gate, including when residual Docker health on that completed container is still starting or unhealthy. Check the named container's logs from the stack page. If the container legitimately restarts during startup (a migration step that should stay up, for example), raise the observation window under **Settings > Infrastructure > Stacks > Deploy Guardrails** so the gate watches past the settle period.
  </Accordion>
  <Accordion title="The health gate result is unknown">
    Unknown means the observation could not finish: Sencho restarted mid-window, a newer update superseded the observation, Docker became unreachable, a long-running healthcheck was still starting when the window ended, or no containers appeared to observe. Residual starting health on a completed one-shot does not produce unknown. The stack timeline records the reason. Check the containers directly; an unknown verdict makes no claim either way.
  </Accordion>
  <Accordion title="The modal stays open saying the gate is observing">
    The modal holds its auto-close while the gate observes so the verdict is not lost. You can close it at any time; the observation continues server-side and the verdict lands on the stack timeline. If the gate result repeatedly cannot be retrieved, the modal gives up with an unknown verdict instead of waiting forever.
  </Accordion>
  <Accordion title="Rollback readiness says my data is not covered">
    That is by design and true for every stack: recovery generations cover the managed authored inventory and held prior images, never named volumes or bind-mounted data. For point-in-time copies of stack files across the fleet, use [fleet snapshots](/features/fleet-backups); for application data, use a backup tool appropriate to the workload (database dumps, volume backups) before risky updates.
  </Accordion>
  <Accordion title="Updates from the sidebar menu now show a dialog first">
    The sidebar's per-stack **Update** action runs the same path as the editor toolbar, so it shows the same readiness dialog and deploy progress. One click on **Update now** proceeds. On nodes that do not advertise the capability, updates run directly without the dialog.
  </Accordion>
  <Accordion title="Why do I see sencho-rb/... images in docker images on the host">
    Those are automatic rollback images: an opaque copy of a service's prior image, held so Sencho can restore it if an atomic deploy, full-stack update, or Git apply-with-deploy path needs the prior runtime after handoff. They are not leftovers. Sencho keeps them out of **Resources → Images** and **Security** on purpose (they are recovery state, not image inventory or scan targets) and lists them in **Resources → Rollback** instead, showing which stack and generation each one belongs to and how soon it clears on its own. If one still carries a normal registry tag too, it also stays visible in the Images tab with a **Rollback protected** badge, and Security continues to scan that registry tag.
  </Accordion>
  <Accordion title="Deleting a rollback-protected image fails">
    That failure is intentional: the image is protected by an active or recently superseded rollback generation. Open **Resources → Rollback**, find the matching generation, and use **Release** there if you are sure you do not need it. Releasing the current generation means Sencho cannot automatically roll that stack back until its next successful atomic deploy, full-stack update, or Git apply capture.
  </Accordion>
</AccordionGroup>