fix: recognize clean one-shot completions in health gate and drift (#1691)

* fix: recognize clean one-shot completions in health gate and drift

Treat exit 0 with restart policy no/absent as successful completion so
init and migration jobs no longer fail post-update observation or show as
service-missing, while long-running restart policies still fail closed.

* fix: ignore residual health on clean one-shots and honor deploy.restart_policy

Completed exit-0 jobs with no-restart intent no longer fail the health gate on leftover starting/unhealthy state, and Drift treats deploy.restart_policy with Compose precedence so any/on-failure services are not mistaken for one-shots.

* fix: require explicit Compose restart no for one-shot recognition

Docker inspect reports restart no for both intentional jobs and bare services that omit restart, so Health Gate and Drift now require declared restart:""no"" (or deploy.restart_policy condition none) and load Compose intent once per gate.
This commit is contained in:
Anso
2026-07-24 09:41:21 -04:00
committed by GitHub
parent 524cc56d2f
commit 79914fe750
22 changed files with 1015 additions and 69 deletions
+9 -7
View File
@@ -49,11 +49,13 @@ The 3-second crash probe from [atomic deployments](/features/atomic-deployments)
During the window, the gate checks every container that came up with the stack:
- all expected containers stay **running**,
- Docker healthchecks report **healthy** wherever a service defines one,
- no container **exits**, **disappears**, or enters a **restart loop**.
- all expected containers stay **running** (or finish cleanly as a one-shot; see below),
- Docker healthchecks report **healthy** on long-running containers wherever a service defines one (residual health on a completed one-shot is ignored),
- no container **crashes** (non-zero exit), **disappears**, or enters a **restart loop**.
A clear failure ends the observation immediately. Passing requires the full window, and a healthcheck still in its start period when the window ends records the verdict as unknown rather than claiming success.
A one-shot or init service that finishes with exit code `0` and explicitly declares `restart: "no"` (or `deploy.restart_policy.condition: none`) counts as a successful completion, not a gate failure. Omitting `restart:` does not qualify: Compose defaults to no-restart, and Docker reports the same inspect value for both intentional jobs and bare long-running services. Residual Docker health state on a completed one-shot (`starting` or `unhealthy` from an inherited healthcheck) does not fail or leave the gate unknown. A clean exit under `unless-stopped`, `always`, or `on-failure`, or with no declared restart, still fails the gate. An exit whose code cannot be read also fails the gate.
A clear failure ends the observation immediately. Passing requires the full window, and a healthcheck still in its start period on a long-running container when the window ends records the verdict as unknown rather than claiming success.
After a **service-scoped** update or restore, the gate still watches the whole project, but it attributes failures: a problem on the updated service is a primary failure; a previously healthy sibling that regresses during the window is a collateral failure. Siblings that were already unhealthy before the update do not fail the gate on their own.
@@ -78,7 +80,7 @@ The gate also runs for updates you did not click: scheduled image updates, webho
Open **Settings > Infrastructure > Stacks > Deploy Guardrails** on the node you want to configure:
- **Observe health after updates** turns the gate on or off for that node. On by default; turning it off only stops the observation and its timeline events, never the update itself.
- **Observation window** sets how long containers are watched, from 15 to 600 seconds. The default is 90 seconds; raise it for stacks that take a while to settle, since a healthcheck still starting at the end of the window records an unknown verdict instead of a pass.
- **Observation window** sets how long containers are watched, from 15 to 600 seconds. The default is 90 seconds; raise it for stacks that take a while to settle, since a long-running healthcheck still starting at the end of the window records an unknown verdict instead of a pass.
<Frame>
<img src="/images/health-gated-updates/settings.png" alt="Stacks settings page showing the Deploy Guardrails subsection, with the Observe health after updates toggle set to On and the Observation window field set to 90 seconds" />
@@ -110,10 +112,10 @@ When a deploy or update fails, Sencho classifies the failure from the compose ou
Unknown means one of the verdict-affecting signals could not be verified, most often because Docker or the node did not answer in time. The update path is never blocked by an unknown verdict; check the node's connection if it persists, and proceed when you are confident in the stack's state.
</Accordion>
<Accordion title="The health gate failed but my app seems fine">
The gate fails on the first clear problem it observes: a container exit, an unhealthy healthcheck, a restart loop, or a container disappearing mid-window. Check the named container's logs from the stack page. If the container legitimately restarts during startup (a migration step, for example), raise the observation window under **Settings > Infrastructure > Stacks > Deploy Guardrails** so the gate watches past the settle period.
The gate fails on the first clear problem it observes: a non-zero exit, an unexpected clean exit of a service that is not an explicit one-shot, an unhealthy healthcheck on a long-running container, a restart loop, or a container disappearing mid-window. A one-shot that exits `0` with explicit `restart: "no"` (or `deploy.restart_policy.condition: none`) does not fail the gate, including when residual Docker health on that completed container is still starting or unhealthy. Check the named container's logs from the stack page. If the container legitimately restarts during startup (a migration step that should stay up, for example), raise the observation window under **Settings > Infrastructure > Stacks > Deploy Guardrails** so the gate watches past the settle period.
</Accordion>
<Accordion title="The health gate result is unknown">
Unknown means the observation could not finish: Sencho restarted mid-window, a newer update superseded the observation, Docker became unreachable, a healthcheck was still starting when the window ended, or no containers appeared to observe. The stack timeline records the reason. Check the containers directly; an unknown verdict makes no claim either way.
Unknown means the observation could not finish: Sencho restarted mid-window, a newer update superseded the observation, Docker became unreachable, a long-running healthcheck was still starting when the window ended, or no containers appeared to observe. Residual starting health on a completed one-shot does not produce unknown. The stack timeline records the reason. Check the containers directly; an unknown verdict makes no claim either way.
</Accordion>
<Accordion title="The modal stays open saying the gate is observing">
The modal holds its auto-close while the gate observes so the verdict is not lost. You can close it at any time; the observation continues server-side and the verdict lands on the stack timeline. If the gate result repeatedly cannot be retrieved, the modal gives up with an unknown verdict instead of waiting forever.
+6 -6
View File
@@ -28,9 +28,9 @@ The status badge at the top of the tab summarizes the comparison between the eff
| Status | Meaning |
|--------|---------|
| **In sync** | Running containers match the effective model: same services, images, and published ports. |
| **Drifted** | Something running differs from the model. Specific findings are listed below the badge. |
| **Not running** | The stack has no running containers. The file is present but nothing is up. |
| **In sync** | Declared services match runtime: same running services, images, and published ports, or a declared one-shot finished cleanly (`Exited (0)` with explicit `restart: "no"` or `deploy.restart_policy.condition: none`). Omitting `restart:` does not count as a one-shot. |
| **Not running** | The stack has no running containers, and no declared service is satisfied by a finished one-shot. The file is present but nothing is up. |
| **Drifted** | Something differs from the model. Specific findings are listed below the badge. |
| **Unreachable** | Docker could not be reached on the active node, so drift cannot be assessed. |
## Since your last deploy
@@ -62,10 +62,10 @@ When a stack is drifted, each reason is listed in the **Findings** section again
|---------|---------------|
| **Image** | A running container uses a different image than the Compose file declares. Expected and actual values are shown side by side. |
| **Ports** | The published ports of a service differ from what the Compose file declares. |
| **Service missing** | The Compose file declares a service, but no running container matches it. |
| **Service missing** | The Compose file declares a service, but no running container matches it and no clean one-shot completion satisfies it. |
| **Undeclared service** | A container is running for the stack but no matching service exists in the Compose file. |
| **Network undeclared** | A running service is attached to a network that is not declared in the Compose file. |
| **Network missing** | A network declared in the Compose file is not used by any running service or is absent from the runtime. |
| **Network missing** | A network declared in the Compose file is not used by any running service or is absent from the runtime. Network usage still counts only running or restarting containers, so a finished one-shot does not keep a dedicated Compose network marked as in use. |
### Image comparison
@@ -157,7 +157,7 @@ Viewing the tab requires read access to the stack; any role that can see a stack
<AccordionGroup>
<Accordion title="A stack I deliberately stopped shows as 'Not running'">
That is expected. Drift compares the file on disk against what is actually running, so a stack with no running containers reports **Not running** even when you stopped it intentionally. Deploy it to return it to **In sync**.
That is expected for long-running services. Drift compares the file on disk against what is actually running, so a stack with no running containers and no finished one-shot jobs reports **Not running** even when you stopped it intentionally. Deploy it to return it to **In sync**. A stack made only of one-shot jobs that already exited `0` with explicit `restart: "no"` (or `deploy.restart_policy.condition: none`) can show **In sync** (or **Drifted** for other findings such as unused networks) with no containers currently running.
</Accordion>
<Accordion title="The image finding flags a stack I just updated">
Click **re-check** after the deploy finishes. During a rolling update, replicas can briefly run different images, and the report captures that moment. Once every container is on the declared image, the finding clears.