gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-10-25 21:20:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
cla-signatures
sencho/docs/docs.json
T
Anso 32fd95dd57 feat: health-gated Blueprint rollout policy (#2018) 
* feat: health-gated Blueprint rollout policy

A rollout policy that reacts to each target's health outcome instead of
deploying the whole fleet blind. Five modes, applied one target at a time
under any non-observe policy:

  observe     record the outcome and keep deploying the rest
  pause       hold the rollout before the next target
  retry_once  deploy the same generation one more time
  stop        hold, and leave the failed target where it is
  rollback    put the failed target back on the generation it ran before

The policy is frozen into the rollout generation at authorization, so a
fleet is never halfway between two policies, and it is carried across
Blueprint edits. Observe is the default and is exactly the old behaviour.

How it works:

  - A health run is reserved, with the whole rollout binding, before the
    apply goes out, so a lost response still leaves a durable record.
  - The next target is the first required node not settled by health, not
    the first unacked one, so a fleet cannot run ahead of a check.
  - The verdict drives a pure decision function; the executor carries the
    result out afterwards, so the state is durable before anything is
    applied or restored.
  - `unknown` never retries, stops, or rolls back. Missing evidence is
    held for a human, not acted on.
  - Remote targets are refused for non-observe policies before the first
    apply, because this instance cannot produce a health verdict for a
    remote node's Docker.

Permissions: the policy write and the controls take `stack:deploy` at the
application, and every target the policy can act on is checked exactly.

* fix: hold before a policy restore and never lose its record

Five review findings on the health-gated rollout policy, all on the paths
the policy exists to make safe.

A completion that could not be recorded was swallowed. `holdRollout` ran
only after `restoreTargetToGeneration` returned, so a `rollbackCompleted`
that threw left the outer catch with nothing to do: no hold, no partial
record, and the target stuck in `restoring` while the fleet carried on as
if the policy had not stopped it. The restore and the record of it are now
one unit, and any failure records a partial failure on the target *and* the
application, since `rollbackInProgress` opened the application into
`restoring`.

The hold now precedes the restore rather than following it. A restore is
external and slow, and until it returns the rollout is still authorized, so
an operator dispatch or an auto-accept could deploy the next target, or
this one again, while it was in flight.

A decided rollback was indistinguishable from a pause, so a crash between
deciding and doing lost it: the fence read as a question, a resume answered
it, and the rollout moved on past a target still on the failed generation.
It now has its own reason, which no resume clears.

A resume re-paused on every restart. A kept fence is also what reconstruction
holds on, so a rollout the operator had already answered for was held again
each time. A resume now records the answer as `stop_acknowledged`, which
still keeps the target out of the queue but no longer holds the rollout.

A policy rollback passed null captured refs, which cleared the source
acceptance the restored generation was bound to and recorded it unprovable.
That is source-authority state, and a health verdict has none over it.

A blocked follow-up dispatch now holds the rollout with the reason it was
refused. There is no scheduler, so that dispatch is the only thing that
brings the next target.

Also: the policy route required the fleet-wide deploy grant on top of the
per-target checks, unlike the pause and resume routes beside it, so an
operator whose grants are scoped to specific stacks could pause and resume
but not configure the policy. The fleet-wide grant is now required only
when there is no target to check.

The rollback happy-path test asserted only that the restore had been
*called*, and passed while `rollbackCompleted` threw inside the executor
because the fixture's generation row was fake. It now seeds a real one and
asserts the completion, plus the completion-failure, restore-throws, and
source-acceptance cases.

* fix: a pending roll back stops the rollout, not just its own target

Excluding a `rollback_pending` target from the queue is not enough. The
queue steps over it and deploys the next one, so a fleet advances past a
target still on the generation that failed, with no restore done and
nothing telling the operator. Resuming did not help either: it left the
fence in place and unpaused the application, so the next dispatch moved on
and the following restart paused the rollout again, repeating for as long
as the operator alternated the two.

A gated dispatch now refuses outright while any target of the current
rollout generation is `rollback_pending`, and the refusal names the manual
restore route as the way out. The fence stays where it was, so the target
card still reports it.

The test now uses two nodes and dispatches after the resume, so the
step-over is visible; it is verified to fail without the change.

Also corrects the schema comment, which claimed a restart mid-rollback
"reconciles the rollback". It fences the unfinished restore and does not
replay it, which is what the comment says now.
2026-09-26 15:34:14 -04:00

385 lines
12 KiB
JSON
Raw Permalink Blame History

{
"$schema": "https://mintlify.com/docs.json",
"theme": "mint",
"name": "Sencho",
"colors": {
"primary": "#00BEC7",
"light": "#007982",
"dark": "#00BEC7"
},
"logo": {
"light": "/images/logo/sencho-logo-light.svg",
"dark": "/images/logo/sencho-logo-dark.svg",
"href": "https://sencho.io"
},
"favicon": "/images/logo/favicon.ico",
"appearance": {
"default": "dark"
},
"fonts": {
"family": "Geist"
},
"background": {
"decoration": "gradient",
"color": {
"dark": "#090909"
}
},
"navbar": {
"links": [
{
"type": "github",
"href": "https://github.com/studio-saelix/sencho"
}
],
"primary": {
"type": "button",
"label": "Get Started",
"href": "https://sencho.io/#pricing"
}
},
"footer": {
"socials": {
"github": "https://github.com/studio-saelix/sencho"
},
"links": [
{
"header": "Product",
"items": [
{ "label": "Website", "href": "https://sencho.io" },
{ "label": "Pricing", "href": "https://sencho.io/#pricing" },
{ "label": "Changelog", "href": "https://github.com/studio-saelix/sencho/releases" }
]
},
{
"header": "Community",
"items": [
{ "label": "GitHub", "href": "https://github.com/studio-saelix/sencho" },
{ "label": "Contributing", "href": "https://github.com/studio-saelix/sencho/blob/main/CONTRIBUTING.md" }
]
},
{
"header": "Legal",
"items": [
{ "label": "Terms of Service", "href": "https://sencho.io/terms" },
{ "label": "Privacy Policy", "href": "https://sencho.io/privacy" }
]
}
]
},
"api": {
"auth": {
"method": "bearer",
"name": "Authorization"
},
"playground": {
"display": "simple"
}
},
"interaction": {
"drilldown": false
},
"navigation": {
"tabs": [
{
"tab": "Guides",
"groups": [
{
"group": "Start here",
"pages": [
"getting-started/introduction",
"getting-started/quickstart",
"getting-started/configuration"
]
},
{
"group": "Product guide",
"pages": [
"features/overview",
{
"group": "Core workflow",
"expanded": true,
"pages": [
"features/stack-management",
"features/editor",
"features/stack-file-explorer",
"features/resources",
"features/networking",
"features/dashboard",
"features/app-store",
"features/global-search",
"features/deep-links",
"features/appearance"
]
},
{
"group": "Stack intelligence",
"pages": [
"features/stack-activity",
"features/stack-dossier",
"features/stack-drift",
"features/compose-doctor",
"features/compose-networking",
"features/environment-guardrails",
"features/docker-label-audit",
"features/compose-storage",
"features/stack-labels",
"features/sidebar"
]
},
{
"group": "Deploy and automate",
"pages": [
"features/deploy-progress",
"features/atomic-deployments",
"features/health-gated-updates",
"features/deploy-enforcement",
"features/git-sources",
"features/git-transport-support",
"features/gitops-portfolio",
"features/gitops-drift-model",
"features/health-rollout-policy",
"features/blueprint-model",
"features/scheduled-operations",
"features/auto-update-policies",
"features/auto-heal-policies",
"features/webhooks"
]
},
{
"group": "Observe and audit",
"pages": [
"features/global-observability",
"features/alerts-notifications",
"features/audit-log"
]
},
{
"group": "Fleet management",
"pages": [
"features/multi-node",
"features/pilot-agent",
"features/sencho-mesh",
"features/fleet-view",
"features/fleet-dossier",
"features/fleet-federation",
"features/fleet-actions",
"features/fleet-sync",
"features/fleet-secrets",
"features/fleet-backups",
"features/remote-updates",
"features/node-compatibility",
"features/host-console"
]
},
{
"group": "Security and access",
"pages": [
"features/security",
"features/vulnerability-scanning",
"features/cve-suppressions",
"features/private-registries",
"features/two-factor-authentication",
"features/rbac",
"features/sso",
"getting-started/sso-quickstart",
"features/api-tokens"
]
}
]
},
{
"group": "Operations",
"pages": [
"operations/self-hosting",
"operations/upgrade",
"operations/backup",
"operations/recovery",
"operations/emergency-cli",
"operations/troubleshooting",
"operations/verifying-images",
"operations/trivy-setup",
"operations/two-factor-admin"
]
},
{
"group": "Reference",
"pages": [
"reference/settings",
"features/licensing",
"reference/security",
"reference/contact"
]
}
]
},
{
"tab": "Tutorials",
"groups": [
{
"group": "Fleet & nodes",
"pages": [
"tutorials/enroll-a-remote-node",
"tutorials/set-up-fleet-secrets",
"tutorials/set-up-fleet-federation",
"tutorials/configure-recovery-vault-backups"
]
},
{
"group": "Deploy & automate",
"pages": [
"tutorials/create-and-approve-a-blueprint",
"tutorials/connect-a-git-source",
"tutorials/schedule-an-operation",
"tutorials/configure-auto-update-policies",
"tutorials/configure-auto-heal-policies",
"tutorials/set-up-deploy-enforcement"
]
},
{
"group": "Secure & integrate",
"pages": [
"tutorials/set-up-sso",
"tutorials/grant-scoped-stack-access",
"tutorials/configure-environment-guardrails",
"tutorials/set-up-a-webhook",
"tutorials/resolve-stack-drift"
]
}
]
},
{
"tab": "API Reference",
"openapi": "openapi.yaml",
"pages": [
"api-reference/overview",
"api-reference/security",
{
"group": "Health & Meta",
"pages": [
"GET /api/health",
"GET /api/meta",
"POST /api/system/update"
]
},
{
"group": "Stacks",
"pages": [
"GET /api/stacks",
"POST /api/stacks",
"GET /api/stacks/readiness-summary",
"GET /api/stacks/{stackName}",
"PUT /api/stacks/{stackName}",
"DELETE /api/stacks/{stackName}",
"GET /api/stacks/{stackName}/envs",
"GET /api/stacks/{stackName}/env",
"PUT /api/stacks/{stackName}/env",
"GET /api/stacks/{stackName}/containers",
"GET /api/stacks/{stackName}/services",
"POST /api/stacks/{stackName}/deploy",
"POST /api/stacks/{stackName}/down",
"POST /api/stacks/{stackName}/start",
"POST /api/stacks/{stackName}/stop",
"POST /api/stacks/{stackName}/restart",
"POST /api/stacks/{stackName}/update",
"POST /api/stacks/{stackName}/rollback",
"GET /api/stacks/{stackName}/backup"
]
},
{
"group": "Containers",
"pages": [
"GET /api/containers",
"GET /api/containers/{id}/logs",
"POST /api/containers/{id}/start",
"POST /api/containers/{id}/stop",
"POST /api/containers/{id}/restart"
]
},
{
"group": "API Tokens",
"pages": [
"POST /api/api-tokens",
"GET /api/api-tokens",
"DELETE /api/api-tokens/{id}"
]
},
{
"group": "Webhooks",
"pages": [
"GET /api/webhooks",
"POST /api/webhooks",
"PUT /api/webhooks/{id}",
"DELETE /api/webhooks/{id}",
"GET /api/webhooks/{id}/history",
"POST /api/webhooks/{id}/trigger"
]
},
{
"group": "Nodes",
"pages": [
"GET /api/nodes",
"POST /api/nodes",
"GET /api/nodes/{id}",
"PUT /api/nodes/{id}",
"DELETE /api/nodes/{id}",
"POST /api/nodes/{id}/test",
"GET /api/nodes/{id}/meta"
]
},
{
"group": "Fleet",
"pages": [
"GET /api/fleet/overview",
"GET /api/fleet/readiness",
"GET /api/readiness/evidence",
"GET /api/fleet/node/{nodeId}/stacks",
"GET /api/fleet/node/{nodeId}/stacks/{stackName}/containers",
"GET /api/fleet/update-status",
"POST /api/fleet/nodes/{nodeId}/update",
"POST /api/fleet/update-all",
"POST /api/fleet/snapshots",
"GET /api/fleet/snapshots",
"GET /api/fleet/snapshots/{id}",
"POST /api/fleet/snapshots/{id}/restore",
"DELETE /api/fleet/snapshots/{id}"
]
},
{
"group": "Scheduled Tasks",
"pages": [
"GET /api/scheduled-tasks",
"POST /api/scheduled-tasks",
"GET /api/scheduled-tasks/{id}",
"PUT /api/scheduled-tasks/{id}",
"DELETE /api/scheduled-tasks/{id}",
"PATCH /api/scheduled-tasks/{id}/toggle",
"POST /api/scheduled-tasks/{id}/run",
"GET /api/scheduled-tasks/{id}/runs",
"GET /api/scheduled-tasks/{id}/runs/export"
]
},
{
"group": "Registries",
"pages": [
"GET /api/registries",
"POST /api/registries",
"PUT /api/registries/{id}",
"DELETE /api/registries/{id}",
"POST /api/registries/{id}/test"
]
},
{
"group": "Image Updates",
"pages": [
"GET /api/image-updates",
"POST /api/image-updates/refresh",
"GET /api/image-updates/status"
]
}
]
}
]
}
}
Reference in New Issue View Git Blame Copy Permalink