Files
sencho/docs/tutorials
Anso 324dd7e1ae fix(gitops): restore only the generation a Blueprint target acknowledged (#2024)
* fix(gitops): restore only the generation a target acknowledged

A Blueprint's Enforce mode restores state; it does not authorize it. The
repair path resolved the identity to restore from the application's
current pointers, so accepting a newer generation for any reason silently
changed what an already-deployed target was repaired to. On a target
acknowledged against an older generation this shipped compose content
that no rollout had authorized for it, pinned to an artifact set from a
different generation, which is a content and artifact substitution
arriving through the one path that is supposed to be safe.

The authority for a repair is now resolved in one place from immutable
rows: the target's acknowledged generation and the artifact set that
generation was qualified under, corroborated for Git-managed content by
the live rollout generation that froze the pair. The application's newest
accepted generation is no longer read on the repair path at all.

Every way the authority can fail to resolve becomes an explicit hold with
its own reason rather than a guess or a generic failure: no acknowledged
generation, a superseded rollout, a node outside the frozen target set, an
unreadable row, or a disagreement between the rollout and the target's
acknowledgements. A hold records its own deployment status and history
stage, repeats no work on the next tick, and never mutates the workload.

The on-node marker now names the generation, artifact set, and rollout that
installed it. Without that a repair could not prove what it was about to
overwrite, so a marker that does not name one holds rather than assuming.
Markers written before this still parse.

Tests cover the acknowledged generation winning over a newer accepted one,
the superseded-rollout hold, the never-acknowledged hold, and digest pins
coming from the acknowledged set.

* fix(gitops): hold every runtime repair the policy must not attempt

A drift repair that cannot prove what it would overwrite must decline
visibly, not fail quietly and not guess. Each way the authority can fail to
resolve now becomes a typed hold with its own deployment status, its own
history stage, and its own attention reason, so it stays visible instead of
reading as converged.

The arms are the states the drift model already treats as uncertainty rather
than divergence: a target that is unreachable or whose connectivity evidence
is stale, a target whose earlier mutation was interrupted, a target an
unfinished rollback or an unavailable LKG owns, and a target outside the
frozen required set of the rollout it is bound to.

Two decisions move ahead of the drift write rather than after it. A stateful
or unclassifiable Blueprint is never auto-repaired, and the classification
check now runs before the drift is committed, so a held target is not first
written as drifted and then rewritten on every tick. The check is repeated at
the site that mutates, so the guard holds even when that path is reached
directly.

A hold is a state, not a latch: a later tick re-checks the target, so clearing
the cause lets the next tick repair it. A hold that has not changed is not
recorded again, which keeps a permanently held target from appending history
forever.

The on-node marker gains the generation, artifact set, and rollout that
installed it, and the rollout path stamps them, so a drift check on a remote
leaf compares against the same authority the hub resolved rather than
inferring it. Both transports serialise the same marker, asserted byte for
byte in the tests.

Tests cover each hold arm, the hold-then-repair transition, replay
suppression, survival across a store reset, local and remote reaching the
same decision, and Observe and Suggest never mutating.

* fix(gitops): re-check held drift targets and report a hold as a hold

The hold work that closed the runtime drift policy made two states one-way,
which is worse than the behaviour it replaced. A repair that re-resolved its
authority and declined was reported as a correction failure and left the row
reading "correcting" with nothing in flight, and a held row matched no branch
in the decision pass, so an ordinary network flap could strand a target
permanently where it had previously self-healed.

Both paths now handle a hold as a decision: a declined repair records the hold
with its own reason and warning, and a held row keeps its drift check, so
clearing the cause lets the next tick repair it. A check that finds the target
already matching returns the row to active through a cause that records no
history, because nothing was written to a node.

Classification holds get their own reason instead of borrowing
evidence_incomplete, which sent operators looking for a missing generation
when the Blueprint was stateful. Their text drops the drift reason, which was
the replay key: a reason changing per tick re-alerted every tick for a hold
that never changed. The recovery arm now also covers an in-progress recovery
and a rollout awaiting its health verdict, both of which own the target.

Docs no longer promise every hold self-clears. Superseding is one-way, so that
hold persists until a later rollout covers the target or an operator deploys.

* fix(gitops): bound the recovery hold to a recovery in progress

The recovery arm tested recovery_phase for non-null, but that column keeps a
terminal value for the life of the target and is never reset, so every target
that had ever been rolled back would have been held from repair permanently,
with only an explicit deploy to clear it. Only capturing, restoring, and
compensating own the target now, matching the in-progress set the projection
already uses.

Clearing a hold without repairing it now advances the observation stage rather
than only the deployment row. A row that went back to active while the
projection still read the hold, or the drift, that had just cleared left the
Drift surface and the attention queue contradicting the deployment table with
no event left to correct them.

Both carry tests that fail without the fix. The projection case asserts a
literal status and requires the target to exist, because the optional chain it
replaced made the whole case pass on a missing target.

* fix(gitops): let a failed recovery keep holding, and rank both new statuses

A failed recovery was the one terminal phase that stopped holding once the arm
was narrowed to in-progress phases, so auto-repair would have performed exactly
the mutation an explicit deploy is already refused on the same target. Only a
recovery that finished successfully releases the target now, and a test pins it.

The severity ladder had neither of the new runtime statuses, and a status it
does not know does not degrade to something reasonable: it reads as
unrecognized and the whole application row becomes unknown. A hold is ranked
alongside drift for that reason.

Clearing a hold advances the observation stage rather than only the deployment
row, so the decision the reconciler made is what the projection reads, instead
of depending on the observed-artifact write that happens to follow it. The test
asserts the stage, because the derived status is identical either way.

A new "converged" runtime status was tried here and reverted: the posture
vocabulary has no slot for "checked but no health verdict", so it landed on
unknown. A matched drift check projects to the status a Blueprint target
already reached by the pointer path, which claims no health verdict either.

Each test was checked to fail with its fix reverted. Two earlier attempts at
the same tests passed with the bug reintroduced, one through an optional chain
and one because a neighbouring check claimed the status first, so both now
assert the mechanism rather than the outcome.

* test(blueprints): render a held repair in the deployment table

The held row is where a declined repair becomes visible, so it is rendered and
asserted rather than only typechecked. A status that reaches the table without a
label renders as a blank chip, and only the label map being total over the
status union would have caught it.

* fix(gitops): classify drift for Inline targets and observe held ones

The repair authority was corroborated against a rollout generation whenever
the target carried a rollout_generation_id, but an Inline target can carry one
that was never the authority for its acknowledged pair, because Inline freezes
its own generation per acceptance. Every such target was held for a disagreement
that could not exist, and a held target reported no drift at all, so a replaced
image on an Inline Blueprint was invisible in every drift surface. The
corroboration is now keyed on the content mode rather than the pointer.

A hold also suppressed the observation, because it returned before the runtime
identity was collected. A hold refuses to mutate, not to observe, and what the
node is running is exactly the evidence an operator needs when a hold blocks
the repair. The identity is now captured before the hold is returned, with the
probe and the record each guarded so neither can turn the hold into a failure.

Both carry tests verified to fail with the fix reverted.
2026-09-27 18:49:20 -04:00
..