mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-26 13:40:26 +00:00
Anso
0ce9754766
* fix(gitops): make automatic placement approvals reach execution Two defects on the bounded automatic placement path. A removal was never auto-approved: the decision read `gitops_targets.connectivity`, a column seeded to null that no producer ever wrote, so a withdrawal resolved to `unknown_connectivity` even though the target projection already derived reachability from the recorded observation. The decision now reads the projection's own `targetConnectivity`, requires the node registry to have the node online (the same gate an addition passes), and carries an observation that cannot be decoded as a limitation, so the refusal names malformed evidence instead of sending the operator to check a node that answered. An Inline Blueprint's policy placement approval was recorded and then never read: an operational edit clears the combined approval, and the reconciler returned at the legacy gate before consulting the policy's approval. The reconciler now executes a plan a current policy placement approval covers, under seven conditions: the application is inline, the placement policy is still the one that could have written the approval, the approval is a policy's, the rollout authorization policy is automatic, at least one node is already placed at the current revision, every retained node runs the current revision, and the approval covers every action in the plan. An uncovered plan waits for the operator rather than partly executing, a first placement always waits for Apply, and a retained node is observed but never repaired under this approval, because the placement policy decided the node set and not the content. A node-driven roster change no longer hides a policy approval behind a current combined one: a label or cordon change moves the node set without moving the combined approval's fingerprint, so the reconciler tries the policy approval first and falls back to the combined approval when the policy cannot run the whole plan. That fallback narrows the operator's blast to the placement's frozen target set instead of invalidating it, so a removal a policy approved no longer stops drift observation on the nodes that stay; the preview and the lightweight list and detail evaluators read the same narrowing, so no surface reports an approval the executor withholds or withholds one it will run. An operator's approval is never discarded by a policy decision. Demoting an application back to Inline takes the Git-era authority with it: the placement pointer is cleared and the rollout authorization policy returns to operator authorization, because neither was an operator's choice about Inline placement. An Inline Blueprint can reach the automatic rollout authorization the policy path requires from the control beside the placement policy. * fix(gitops): say when automatic placement waits for Apply A selector edit that moves one stateless node runs on the next tick, and Enforce holds a policy-approved plan until Apply. The docs described both as waiting, and the preview showed a pending plan with no reason. * fix(gitops): keep a cleared pin from automatic placement A pin clear left the new intent unpinned, so bounded auto could place the freed node without Confirm Apply. The decision now treats a clear as pin-driven when the standing approval was recorded on a pinned intent, or when a pin was written after an older approval and never confirmed. * fix(gitops): keep a detached snapshot off automatic placement Git-managed deploys stamp the Blueprint revision, and Git content never bumps it, so after Detach a retained node still looked current. Recording a new revision on Detach makes that stamp stale until Apply. * fix(gitops): hold Enforce plans that drop a mid-deploy repair A retained node that is correcting or deploying leaves the plan with no repair, so a policy approval could place an added node in Enforce. The mode itself now holds that plan until Apply, and the preview says so only when Enforce is the reason it is waiting. * fix(gitops): keep a pin chain and a re-armed policy from placing on their own Three authority gaps, all on the automatic placement path, found by the pre-merge audit. A pin that was set and cleared with no Apply between them read as no pin at all once a third intent followed: the intent before the current one was the clear, so the pin that moved the work was out of view and the freed node was placed without Apply. With no decomposed placement approval to anchor on there is no record of any confirmation, so the whole intent history now counts. A Confirm Apply mints the approval the decision anchors on from, and the next change is judged against that, so the wait ends at the next operator decision rather than never. Re-enabling a policy revived a decision the operator had taken back: execution stopped the moment the policy went, but the approval the policy wrote stayed on the row, so setting it back handed the next tick a decision made days earlier on evidence nobody had read since. Arming either policy again now withdraws what that policy decided, with the rollout authorization that only resolves through it. An operator's approval is untouched, the approval rows stay as the record of what was decided, the rollout generation stays so a fresh approval re-drives it, and the history row records the pointers that went with the policy. The refusals a withdrawal produces now read as what the backend checks: a node that has not reported, or that is not a node this Sencho knows about, is a different answer from a node that is not answering, and a withdrawal needs the node to have reported the workload it was running. * fix(gitops): clear the attention a policy placement left behind A Blueprint the placement policy placed a node for sat at Needs attention with a stale acknowledgement and a runtime drift, and neither Confirm Apply nor an API apply cleared it: the retained nodes' plan is observation-only, so nothing ever re-acknowledged them. Two causes, both fixed. The Inline freeze is now keyed on the compose digest rather than the intent revision, so a roster change no longer drops the accepted generation and re-points every placed target at a new generation id for the content it already runs. And the projection's identity test yields when both intent revisions resolve with the same compose digest and the current candidate still requires that node: a node the candidate dropped keeps its signal, because a withdrawal the plan is asking for is not a superseded acknowledgement. Two roster changes inside one tick also left both additions unplaced with no refusal recorded. The reconciler runs one approval and refuses a plan it does not cover whole, so the second approval named only its own addition while the plan still wanted the first. The blast now names every node the fleet does not hold, carrying the earlier decision's unlanded placement forward, while the decision still reads the approved set for its size and safety reasons and the evidence is read for every node the blast names. Removals stay on the approved set, because a withdrawal is judged from the observation of the workload that ran there. The Confirm rollout dialog no longer claims the fleet cannot change until it is confirmed when a configured policy has already approved the plan. * fix(gitops): write the placement blast in canonical order and name a withdrawn node again Two findings from the QA run of this PR, both now pinned by failing-first tests. The first: a roster move that adds a higher node id while withdrawing a lower one left the approval unwritable and the operator looking in the wrong place. The blast effect was assembled places-then-removals, while the decoder requires the node ids strictly increasing, so the write was refused as out of canonical order. The refusal reason then reported that refusal as `conflicting_operation`, so the operator was told an operation was in flight when none was, the plan read `reapproval_required`, and nothing executed. The effect is now sorted by node id, which is the order the decoder wants and carries no meaning of its own: the reconciler reads the effect as a map keyed by node id and the transition re-encodes it to canonical form anyway. Blast decode failures are also recorded as `malformed_evidence` rather than falling through to the conflict default, because a blast that did not decode is a data fault. The roster window that reaches it: the reconciler executes one approval per tick, so an addition approved but not yet landed is carried into the next approval's blast, which then names a placed node alongside the withdrawal. The second: a withdrawn node that the selector wants again produced no row in the plan at all. The tombstone holds deliberately, since only an explicit deploy re-opens a severed placement, but the node stayed online and labelled with nothing to say about why it was skipped, so the plan read as nothing to do. A new informational `skip_withdrawn` row now names what the operator has to do. It is excluded from the confirmable set and from both executor groups, so it authorizes nothing and cannot be confirmed into a mutate, and it yields to the status-precedence pass so an in-flight or guard row for the same node wins rather than leaving two explanations stacked on one node. Follow-up, not addressed here: the replay-guard refusal (`placement approval already recorded for this intent and candidate`) still maps to `conflicting_operation`, which is the same class of untruthful reason on the crash-retry path. It predates this change and is left for its own pass. Tests: a new case drives the add-then-withdraw window end to end and asserts the approval lands with `stateless_removal` and no refusal recorded; a second drives a severed target the selector wants again and asserts the informational row; a third pins that a severed node carrying a `withdrawing` row reports once, as the in-flight withdrawal. Backend: `npx tsc --noEmit`, `npm run lint` (0 errors), `npm test` (11013 passed; the single file-level failure is `git-transport-ssh.integration.test.ts`, which requires loopback port 22 free and fails identically on an unmodified tree because this host runs sshd on 22, freed in CI by `scripts/ci-free-loopback-ssh-port.sh`). Frontend: `npx tsc -b --noEmit`, `npm run lint` (0 errors).
181 lines
19 KiB
Plaintext
181 lines
19 KiB
Plaintext
---
|
|
title: "GitOps Drift"
|
|
description: "The drift classes Sencho derives for every Git-managed stack and Blueprint, what each one proves, and which policy decides what happens next."
|
|
---
|
|
|
|
Sencho reports drift in two layers. The **spatial** layer compares the effective Compose model against the containers running right now and is covered in [Drift Detection](/features/stack-drift). The **GitOps** layer compares what you intended against what Sencho has proven, accepted, and observed over time, for anything managed by a Git source or a [Blueprint](/features/blueprint-model).
|
|
|
|
This page is the reference for that second layer: the classes it reports, the evidence each one rests on, and the policy that decides what happens after a class is reported.
|
|
|
|
## The seven classes
|
|
|
|
Every GitOps drift item carries a class. The class says which part of the chain diverged, and that decides who owns the fix.
|
|
|
|
| Class | The comparison it is made from | A mismatch is reported when |
|
|
|---|---|---|
|
|
| **Source** | The accepted generation's commit against the commit the configured ref names. | The two differ, or the last fetch or validation failed. |
|
|
| **Managed project** | The stack's recorded managed-project manifest against the accepted generation. | They differ, or the manifest is missing or unreadable. |
|
|
| **Placement** | The approved blast against the nodes that hold applied authority. | A recorded approval no longer binds the current intent, or a node still holds authority outside the required set after the rollout settled. |
|
|
| **Health** | The latest stack-scoped health run for a node against the generation that node runs. | The run failed for exactly that generation. |
|
|
| **Runtime** | What is running against what was approved: the artifact identity, and the generation each node was asked to run. | A node reports another artifact identity, or runs a generation other than the one it was asked for. |
|
|
| **Rollout** | A required rollout target's observed artifact identity against the authorized rollout set. | A required target reports another identity. |
|
|
| **Invocation** | The compose invocation a node recorded when the stack was applied, against the invocation that generation was applied with. | The node's compose file order, project name, project directory, or env file order is not the one it was given. |
|
|
|
|
This is the model's vocabulary: the boundaries a drift item can fall into, and what each one would be measured against. A class appears only where Sencho holds the evidence for it and the rows prove a mismatch, so a class whose evidence is not held is absent rather than approximated. The table does not promise that every class is reported today; it says what each class means when it is.
|
|
|
|
The [Drift tab](/features/stack-drift) and the application sheet show the classes they have evidence for. The portfolio's drift summary counts them by class, and its attention queue names the ones that are waiting on you.
|
|
|
|
## Reading an item
|
|
|
|
Each item is one line of comparison plus the facts you need to act on it. The drift row shows the class, the owner, the reason, and the expected identity beside the observed one.
|
|
|
|
- **Class** names the part of the chain that diverged.
|
|
- **Expected** is the identity Sencho was meant to run: a commit, a generation, an artifact set, an intent, or a health run.
|
|
- **Observed** is what it found: the same kind of identity, or `unknown` when Sencho found nothing to compare.
|
|
- **Reason** says in one line what differs.
|
|
- **Owner** names the surface that owns the fix, so you know whether to go to the Git Source sheet, the Blueprint, or the node.
|
|
- **Affected targets** names the nodes involved. A direct Git source has one; a Blueprint can have many.
|
|
- **Freshness** is when the evidence was recorded, when the evidence carries a time. An old observation is weaker proof than a new one, so the timestamp travels with the item; an item derived from two pointers rather than a recorded observation has no timestamp to show.
|
|
- **Action** is the one step that would resolve the item for the affected target, and only when that step is legal right now. A paused or failed node gets no action rather than one that cannot run.
|
|
|
|
## When Sencho will not repair it
|
|
|
|
A Blueprint whose drift mode is **Enforce** can restore runtime drift on its own, but only the state the target was already authorized to run: the generation that target acknowledged and the image identity approved for it. A newer accepted generation never changes what an older target restores, because the repair answers "put this node back" rather than "deploy the newest thing".
|
|
|
|
Holding is an **Enforce** decision, made fresh each tick. Observe and Suggest still detect and report the same drift, because "auto-fix was declined" is not an answer for someone who never asked for one. And a hold never suppresses detection: Sencho still inspects the node and still records what is running, it only declines to overwrite it.
|
|
|
|
Two things a held row does not tell you. Its reason is the hold, not the drift, so read the Drift panel for what is actually running. And a row held under Enforce keeps reading **Repair held** if you switch to Observe or Suggest while the drift cannot be classified, because the next check has nothing to record until the cause clears. The deployment row reads **Repair held** with the reason, the attention queue names it, and the workload is left untouched. A held target is still drifted: it is never counted as up to date.
|
|
|
|
The reasons a repair is held are the states this model already treats as uncertainty rather than divergence: a stateful or unclassifiable Blueprint, a rollout that has moved on, on-node evidence that does not name a generation, an unreachable target or stale connectivity, an interrupted change, a recovery or rollback that owns the target, and a disagreement between the rollout's frozen pair and the target's own expectation.
|
|
|
|
Most holds clear themselves. Sencho keeps re-checking the target, so once the cause is resolved, by a node returning or an interrupted change finishing, the next reconciliation repairs the drift normally.
|
|
|
|
One does not: a hold because the rollout moved on stays until a later rollout includes that target. Superseding is a one-way fact, so nothing reopens that rollout on its own, and Git-managed Blueprints have no manual redeploy to fall back on, so the rollout is the only way out.
|
|
|
|
## Where drift is shown
|
|
|
|
| Surface | What it shows |
|
|
|---|---|
|
|
| **Drift** tab on a stack | The spatial findings plus the GitOps state for that stack, one card per node, with the per-service digest comparison under each node. |
|
|
| **Application sheet** in the portfolio | The same drift items for one application, alongside its source, artifact, placement, and rollout state. |
|
|
| **Portfolio application table** | A count and the classes involved, so you can narrow the list to one class. |
|
|
|
|
The same projection feeds all three, so a class never reads differently depending on where you look.
|
|
|
|
## Per-service digests
|
|
|
|
When the approved artifact set and the node's observation both carry per-service image evidence, each target's card shows the approved digest beside the running one, per service, and marks each one matched, drifted, or unverified. The comparison uses the child digest for the platform that node runs, because a multi-platform image has one child per platform and a node only ever proves its own.
|
|
|
|
Where evidence is missing, the card says so. It never renders an empty comparison, because an empty table reads as agreement.
|
|
|
|
When a Blueprint is first applied, Sencho freezes the exact image identity it was approved at by resolving each image against its registry. If that resolution cannot reach the registry, the target has no approved identity to compare against and its drift reads as unverified rather than as agreement.
|
|
|
|
For a Blueprint applied from stored content, Sencho retries that resolution on its own, so the target recovers on its own schedule instead of waiting for you to redeploy it. A caveat under the state names the condition while it lasts, and clears once the identity is provable. The retry interval is yours to set under **Settings → GitOps**, and it has no effect while every target's identity is already resolved.
|
|
|
|
The retry resolves the images the Blueprint was approved at, not whatever compose happens to be in the stack directory at the time. If someone edits a file on the node directly, that change is drift to be reported, never an approved identity to be adopted.
|
|
|
|
Some Compose shapes are not recovered this way, and Sencho says so rather than guessing. The approved images are read from what you wrote, while a deploy resolves them the way Compose does, and the two can disagree for `profiles`, `extends`, `include`, and YAML merge keys. For those, the retry stands down and the target stays unverified until you redeploy the Blueprint.
|
|
|
|
The tick that resolves a target waits before judging it. It reports unverified that pass, and compares on the following one. That is one reconciler interval, not a review window: if the registry resolves a tag that moved while the identity was unproven, the next tick reports it as drift, and Enforce mode repairs against it.
|
|
|
|
<Note>
|
|
A digest is compared, never pulled. When Sencho repairs a digest divergence it redeploys against the approved identity, so authored tags on disk stay as you wrote them.
|
|
</Note>
|
|
|
|
## What is not drift
|
|
|
|
Work in progress is not drift, and the classes above stay silent while it happens:
|
|
|
|
- a commit that has been fetched but not yet accepted, including one waiting on review
|
|
- a placement or authorization decision that is still open
|
|
- a rollout that is queued, canarying, batching, or waiting on its health verdict
|
|
- a manifest being written as part of an apply that has not landed on the node yet
|
|
- a node that is deploying, interrupted, or recovering
|
|
- a drift that Enforce is repairing on its current pass, which is why the Blueprint deployment row reads as correcting rather than drifted
|
|
- a target that has not been deployed yet, or has never had one recorded
|
|
|
|
Those states have their own surfaces, and each one names the next step. A node running a different generation than the one it was asked for is a separate class of its own, and is reported whether or not something else about that node is in flight.
|
|
|
|
One more case sits here by decision rather than by accident. Source drift compares the accepted commit against the commit the ref names. When those still agree but the materialization settings changed, for example a compose path, an env path, or a context directory, there is no expected-versus-observed pair of commits to report, so no source drift item is created. Instead the source shows as **reconcile required** in the attention queue, and for a Direct application that is where you fetch again: the fetch rebuilds the candidate under the settings now in force, and accepting that candidate produces the new generation. This is a choice about what the word drift means, not a claim that the materialized content was proven unchanged. Changing where Compose is read from can change the content, which is why Sencho asks rather than assumes.
|
|
|
|
## Invocation
|
|
|
|
Invocation drift answers one question: is this stack being brought up the way this generation said it would be?
|
|
|
|
When a generation is applied, Sencho reads back how Compose was actually invoked on the node: the compose files in order, the project name, the project directory, and the env files in order. That record is compared against the invocation the accepted generation was applied with. It is the only evidence in the model that can contradict the authored invocation at all, because reading the command back would only confirm what Sencho already believed.
|
|
|
|
What it catches in practice is a project that came up differently from the way the generation named it. The common case is a stack directory holding a compose file the generation did not name: Compose resolves it, the node records more files than were asked for, and the difference is reported. A node that answers with a different project name, project directory, or file order is reported the same way.
|
|
|
|
Two limits worth knowing, both about when the evidence is taken:
|
|
|
|
- **The record is read after the apply finishes.** A stack brought up by hand *before* an apply is overwritten by that apply, so the apply reads back its own work. A hand-made invocation that the apply replaced is not what you see here.
|
|
- **It is a snapshot, not a watch.** Nothing re-reads it on a timer. An invocation changed on the node after the apply is not reported until the next apply records a new one. Every item carries the time its evidence was taken, so you can see how old it is.
|
|
|
|
A stack whose invocation has not been recorded yet reports nothing and says why, rather than showing you an empty comparison that would read as agreement. The first apply records it and the caveat clears.
|
|
|
|
## What Sencho cannot prove
|
|
|
|
Drift is derived from what Sencho has recorded, so missing evidence is never rounded up to agreement. A few cases worth knowing:
|
|
|
|
- A target on a node Sencho does not host has no health verdict here, so health drift stays silent for it rather than claiming a verdict nobody wrote.
|
|
- A Git-managed Blueprint deployment that opens no generation-bound health gate produces no health item.
|
|
- A required-target set Sencho cannot read records a caveat naming the unreadable record, instead of quietly reporting no placement drift.
|
|
- A stack whose invocation has not been recorded yet records a caveat saying so, rather than reporting no invocation drift.
|
|
- A stack whose generation names no compose file records a caveat, rather than reporting a difference against a file list that was never written.
|
|
- A node that was withdrawn while a recovery on it had failed records a caveat, because the withdrawal ended what Sencho tracks there and the failed recovery is no longer reported as a state.
|
|
- A Blueprint deployment is not compared at all. Each of its targets is brought up from the Blueprint rather than from a recorded compose invocation, so there is no expected invocation to compare against, and the class stays silent for them rather than reporting every one as different.
|
|
- A Blueprint target whose approved image identity could not be resolved records a caveat and is not compared against what is running, rather than reporting no drift. For a Blueprint applied from stored content, Sencho retries the resolution on its own, so this clears without a redeploy.
|
|
- A service built on the node has no published digest to compare, so the per-service view labels it **not compared** rather than matched or drifted.
|
|
- A service running something the approved set does not name is outside the comparison, so it reads **not compared** rather than drift.
|
|
|
|
Each of those is listed as a caveat under the state it applies to, on the Drift tab and in the application sheet, saying what could not be proven. The per-service digest view says the same in its own words: a target with no per-service evidence on either side says nothing rather than showing an empty comparison that would read as agreement.
|
|
|
|
## The policy domains
|
|
|
|
A drift class reports; a policy decides. Each decision below is independent, and none of them authorizes another. Three of them answer the same question at three points in the chain: who is allowed to decide. The other two answer what happens after something has already gone wrong or already drifted.
|
|
|
|
| Policy | Where it lives | What it governs |
|
|
|---|---|---|
|
|
| **Source policy** | Per Git source: Manual, Review only, Auto-write files, or Auto-deploy. | How far Sencho goes on its own after a fetch. Manual takes the source out of the cadence entirely, so nothing happens until you ask for a pull. Review only records the result and waits for you. A withdrawn stateful target always waits for you, whatever the source policy says. |
|
|
| **Placement policy** | Per application, beside the placement approval. | Which placement changes are approved without you. **Operator approves each change** is the only option unless you choose **One stateless change at a time**, and that option approves exactly one stateless addition or one stateless removal, falling back to you with a stated reason for everything else. |
|
|
| **Rollout authorization policy** | Per application. On an Inline Blueprint it sits beside the placement policy. On a Git-managed Blueprint it sits in the rollout overflow. | Which rollouts are authorized without you. On an Inline Blueprint it also decides whether a placement the placement policy already approved runs on the next tick. A rollout cannot start before its placement is approved. |
|
|
| **Drift mode** | Per Blueprint: Observe, Suggest, or Enforce. | What happens when the Blueprint's own drift is detected: surface it, notify you, or redeploy it. |
|
|
| **Health rollout policy** | Per application, beside the two above. | Whether a deploy must prove healthy before it is allowed to finish, and what a failure does next: observe, pause, retry once, stop, or roll back. |
|
|
|
|
Two rules keep them honest. A digest repair redeploys against the approved identity rather than pulling whatever the tag now points at, and a stateful change always re-enters operator confirmation, whatever the drift mode says.
|
|
|
|
Every rollout freezes the policy snapshot it executes, and every placement approval carries the snapshot that decided it. That is why a policy edit changes what the next decision may do and nothing else: no standing approval is withdrawn, no generation is invalidated, and the card for work already in flight says which value it was actually decided under rather than the one configured now.
|
|
|
|
The three "who decides" policies are recorded together as one snapshot, so no reader can reach a single value that appears to govern more than one of them.
|
|
|
|
### What a new application starts on
|
|
|
|
A new or newly adopted application waits for an operator at every step. That is
|
|
the safe default for something with no history: nothing about it has yet proved
|
|
that a node can take a generation, and the placement policy starts on operator
|
|
review for the same reason.
|
|
|
|
An application that already existed before these settings were introduced keeps
|
|
what it was doing, and only that. If you adopt a Blueprint after the fact, or
|
|
convert one twice, it starts on operator review rather than inheriting the
|
|
behavior of applications that were around earlier. The setting is beside the
|
|
rollout authorization action it governs, so turning it on is one click rather
|
|
than a request.
|
|
|
|
## Related
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Drift Detection" icon="magnifying-glass" href="/features/stack-drift">
|
|
The spatial layer: effective Compose model against the containers running now.
|
|
</Card>
|
|
<Card title="Git Sources" icon="git-branch" href="/features/git-sources">
|
|
Where the source policy, commits, and acceptance live.
|
|
</Card>
|
|
<Card title="Blueprints" icon="layers" href="/features/blueprint-model">
|
|
Who decides each step, drift modes, and digest drift on a fleet template.
|
|
</Card>
|
|
<Card title="GitOps portfolio" icon="chart-pie" href="/features/gitops-portfolio">
|
|
Every application in one table, with drift by class and the attention queue.
|
|
</Card>
|
|
</CardGroup>
|