mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-27 05:57:40 +00:00
Anso
b6b059a3fb
A tombstoned target reports the tombstone before the runtime facet reads any recovery field, so a target withdrawn while its recovery had failed reported nothing anywhere. Both retirement transitions clear the failure and LKG columns but leave recovery_phase and recovery_ref on the row, and no projection read them back, so the failed recovery was gone from the runtime status, the rollout status, and the attention queue in one step. Mint a withdrawal_residue limitation in deriveTarget for a tombstoned row carrying a failed recovery, on the same condition deriveRuntime reads, so the caveat and the status cannot disagree about a row. It rides the limitations array deriveTarget already threads, so it reaches the caveats surface with no wire change, no new route, and no new broadcast. Tombstone precedence is unchanged: the runtime status stays tombstoned and the rollout status never sees the recovery. Only a retirement that carried a failed recovery is qualified. After any withdrawal Sencho cannot prove the compose project came down, because the teardown is best effort and swallows its own error, but qualifying every retirement would train a reader to skip the block.
170 lines
17 KiB
Plaintext
170 lines
17 KiB
Plaintext
---
|
|
title: "GitOps Drift"
|
|
description: "The drift classes Sencho derives for every Git-managed stack and Blueprint, what each one proves, and which policy decides what happens next."
|
|
---
|
|
|
|
Sencho reports drift in two layers. The **spatial** layer compares the effective Compose model against the containers running right now and is covered in [Drift Detection](/features/stack-drift). The **GitOps** layer compares what you intended against what Sencho has proven, accepted, and observed over time, for anything managed by a Git source or a [Blueprint](/features/blueprint-model).
|
|
|
|
This page is the reference for that second layer: the classes it reports, the evidence each one rests on, and the policy that decides what happens after a class is reported.
|
|
|
|
## The seven classes
|
|
|
|
Every GitOps drift item carries a class. The class says which part of the chain diverged, and that decides who owns the fix.
|
|
|
|
| Class | The comparison it is made from | A mismatch is reported when |
|
|
|---|---|---|
|
|
| **Source** | The accepted generation's commit against the commit the configured ref names. | The two differ, or the last fetch or validation failed. |
|
|
| **Managed project** | The stack's recorded managed-project manifest against the accepted generation. | They differ, or the manifest is missing or unreadable. |
|
|
| **Placement** | The approved blast against the nodes that hold applied authority. | A recorded approval no longer binds the current intent, or a node still holds authority outside the required set after the rollout settled. |
|
|
| **Health** | The latest stack-scoped health run for a node against the generation that node runs. | The run failed for exactly that generation. |
|
|
| **Runtime** | What is running against what was approved: the artifact identity, and the generation each node was asked to run. | A node reports another artifact identity, or runs a generation other than the one it was asked for. |
|
|
| **Rollout** | A required rollout target's observed artifact identity against the authorized rollout set. | A required target reports another identity. |
|
|
| **Invocation** | The compose invocation a node recorded when the stack was applied, against the invocation that generation was applied with. | The node's compose file order, project name, project directory, or env file order is not the one it was given. |
|
|
|
|
This is the model's vocabulary: the boundaries a drift item can fall into, and what each one would be measured against. A class appears only where Sencho holds the evidence for it and the rows prove a mismatch, so a class whose evidence is not held is absent rather than approximated. The table does not promise that every class is reported today; it says what each class means when it is.
|
|
|
|
The [Drift tab](/features/stack-drift) and the application view show the classes they have evidence for. The portfolio's drift summary counts them by class, and its attention queue names the ones that are waiting on you.
|
|
|
|
## Reading an item
|
|
|
|
Each item is one line of comparison plus the facts you need to act on it. The drift row shows the class, the owner, the reason, and the expected identity beside the observed one.
|
|
|
|
- **Class** names the part of the chain that diverged.
|
|
- **Expected** is the identity Sencho was meant to run: a commit, a generation, an artifact set, an intent, or a health run.
|
|
- **Observed** is what it found: the same kind of identity, or `unknown` when Sencho found nothing to compare.
|
|
- **Reason** says in one line what differs.
|
|
- **Owner** names the surface that owns the fix, so you know whether to go to the Git Source sheet, the Blueprint, or the node.
|
|
- **Affected targets** names the nodes involved. A direct Git source has one; a Blueprint can have many.
|
|
- **Freshness** is when the evidence was recorded, when the evidence carries a time. An old observation is weaker proof than a new one, so the timestamp travels with the item; an item derived from two pointers rather than a recorded observation has no timestamp to show.
|
|
- **Action** is the one step that would resolve the item for the affected target, and only when that step is legal right now. A paused or failed node gets no action rather than one that cannot run.
|
|
|
|
## When Sencho will not repair it
|
|
|
|
A Blueprint whose drift mode is **Enforce** can restore runtime drift on its own, but only the state the target was already authorized to run: the generation that target acknowledged and the image identity approved for it. A newer accepted generation never changes what an older target restores, because the repair answers "put this node back" rather than "deploy the newest thing".
|
|
|
|
Holding is an **Enforce** decision, made fresh each tick. Observe and Suggest still detect and report the same drift, because "auto-fix was declined" is not an answer for someone who never asked for one. And a hold never suppresses detection: Sencho still inspects the node and still records what is running, it only declines to overwrite it.
|
|
|
|
Two things a held row does not tell you. Its reason is the hold, not the drift, so read the Drift panel for what is actually running. And a row held under Enforce keeps reading **Repair held** if you switch to Observe or Suggest while the drift cannot be classified, because the next check has nothing to record until the cause clears. The deployment row reads **Repair held** with the reason, the attention queue names it, and the workload is left untouched. A held target is still drifted: it is never counted as up to date.
|
|
|
|
The reasons a repair is held are the states this model already treats as uncertainty rather than divergence: a stateful or unclassifiable Blueprint, a rollout that has moved on, on-node evidence that does not name a generation, an unreachable target or stale connectivity, an interrupted change, a recovery or rollback that owns the target, and a disagreement between the rollout's frozen pair and the target's own expectation.
|
|
|
|
Most holds clear themselves. Sencho keeps re-checking the target, so once the cause is resolved, by a node returning or an interrupted change finishing, the next reconciliation repairs the drift normally.
|
|
|
|
One does not: a hold because the rollout moved on stays until a later rollout includes that target. Superseding is a one-way fact, so nothing reopens that rollout on its own, and Git-managed Blueprints have no manual redeploy to fall back on, so the rollout is the only way out.
|
|
|
|
## Where drift is shown
|
|
|
|
| Surface | What it shows |
|
|
|---|---|
|
|
| **Drift** tab on a stack | The spatial findings plus the GitOps state for that stack, one card per node, with the per-service digest comparison under each node. |
|
|
| **GitOps application view** in the portfolio | The same drift items for one application, alongside its source, artifact, placement, and rollout state. |
|
|
| **Portfolio application table** | A count and the classes involved, so you can narrow the list to one class. |
|
|
|
|
The same projection feeds all three, so a class never reads differently depending on where you look.
|
|
|
|
## Per-service digests
|
|
|
|
When the approved artifact set and the node's observation both carry per-service image evidence, each target's card shows the approved digest beside the running one, per service, and marks each one matched, drifted, or unverified. The comparison uses the child digest for the platform that node runs, because a multi-platform image has one child per platform and a node only ever proves its own.
|
|
|
|
Where evidence is missing, the card says so. It never renders an empty comparison, because an empty table reads as agreement.
|
|
|
|
<Note>
|
|
A digest is compared, never pulled. When Sencho repairs a digest divergence it redeploys against the approved identity, so authored tags on disk stay as you wrote them.
|
|
</Note>
|
|
|
|
## What is not drift
|
|
|
|
Work in progress is not drift, and the classes above stay silent while it happens:
|
|
|
|
- a commit that has been fetched but not yet accepted, including one waiting on review
|
|
- a placement or authorization decision that is still open
|
|
- a rollout that is queued, canarying, batching, or waiting on its health verdict
|
|
- a manifest being written as part of an apply that has not landed on the node yet
|
|
- a node that is deploying, interrupted, or recovering
|
|
- a drift that Enforce is repairing on its current pass, which is why the Blueprint deployment row reads as correcting rather than drifted
|
|
- a target that has not been deployed yet, or has never had one recorded
|
|
|
|
Those states have their own surfaces, and each one names the next step. A node running a different generation than the one it was asked for is a separate class of its own, and is reported whether or not something else about that node is in flight.
|
|
|
|
One more case sits here by decision rather than by accident. Source drift compares the accepted commit against the commit the ref names. When those still agree but the materialization settings changed, for example a compose path, an env path, or a context directory, there is no expected-versus-observed pair of commits to report, so no source drift item is created. Instead the source shows as **reconcile required** in the attention queue, and for a Direct application that is where you fetch again: the fetch rebuilds the candidate under the settings now in force, and accepting that candidate produces the new generation. This is a choice about what the word drift means, not a claim that the materialized content was proven unchanged. Changing where Compose is read from can change the content, which is why Sencho asks rather than assumes.
|
|
|
|
## Invocation
|
|
|
|
Invocation drift answers one question: is this stack being brought up the way this generation said it would be?
|
|
|
|
When a generation is applied, Sencho reads back how Compose was actually invoked on the node: the compose files in order, the project name, the project directory, and the env files in order. That record is compared against the invocation the accepted generation was applied with. It is the only evidence in the model that can contradict the authored invocation at all, because reading the command back would only confirm what Sencho already believed.
|
|
|
|
What it catches in practice is a project that came up differently from the way the generation named it. The common case is a stack directory holding a compose file the generation did not name: Compose resolves it, the node records more files than were asked for, and the difference is reported. A node that answers with a different project name, project directory, or file order is reported the same way.
|
|
|
|
Two limits worth knowing, both about when the evidence is taken:
|
|
|
|
- **The record is read after the apply finishes.** A stack brought up by hand *before* an apply is overwritten by that apply, so the apply reads back its own work. A hand-made invocation that the apply replaced is not what you see here.
|
|
- **It is a snapshot, not a watch.** Nothing re-reads it on a timer. An invocation changed on the node after the apply is not reported until the next apply records a new one. Every item carries the time its evidence was taken, so you can see how old it is.
|
|
|
|
A stack whose invocation has not been recorded yet reports nothing and says why, rather than showing you an empty comparison that would read as agreement. The first apply records it and the caveat clears.
|
|
|
|
## What Sencho cannot prove
|
|
|
|
Drift is derived from what Sencho has recorded, so missing evidence is never rounded up to agreement. A few cases worth knowing:
|
|
|
|
- A target on a node Sencho does not host has no health verdict here, so health drift stays silent for it rather than claiming a verdict nobody wrote.
|
|
- A Git-managed Blueprint deployment that opens no generation-bound health gate produces no health item.
|
|
- A required-target set Sencho cannot read records a caveat naming the unreadable record, instead of quietly reporting no placement drift.
|
|
- A stack whose invocation has not been recorded yet records a caveat saying so, rather than reporting no invocation drift.
|
|
- A stack whose generation names no compose file records a caveat, rather than reporting a difference against a file list that was never written.
|
|
- A node that was withdrawn while a recovery on it had failed records a caveat, because the withdrawal ended what Sencho tracks there and the failed recovery is no longer reported as a state.
|
|
- A Blueprint deployment is not compared at all. Each of its targets is brought up from the Blueprint rather than from a recorded compose invocation, so there is no expected invocation to compare against, and the class stays silent for them rather than reporting every one as different.
|
|
- A service built on the node has no published digest to compare, so the per-service view labels it **not compared** rather than matched or drifted.
|
|
- A service running something the approved set does not name is outside the comparison, so it reads **not compared** rather than drift.
|
|
|
|
Each of those is listed as a caveat under the state it applies to, on the Drift tab and in the application view, saying what could not be proven. The per-service digest view says the same in its own words: a target with no per-service evidence on either side says nothing rather than showing an empty comparison that would read as agreement.
|
|
|
|
## The policy domains
|
|
|
|
A drift class reports; a policy decides. Each decision below is independent, and none of them authorizes another. Three of them answer the same question at three points in the chain: who is allowed to decide. The other two answer what happens after something has already gone wrong or already drifted.
|
|
|
|
| Policy | Where it lives | What it governs |
|
|
|---|---|---|
|
|
| **Source policy** | Per Git source: Manual, Review only, Auto-write files, or Auto-deploy. | How far Sencho goes on its own after a fetch. Manual takes the source out of the cadence entirely, so nothing happens until you ask for a pull. Review only records the result and waits for you. A withdrawn stateful target always waits for you, whatever the source policy says. |
|
|
| **Placement policy** | Per application, beside the placement approval. | Which placement changes are approved without you. **Operator approves each change** is the only option unless you choose **One stateless change at a time**, and that option approves exactly one stateless addition or one stateless removal, falling back to you with a stated reason for everything else. |
|
|
| **Rollout authorization policy** | Per application, in the rollout overflow. | Which rollouts are authorized without you. Nothing about placement is decided by it: a rollout cannot start before its placement is approved, whichever way this is set. |
|
|
| **Drift mode** | Per Blueprint: Observe, Suggest, or Enforce. | What happens when the Blueprint's own drift is detected: surface it, notify you, or redeploy it. |
|
|
| **Health rollout policy** | Per application, beside the two above. | Whether a deploy must prove healthy before it is allowed to finish, and what a failure does next: observe, pause, retry once, stop, or roll back. |
|
|
|
|
Two rules keep them honest. A digest repair redeploys against the approved identity rather than pulling whatever the tag now points at, and a stateful change always re-enters operator confirmation, whatever the drift mode says.
|
|
|
|
Every rollout freezes the policy snapshot it executes, and every placement approval carries the snapshot that decided it. That is why a policy edit changes what the next decision may do and nothing else: no standing approval is withdrawn, no generation is invalidated, and the card for work already in flight says which value it was actually decided under rather than the one configured now.
|
|
|
|
The three "who decides" policies are recorded together as one snapshot, so no reader can reach a single value that appears to govern more than one of them.
|
|
|
|
### What a new application starts on
|
|
|
|
A new or newly adopted application waits for an operator at every step. That is
|
|
the safe default for something with no history: nothing about it has yet proved
|
|
that a node can take a generation, and the placement policy starts on operator
|
|
review for the same reason.
|
|
|
|
An application that already existed before these settings were introduced keeps
|
|
what it was doing, and only that. If you adopt a Blueprint after the fact, or
|
|
convert one twice, it starts on operator review rather than inheriting the
|
|
behavior of applications that were around earlier. The setting is beside the
|
|
rollout authorization action it governs, so turning it on is one click rather
|
|
than a request.
|
|
|
|
## Related
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Drift Detection" icon="magnifying-glass" href="/features/stack-drift">
|
|
The spatial layer: effective Compose model against the containers running now.
|
|
</Card>
|
|
<Card title="Git Sources" icon="git-branch" href="/features/git-sources">
|
|
Where the source policy, commits, and acceptance live.
|
|
</Card>
|
|
<Card title="Blueprints" icon="layers" href="/features/blueprint-model">
|
|
Who decides each step, drift modes, and digest drift on a fleet template.
|
|
</Card>
|
|
<Card title="GitOps portfolio" icon="chart-pie" href="/features/gitops-portfolio">
|
|
Every application in one table, with drift by class and the attention queue.
|
|
</Card>
|
|
</CardGroup>
|