mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-26 13:40:26 +00:00
Anso
91648b049d
* fix(gitops): keep Stack Drift, notifications, and metrics on the canonical posture Three surfaces consumed GitOps application status without going through the posture read model, so each could contradict the portfolio about the same application. The portfolio, the application detail panel, and the Blueprint surfaces were already on it; these three were not. Stack Drift rendered only the node-local revision, which has no posture and no evidence freshness. Every facet card could read satisfied with no drift listed while the portfolio reported the application unsettled, because the evidence was stale, a node had gone dark, or a target was unreachable. The tab now reads the hub-owned portfolio row by application id and renders the posture, the evidence notices, and per-target evidence freshness, and each drift item shows the age of the observation behind it. The read is keyed on the application id the projection already carries rather than on a stack name, because a Blueprint application has no stack name of its own and two applications can share a deploy stack. A settled source attempt whose outcome is unproven notified as a pull failure at error level, while the portfolio reported the same application as in-progress. Unproven is now its own category at info level, saying the attempt settled without proving a result. Genuine failures are unchanged. A recorded health-check failure produced no notification at all, because the finalize stage was not notifiable, so a known-bad workload left the portfolio reporting a failure with nothing behind it in the bell. A recorded failure now notifies, and only a recorded failure: a passing check, a check that could not reach a verdict, a verdict that could not be tied to the running stack, and a failure recorded against a generation a rollout has already superseded are all silent. The health transition now records the verdict it actually persisted, which is what the notification keys on, so the bell and the projection cannot disagree about which failures count. Metrics was audited rather than changed. Its counters key only on transition stage and outcome, both closed unions carrying no application identity, so no count derived from them can be read as a claim about an application. That is now pinned by a compile-time exhaustiveness assertion and a behavioural test, because a settled word reaching that union is the only way it could start lying. * fix(gitops): stop the posture consumers contradicting the portfolio Follows review of the first attempt at this change. Four defects, three of them introduced by that attempt and one pre-existing. The health notification was a duplicate. The only producer of a recorded GitOps health verdict is the health gate, which announces the failure as it records it, so a failed run produced two bell entries saying the same thing. The GitOps outbox now stays out of it, and a test pins that a recorded failure yields exactly one notification and names where it comes from. The unproven reconcile outcome now does not notify at all, rather than notifying at a lower level. A settled attempt that proved nothing is evidence the portfolio reports as in progress or unknown, and the criterion is that the bell must not report on such evidence. What is unproven is the portfolio's to show, not the bell's, so the row is suppressed at the point the outbox decision is made. The insert and the drain carry the same transition record, so the two cannot disagree about which rows exist. The posture card warned falsely in two cases. A detached stack still projects a revision, so an identity was built for it and the portfolio route answered 404, producing a permanent warning about an application the portfolio does not list. A Blueprint stack read by a caller without the fleet-wide node grant answered 403 and produced the same permanent warning for a stack that caller is entitled to read. Both now say nothing: the route answers 404 for missing and not-granted alike on purpose, so neither status can honestly be reported as a fault, and the tab keeps the node-local view it always had. The unreadable warning is now reserved for the one case that is a real fault, a permitted read the server could not answer. The metrics invariant test claimed to be an exhaustiveness tripwire while only checking that its listed values were allowed. It now also asserts no unlisted member exists, verified by adding a value and watching it fail. Tests for the new hook cover the answers per outcome, the hub-only read, a slow answer for a previous identity being discarded, and the announced refetch with its debounce. Backend 1025 GitOps tests, 479 neighbouring tests, and 3506 frontend tests pass; both typechecks, both builds, and both lints are clean. * fix(gitops): scope the posture read and drop the report that no longer fits Follows the non-blocking findings on the posture work. Six items; four changed code, two are recorded as needing no change. The change announcement names the application it came from, so the posture read filters on it rather than treating every GitOps event in the fleet as this application changing. That was wasted work and a card that flickered on activity it had nothing to do with. An announcement that names no application is not filtered, because it cannot be attributed and dropping it could miss a change to this very application. A node change left the previous node's report in state for the length of the new request, so the new node was composed with the previous node's application and one identity that cannot exist was probed. Clearing the report was not sufficient on its own: the render that follows the node change already sees the old report, because the effect that clears it runs after that render. The report now records the node it was read from, and the posture read is skipped unless that is the node now selected. A test drives the switch and asserts the mismatched request is never sent. Work in flight and cannot-prove shared a glyph. They are the two postures most easily read as one state from a distance, and they are exactly the two that must not look alike on the surface an operator opens to find out which it is, so each tone now resolves its own icon. Boot repair drains rows an earlier process inserted, so a build predating the unproven suppression can leave one behind. The repair now declines it too, rather than announcing a failed pull the current insert would refuse to write: the one path meant to reproduce the original outcome must not be the path that diverges from it. The row is still drained, and the suppression is logged so a silent entry is a recorded decision rather than something a reader later has to explain. The health gate being disabled mid-run, which leaves a failed verdict announced while the projection reports health as not applicable, needs no change here. The entry it describes is the health gate's own, and the GitOps notification that would have doubled it is gone. The reclassification of a stored row is no longer intended but is no longer reachable either, since the category it was reclassified into does not exist.
223 lines
18 KiB
Plaintext
223 lines
18 KiB
Plaintext
---
|
|
title: Drift Detection
|
|
description: See at a glance whether a stack's running containers still match the effective Compose model, with specific, actionable reasons when they have diverged.
|
|
---
|
|
|
|
The **Drift** tab answers the core day-two operations question: does what is actually running still match what Compose would deploy?
|
|
|
|
Sencho treats your Compose file as the source of truth. Every time you open the Drift tab, it renders the effective model (variable substitution, multi-file merge, active profiles) and compares the live Docker runtime against that result. If you have deployed through Sencho, it also compares the current file against the baseline it recorded at deploy time, so you know whether the file has already changed for the next deploy.
|
|
|
|
The check is always read-only. It reports drift; it never fixes it on its own.
|
|
|
|
<img
|
|
src="/images/stack-drift/drift-in-sync.png"
|
|
alt="The Drift tab showing In sync status with Matches last deploy below it and a resolved network finding in the history"
|
|
/>
|
|
|
|
## How it works
|
|
|
|
Drift detection is split into two layers that work together.
|
|
|
|
**The spatial engine** performs a stateless diff between the effective Compose model and the containers currently running in Docker, then returns a report. It never writes anything to the database, so opening the tab is always side-effect-free.
|
|
|
|
**The drift ledger** sits on top of the spatial engine and provides persistence. When you click re-check, or when Sencho automatically reconciles after a deploy, it records new findings with timestamps, marks resolved findings as cleared, and writes events to the stack's Activity timeline. This turns a point-in-time snapshot into a short history of when drift appeared and when it cleared.
|
|
|
|
## Status
|
|
|
|
The status badge at the top of the tab summarizes the comparison between the effective Compose model and the running containers.
|
|
|
|
| Status | Meaning |
|
|
|--------|---------|
|
|
| **In sync** | Declared services match runtime: same running services, images, and published ports, or a declared one-shot finished cleanly (`Exited (0)` with explicit `restart: "no"` or `deploy.restart_policy.condition: none`). Omitting `restart:` does not count as a one-shot. |
|
|
| **Not running** | The stack has no running containers, and no declared service is satisfied by a finished one-shot. The file is present but nothing is up. |
|
|
| **Drifted** | Something differs from the model. Specific findings are listed below the badge. |
|
|
| **Unreachable** | Docker could not be reached on the active node, so drift cannot be assessed. |
|
|
|
|
## Since your last deploy
|
|
|
|
Below the status badge, a second signal compares the Compose file on disk against the version that was deployed through Sencho most recently.
|
|
|
|
<img
|
|
src="/images/stack-drift/drift-drifted.png"
|
|
alt="The Drift tab showing Drifted status with Source changed below it, and an image finding in the Findings section showing expected vs running image"
|
|
/>
|
|
|
|
| Signal | Meaning |
|
|
|--------|---------|
|
|
| **Matches last deploy** | The Compose file is unchanged since the last deploy. |
|
|
| **Source changed** | The Compose file has changed since the last deploy. If the change is structural (an image, port, or service definition), Sencho says so. A formatting-only or comment-only edit is noted separately. |
|
|
| **No deploy baseline** | This stack has not yet been deployed through Sencho, so there is no recorded baseline. Deploy it once to start tracking. |
|
|
|
|
This signal is independent of the runtime status. A stack can be **In sync** with its running containers while its file has **already changed** for the next deploy, or it can be **drifted** at runtime while the file itself is unchanged from what was deployed.
|
|
|
|
<Note>
|
|
Sencho records two hashes at deploy time: a raw-file hash (catches any text change, including formatting) and a parsed-model hash (only changes when structure changes, ignoring comments and whitespace). The "source changed" signal uses both: if only the raw hash differs, the edit was formatting or comments. If the model hash also differs, a structural change has been made.
|
|
</Note>
|
|
|
|
## GitOps state
|
|
|
|
For a stack managed by a Git source or a Blueprint, a **gitops** section sits below the deploy signal. Its first line is the application's overall state, then the per-domain states. For a Git-backed stack the source state uses the same phrase the Git Source panel shows; a Blueprint-delivered stack has no Git source of its own, so it shows only its per-node lines. Those say what Sencho has established about the stack on each node: whether a generation has been applied, whether it is deployed, whether its health verdict is still outstanding, and whether an operation was interrupted without a confirmed outcome.
|
|
|
|
The first line, labelled **application**, is the one-line summary the GitOps workplace shows for the same application, and it is the same value in both places. It reads **converged** only when every target was reached and complete, current evidence confirms the intended generation is running. Every other value says Sencho cannot currently claim convergence; **unknown** in particular says nothing more than that, rather than naming a fault.
|
|
|
|
This is a different question from the two signals above it. Those compare files and containers as they are right now. The gitops section reports what Sencho has proven and accepted over time, so a stack can be **In sync** at runtime while its application state reads **unknown**, meaning the containers look right but Sencho cannot prove that every target reported for the generation now intended.
|
|
|
|
Once that summary has loaded, each per-node line also carries an **evidence** note saying how much Sencho could read from that node:
|
|
|
|
| Evidence | What it means |
|
|
| --- | --- |
|
|
| **evidence current** | The node answered with its own runtime observation. |
|
|
| **evidence stale** | The node has an observation on record, but it is known to be out of date. |
|
|
| **evidence unknown** | Sencho could not read the node's runtime observation, so it cannot say what is running there. |
|
|
|
|
The note describes the evidence, not the application. Whether the intended generation is actually deployed is the per-node runtime state above it, and whether that is enough to settle the application is the summary.
|
|
|
|
Anything Sencho could not prove is listed underneath, one line per item, in the same form the Git Source panel uses. A stack with no Git source and no Blueprint shows no gitops section at all. If Sencho can see the application but cannot read its summary, the section says so in place of the summary rather than omitting it, so a moment of lost visibility is never mistaken for a healthy application.
|
|
|
|
## Findings
|
|
|
|
When a stack is drifted, each reason is listed in the **Findings** section against the service it affects.
|
|
|
|
| Finding | What it means |
|
|
|---------|---------------|
|
|
| **Image** | A running container uses a different image than the Compose file declares. Expected and actual values are shown side by side. |
|
|
| **Ports** | The published ports of a service differ from what the Compose file declares. |
|
|
| **Service missing** | The Compose file declares a service, but no running container matches it and no clean one-shot completion satisfies it. |
|
|
| **Undeclared service** | A container is running for the stack but no matching service exists in the Compose file. |
|
|
| **Network undeclared** | A running service is attached to a network that is not declared in the Compose file. |
|
|
| **Network missing** | A network declared in the Compose file is not used by any running service or is absent from the runtime. Network usage still counts only running or restarting containers, so a finished one-shot does not keep a dedicated Compose network marked as in use. |
|
|
|
|
### Image comparison
|
|
|
|
Drift compares running images against the **effective** compose model: the same resolved result Docker produces at deploy time, including variable substitution (for example `${VAR:-default}`), multi-file merge, and active profiles.
|
|
|
|
Image references are normalized before comparison, so equivalent forms do not produce false positives. `nginx`, `docker.io/nginx`, and `docker.io/library/nginx:latest` all resolve to the same reference.
|
|
|
|
If a service has multiple replicas, any replica running a different image than the declared one triggers an image finding. This catches stacks that are mid-update with mixed versions running simultaneously.
|
|
|
|
If the Compose file declares a tag (`:latest`, `:1.25`) and the running container was pulled from a digest pin, the two forms are compared as-is. A digest-pinned container reads as an image mismatch against a tag declaration. This is intentional: the two references are not equivalent. Sencho uses the same distinction in scan history and update detection; see [Tags vs digests](/features/vulnerability-scanning#tags-vs-digests).
|
|
|
|
### Port ranges
|
|
|
|
A Compose port range such as `8000-8002:8000-8002` is compared conservatively. Because the runtime expands ranges into individual ports, the comparison can read as a difference even when the deployment is correct. Sencho errs toward surfacing the possible difference rather than hiding it.
|
|
|
|
### Network findings and the Networking tab
|
|
|
|
The **network-undeclared** and **network-missing** findings reuse the same comparison the stack's [Networking](/features/compose-networking) tab uses, so the two surfaces never disagree about a network attachment. Sencho-verified `sencho_mesh` attachments for its own container and Mesh-opted-in stacks are excluded, while a manual attachment from an opted-out stack remains actionable. The difference is history: this tab persists findings in the drift ledger over time, while the Networking tab always shows the current live state with no history. A stack with an open network finding also counts toward the **Drift** chip in the Fleet Overview's Networking filter group, so a network-attachment mismatch is visible both per-stack here and across the fleet there.
|
|
|
|
## When drift is recorded
|
|
|
|
**Opening the tab** runs the spatial engine and displays the current state. It does not write anything to the ledger or update the history.
|
|
|
|
**Clicking re-check** runs the spatial engine and then reconciles the result into the ledger: new findings are recorded with a detection timestamp, cleared findings are marked resolved, and Activity timeline events are written for any change. The history and "checked" timestamp update after a re-check.
|
|
|
|
**After every deploy or update**, Sencho automatically records a new baseline hash and runs a full reconciliation. You do not need to click re-check after deploying; the ledger is updated as part of the deploy pipeline.
|
|
|
|
An open ledger entry for an attachment that is no longer reported clears during the next re-check or post-deploy reconciliation. Opening the tab refreshes the live report but does not change persisted history.
|
|
|
|
<img
|
|
src="/images/stack-drift/drift-history.png"
|
|
alt="The Drift tab showing both the Findings section and the Drift history section with an open finding marked just now"
|
|
/>
|
|
|
|
## Drift history
|
|
|
|
The **Drift history** section shows up to the 20 most recent findings, both open and resolved. Open findings appear first.
|
|
|
|
Each entry shows:
|
|
- The service name and finding type (image, ports, service, or network)
|
|
- A description of the specific difference
|
|
- When the finding was first detected
|
|
- When it was resolved, if it has cleared
|
|
|
|
The history header shows when the ledger was last reconciled ("checked X ago"). The status badge at the top always reflects the current runtime state, computed fresh each time the tab is loaded. If the two disagree, a re-check will bring the history up to date.
|
|
|
|
### Activity timeline
|
|
|
|
Drift events also appear in the stack's **Activity** tab alongside deploys, restarts, and updates:
|
|
|
|
- **Drift detected**: recorded when a re-check or post-deploy reconciliation finds new open findings
|
|
- **Drift resolved**: recorded when findings that were previously open have all cleared
|
|
|
|
## Accessing the Drift tab
|
|
|
|
<img
|
|
src="/images/stack-drift/drift-tab-location.png"
|
|
alt="The tab bar in the stack detail view showing Anatomy, Activity, Drift, Dossier, Environment, and Compose Labels tabs with Drift selected, and separate Files and Edit compose buttons to the right"
|
|
/>
|
|
|
|
1. Click any stack in the sidebar to open it.
|
|
2. In the right-hand panel, switch to the **Drift** tab. If the tab row doesn't fit the panel width, scroll it to reveal Drift alongside the stack's other tabs (Anatomy, Activity, Doctor, Dossier, Environment, Networking, and so on).
|
|
3. Read the status badge and any findings.
|
|
4. Click **re-check** after editing the Compose file or after a manual Docker operation to refresh the ledger.
|
|
|
|
**Files** and **Edit compose** are separate buttons next to the tab row, not tabs themselves; they open the file browser and the compose editor respectively.
|
|
|
|
On mobile, the same report appears under the **Compose** segment of the stack detail view.
|
|
|
|
## Requirements
|
|
|
|
Drift detection is available on all plans with no additional configuration. As long as the stack has a Compose file and Docker is reachable on the node, the tab is functional.
|
|
|
|
Viewing the tab requires read access to the stack; any role that can see a stack (including read-only **viewer** and **auditor** roles) can open the Drift tab. Clicking **re-check** requires the same read-level access, so a read-only role can still trigger a ledger write (new findings recorded, resolved findings cleared, an Activity event logged) even though it cannot edit or deploy the stack.
|
|
|
|
## Limitations
|
|
|
|
**Read-only and advisory.** Drift detection never alters a stack. It reports differences and records them in the ledger, but deploying to resolve drift is a separate, explicit action.
|
|
|
|
**No automatic background scanning.** Drift is checked when you open the tab, when you click re-check, and after every deploy or update through Sencho. There is no automatic background poll that checks stacks on a schedule outside of those events.
|
|
|
|
**History cap.** The ledger keeps the 20 most recent findings per stack. Older entries are not shown in the UI.
|
|
|
|
**Port ranges.** Compose port ranges are compared conservatively and may report a difference even when the deployment is correct.
|
|
|
|
**Unreachable Docker.** When Docker cannot be reached, the status shows as Unreachable and no findings are reported. Open findings in the ledger are not resolved just because the check could not run; they remain open until a successful check confirms they have cleared.
|
|
|
|
**Parse errors.** If the Compose file cannot be parsed, no findings are extracted from it and the ledger is not updated. Resolve the parse error and re-check.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="A stack I deliberately stopped shows as 'Not running'">
|
|
That is expected for long-running services. Drift compares the file on disk against what is actually running, so a stack with no running containers and no finished one-shot jobs reports **Not running** even when you stopped it intentionally. Deploy it to return it to **In sync**. A stack made only of one-shot jobs that already exited `0` with explicit `restart: "no"` (or `deploy.restart_policy.condition: none`) can show **In sync** (or **Drifted** for other findings such as unused networks) with no containers currently running.
|
|
</Accordion>
|
|
<Accordion title="The image finding flags a stack I just updated">
|
|
Click **re-check** after the deploy finishes. During a rolling update, replicas can briefly run different images, and the report captures that moment. Once every container is on the declared image, the finding clears.
|
|
</Accordion>
|
|
<Accordion title="A stack that uses a published port range shows a ports finding">
|
|
A Compose port range such as `8000-8002:8000-8002` is compared conservatively and can read as a ports difference even when the deployment is correct. Sencho errs toward surfacing a possible difference rather than hiding one.
|
|
</Accordion>
|
|
<Accordion title="The status says 'Unreachable'">
|
|
Sencho could not reach Docker on the active node, so it cannot compare the runtime. Confirm the Docker engine is running and the node is online, then use **re-check**. Other stacks on the same node will show the same state until Docker responds. While Docker is unreachable, Sencho does not update the drift history, so open findings are never cleared just because the check could not run.
|
|
</Accordion>
|
|
<Accordion title="The tab says 'No deploy baseline'">
|
|
Sencho records a baseline the first time you deploy a stack through it. A stack that was imported or has not yet been deployed from Sencho has nothing to compare against. Deploy it once from Sencho and the signal changes to **Matches last deploy**.
|
|
</Accordion>
|
|
<Accordion title="The Activity tab shows a drift event but the Drift tab now shows In sync">
|
|
The Activity event was written when drift was detected or resolved during a previous reconciliation. The Drift tab always shows the current live state, which may already be back in sync. If the history and badge disagree, click **re-check** to reconcile the ledger with the current runtime.
|
|
</Accordion>
|
|
<Accordion title="Opening the tab does not update the history">
|
|
Opening the tab runs the spatial engine (read-only) and updates the status badge, but does not write to the ledger. Only **re-check** and post-deploy reconciliation persist findings and update the history. Click re-check explicitly when you want the history to reflect the current state.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Related
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Stack Activity" icon="clock-rotate-left" href="/features/stack-activity">
|
|
Drift detected and resolved events appear in the Activity timeline alongside deploys and restarts.
|
|
</Card>
|
|
<Card title="Stack Dossier" icon="book-open" href="/features/stack-dossier">
|
|
The Dossier tab stores documentation and access URLs; its documentation drift feature checks whether URLs still match the stack's published ports.
|
|
</Card>
|
|
<Card title="Atomic Deployments" icon="rocket" href="/features/atomic-deployments">
|
|
Understanding how Sencho deploys stacks explains when and how drift baselines are recorded.
|
|
</Card>
|
|
<Card title="Blueprint Model" icon="drafting-compass" href="/features/blueprint-model">
|
|
Blueprints have a separate policy-based drift mode (observe, suggest, enforce) distinct from the runtime drift detection described on this page.
|
|
</Card>
|
|
<Card title="Compose Networking" icon="diagram-project" href="/features/compose-networking">
|
|
The Networking tab's runtime drift section uses the same network comparison as the findings on this page, without the persisted history.
|
|
</Card>
|
|
</CardGroup>
|