mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-26 05:30:18 +00:00
docs/tutorials-batch-3
2180 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Anso
|
84f335097c |
refactor(dashboard): give the Home stack health table's GitOps state its own column (#2068)
The GitOps source-state tag (accepted, review required, pending update, and so on) sat beside the stack name and crowded it. It now has a GITOPS column after SOURCE, the name cell keeps just the name and the update icon, and clicking the tag still opens that stack in the GitOps portfolio. The column only appears while at least one stack in the table has a state the tag can show, so an instance with no Git-managed stacks keeps the table as it was. Stacks without a state read "--" in it, and a state this build cannot name no longer leaves an empty clickable button behind. |
||
|
Anso
|
e34948bb64 |
fix(gitops): announce local Git source changes on the channel GitOps views listen to (#2067)
Retry and Resume from the attention queue, and changes made in the Git source sheet over GitOps, raised the refresh signal as a bare event. The portfolio, the application sheet and the dashboard's Git source states only react to a signal tagged as a gitops change (and the dashboard also needs the node), so those views refreshed only if the server happened to push its own announcement. If it did not, the queue stayed stale after an action. Both places now announce through one helper that tags the change as gitops on the application's node, the same shape the server announces. |
||
|
Anso
|
67889e168f |
refactor(gitops): put the workplace masthead on the shared hero and drop the duplicate attention column (#2066)
The GitOps masthead is now the shared hero masthead with a GITOPS kicker. Its meta line carries how many nodes are reporting and when the data was updated, and reads "refreshing" while a refresh is in flight, so the floating refresh pill over the page is gone. A failed refresh keeps the last-known data with the same qualifier as before. Attention is one of the masthead counts. The application table no longer repeats attention in its own column: the attention list above it, the row tint and the state dot already carry it. The dot now has an accessible name (and a tooltip) giving the most urgent reason, so a row beyond the list's cap is still explained and screen readers get the reason too. Facet chips read the shared tone classes, and the portfolio load-error Retry is a regular button. |
||
|
Anso
|
2d89f13ef2 |
feat(gitops): resolve attention in place (#2065)
The attention queue now lists each application once, with its most urgent reason inline and a count of the rest, and every entry carries the step that resolves it where the operator already is. Retry and Resume on a Direct source run from the queue on the application's own node; Review update opens that stack's Git source sheet already pulled to its diff, so applying an update is two clicks from the GitOps page. A step the session may not run, or the server does not offer, opens the Git source sheet instead of a button that would be refused. The full-page application view is replaced on desktop by a sheet over the portfolio list, which stays mounted underneath, so closing it lands back on the same filters and page. A Direct application is its stack's Git source, so a row or a link that names one opens that sheet and never a second read-only copy. Blueprint applications keep their authority and rollout controls in their own sheet. The whole table row is now clickable; the actions menu is unchanged. The Retry, Resume and Suspend request now lives in one shared handler used by the sheet and the queue, and names the stack in its toasts. The phone screen keeps its application page. |
||
|
Anso
|
ee62a449b2 |
refactor(gitops): organize the Git source sheet into tabs and make editing deliberate (#2064)
The sheet was one long scroll with the repository form always open and a toolbar of five buttons. A linked source now has tabs: - Overview: the status block, with Review update, Retry, or Resume in it. - Source: a read-only summary of the configuration, an Edit button into the form (with Cancel), Adopt onto Blueprint, and the manifest. - Automation: the apply behavior, Suspend or Resume, and the provider hooks. - Secrets. - Drift, only while the application has drift. An unlinked stack gets no tabs, only the form. The toolbar keeps Pull now and Detach (named the same as its confirmation); Update appears only while something differs from what is stored, so it never sits there doing nothing. Reading and refreshing are more careful. A failed read now says so with a Retry instead of presenting an empty "link a source" form; a change that was saved but could not be re-read keeps the linked source on screen; a pull, apply, suspend, or resume refreshes the status without discarding a change that is still being made; and a slow answer for a stack the sheet has left can no longer overwrite the one it shows. Detaching empties the form instead of leaving the removed source prefilled. Docs, the Git sources page, and the two end-to-end specs that drive the sheet are updated. |
||
|
Anso
|
b689733447 |
refactor(gitops): one layered status across the Drift tab, Git source sheet, and application view (#2063)
* refactor(gitops): one layered status across the Drift tab, Git source sheet, and application view The GitOps state used to be a stack of tinted cards on five surfaces: a posture card, approval chips, a card per facet, a card per target, and a caveats box, restating the same state several times. Each surface now shows one status: a single answer (with its resolving button, such as Review update or Retry), a quiet row naming each stage that applies, and an Evidence control holding recorded authority, policy lines, targets, and anything Sencho could not prove. - Stages that say nothing about an application (unbound placement, an absent rollout) are left out of the row instead of printed as "n/a". - Evidence is never silently lost: the answer carries a marker for partial evidence, unreached nodes, and unproven items, and every attention reason is kept, the first in the answer and the rest in the evidence. - The Git source sheet's Status section drops from about 450px to about 170px, and its separate source-state and last-applied rows go; the header now says "never applied" until something is. - The application view no longer shows the posture twice. Adds a domain-free StatusPath primitive, a pure status model, and a GitOpsStatus composer; retires the facet-card and approval-chip components and keeps the policy lines. Docs, the tutorial, and the two affected screenshots are updated. * test(gitops): open the status evidence in the digest drift spec The per-service digests now sit under the Evidence control of the GitOps status, so the spec opens it before looking for them. |
||
|
Anso
|
bba9d30204 |
feat(gitops): open the Git source in place and animate Pull (#2062)
Opening a Direct application's Git source from the GitOps portfolio (attention queue, row menu, application header) now opens the Git Source sheet over the portfolio instead of switching to the stack editor. The sheet targets the application's own node, so reads, pulls, applies, provider hooks, repository secrets, and the manifest all go to that node even when another node is active. Changes made in the sheet refresh the portfolio and the open application view. Adopt onto Blueprint is offered only when that node is the hub, where adoption runs. Sheet toolbar actions gain a pending state: the button locks at once and, once the work outlasts the busy delay, its own icon spins. Pull now and Save use it, so the refresh circle turns while a pull runs. |
||
|
Anso
|
d2d2a5d347 |
fix(gitops): match Blueprint repo claims on repository identity (#2054)
* fix(gitops): match Blueprint repo claims on repository identity The guard that keeps one live Blueprint-mode application per repository compared configured_repo_url as an exact string. A repository has an unbounded number of valid spellings, so the HTTPS URL and the SSH URL for the same clone each bound their own Blueprint, with or without a .git suffix, with a trailing slash, and with host casing either way. Compare a canonical repository key instead: host lowercased, trailing slashes dropped, and one trailing .git stripped from the last path segment, with the path left case-sensitive because Git hosts treat it that way. The key is derived through the same RepoIdentity shape already stored per application, so the stored identity and a recomputed key agree by construction. Fail-closed. A URL naming no repository cannot be compared against anything, so it refuses rather than passes, which means one unidentifiable live claimant blocks every later conversion until it is retired, detached, or tombstoned. Ingress now refuses such a URL outright so the state cannot be created, and the refusal names the Blueprint that holds the claim so the operator can act on it. GitOpsTransitionError carries an optional code, because the binding service was telling two refusals apart by testing a message for the word "repo". * fix(gitops): drop the port from the repo claim key and align ingress Two ways a Blueprint could still bind a repository another Blueprint already claimed, both found reviewing the claim guard. RepoIdentity folds an SSH port into the host it serializes, so a forge serving HTTPS on 443 and SSH on 2222 gave the HTTPS and SSH spellings of one repository two different keys, and both converted. The port is now not part of the key. Two repositories on two ports of one host therefore collide, which costs a refusal rather than a second claim, which is the only direction this key is allowed to be wrong in. Ingress judged the submitted URL while the guard judges the stored one, and those can disagree. An scp-style URL carries `?` and `#` through verbatim, so it keyed fine as submitted while the ssh:// form actually persisted could not be parsed back at all, and a host holding a slash reparsed as a different repository. Those rows were accepted on the way in and refused at conversion for a URL the operator never sees. Ingress now requires the stored form to name the same repository the submitted one did, so a source is never accepted and then unclaimable. The transition fixture now stores the derived ssh:// form, which is what every production writer persists and what the guard reads, so it no longer asserts against a row shape the code cannot create. |
||
|
Anso
|
ae22a222ab |
perf(gitops): bound the Blueprint detail reachability probe with a short-lived verdict window (#2052)
* perf(gitops): bound the Blueprint detail reachability probe with a short-lived verdict window Opening a Blueprint application detail asks the application's remote target nodes whether they answered, so the detail panel agrees with the portfolio row instead of contradicting it. The panel is not polled on an interval, but it is read once per mount and again for every published GitOps transition while it is held open, the stack drift panel reads the same endpoint through a second hook, and the list read does its own fleet fan-out. A node that does not answer cost the full probe budget on each of those reads. A "did not answer" verdict is now reused per node id for a window strictly shorter than the probe budget, in the existing process cache. Only that direction is reused, because serving it can withhold a settled claim but never assert one, so a cached entry cannot report a node as reachable after it went dark. The portfolio fan-out records and retires the verdict and never serves it: it needs the rows themselves, and a cached verdict must not override a live answer. Both writers retire the entry on an answer, including when the other writer recorded dark while a leg was in flight, so the row and its own detail panel cannot disagree about a node the hub has just reached. The probe timeout, what a verdict means, the projection, and the response shape are unchanged. * docs(gitops): pin the reachability reuse window to the shared probe ceiling The reuse window was pinned to this read model's own probe budget, which left one drift open: raising that budget on its own would let a widened window still look compliant while the fleet overview and fleet readiness surfaces keep treating 3 s as the bound they assume. The test now also asserts the window against that shared ceiling as a literal, and the constant's docblock names the two budgets it mirrors. No behaviour change. The window is unchanged at 1500 ms. * fix(gitops): share one reachability probe per node and stop a stale dark write Opening a Blueprint application detail, the stack drift panel and the portfolio list all ask about the same node, and they overlap: a panel opened over the list probes twice in the same instant, and a held-open panel re-probes on every published GitOps transition. The verdict is only written once a probe settles, so every reader arriving during a probe started its own, and against a node that really does time out that is a full probe budget each time. The reuse window did nothing for the overlapping case, which is the case the reads that prompted it actually produce. One probe leg per node is now in flight at a time, so a reader arriving mid-probe joins it. The list fan-out joins the same leg, so opening a panel over the list no longer probes the same node twice. That sharing is also what orders the two writers. Only one leg per node exists, so a probe that saw a node answer settles and writes before the next probe of that node can start, and a dark verdict landing after a newer answer would need two overlapping probes of one node. The alternative, comparing probe timestamps, would have been state that can never fire. A thrown leg is now evidence in neither direction for both writers: it records no verdict and retires none, because a throw says this build could not read the node rather than that the node is down. Retirement moved to the moment a payload arrives, before the payload is walked, so a node that answered is never left reading unreachable because its payload could not be parsed. The probe timeout, the reuse window, what a verdict means, and the response shape are unchanged. * refactor(gitops): let the probe leg own its verdict write The reachability probe's leg now records the verdict instead of each reader recording it after its own await. The ordering guarantee no longer depends on both writers writing synchronously in the settling continuation, which was a rule a later edit could break silently: three readers sharing one leg recorded the same verdict three times, and a reader arriving late could re-arm the reuse window once per read rather than once per probe. `probeNodeOnce` also releases its slot in a finally, so a verdict write that ever throws cannot wedge every later read of that node on a settled promise, and the test-only reset attaches a rejection sink to the legs it drops rather than creating an unhandled rejection. A dropped leg cannot be cancelled, so one that settles after a reset still records its single verdict; flushing the cache after the reset clears that. Docs and comments follow the code: `probeSilentNodeIds` and the fan-out are readers now, and say so, so a later editor does not re-add reader-side writes. The shared-rows contract is stated with its fragility rather than as settled, and the rejection comment no longer reads as one log line per joiner. * test(gitops): make the shared-leg test order-independent and honest about what it shows The shared-leg filtering test depended on other tests having run first. It waited for a probe count derived from the whole node roster, so with only its own node the wait passed immediately, the second read started after the leg was released, and the test hung until the timeout. It now waits on a marker node the test creates itself, which says when a read has reached its fan-out regardless of what else is in the database. That test also only caught an in-place filter when the unrestricted read went first, because the restrictive read has nothing to remove and an admin sees everything. The viewer now goes first, so a filter that spliced the shared array would cost the admin a row the test asserts it must see. Verified: an in-place filter mutation now fails the test. The late-reader test is renamed to what it shows, which is that a settled leg frees its slot so a later probe can retire the verdict. It never tested that a reader writes nothing late, because under a leg-owned write there is no reader left to intercept; the per-probe spy test carries that guarantee alone, and says so. The duplicate overlap test, whose only distinction was one microtask wait, is gone. Comment fixes from the same review: a garbled sentence in the detail-probe docblock, a wrong claim that flushing after the test reset clears a dropped leg's late write, a worst-case example that named a node going dark rather than one recovering just after its probe started, and two-writer phrasing left in the test file header. Tests now build their cache key from the exported namespace rather than repeating the string. |
||
|
Anso
|
1426b79bb2 |
fix(sso): accept the issuer GitHub now returns on the OAuth callback (#2061)
GitHub began sending an RFC 9207 `iss` parameter on the authorization response (https://github.com/login/oauth). The GitHub provider was configured with issuer https://github.com, so the callback validation rejected every login with "unexpected iss (issuer) response parameter value" and users saw "Authentication failed. Please try again.". The settings test passed because it never exercises the callback. Set the issuer to the value GitHub advertises. Issuer validation stays on, so a callback carrying a different `iss` is still rejected, and a callback without `iss` keeps working. |
||
|
Anso
|
c0b5708dd7 |
fix(gitops): date a recorded placement refusal instead of presenting it as current (#2042)
A placement decision that refuses records its reason once, and the reason is cleared only when an operator approves the placement, edits the policy, or a later change re-decides. A decision runs only when a placement change is pending, so a refusal recorded because an operation was in flight outlives that operation and nothing corrects it on its own. The facet card rendered the reason in the present tense. An operator arriving after a deploy finished was told the system was withholding a placement because a conflict was in flight, and had to work out that the conflict was over. The record is kept and the moment it was made is shown. The timestamp was already stored and already projected as the decision's own date; the frontend carried the field and rendered nothing. Re-deciding instead would need a trigger that does not exist today, and would let an automatic approval land at a moment the operator is not watching, which is a larger claim than the evidence supports. Both tests fail if the behavior is reverted. The backend case drives the real transitions: an operation in flight, a refusal recorded for it, the operation finishing, and the projection still reporting the decline carrying the moment it was recorded rather than the moment it was read. Verified against a running instance at desktop and phone width: the card carries the date in both, and the only desktop delta is the one added line. |
||
|
Anso
|
1e1bea0af5 |
fix(gitops): read a redeployed failed generation as work in progress (#2045)
* fix(gitops): read a redeployed failed generation as work in progress A recorded health failure keeps being reported until a newer verdict lands, which is right, but nothing knew that a redeploy of that same generation was in flight. The attention reason carrying failure tone outranks the portfolio's in-flight rule, so redeploying a failed generation reported the application as failed for the whole duration of the redeploy, naming a settled state nobody was working on. The projection now carries whether a live operation is running against the generation a recorded failure is about, and the attention classifier withholds the failure reason when it is. The health facet still reports failed, which stays true: the last verdict did fail, and its promotion was withdrawn. The runtime facet already reported the same target as deploying, so the portfolio reads the application as work in progress instead, and a redeploy that fails again reports its own verdict. The generation match is the rule, and it is compared against the failure's own record rather than the running pointer, so a redeploy of a different generation cannot silence a failure that belongs to what is still running. The scope is a target whose operation recorded a generation, which only a Direct deploy writes, so a redeployed Blueprint target keeps reporting its failure as before. * fix(gitops): keep a redeployed failed generation in progress through its health window The in-progress reading ended at deploy-bind, which clears the live operation. The health gate reserves its run before the deploy and leaves it open while it observes, so the application dropped back to failed for the whole observation window: failed, in progress, failed, then the new verdict. An operator watching it, or a UI that polls, sees a flip-flop and can miss the progress entirely. A second moment now qualifies alongside the live operation: a stack health run for the same generation that is still observing. That reads a run that already exists and changes no gate behavior, window, or decision. The two arms cover different populations, which is why neither is redundant. A live operation on the failed generation needs an active_generation_id, which only a Direct deploy writes, so a Blueprint retry is visible only through its reserved run. The generation match is the failure's own record on both arms, so an observation of some other generation cannot supersede it either. Two corrections to how this was described. The automatic retry the health rollout policy performs dispatches through the Blueprint adapter, which refuses a Direct application, so the Direct deploy step covers an operator's own redeploy rather than the automatic one. And the mixed-version contract is now enforced rather than described: the remote boundary accepts only an absent field or a boolean, and the classifier reads it as true, so a peer cannot suppress a real failure by sending a truthy non-boolean. Test pins cover both arms independently, including the case that distinguishes judging an observation against the failure's generation from judging it against the running pointer, an observation of another generation, an observation that has already settled, one that names no generation, the Blueprint path, and a projection arriving with the field absent. * fix(gitops): stop the health supersede rule outliving the facts it rests on Four corrections to the rule that keeps a redeployed failed generation reading as work in progress. The flag was computed for every target, including ones whose own last verdict passed and ones a recovery is restoring. It now answers false from the row before any lookup, so a healthy target costs nothing and the flag cannot be true where the comment on it says it is meaningless. A Direct redeploy has a handover gap the previous change papered over. A Direct deploy binds the generation, then does its baseline, drift and exposure bookkeeping, and only then does the route open the observation, so for that window neither moment has anything to read and the recorded failure shows again. Blueprint rollouts reserve the run before the apply and never have that gap. The gap is now stated and pinned by a test rather than described as covered, because closing it would mean opening the observation before the bind, which reorders the deploy path. The catch around blueprintDeployStarted returned blocked without abandoning the run it had reserved, unlike its refusal, ack and outcome siblings. Left open, that row reads as observing for a deploy that never started, and the supersede rule would read it as this failure's successor. It is abandoned now, for the reason the ack catch already gives. Two claims were wrong. The mixed-version protection is the shape check at the remote boundary, not the comparison in the classifier: for the values that can reach it, a bare negation behaves identically, so the comments now say what actually holds. And a redeploy that settles unknown does restore the recorded failure, because the rule only ever stood in for a verdict that was coming, so that is now pinned rather than assumed. Posture is now asserted wherever the flag is, not just the reason, since the suppression only reaches in_progress because the runtime facet already reads as in flight. * test(gitops): pin the abandoned reservation when a deploy cannot be opened The abandon in the deploy-start catch was the only thing standing between a refused dispatch and a run row left observing for a deploy that never opened, and deleting it failed no test. That path now has one. A dispatch whose deploy-start transition throws leaves nothing pending, because the throw happens inside the transaction before the target row is written, so what is left behind is the run row itself. The test makes the transition throw, and pins that the reservation ends with a terminal status and an end time rather than reading as an open observation of the accepted generation. The comment above the catch said the run would make a later dispatch skip the target, which is the ack catch's mechanism and not this one. It now names the hazard that is actually here: an unarmed observing row that the health projection would read as work about to produce a replacement verdict. Also pins the tombstone guard, which was the other untested branch, and settles the run under the same id the verdict reports rather than a second one. |
||
|
Anso
|
50a9450abb |
refactor(fleet): flatten the Readiness tab chrome and start its check when Fleet opens (#2060)
The tab stacked its own title block and a hero summary card under the Fleet masthead. Replace both with one flat status row and put the findings table first. The readiness check now runs from the Fleet shell, so it is usually done by the time the tab is opened and survives tab switches. The skeleton only appears when the check is actually slow, and a failed or stale result is re-checked when the tab is reopened. |
||
|
Anso
|
8eaa954927 |
feat(updates): table view as the default for the update page (#2059)
The Update page now lists pending updates as a sortable table, one row per stack across nodes, with the node-grouped card board kept behind a Table / Cards switch. The choice is remembered per browser. Row details hold the changelog, blocked reason, and per-service Apply. |
||
|
Anso
|
3667651a2b |
refactor(fleet): trim node card badges to what matters (#2058)
Reachable nodes no longer carry an Online badge; Offline shows only when a node is unreachable. The local/remote type badge is gone, since the local card already has its pin. Remote cards now show a Proxy or Pilot marker in the same corner slot so the connection type is clear at a glance. The persistent Networking exposure badge is removed from the card. It fired on most nodes without detail; exposure and drift stay available in the node details sheet and the Networking filter. |
||
|
Anso
|
3f02e7f77c |
fix(gitops): name what a Blueprint acknowledgement actually means (#2037)
* fix(gitops): report a Blueprint generation divergence in the drift list The runtime facet and the canonical drift list are two readings of the same rows, and the drift list still required a populated `deployed_generation_id` to emit its generation-mismatch item. A Blueprint target has no deploy-bound writer, so that pointer is null by construction, and the item was unreachable for every Blueprint target. A superseded acknowledgement therefore showed in the runtime facet and contributed nothing to the list an operator's triage queue reads, which is the same disagreement this change exists to remove, reintroduced through the other surface. Both now resolve the per-mode running generation through the helper `deriveRuntime` already used, so the two cannot disagree about which generation a target is running. A staleness claim also now requires the application to have something to have moved on from. A target that recorded an intent against an application that never established one is running an unplaced generation, not a superseded one, and reads as the unconfirmed acknowledgement it is. * fix(gitops): correct the Blueprint acknowledgement statuses Five defects in the status derivation, all found by audit. A superseded acknowledgement was tested for evidence before it was tested for identity, so a health verdict or a later drift check made a node that acknowledged something the application has moved past read `synced_and_healthy`. The identity test now runs first and on its own: whether a node was asked to run something stale is a fact about the request, and no evidence recorded since can make it current again. An unconfirmed acknowledgement had no exit. A Blueprint whose drift policy never observes, with no health contract, has no confirmation coming, so the status pinned the target, and every application built from it, to a failure-toned state for ever. It is now gated on a confirmation actually being expected, resolved once per application. An acknowledgement binding a new artifact set left the previous observation in place, so the projection compared a stale identity against a new expectation and reported confirmed artifact drift for a stack that had simply not been re-checked. The observation is dropped when the expected set changes, and a deploy that does not change the set keeps it. The stateful hold outranked the authorization statuses, which removed the affordance that resolves them: an operator whose rollout was unauthorized and who also had one node held was shown a confirmation to perform and no way to authorize it. It now replaces the settled answer only. Clearing a stale guard also retires the target behind it, which previously outlived the row that cleared it and could never be resolved again; the retirement runs first so a failure leaves both in place for the next tick. A superseded acknowledgement had no attention reason, so a real stale target never reached the queue. It has one now, with its own pending tone and copy: nothing has gone wrong, and the newer rollout resolves it. * fix(gitops): make the confirmation rule and the re-hold reachable Two defects found by a second audit. Whether a Blueprint target's outcome will ever be confirmed had one leg too few. A health verdict reaches a Blueprint target only through a run the rollout executor reserved, which needs a policy that gates advancement; the default observes and arms nothing. The rule now reads both policies it has to: the one frozen into the authorized rollout generation, which is what is arming verdicts now, and the configured one on the intent, which is what the next authorization would freeze. Reading only the configured one let a policy change flip the answer before any run existed to justify it. Clearing a stale stateful guard severs the target, and a state-review hold recorded against a severed node could never reach the projection: the row said awaiting confirmation while the projection said the placement was settled. A hold now revives the target the way an explicit deploy already does, and only a hold, because every other stage is a report about what was seen rather than a request for the node. The interruption left behind by the severance is cleared with it, since an interruption outranks the observation and nothing is going to finish the operation it described. |
||
|
Anso
|
e0c4c1013b |
fix(gitops): record the policy on a policy-authorized rollout (#2040)
* fix(gitops): record the policy on a policy-authorized rollout A rollout authorization is the record of who authorized a rollout, and for a policy-authorized one that record could not name the policy it acted under. The approval row stored a hard-coded null, so the snapshot survived only on the generation the approval opened, and the projection could answer "what authorized this" only while that generation was still reachable. The agreement half mattered more. Placement already refuses an approval whose snapshot no longer matches the configured policy, so an approval cannot be granted under a policy the operator has just changed. Rollout authorization had no such check, so the same race was accepted silently and the generation froze whatever was configured at write time. rolloutAuthorized now takes the snapshot and enforces the same three checks placementApproved enforces: a policy mint with no snapshot, an operator mint claiming one, and a policy mint whose snapshot is no longer configured. authority becomes required rather than defaulting to configured_policy, which is how an operator's approval could be recorded as a policy's. Both mint paths read the application row immediately before the write, so the window the check guards is the write rather than the preflight await, and the race-retry path re-reads rather than reusing the row it read before the failed attempt. The comparison is over the whole snapshot, not the rollout domain alone. The approval and the frozen generation are two records of one decision, so letting them disagree about the source or placement value is the same class of defect. A refused mint is a one-pass delay, not a stranded rollout: the caller reports the refusal and the next dispatch mints under the policy configured then. * test(gitops): pin the racing-writer retry in the rollout mint The retry branch in ensureRolloutAuthorization was changed by the previous commit and nothing reached it. Forcing its snapshot argument to null passed the whole suite, so a policy-minted remint losing its snapshot would have been invisible. The test drives the branch with the real writer rather than a stubbed refusal. The racing writer plants a live authorization under a different preflight fingerprint, so the real mint refuses as already live, the drifted binding is invalidated, and the retry runs. The assertion is the retry's own approval: it decodes to the configured policy and matches the snapshot frozen on the generation it opened. |
||
|
Anso
|
adab9f38f8 |
fix(gitops): treat an unresolved health run as an operation in flight (#2041)
* fix(gitops): treat an unresolved health run as an operation in flight A health-gated rollout reserves its health run before the apply, and the acknowledgement that lands afterwards clears the target's operation stage without touching that pointer. The target is then acked but unsettled for the rest of the observation window, and the only evidence the placement conflict check had was the stage, so a bounded-automatic placement decided in that window read the target as idle. The approval supersedes the live rollout generation, so a fleet rollout stops partway instead of finishing. `hasTargetOperationInFlight` now also counts a target that still points at a health run it is awaiting. The pointer is the target's own record of the run, the same one the dispatch queue and the startup reconciliation read, so the placement gate now agrees with the queue it protects, and a run that turned terminal without a delivered verdict is released by that reconciliation rather than held. Tests: the placement end-to-end case refuses with the conflicting-operation reason and mints no approval, the real dispatch path is driven to the acked state to show the window is reachable, and each piece of evidence is asserted on its own, including the tombstoned target that must not hold anything up. * test(gitops): pin the in-flight rule for every health policy and for minting An audit of the conflict rule asked two questions the tests did not answer. The first is which policies the rule covers. It covers all of them, and that is now a decision on the record rather than a side effect: observe reserves a run like every other policy, so its targets hold the same pointer for the same observation window. The gate does not narrow to the policies that act on a verdict, because the run is bound to the rollout generation an approval would supersede whichever policy reserved it, and narrowing would mean reading the frozen strategy inside a predicate that is otherwise a pure read of the target rows. The placement side is only reachable on bounded_auto, which is opt-in. The second is the other reader of the predicate. Minting a rollout authorization is the moment a placement change would displace a rollout, so an unsettled target has to hold it there too. The refusal and the live-authorization path are both pinned now, the second because a gated rollout is authorized before it deploys anything and reaches its own verdicts in exactly that state. Also correct the predicate's comment: the startup reconciliation replays a terminal run's recorded verdict rather than releasing its pointer, and releases only when the run row is gone. |
||
|
Anso
|
194b8a5263 |
fix(gitops): read a failed marker as unverified rather than as drift (#2043)
* fix(gitops): read a failed marker as unverified rather than as drift
`readMarker` returned null for four different situations: an absent marker
file, a non-200 answer from a remote node, a body that does not parse, and a
thrown transport error. `checkForDrift` read that null as proof of drift, so a
remote node that could not be reached for a read was reported as `marker file
missing on node` on the `revision` cause and Enforce mode attempted a repair
against it. The check's own catch refuses to answer a transport failure with
drift for exactly that reason, and the marker arm reached a `drifted` verdict
before the catch could apply.
The local disk read already returned a discriminated result with missing,
present, and failed arms, and `readMarker` discarded it. The remote arm had no
discrimination at all.
`readMarker` now returns that discriminated result, and the remote arm
produces it the same way the local one does. `checkForDrift` maps a failed
read to `unverified`, which attempts no repair and writes no drift row, so
the reconciler leaves the target alone and announces nothing. An absent marker
is unchanged and still reports as revision drift.
A bare 404 is not a missing marker. Only a 404 carrying the leaf's NOT_FOUND
code counts as absent: a leaf predating the files route, an api_url pointing
at the wrong base path, and any middlebox also answer 404, and each is a node
that could not be read. Accepting the status alone would let the same defect
back in through a different door. The marker read sends no rootId, so it
always resolves the stack-source root, whose only absent path is ENOENT.
The three callers that are not the drift check keep treating absent as
absent and now name the new arm rather than letting it collapse. The
ownership probe throws on a failed read, as its local arm already did and as
its own contract already promised, so an unverifiable create is no longer
reported as an unmanaged stack that exists. The classification guard stops
claiming a Blueprint lost its marker when the read merely failed. The binding
preview reads a failed read as unproven on every action, including create.
Removing the failed-read mapping fails the new cases and leaves the
absent-marker companion passing.
* test(gitops): make the ownership-probe refusal actually reach the marker read
The probe test stubbed the stack listing on a URL containing "/api/stacks?",
but the listing is requested as `${baseUrl}/api/stacks` with no query string.
The listing call therefore rejected too, and the throw came from the probe's
own list-failure catch rather than from the failed marker read the test is
named for. Asserting only the error type could not tell those apart, since
both paths throw the same one, so the test passed with the fix deleted.
It now stubs the listing by exact URL, asserts inside the stub that the only
call allowed to fail is the marker read, and checks the message: the listing
failure names the node, the marker failure names the stack and carries the
underlying read error. Reverting the failed arm to returning true fails the
test.
Also corrects the hasNameConflict docstring, which described a remote
malformed marker as a conflict. A corrupt marker on local disk is a fact
about the node, but a remote marker that could not be read is a failure to
find out and throws.
* test(gitops): carry the marker read shape into the artifact freeze retry suite
That suite stubbed the marker read with a bare marker object, the shape the
read no longer returns, so it stopped compiling once this branch landed. The
stub is now the present arm, the same conversion every other readMarker stub
in the suite already has.
|
||
|
Anso
|
e4ad9029a5 |
fix(gitops): retry an unresolved Blueprint artifact freeze from the drift check (#2039)
* fix(gitops): retry an unresolved Blueprint artifact freeze from the drift check
A Blueprint freeze resolves exact image digests for the generation a target was
approved at. When that resolve cannot reach the registry, the expected artifact
set stays unresolved and every drift check reports unverified until an unrelated
redeploy of the same revision happens to re-run it. That made a redeploy the
price of proving what a node is running, on a condition that is often a
transient registry blip.
The drift check now re-resolves the frozen generation itself, throttled by a new
operator-set interval so a registry that is down is not asked once per target
per tick. The expectation can only advance through the existing transition, so
a retry cannot redefine an identity that was ever approved. Throttling reads the
target's latest evidence row, which advances on every attempt including a failed
one, rather than the expected set, which does not; the retry takes the target's
deploy lock and skips itself when a deploy is already resolving the same
generation.
A target whose expected identity is unresolved now also records a caveat saying
why nothing is being compared, and clears it when the identity becomes provable.
The retry interval lives in a new GitOps settings section, alongside the Git
source polling control it now sits next to. It is instance-scoped rather than
node-scoped, because the reconciler that reads it runs the checks for every node
the instance manages.
* fix(gitops): resolve the approved intent on a freeze retry, not the node disk
The retry introduced to recover an artifact freeze resolved whatever compose
was in the stack directory at the time it ran, and recorded that as the
approved identity. A reconcile tick can run days after the deploy, by which
point that directory may have been hand-edited or restored from a backup, so a
local change could become the approved identity for every target on the
generation, and a target restarted onto the edit read as matched against content
no approval ever saw.
The retry now parses the compose text the intent revision hashed, after checking
that hash against the live Blueprint row so an edit since approval refuses the
retry outright, and reads only the node's Docker platform from the node. Which
manifest child is correct is a property of the machine; which images were
approved is not. The deploy-time freeze still reads the directory it just wrote,
and both now share one implementation so the two paths cannot drift.
Also from review:
- A permanent failure is no longer retried. `unsupported_registry` and
`platform_ambiguity` record `unavailable`, so gating on the qualification
alone retried them forever, spending registry traffic and appending a row to a
table nothing prunes. The decision is now made on the recorded failure classes.
- The tick that produces an expectation does not judge by it. Enforce would
otherwise redeploy a node onto a digest resolved from a tag that may have
moved while the identity was unproven, which is the automatic advancement a
retry must not cause. The next tick compares instead.
- The lost-write-race rethrow is reverted. The race it claimed to handle cannot
occur, the two swallow sites it needed were unjustified, and nothing tested
them.
- A remote leaf is asked for its platform over its own route rather than for a
rendered compose context, so the retry never pulls a remote node's directory
into the answer. Registered as a static path so it is not read as a stack named
`platform`.
The evidence advances in the retry tests now go through the real transition
rather than by moving pointers, so the advance rule under test is the one
production applies, and the deploy-time freeze is asserted to still read the node
it just wrote.
* test(e2e): match the role the settings sidebar actually renders
The section-discovery test looked for a link named GitOps, but
SettingsSidebar renders every settings section as a button that calls
onSectionChange, with no anchor behind it, so the locator could never
resolve and the test failed in CI before reaching its URL or viewport
assertions.
The section itself is fine: the registry entry, the URL scheme built in
senchoRoute, and the Drift verification and Retry an unresolved image
identity copy were all checked against the source while tracing this.
Only the role in the locator was wrong.
Locator only. No product code changes.
* test(e2e): drop settings-nav assertions that were guesses, keep the behaviour
Two CI runs failed on this spec's section-discovery test, and both failures
were the test's fault rather than the section's. It asserted the GitOps entry
is in the settings navigation and that the retry control renders at desktop and
phone width, written from the settings registry rather than from the rendered
DOM. The first locator matched nothing. The corrected one matched nothing
either, because reading that the sidebar renders a button was never the same as
checking that the entry renders at all.
Two attempts and two guesses cost a CI cycle each and proved nothing, so the
assertions come out rather than getting a third guess. What they were standing
in for needs a browser, and the reason is recorded in the spec so the next
person does not reinstate them as though they had passed.
The remaining test is settings API traffic with no deploy in it, so the Docker
gate and the six-minute timeout that came with the dropped deploy assertions go
too. Left in place they would have skipped this spec on any machine without
Docker despite not needing it.
Kept: the seeded default, an in-range round trip, and the refused bounds. The
resolve behaviour is covered by the backend suite against the real transition.
* fix(gitops): refuse compose shapes the retry cannot faithfully resolve
The retry parsed the authored compose and treated every key under `services:`
as a service, but the freeze resolves the *rendered* model. Anything Compose
expands can make the two disagree, and a service the authored text declares but
the rendered model omits enters the expected set, is absent from the
observation, and `observationMatchesExpected` reads a missing expected service
as drift. That is a healthy stack reported as drifted, and under Enforce the
repair cannot converge it, so it would redeploy on every tick.
Four constructs now refuse the retry instead, each logged with the service that
carries it: `include`, `profiles`, `extends`, and `<<` merge keys. Merge keys are
included because the YAML parser does not resolve them at all -- it leaves a
literal `<<`, so merged fields vanish silently, and a service can carry its own
`image` while a merge key imports `profiles` invisibly. Refusing leaves the
target unverified, which is where it stood before this retry existed.
Also from review:
- The gate and the throttle now read the latest recorded row, not the expected
one. The expected pointer only advances for exact/qualified, so a failed
resolve wrote a row the gate never saw: a build-only stack stayed `unresolved`
there while every recorded row said `local_build_unverified`, and it retried
and appended a row every interval. Both now read one row, fetched once.
- A throwing retry is attempted once. It previously logged the failure and fell
through to a second attempt after `finally` had released the lock, doubling
registry traffic during an outage and running the repeat concurrently with a
deploy.
- The caveat fires when the expectation is not comparable, and the latest row
only suppresses the permanent build case. Keying it purely on the latest row
would have claimed drift was unchecked whenever newer non-advancing evidence
followed a comparable expectation, which is a state a stale probe produces.
- The removed categories are honest now: the tick hold is one reconciler
interval rather than a review window, and the two GitSourceService catches for
a race that cannot occur are gone, along with the comment claiming a rethrow
that no longer exists.
- A null platform records `platform_ambiguity` rather than a guessed child, so a
hub running ahead of its leaves costs one row instead of one per interval.
- Retry evidence carries its own history trigger, so an audit trail can tell a
deploy-time pin from a later recovery.
* fix(gitops): stop the retry looping on refusal, and unify the failure-class list
Three findings from the audit of the previous commit, plus one the review of
this one turned up.
An interpolated image reference is now refused rather than resolved. The
intuition that `${TAG:-1.25}` fails safely is wrong and the code comment said
so: `parseImageRef` accepts the literal text, splits it on the last colon, and
the registry is asked about a reference that cannot exist. That comes back as an
ordinary retryable error, so the retry ran forever instead of never. `$` cannot
appear in a legal image reference, so the check has no false positives, and a
mixed compose is refused as a unit because the set's qualification is the
weakest across services: resolving the other services would leave the set
uncomparable while writing a row that looks like progress.
A node that could not report its platform is no longer treated as a node that
never can. Both situations used to record `platform_ambiguity`, which is
permanent, so a single timeout on the hub-to-leaf platform read switched the
retry off for that generation until an unrelated redeploy, which is the exact
failure this path exists to remove. A 404 is now `route_missing` and stays
permanent, because a leaf older than this hub will not grow the route on its
own; a timeout, a 5xx, an unreachable daemon, or a missing proxy target records
the new `platform_unavailable` and is retried.
A refused retry is remembered per generation. It writes no evidence row, so the
latest row's stamp never moves, the time-based throttle never closes again, and
the refusal was re-derived and re-logged every reconcile tick forever rather
than once per interval. The retry now reports a three-state outcome and the
caller keeps a process-local map of refused generations, checked before
anything else. Nothing is persisted on purpose: a refusal is a stable property
of a generation, an edit clears the freeze pointers and a redeploy re-resolves
from the rendered model without consulting the map, and a restart costs one
redundant refusal per affected target.
From the review of this delta: the failure-class allowlist existed twice.
`portfolioAggregator.ts` had its own copy, so adding a class to `json.ts` alone
left the fleet projection rejecting any record that carried it and degrading a
remote row to unknown evidence. Both call sites now use one exported predicate
that reads the same set the type is declared beside.
GitSourceService.ts is reverted to match main: it had accumulated comment-only
lines here, and the file is not this PR's concern.
|
||
|
Anso
|
18d6aba11a |
fix(docker): floor pcre2 at the version that fixes CVE-2026-103111 (#2057)
The Docker Build & Scan job went red with a HIGH finding for pcre2 10.48-r0 (CVE-2026-103111) in the runtime image, while no dependency or base-image input had changed to produce it. The package is not in the base image at all: pcre2 arrives transitively through `apk add git`, so its version is decided entirely by the runtime stage's `apk upgrade --no-cache`. That layer is cache-busted by a date, so a fix landing in Alpine's repo during the day is invisible to any build reusing a layer cached earlier the same day, and the built image kept shipping 10.48-r0 after Trivy's advisory database learned that 10.49-r0 is the fixed version. Naming the floor does two things: it changes the RUN command so the stale layer cannot be reused, and it keeps a future cache from serving a version below the fix. It follows the existing openssh-client floor on the same line, which is there for the same reason. Alpine's v3.24/main index already carries 10.49-r0, and the floating node:26-alpine tag resolves to the digest already pinned, so there is no newer base image to move to and the floor is the fix rather than a workaround. Verified the exact apk line in a throwaway container against the base image digest: with the floor it installs pcre2 10.49-r0. The full image build and the Trivy scan are CI's to confirm, since a local multi-stage build is expensive. |
||
|
Anso
|
b2155bd6dd |
fix(gitops): observe a remote deploy's runtime identity on the leaf (#2038)
observeStackRuntimeArtifact had no node dispatch, so a remote node id
rendered the compose model hub-locally and then called
DockerController.getInstance(remoteId).getDocker(), which throws by
design. The catch turned that into {kind:'unavailable'}, so a remote
GitOps target recorded "cannot tell" instead of what the leaf runs.
Add fetchRemoteRuntimeArtifactIdentity, which asks the leaf over the node
proxy via its existing GET /api/stacks/:name/runtime-artifact-identity,
and dispatch on node type at the top of observeStackRuntimeArtifact so
nothing hub-local is touched for a remote id. No proxy target, a
non-200, a transport error, and an unreadable payload all resolve to an
explicit unavailable rather than a hub-local guess.
The leaf's observedAt is kept: that is when it actually looked. Stamping
it with the hub's envelope time would fabricate evidence.
BlueprintService.observeRuntimeIdentity now calls the same fetch instead
of keeping its own copy of it. Behaviour is unchanged: an unreachable or
unreadable leaf still yields null, and the drift check still reports
unverified without recording. Deploy-provenance headers are no longer sent
on that GET, but the route is a stack:read endpoint that never reads them.
A remote Direct target can now surface a real runtime artifact drift
where it previously could only read as pending verification, which is the
point of observing the leaf rather than guessing.
|
||
|
Anso
|
d57d2a3ee9 |
fix(gitops): compare stale artifact probes by platform membership (#2034)
* fix(gitops): compare stale artifact probes by platform membership The Direct dispatch probe decided that an accepted generation's executable identity had moved by comparing artifact set fingerprints. A fingerprint pins one platform child per service, so it only compares resolutions made on the same platform. A target whose architecture differs from the one that produced the frozen expected set resolved a different fingerprint for the same approved child of a multi-arch tag, recorded stale for a converged target, advanced the newest-evidence pointer to that row, and surfaced as artifact stale in the projection and in attention. Route the probe through the same per-target platform membership the Blueprint drift path applies, so a target that matches the approved child for its own platform is no longer stale, a real move of that child still is, and an expected set with no per-service evidence keeps the fingerprint verdict. * fix(gitops): keep the stale probe fail-closed on a changed service set The membership test only walks the expected services and weighs the registry ones, so it cannot tell an extra or renamed service from an unchanged set, and with nothing to weigh it answers vacuously. The probe resolves the live stack rather than the staged candidate, so a stack edited out of band can carry a different set than the one that was frozen, and that is a moved identity rather than a matching one. Ask membership only when both sides hold the same services. A set that grew or shrank keeps the fingerprint verdict, as does an expected set that recorded no per-service evidence. * fix(gitops): tell a swapped service apart from an unchanged set The set-equality check compared service names, so a service that kept its name while swapping a local build for a published image still looked unchanged, and the membership test had no registry service left to weigh. Compare the kind of each service as well as its name, and cover that flip. Also fold the same-platform move into the moved-child test, since it shares the service set and belongs beside it rather than as its own case. |
||
|
Anso
|
bd6a7eb61e |
docs: remove the stale permission inventory copy from backend/src/routes (#2055)
The operational permission inventory lives with the architecture docs, and that
is the maintained copy. The one sitting beside the routers stopped tracking
route families in
|
||
|
Anso
|
b6b059a3fb |
fix(gitops): surface withdrawal residue for a tombstoned target (#2053)
A tombstoned target reports the tombstone before the runtime facet reads any recovery field, so a target withdrawn while its recovery had failed reported nothing anywhere. Both retirement transitions clear the failure and LKG columns but leave recovery_phase and recovery_ref on the row, and no projection read them back, so the failed recovery was gone from the runtime status, the rollout status, and the attention queue in one step. Mint a withdrawal_residue limitation in deriveTarget for a tombstoned row carrying a failed recovery, on the same condition deriveRuntime reads, so the caveat and the status cannot disagree about a row. It rides the limitations array deriveTarget already threads, so it reaches the caveats surface with no wire change, no new route, and no new broadcast. Tombstone precedence is unchanged: the runtime status stays tombstoned and the rollout status never sees the recovery. Only a retirement that carried a failed recovery is qualified. After any withdrawal Sencho cannot prove the compose project came down, because the teardown is best effort and swallows its own error, but qualifying every retirement would train a reader to skip the block. |
||
|
dependabot[bot]
|
e39aada173 |
chore(deps): bump @grpc/grpc-js (#2051)
Bumps the all-npm-backend-security group with 1 update in the /backend directory: [@grpc/grpc-js](https://github.com/grpc/grpc-node). Updates `@grpc/grpc-js` from 1.14.4 to 1.14.5 - [Release notes](https://github.com/grpc/grpc-node/releases) - [Commits](https://github.com/grpc/grpc-node/compare/@grpc/grpc-js@1.14.4...@grpc/grpc-js@1.14.5) --- updated-dependencies: - dependency-name: "@grpc/grpc-js" dependency-version: 1.14.5 dependency-type: indirect dependency-group: all-npm-backend-security ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
1ef06136bb |
chore(deps): bump the all-actions group across 1 directory with 4 updates (#2050)
Bumps the all-actions group with 4 updates in the / directory: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action). Updates `github/codeql-action/init` from 4.38.1 to 4.38.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1c5b675653bb5c22dbe9b12b556ec555138e09fd...2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2) Updates `github/codeql-action/autobuild` from 4.38.1 to 4.38.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1c5b675653bb5c22dbe9b12b556ec555138e09fd...2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2) Updates `github/codeql-action/analyze` from 4.38.1 to 4.38.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1c5b675653bb5c22dbe9b12b556ec555138e09fd...2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2) Updates `github/codeql-action/upload-sarif` from 4.38.1 to 4.38.2 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/1c5b675653bb5c22dbe9b12b556ec555138e09fd...2892aa5e19bbd11bc0cff5427e3b750a04d9e3c2) --- updated-dependencies: - dependency-name: github/codeql-action/init dependency-version: 4.38.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-actions - dependency-name: github/codeql-action/autobuild dependency-version: 4.38.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-actions - dependency-name: github/codeql-action/analyze dependency-version: 4.38.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-actions - dependency-name: github/codeql-action/upload-sarif dependency-version: 4.38.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
a13ac12e73 |
chore(deps): bump the all-npm-backend group in /backend with 9 updates (#2049)
Bumps the all-npm-backend group in /backend with 9 updates: | Package | From | To | | --- | --- | --- | | [undici](https://github.com/nodejs/undici) | `8.10.2` | `8.11.2` | | [ws](https://github.com/websockets/ws) | `8.21.3` | `8.22.0` | | [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.6.2` | `26.6.3` | | [supertest](https://github.com/ladjs/supertest) | `7.2.2` | `7.3.0` | | [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) | `8.70.0` | `8.70.1` | | [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.3.0` | `8.3.1` | | [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `5.0.1` | `5.0.2` | | [@aws-sdk/client-ecr](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-ecr) | `3.1136.0` | `3.1141.0` | | [@aws-sdk/client-s3](https://github.com/aws/aws-sdk-js-v3/tree/HEAD/clients/client-s3) | `3.1136.0` | `3.1141.0` | Updates `undici` from 8.10.2 to 8.11.2 - [Release notes](https://github.com/nodejs/undici/releases) - [Commits](https://github.com/nodejs/undici/compare/v8.10.2...v8.11.2) Updates `ws` from 8.21.3 to 8.22.0 - [Release notes](https://github.com/websockets/ws/releases) - [Commits](https://github.com/websockets/ws/compare/8.21.3...8.22.0) Updates `@types/node` from 26.6.2 to 26.6.3 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `supertest` from 7.2.2 to 7.3.0 - [Release notes](https://github.com/ladjs/supertest/releases) - [Commits](https://github.com/ladjs/supertest/compare/v7.2.2...v7.3.0) Updates `typescript-eslint` from 8.70.0 to 8.70.1 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.70.1/packages/typescript-eslint) Updates `vite` from 8.3.0 to 8.3.1 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.3.1/packages/vite) Updates `vitest` from 5.0.1 to 5.0.2 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v5.0.2/packages/vitest) Updates `@aws-sdk/client-ecr` from 3.1136.0 to 3.1141.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-ecr/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1141.0/clients/client-ecr) Updates `@aws-sdk/client-s3` from 3.1136.0 to 3.1141.0 - [Release notes](https://github.com/aws/aws-sdk-js-v3/releases) - [Changelog](https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-s3/CHANGELOG.md) - [Commits](https://github.com/aws/aws-sdk-js-v3/commits/v3.1141.0/clients/client-s3) --- updated-dependencies: - dependency-name: undici dependency-version: 8.11.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: ws dependency-version: 8.22.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: "@types/node" dependency-version: 26.6.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: supertest dependency-version: 7.3.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: typescript-eslint dependency-version: 8.70.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: vite dependency-version: 8.3.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: vitest dependency-version: 5.0.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-backend - dependency-name: "@aws-sdk/client-ecr" dependency-version: 3.1141.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend - dependency-name: "@aws-sdk/client-s3" dependency-version: 3.1141.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-backend ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
bc29689d00 |
chore(deps): bump the all-npm-frontend group in /frontend with 9 updates (#2048)
Bumps the all-npm-frontend group in /frontend with 9 updates: | Package | From | To | | --- | --- | --- | | [@xyflow/react](https://github.com/xyflow/xyflow/tree/HEAD/packages/react) | `12.11.6` | `12.12.0` | | [lucide-react](https://github.com/lucide-icons/lucide/tree/HEAD/packages/lucide-react) | `1.47.0` | `1.48.0` | | [monaco-editor](https://github.com/microsoft/monaco-editor) | `0.56.0` | `0.57.0` | | [motion](https://github.com/motiondivision/motion) | `13.4.0` | `13.4.4` | | [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.6.2` | `26.6.3` | | [jsdom](https://github.com/jsdom/jsdom) | `30.1.0` | `30.1.1` | | [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) | `8.70.0` | `8.70.1` | | [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) | `8.3.0` | `8.3.1` | | [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `5.0.1` | `5.0.2` | Updates `@xyflow/react` from 12.11.6 to 12.12.0 - [Release notes](https://github.com/xyflow/xyflow/releases) - [Changelog](https://github.com/xyflow/xyflow/blob/main/packages/react/CHANGELOG.md) - [Commits](https://github.com/xyflow/xyflow/commits/@xyflow/react@12.12.0/packages/react) Updates `lucide-react` from 1.47.0 to 1.48.0 - [Release notes](https://github.com/lucide-icons/lucide/releases) - [Commits](https://github.com/lucide-icons/lucide/commits/1.48.0/packages/lucide-react) Updates `monaco-editor` from 0.56.0 to 0.57.0 - [Release notes](https://github.com/microsoft/monaco-editor/releases) - [Changelog](https://github.com/microsoft/monaco-editor/blob/main/CHANGELOG.md) - [Commits](https://github.com/microsoft/monaco-editor/compare/v0.56.0...v0.57.0) Updates `motion` from 13.4.0 to 13.4.4 - [Changelog](https://github.com/motiondivision/motion/blob/main/CHANGELOG.md) - [Commits](https://github.com/motiondivision/motion/compare/v13.4.0...v13.4.4) Updates `@types/node` from 26.6.2 to 26.6.3 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `jsdom` from 30.1.0 to 30.1.1 - [Release notes](https://github.com/jsdom/jsdom/releases) - [Commits](https://github.com/jsdom/jsdom/compare/v30.1.0...v30.1.1) Updates `typescript-eslint` from 8.70.0 to 8.70.1 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.70.1/packages/typescript-eslint) Updates `vite` from 8.3.0 to 8.3.1 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.3.1/packages/vite) Updates `vitest` from 5.0.1 to 5.0.2 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v5.0.2/packages/vitest) --- updated-dependencies: - dependency-name: "@xyflow/react" dependency-version: 12.12.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-frontend - dependency-name: lucide-react dependency-version: 1.48.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-frontend - dependency-name: monaco-editor dependency-version: 0.57.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: all-npm-frontend - dependency-name: motion dependency-version: 13.4.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-npm-frontend - dependency-name: "@types/node" dependency-version: 26.6.3 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-frontend - dependency-name: jsdom dependency-version: 30.1.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-frontend - dependency-name: typescript-eslint dependency-version: 8.70.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-frontend - dependency-name: vite dependency-version: 8.3.1 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-frontend - dependency-name: vitest dependency-version: 5.0.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: all-npm-frontend ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
001eeb50da |
chore(deps): bump golang from cf6fca6 to 8a5910f (#2047)
Bumps golang from `cf6fca6` to `8a5910f`. --- updated-dependencies: - dependency-name: golang dependency-version: 1.27-alpine dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
32d9ca7c67 |
chore(deps): bump node from ef24c50 to 0b36e8c (#2046)
Bumps node from `ef24c50` to `0b36e8c`. --- updated-dependencies: - dependency-name: node dependency-version: 26-alpine dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
5c113ee6b6 |
chore(deps): bump the all-npm-backend-security group across 1 directory with 2 updates (#2044)
* chore(deps): bump the all-npm-backend-security group across 1 directory with 2 updates Bumps the all-npm-backend-security group with 2 updates in the /backend directory: [brace-expansion](https://github.com/juliangruber/brace-expansion) and [fast-uri](https://github.com/fastify/fast-uri). Updates `brace-expansion` from 5.0.9 to 5.0.12 - [Release notes](https://github.com/juliangruber/brace-expansion/releases) - [Commits](https://github.com/juliangruber/brace-expansion/compare/v5.0.9...v5.0.12) Updates `fast-uri` from 3.1.7 to 3.1.8 - [Release notes](https://github.com/fastify/fast-uri/releases) - [Commits](https://github.com/fastify/fast-uri/compare/v3.1.7...v3.1.8) --- updated-dependencies: - dependency-name: brace-expansion dependency-version: 5.0.12 dependency-type: indirect dependency-group: all-npm-backend-security - dependency-name: fast-uri dependency-version: 3.1.8 dependency-type: indirect dependency-group: all-npm-backend-security ... Signed-off-by: dependabot[bot] <support@github.com> * fix(deps): bump brace-expansion to 5.0.12 in the frontend lockfile The dependency audit step in the frontend build gate fails on brace-expansion 5.0.9, which falls inside the affected range 4.0.0-5.0.11 of three high severity advisories (GHSA-q2hr-2g5m-vwhr, GHSA-qhr7-859c-m2p7, GHSA-6j4f-fj2g-mc7p). 5.0.12 is the first fixed release. The backend lockfile in this branch already carries the same bump, so the backend gate passed while the frontend gate, which audits its own lockfile, stayed red. Bumping the frontend pin the same way brings the lockfile to the fixed release and clears the gate. brace-expansion is a dev-only transitive dependency of minimatch, used by the lint and test toolchain, so nothing in the runtime image changes. Verified locally: npm ci, vite build, 3712 unit tests, eslint (0 errors) and npm audit --audit-level=high, which now reports 0 vulnerabilities. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anso <dev@anso.codes> |
||
|
dependabot[bot]
|
49175ae436 | chore(deps-dev): bump @humanfs/node (#2036) | ||
|
dependabot[bot]
|
e5664b66bb | chore(deps): bump ip-address (#2035) | ||
|
Anso
|
2358a07b83 |
fix(gitops): attribute a health verdict to an unacked rollout attempt (#2033)
* fix(gitops): attribute a health verdict to an unacked rollout attempt A health-gated rollout reserves a health run before dispatching its apply, and a target only records its `rollout_generation_id` when the node acks. So after a lost apply response and a restart, the guard that skips a verdict from a rollout the target has left could not tell that apart from a superseded rollout: the swept `unknown` was recorded as evidence, no policy decision was made, and the reconstruction re-applied the same generation. An interrupted rollout looked like a slow one, and nothing reported the apply as unconfirmed. The application is what separates the two cases. While it is still on the run's rollout generation, an unacked target is that rollout's own attempt, so the verdict is attributed and the frozen policy decides. Only `unknown` is attributable: the ack is the only proof the apply landed, and `unknown` is the verdict that never retries, stops, or rolls back. A `passed` would claim convergence nothing proved, and a `failed` would spend the policy on whichever generation the observation turned out to be about, so both stay evidence and fall through to the existing release-and-redrive path. An unacked target's fence cannot be scoped to a rollout, because it names no rollout generation, so the queue would not honour it. The application hold therefore commits in the same transaction as the fence, leaving the executor's fire-and-forget write a no-op for this case. A resume clears the unacked case explicitly, so the target card stops reporting a stopped target while the rollout re-drives it. Also adds the coverage the health-policy route's per-target grant never had: the shared fixture created no target rows, so the resolved target set was always empty and only the fleet-wide fallback could run. * fix(gitops): read an unacked rollout from a pointer mismatch, not a null A target's `rollout_generation_id` is written only by an acknowledgement, so "has not acked this rollout" covers two states: none at all on a first rollout, and the previous rollout's value on a re-rollout until it is re-acked. The resume path tested for the null and so handled the first, leaving the second reporting a stopped target on the card while the queue re-drove it. The clearing is now by fence class, so any resumable fence goes and a finished outcome is still preserved. The attribution predicate had the matching gap. It asked whether the target ran the observed generation before asking whether it had acked, and a re-rollout of the same accepted generation answers the first question yes, so a `passed` or a `failed` was attributed to an unacked target: the pass would promote, the fail would spend the previous rollout's retry budget, and either could write a fence under a pointer the queue does not honour, so a stop decided that way would hold nothing. The unacked case is now decided from the verdict alone, and only `unknown` attributes. Also closes the gaps the same audit found around this change: a replayed verdict is asserted to record one hold and change nothing the second time, the refusal the in-transaction hold takes when a teardown races it is covered, the recorded `unknown` is asserted not to project as a pass or a failure, and the route tests gain a read-only persona, a single-target grant on another node, and the role-based positive case for an empty target set. * fix(gitops): scope the resume acknowledgement to the rollout that wrote it The previous commit cleared the resume loop's pointer check to fix unacked fences on a re-rollout, and that check was doing more than one job. A finished outcome under a stale pointer is a record of what an earlier rollout did to that node, and the current rollout is about to re-drive it, so acknowledging the outcome here replaced a "rolled back" record with "stopped, resumed" and told the operator the policy was finished with a target it was about to deploy to. Main skipped those targets, so the rewrite is scoped to the rollout that wrote the outcome again, while a resumable fence still goes whatever the pointer says because an unacked target's fence is one no queue can honour. A roll back still in flight is in neither set and is now covered by the same test, so all three non-resumable fences are asserted to survive a resume under both pointers. The projection test that claimed to show an `unknown` verdict cannot be read as a pass or a failure was asserting `unbound` on a target with no running generation, which `deriveHealth` returns before it reads the status at all. It now runs against a target that is still running the previous generation with the observation about the next one, which is the state where the two would differ. |
||
|
Anso
|
3623ce2d89 |
feat(gitops): persist the observed apply invocation and classify invocation drift (#2032)
The invocation drift class had no producer because the observed compose invocation was never persisted anywhere the deriver could read it. Emitting the authored invocation against a permanent unknown would have been constant fabricated drift, so the class was absent rather than wrong. This records the observation and adds the producer that compares it. What Compose was invoked with is read back off the running project's container labels at apply time, not off the argv Sencho built, because reading the argv back would only ever confirm what Sencho already believes. It lands on a new nullable per-target column, and the column stays null when the node cannot be reached, which is the same absence as never having looked. A new producer compares that record against the invocation the accepted generation was authored with, and reports nothing while either side is missing, unreadable, or not comparable, naming each case in a caveat rather than reporting agreement. The single-file case is what the design turns on. Compose omits every flag for a single-file selection, so a generation's argv is empty while the observed label always names at least one file. The authored file order therefore falls back to the application's configured compose selection, which is what makes a true negative reachable on the majority of stacks instead of a standing false positive. The comparison is Direct-only: a Blueprint target is deployed under its own stack name from a generation whose argv was authored for the source stack, so there is no correct expected side to compare against and the class stays silent for it. The read requires the project's containers to agree. An ordinary up recreates only the services whose configuration changed, so a half-recreated project carries labels from two invocations; a disagreement records nothing rather than sampling whichever container the daemon listed first, which would flap across applies and could report agreement while a sibling container disagrees. The history column for this observation already existed in the schema without a writer, so the audit row now records the evidence rather than only the fact that a column moved. A new target column is added additively for existing installs, and a migrated target stays unobserved rather than being backfilled with a value no apply recorded. Validated over the live route: a differing invocation produces one item with both identities real, a matching one produces nothing, and a missing observation produces nothing plus a caveat. Unit coverage spans the canonicalization, the label read, the three acceptance states, the additive migration, and the drift row rendering. |
||
|
Anso
|
b833e554b4 |
fix(security): resolve the WebSocket client IP through the trusted-proxy predicate (#2031)
The host console WebSocket handler took the audit row's client IP from the first X-Forwarded-For hop ahead of the socket address, without checking that the peer was a trusted reverse proxy. Express's `trust proxy` setting does not apply on an upgrade, so the header was taken at face value and any principal able to open a host shell could write an arbitrary IP into the host-console audit trail. CWE-348, audit-log integrity only: the address is not used for an access decision in that handler. Client-IP resolution on the upgrade path now goes through a single helper that applies the same trusted-proxy predicate as the HTTP path and mirrors proxy-addr: a forwarded hop counts only when the direct peer matches SENCHO_TRUSTED_PROXY_CIDRS, and the recorded address is the leftmost hop that no trusted proxy supplied. Unset, malformed, or duplicate proxy configuration fails closed to the socket address, as it already does for `req.ip`. The host console audit row still records the real client address behind a properly configured reverse proxy, and now matches what the HTTP audit path records for the same request. A remote node records the forwarding instance's address unless that instance is allowlisted, which the configuration reference now states. |
||
|
Anso
|
1f1cb33ea4 |
test(gitops): pin the managed-project drift suppression windows (#2030)
The managed-project drift class compared the stack's manifest cache against the accepted generation, and suppressed a set of windows where the cache legitimately disagrees: an application mid-apply, interrupted, or recovering, a live target that has not yet applied the accepted generation, a target mid-deploy, interrupted, or recovering, a cache field nobody has written, and a tombstoned target. One combined test covered the class, so no individual window was pinned and any of them could be dropped without a test noticing. Each window now has a named case, each on its own fixture so no window can be covered by another's guard. Six of the eleven guards these cases pin had no other coverage. A tombstoned target is excluded from the settled check rather than holding the item back, and each manifest state that cannot be read is a reportable item, so both are asserted directly. |
||
|
Anso
|
3c5d4aafc8 |
fix(gitops): project Blueprint deploys, withdrawals, and their failures (#2029)
`deriveRuntime` read `active_operation_stage` only against the Direct value `deploy_started`, and `failure_stage` only against `deploy`. The Blueprint transitions write their own vocabulary into those same columns, so three of the runtime branches could never match them. The interruption branch already handled all three stages, which is what made the gap easy to miss. The consequences were: - A Blueprint deploy or withdrawal in flight projected as a pointer state (`never_applied`, `applied_not_deployed`, or `synced_and_healthy`) rather than as work in progress. A Blueprint target has no deploy-bound writer, so its deployed pointer is null and the pointer checks answered for it. The portfolio classifies in flight work by `deploying` and `withdrawing`, so a rollout under way did not read as in progress either. `withdrawing` was declared, given an evidence-source classification, ranked, consumed by that in-flight check, and given operator copy, and no code path could produce it. - A recorded Blueprint failure projected as a pointer state too, producing no `failed_after_mutation`. The attention classifier keys `deploy_failed` on the two failure statuses, so a failed Blueprint deploy or withdrawal reached no attention queue at all. The fix is the vocabulary, not new statuses. `deriveRuntime` is an ordered chain where each branch claims a status, so both matchers sit in the slots the Direct ones already occupied, which keeps the live operation ahead of the pointer checks, ahead of the observation lookup, and the failure ahead of the observation and after pause and partial state, exactly as Direct reads today. `post_mutation` and `deploy_failed` are the classes that mean the mutation reached the node, and they escalate to `failed_after_mutation`. Every other class reports the workload as intact. That covers `name_conflict` and `target_missing`, which only the Blueprint path records and which a mapping written against the Direct class vocabulary matched neither arm for, as well as a class a future producer invents. The split is directional on purpose: over-claiming a mutation is corrected by looking at the node, while under-claiming is how a half-replaced workload goes on reporting healthy. Widening the failure matchers made one writer change necessary. `blueprintAckRecorded` cleared only a deploy failure, so a node whose withdrawal failed and which the reconciler then re-deployed would keep reporting that old withdrawal as a standing failure for ever. It now clears either stage, which is what `deployBound` already does on the Direct side. Validated by driving the real transitions from a Blueprint create through to the projected status: 12 cases covering both live stages, all five Blueprint failure classes across deploy and withdrawal, both precedence boundaries, the portfolio posture, the attention queue, and the writer change. All 12 fail against the previous projection and pass against this one. Direct coverage is unchanged (10350 backend tests pass), and the pre-existing SSH transport suite still fails its loopback-port precondition in this environment. |
||
|
Anso
|
c92004dc6a |
fix(blueprints): retry a refused drift repair instead of stranding the row (#2028)
* fix(blueprints): retry a refused drift repair instead of stranding the row A drift repair that could not finish left the deployment row reading `correcting` forever, so nothing ever retried it. `drift_enforce_start` writes `correcting` before the repair runs, and a repair that then answered without ever reaching a deploy wrote no status of its own. That state is in-flight as far as the decision is concerned, so `computeDecision` skipped it and the preview projection rendered it as the informational `in_flight_correct`. No later tick re-checked the node: the workload kept running the moved image, one correction-failed alert fired, and then nothing, for as long as the drift lasted. Four ticks on a refused repair recorded one drift check, one repair attempt and one alert, and did nothing on the rest. A refusal, or a repair that throws, now settles the row back to `drifted` with the reason in `last_error`, so the next tick checks and retries it. A throw counts because resolving a remote node's platform label is a network call made before any deploy, so it can reject with nothing written and used to escape to the per-blueprint catch with the row still `correcting`. The row deliberately does not become `failed` or `pending`. Both are retry buckets for an ordinary deploy, so the next tick would repair by tag and adopt whatever the tag points at now, which is the one thing a digest-pinned repair exists to prevent. The failure alert fires when the row changed across the attempt, compared against a snapshot read before the repair ran. It cannot be decided from the row afterwards: a repair that reaches a deploy writes its own `last_error` and returns the very same string, so asking the row afterwards finds the failure already sitting there and mistakes a new incident for a repeat, dropping the first failure of every kind silently. Losing the deploy lock is a deferral rather than a failure, so the row still leaves `correcting` with a reason and the operator is not paged about a race they cannot act on. A converged row now forgets its last failure, as the deploy acknowledgement already did, so a failure that returns after a repair counts as a new incident. computeDecision also treats a `drifted` row at the current revision as a drift-check target. The projection already reached it through its own status fallback with the same action, severity and confirmable membership, so nothing an operator sees changes; it is here so the decision a tick acts on is not two files away in the preview projection. Verified: 10333 backend tests pass with no failures, typecheck and lint clean, and an e2e case drives drift, repair and reconvergence on a real Inline Blueprint, asserting the row returns to Active with no drift summary and the workload runs the approved digest again. * fix(blueprints): correct the drift repair settle found in review Four corrections to the refused-repair settle, all found by reviewing the change rather than by a failing test. The deferral note was the serious one. Losing the deploy lock is silent, and it wrote a note about the race into `last_error`, which is the replay key. One lost-lock tick between two identical failures therefore displaced the failure being replayed, and the next tick reported that same known failure again as though it were new. A deferral now leaves `last_error` alone: the row keeps saying what it last actually failed with, which is worth more to an operator than a note about a race that is already over. An alert now quotes the row rather than the outcome where the outcome carries only a shorthand. The name-conflict paths return the bare word `name_conflict` and then park the row for good, so that one alert is the only signal the operator will ever get, and it has to name the directory in the way. The row's own text is only preferred when the repair actually wrote a status: when the row is still `correcting`, nothing was written and the row is still carrying the previous failure, which must not be reported as the current one. A target that converges while its repair is in flight is no longer reported. The drift the failure belonged to no longer exists, so paging the operator about it is stale news. A repair that throws is now logged with its stack as well as alerted. This boundary cannot tell an operational failure from a bug in the repair and both arrive identically; an alert alone would repeat a bug every tick and never reach a developer, and re-throwing would strand the row again. Separately, the drift observation restamped `last_drift_at` every time it re-observed the same drift, so a drift outliving one tick reported itself as just found, forever. It is stamped only when the row was not already drifted. Both observation sites did this, the Inline one and the Git-managed one. Coverage added for the Git-managed repair path, which is the third dispatch and had no case, for the timestamp, and for the three alert rules above. 19 cases now, 5 of which fail with the whole change reverted. |
||
|
Anso
|
709c7e0269 |
fix(gitops): report a Blueprint reconciler drift in the canonical drift list (#2025)
* fix(gitops): report a Blueprint reconciler drift in the canonical list
A Blueprint digest drift was detected, recorded, and visible everywhere
that reads the deployment row or the per-target digest comparison, while
the canonical GitOps drift list stayed empty for the same application. The
portfolio's drift summary by class never counted it and the attention
queue never named it, so every surface that read the model disagreed with
the two that read the deployment.
The deriver emits a runtime-family item from two statuses, and a Blueprint
observation is neither. The reconciler records its observation as a third
status, which outranks the pointer and artifact comparisons in
deriveRuntime precisely so it cannot be masked by them, and that ordering
meant the artifact comparison was never reached. The item is now built from
that status as well.
The class stays a fact about the target rather than a spelling of the
status, read from the same binding deriveRuntime uses, so a rollout
target is reported once as rollout instead of also as runtime. The
comparison against the expected set is what still decides whether anything
is reported, so a stale stage, an unverified observation, and a digest
Enforce is repairing on this pass all report nothing.
The item's observed identity now carries the per-service evidence, because
the identity string is a fingerprint over the whole set and cannot name
which service moved to which digest on its own.
The reconciler records the same status for a revision mismatch and for a
container that is not running, and refreshes the observation only on the
digest path, so that identity is the last one observed rather than
necessarily a current reading. The cause is not held on the target row, so
the reason states the comparison that was made and leaves the cause to the
facet that does report it, with the evidence's own age on the item.
* fix(gitops): require a comparable approved set before reporting Blueprint drift
A Blueprint observation reaches the drift list with the runtime status the
reconciler recorded, and that status is assigned before deriveRuntime
checks the approved set's qualification. The two artifact-identity statuses
are assigned after it, so those two arrived at the collector already known
to be comparable and the collector never checked. The observation path has
no such guarantee, and the collector's only test was that the approved
identity is not null.
That is not enough. A set Sencho could not resolve, or resolved against a
stale resolution, still carries a real fingerprint identity, so a Blueprint
whose stack builds locally, or whose approved set has since gone stale, could
report a confirmed runtime divergence against an expectation nothing
verified, while the same target with no recorded observation reads the
comparison as pending. The facet and the canonical list disagreed.
The collector now requires the approved set to be exact or qualified, which
is the same test deriveRuntime applies, so the two agree on every case. Also
spell the rollout-binding test the way deriveRuntime spells it, so the
facet and the class cannot answer differently about the same target, and
correct the comment, which had claimed the guards were complete when they
covered only the observed side.
* fix(gitops): anchor a drift comparison to the generation being served
A recorded acknowledgement rebinds a target to the accepted generation's
approved set and leaves the stored observation as the previous generation
recorded it. The two are then facts about different generations, and the
observation says nothing about the set it is being compared against.
Reading an acknowledgement rebinds the target to the new generation's set
while still deploying the previous one, and a drift recorded on the next
tick through a cause that returns before re-observing, a stopped container
or a marker or revision mismatch, leaves that observation untouched. The
comparison then reports a digest divergence that was never measured, and it
reports it as confirmed: a class, a reason, and per-service digests that all
read as real evidence.
The two artifact-identity statuses cannot reach that state, because
deriveRuntime returns applied_not_deployed before it compares identities
whenever desired and deployed disagree. The reconciler observation is
assigned before that pointer check, so it is the only path that can, and the
collector now refuses to compare across generations. The approved set names
the generation it belongs to, so that is what anchors the comparison, and it
fails closed when the set cannot be read or the target has never recorded a
deployed generation, which is the direction that reports nothing.
The same anchor applies to the convergence gate, which was reaching the same
comparison with the same rows, so a target part-way between generations can
no longer read as exactly converged.
* Revert "fix(gitops): anchor a drift comparison to the generation being served"
This reverts commit
|
||
|
Anso
|
e686f55145 |
fix(gitops): address the runtime drift repair audit (#2026)
* fix(gitops): detect drift even when the repair is blocked, and hold only in Enforce A pre-merge audit of the drift-repair work found that the hold was evaluated before any drift, in every mode. Two consequences, both fleet-wide on upgrade: every existing Git-managed target read "Repair held" because its marker predates the generation fields, including targets with no drift, and a genuinely stopped container was never reported as drifted at all. Separately, the stateful classification hold ran in Observe and Suggest, so Suggest stopped alerting, which is the one thing Suggest exists to do. Detection and the repair decision are now separate. The check always completes and carries the block with it, so an unprovable repair authority no longer blinds the drift surfaces. A block is consulted only in Enforce, because it is a decision about auto-repair and means nothing to someone who chose not to have one. A held target takes exactly one write per tick. Writing the drift and then the hold flipped the row between the two states, so neither write was ever suppressed and the hold appended history every minute. Keeping the hold's stage on the row is also what stops the runtime observation from overwriting it and making the hold vanish from the projection on the next tick. * fix(gitops): a drift-mode change no longer supersedes the rollout The audit found the edit that turns Enforce on was the one edit guaranteed to stop Enforce working. drift_mode was classified as an operational key, so changing it opened a rollout candidate, which invalidated the placement approval and superseded the live rollout generation, which held every target with nothing left to advance it. The issue this work belongs to required a runtime policy-only change not to reopen placement approval, so the requirement was unmet rather than merely inconvenient. The drift mode chooses what Sencho does about divergence it already observes and changes nothing any node runs, which is the same test every non-operational key already met. It is now metadata_only, so a mode edit writes the Blueprint and stops. The end-to-end test needed the fixture to persist the application's live rollout pointer, which the pointer writer deliberately does not carry and the authorization transition normally writes. Without it the test could not see a supersede production really performs. * fix(blueprints): carry a held repair through every surface that lists drift The audit found repair_held missing from the three places that decide what an operator is shown and what they are allowed to do. A Blueprint could be disabled while its containers kept running, because the disable guard did not count a held deployment. A held row offered no Withdraw, so the documented way out of that state did not exist. And the catalog left the status out of its priority list, its tile colour and the Drifted filter, so a Blueprint whose only problem was a declined repair rendered as a grey dot that the Drifted chip could not find. A hold is a decision not to overwrite a running workload, so it belongs with drift everywhere drift is counted, and the workload is still there everywhere lifecycle actions are gated. * docs: correct the drift-repair claims the audit found untrue Three things the previous change documented were not what the code did. It said Enforce "restores the older one" when a newer generation has been rolled out. Accepting and rolling out a newer generation supersedes the rollout that owned the target, so the honest outcome is a declined repair, not a quiet restore of the generation the fleet has moved past. It said a superseded hold clears "or until you deploy to it". A Git-managed Blueprint has no manual redeploy to fall back on, so the rollout is the only way out, and saying otherwise sends an operator looking for a button that is not there. It never documented the repair-held notification at all, in the category table or the message list. * fix(gitops): carry the repair block on every drifted return, and keep the hold in the projection A re-audit of this work found two blocking problems. The block was dropped on three early drifted returns: a missing marker, a marker belonging to another Blueprint, and a revision mismatch. Each returned a verdict with no block, so the reconciler treated the target as repairable and Enforce overwrote exactly the targets the hold exists to protect: an Inline target a recovery owned was redeployed by a path that uses the binding only to stamp the marker and never refuses, and a Git-managed node on a legacy marker was overwritten by a reapply that had a valid binding. All drifted returns now go through one helper that attaches the block, so a fourth path cannot forget it. The hold still dropped out of the GitOps projection on the second tick. The runtime observation stamps a stage on every check, and the observation stage maps to no runtime status, so it silently replaced the hold's stage: the deployment row read held while the Drift panel and the attention queue said nothing was wrong. Evidence does not supersede a decision, so recording the observed artifact now leaves a Blueprint observation stage standing until a decision changes it. * fix(gitops): report the drift mode the Blueprint is configured with Making drift_mode non-operational meant a mode-only edit mints no intent, so the intent's runtime_drift_policy snapshot went stale at exactly the moment an operator changed it. The reconciler reads the live Blueprint, so behaviour was right, but the drift surfaces and the next rollout's frozen record reported the old mode, and an explanation of what Sencho will do here that names the wrong mode is worse than none. Intents are immutable revisions, so this is not fixed by rewriting the snapshot. The projection and the freeze read the live Blueprint and fall back to the snapshot only when the application is not a Blueprint. * test(blueprints): cover the held row's status, dot and Withdraw control The catalog's priority order, its dot colour and the deployment table's Withdraw control were all changed for the held status without a render test, so a later edit could drop the status from any of the three and only the types would notice. The tile's status dot is asserted specifically rather than by "any warning class": the tile carries other dots, and an assertion loose enough to match the wrong one would have passed on a tile that showed nothing useful. * chore: drop the leftovers this work left behind, and correct its docs A dead best-effort observation helper from the earlier hold-first design, a count computed and never rendered, a typo, and a doc block that had drifted onto the wrong method and still claimed the drift was recorded when it is not. The docs also made four claims the code does not meet. A held row shows the reason the repair was declined, not the drift, so the Drift panel is where the drift is read. A row held under Enforce keeps reading held after a switch to Observe or Suggest while the drift cannot be classified. And on an Inline Blueprint a mode change is still an edit to the Blueprint, so it re-enters Awaiting confirmation before the new mode takes effect. * fix: settle cleared drift states and keep holds on unclassifiable checks Two ways a hold or a cleared state could be lost on a later tick. The observation write preserved every Blueprint decision stage, because the projection reads the target's latest stage ahead of the pointer path. That correctly protected a hold, but it also protected a cleared drift, which records that the drift is gone and projects to deployed-health-pending. Nothing moves that stage on, so every recovered target stayed health-pending for good and the portfolio reported a healthy application as in progress until the next rollout. The preserved set is now the stages that record a decision someone still has to act on. Evidence may supersede a cleared drift; it may not supersede a hold. Separately, the returns that cannot classify the target dropped the repair block, while the reconciler records its hold off an unverified result too. A held target therefore lost its hold for exactly as long as the node stayed unobservable, which is the state a hold most needs to cover. Both result builders are hoisted above the try so the catch reaches them, and every return made after the binding is resolved now routes through them. The two returns before it carry no block, because there is nothing to carry yet. Also corrects two assertions that proved less than they claimed. The Inline hold was pinned to the digest repair path, which an Inline Blueprint never takes; it now asserts on the binding the repair dispatch actually consults. The no-new-decisions check counted every history row under a loose bound, where the evidence written each tick drowned out the thing being measured; it now counts the decision stages and expects none. Each change has a test that fails with it reverted. * test: cover the live drift mode reported on a placement drift item The reported drift policy is read from the Blueprint row rather than the mode frozen in the intent, because a mode-only edit mints no intent by design, so the snapshot goes stale while the operator's choice does not. That was already fixed and had no test: the derive suite never seeded a Blueprint row, so the live read always fell back to the intent and any test of it would have passed with the stale read restored. This leaves the intent snapshot on one mode and the Blueprint on another, so reading the wrong one is visible. The neighbouring placement-drift fixture is now shared between the two tests, which leaves the difference between them as the one thing each has to say. * fix: keep a marker that cannot prove the repair authority out of unverified holds The previous commit attached the repair block to every return the drift check makes, which is right for the returns that lead to a write and wrong for the ones that do not. Two different things suppress a repair, and only one of them is a decision. A binding hold says Sencho must not write to this target at all, whatever the check can or cannot classify. A marker that names no generation, or a different one, says only that a repair could not prove what it would overwrite, and nothing about whether the workload is drifted. The reconciler's only use of the block on a result it could not classify is recording a hold and raising an alert, so attaching the marker case there reported every target whose marker predates the generation fields as Repair held on any tick the check could not classify, and announced a declined auto-fix that was never due. It flapped too: a transient unreachable node wrote a hold and an alert, the next healthy tick cleared it, and the next blip alerted again. They are now separate. A drifted result carries whichever applies, because a repair is about to be attempted and the mutation sites trust that result. An unverified result carries the binding hold alone. Two tests pin it, one for each way a check fails to classify on a legacy marker: an unreachable node, and a running image that is a local build with no comparable digest. The second never recovers on its own, so without this it would hold the target for good. Both assert the deployment stays active, that nothing is written to the node, and that no repair-held alert fires. Also converts one test's direct assignment on the service singleton into a spy. A direct assignment outlives the test and would have left every later test in the file looking at an unreachable node. |
||
|
Anso
|
ab34e85b86 |
feat(gitops): independent source, placement, and rollout authority policies (#2020)
* feat(gitops): add the versioned authority policy contract
Source acceptance, placement approval, and rollout authorization each get
their own value in one closed, versioned snapshot, so no reader can reach a
single value that looks like it governs all three.
Placement policy and rollout authorization policy are per application, each
with its own CHECK. Both default to the safe answer on a new install: an
operator approves placement, and rollout authorization waits.
Existing installs are a different case and the migration is the delicate part.
Blueprint applications already authorized their own rollouts on the acceptance
handoff, so leaving rollout authorization at its manual default would strip
that behavior from installs that never asked for the change. The backfill
restores it, scoped to live Blueprint applications, and runs only on the boot
that created the column. Every additive column here runs on every start, so an
ungated backfill would re-apply on every restart and quietly re-grant
authority an operator had turned off.
A rollout generation freezes the snapshot it executes, in a nullable column
with no default. NULL is what makes absence readable: a row predating the
column reconstructs as the behavior it actually had, which for a generation
that authorized itself was automatic, not the fresh-install default. An
empty-object default would make absence impossible and turn every existing
generation into a decode failure.
Decoding fails closed on anything present but unreadable, and refuses an
unrecognized key rather than dropping it, because a fourth key means the
writer and the reader disagree about what governs.
* feat(gitops): bound automatic placement to one stateless change
Placement policy decides whether a change in Blueprint target intent needs an
operator. Under bounded auto the only approved shapes are one stateless
addition or one stateless removal, and every other case falls back to explicit
operator review with a named reason rather than a silent skip.
The baseline is the last approved target set, never the previous candidate. An
intent revision clears the placement approval before a new candidate opens, so
the only set still in hand at decision time is a superseded one nobody
approved. Diffing against it lets a pending candidate absorb a later edit: a
candidate asking for one set sits awaiting review, a selector change mints a
larger one, the difference reads as a single addition, and the approval
authorizes a node on the strength of a decision no operator made. A test pins
that case in the refusing direction, and the mutation that would reintroduce
the bug is shown to return the approval.
Statelessness is derived from the compose content that would be placed rather
than from the Blueprint's stored classification. Compose edits are refused for
a Git-managed Blueprint, so that column says what the editor last wrote and a
repository push that adds a stateful service never updates it. An absent
reading is treated as stateful, because a missing field defaulting to the one
answer that approves is the failure this policy exists to prevent.
The decision is pure and total. Node lists are normalized on the way in, since
they arrive from a JSON column the compiler never sees, so damaged evidence
produces a refusal rather than a TypeError thrown from inside a judgement made
inside somebody else's write.
Directory-name ownership is left out of the reason union on purpose. Whether a
stack directory on some node carries this Blueprint's marker is a question
about who owns a name, not about whether moving a stateless workload is safe,
and the deploy path already refuses an ownership conflict by name.
* feat(gitops): apply bounded placement decisions outside the producer write
The decision is applied after the producer's transaction commits, in a
transaction of its own. Evaluating inside the producer would nest the
single-writer transition inside the caller's write, and a refusal or a defect in
the decision code would roll back the operator's own Blueprint create, update, or
pin. Nothing in the path throws at the caller for the same reason: the Blueprint
edit already succeeded, and an unapproved candidate waits for an operator
either way, so the failure direction is safe without a rollback.
Atomicity is therefore not the mechanism. Idempotency is. An approval now
refuses when one already exists against the same intent revision, because the
currency checks only catch a stale approval: replaying the same current one
passed all of them, minted a second approval row, and superseded the generation
the first had just opened. The guard is keyed on the intent revision, which is
the precise key, since a placement approval records no candidate and an intent
opens exactly one.
Authority and the snapshot that produced it are now required together rather
than trusted separately. A policy-authorized approval that records no snapshot
cannot be reviewed later, and an operator approval claiming a policy decided it
is a mislabelled authority record; both are refused at the write.
The two policy edits are configuration, not work. Each refuses while an
operation is in flight and refuses a write to the value already set, mints no
intent or candidate, and clears no source acceptance, placement approval, or
rollout authorization. An edit decides what the next decision may do; an
approval already standing keeps running under the snapshot that authorized it.
That is why a policy change never appears as drift, and it is pinned in the
direction that matters: a fully approved application keeps every reference and
its generation byte for byte.
* fix(gitops): refuse rollout authorization while an operation is in flight
The pause check only knows about a deliberate pause, so an authorization could
be minted while a fetch, apply, deploy, or recovery was still running for the
application. The two then disagree about what the next operation acts on, and
the authorization is the record everything downstream treats as settled.
Two refusals, both with a named reason and both failing closed. The application
guard reads the application's own active operation. The target guard reads the
target rows, because a target can be mid-apply while the application has nothing
running at all: they are separate machines and only the target knows about its
own apply. Reading only the application pointer would have missed exactly that
case, which is why the second refusal exists separately rather than folded into
the first.
* feat(gitops): add the placement and rollout authorization policy writes
Two endpoints, each naming in advance what the system may do, so each is
authorized by exact deploy on every frozen target before anything is written. A
policy that half-applied would leave the fleet split across two policies, half of
which nobody chose, so a caller entitled for one target and not another is refused
outright rather than applied where it happens to be allowed.
Both require only the authority a placement approval or a rollout authorization
already needs, and neither requires a global create grant. That grant is held
only by roles that hold a global deploy grant as well, so demanding it would
silently upgrade the gate to application-wide and make the per-target loop
unreachable for every caller that got past it. The create authority belongs to
the approval the policy authorizes, which already requires it.
Before the first placement there is no target to scope an exact grant to, and a
stack-scoped assignment is node-qualified, so it cannot be resolved for a node
that does not exist yet. The application-wide grant is the only gate evaluable
in that state, and the limitation is pinned as a test rather than worked around:
a scoped operator configures the policy once a target exists.
Each write refuses a value outside its own vocabulary, a write to the value
already set, and a Direct or Inline application, which has no placement or
rollout domain to configure.
* refactor(gitops): drop the unreachable marker conflict from placement
The reason could never mean what it said. The query behind it only ever looked
at this application's own intent revisions, so it could not detect a conflict
with any other Blueprint, and the one thing it did detect, a pinned node inside
the candidate set, is already reported as pin-driven placement. A reason in a
closed union that either never fires or duplicates another advertises a check
that does not exist, which is worse than not having it.
There is no database signal to replace it with. Blueprint names are unique and
deployments are keyed by Blueprint, so whether a stack directory on some node
carries this Blueprint's marker can only be answered by probing that node, which
the deploy path already does and refuses with a named error. Content-binding
contention is refused on the convert and adopt paths instead, and placement
cannot create the contention it would have reported.
The permission tests are reframed to state what the gate is for. Before anything
is placed, a bounded_auto policy authorizes work on whatever nodes the selector
matches later and that set is not known, so the application-wide grant is the
conservative answer and a narrowly-scoped operator is correctly refused. Once
targets exist the per-target loop applies, and an operator scoped to exactly
those targets is authorized, which is now asserted in the passing direction.
* fix(gitops): address the policy review findings
Seven findings, four of them load-bearing.
The rollout authorization policy was never read. The column was stored, the
route wrote it, the migration backfilled it, and generations froze it, but no
enforcement path consulted it, so setting it to manual returned success and the
next dispatch minted anyway. A policy-authorized mint now requires a policy that
says so; an operator authorizing by hand is unaffected, because an operator is
themselves the authority.
The placement baseline was unreachable. It resolved through the application's
placement approval pointer, which an intent revision clears before a new
candidate opens, so the pointer branch never ran. Every application therefore
looked un-approved and every multi-node candidate was refused as a first
placement, which is the opposite of the stated bound of one stateless change. It
now resolves the most recent approval row, which survives the pointer and is
still the record of the last set an operator accepted.
A cordon was never consulted. The refusal existed and the evidence gathering
passed a literal false, so an automatic approval could place a workload onto a
node an operator had cordoned. The flag is now read for the nodes being added.
Node evidence is also read for removals, not only for the candidate set, and a
stale target counts as unreachable rather than as reachable, which had made the
stale-node refusal unreachable for the targets it exists for.
The migration backfill could be stranded. Gating it on the boot that created the
column meant a process dying between the ALTER and the UPDATE skipped it for
ever, because every later boot sees a duplicate column. A completion marker now
decides, so the backfill is offered on every boot and runs only once. A row an
operator moved back to manual is indistinguishable from a never-backfilled row,
which is exactly why re-running it on its own predicate would have been wrong.
An approval's snapshot and the generation it opens are now compared inside the
write transaction. The authorizing snapshot is read by the caller beforehand, so
a policy edit landing in between could leave the two records disagreeing about
one decision, under a policy the operator had already revoked. The replay guard
is keyed on the intent and candidate together, since nothing enforces one
candidate per intent, and a placement approval now records the candidate it was
made against.
A refusal keeps its own cause. Every approval failure was reported as a
conflicting operation, which sent an operator looking for work that was not
running and turned the replay guard's precise signal into a lie on the retry
path. Also: an unreadable candidate target set is logged rather than swallowed,
and the empty-target-set permission comment no longer claims to be stricter than
the per-target path when it is the same shape as the health policy write beside
it.
The GitOps suite passes apart from two files that fail identically on an
unmodified checkout, one on a filesystem exhaustion this host is hitting and one
on a cold-start timeout. Typecheck and lint are clean.
* fix(gitops): report the unmet rollout prerequisite, not the policy
The rollout authorization policy gate ran before the authorization ingredients,
so it short-circuited with its own reason before the ingredient block could say
what was actually missing. An application with no placement approval was
reported as needing an operator rather than as needing a placement approval.
The gate answers "may policy authorize this, or must a human", which has no
meaning until there is something to authorize, so it now runs after the
ingredients pass and before the artifact qualification check.
`authorizationIngredients` is a pure read with no side effects, so nothing
mutates between the two positions and the only behaviour that changes is which
reason is reported.
This was the same class of misdiagnosis the previous review round flagged, where
every approval failure was relabelled with one fixed reason. The prerequisite
that is missing is the more specific and more actionable fact, and reporting
anything else sends an operator looking in the wrong place.
The operator path is unaffected either way: an operator asking to authorize is
themselves the authority, and no policy stands in the way of one.
Verified: the previously failing assertion returns to the placement reason, the
two suites covering the gate pass, and removing the gate from its new position
still fails the suite, so the move did not make it unreachable.
* fix(gitops): make bounded automatic placement reachable
Automatic placement could never approve anything, and never had. Two defects
compounded, and the first one hid the second.
Node evidence was read from the target row. An added node has no target row and
never will until the placement lands, so every addition resolved to unknown and
was refused. Nothing writes connectivity on a target row either: the only
constructor seeds it to null, so the column is null for every node for ever. The
pure decision was exhaustively correct and completely unreachable.
The effect was also hand-assembled as a JSON object where the schema wants an
array, so the transition's own decode rejected every approval. Because that
refusal is the safe direction it surfaced only as an operator review, which is
how a feature can look wired up end to end while never once approving anything.
Each side of a change is now judged by the evidence that actually exists for it.
A node being added is known from the node registry, which is maintained rather
than seeded, and is refused when it is unregistered or not answering. A node
being left is judged by the observation of the workload that ran there, where a
stale observation does not count as reachable. Cordoned nodes are refused for
additions, which the previous code could detect but never supplied.
The eight new tests drive the real path rather than a hand-built input: a
stateless addition reaches auto_approved and records the snapshot that decided
it, and each refusal is reachable from real state. Two of them fail if the node
evidence regresses to the target row, and two fail if the blast encoding
regresses, so neither defect can come back silently.
* test(gitops): drop an unused binding from the placement end-to-end suite
* fix(gitops): do not re-decide an authorization that already exists
An operator authorized a rollout, got a success response, and nothing deployed.
The mint succeeded because an operator is the authority. The dispatch that
follows re-enters the same function on the automatic path, where the policy gate
saw the fresh-install default of manual and refused, so the policy governed the
execution of authority that had already been granted.
The policy governs whether authority may be created, not what may be done with
it. A grant is not withdrawn by a later configuration change, and it should not
be re-litigated on every dispatch. The same reasoning applies to the in-flight
guard: it exists so an authorization is not minted while a fetch, apply, deploy,
or recovery is running, and a routine background source fetch must not pause a
rollout that is already authorized and running.
Both gates now stand down when a live binding already exists, using the existing
live-binding helper rather than a new notion of what counts as authorized. The
preflight evaluation and the drift remint still run either way, since that is
this function's other job and it is about evidence rather than authority.
The gates keep their teeth for the case they exist for: with nothing granted,
the policy is the only thing that could grant authority, and on manual it
refuses. Both directions are asserted, for the application-level and the
target-level in-flight case, and each fails if the live-authorization bypass is
removed.
* fix(gitops): let an operator re-approve a placement on the same intent
The replay guard refused any second placement approval for an intent and
candidate that already had one. That is correct for the automatic path, where a
duplicate is a replay, and wrong for an operator.
The Inline Apply path reuses the application's current intent rather than minting
a new one, so an operator pressing Apply again lands on exactly the same intent
and candidate. The guard refused it as a replay and regressed a flow that had
worked for ever, reporting it as a duplicate decision when it was a deliberate
second one.
The guard is now scoped to policy-authorized approvals, which is the only caller
that can produce a duplicate by being invoked twice. The operator path is
unrestricted, and the automatic path keeps its protection.
The test that previously asserted the refusal now asserts the operator's second
approval succeeds, and a separate policy-authorized case keeps the guard
load-bearing. Widening the guard back to all authorities fails two tests, and
removing it fails one.
* fix(gitops): enforce the mint-time gates in the single writer
The previous fix let a live authorization bypass the policy and in-flight gates,
which was right for executing authority that already exists and wrong for
creating it. The distinction I drew was between a binding existing and a mint
happening, and I enforced it on the former.
So the remint path was ungated. When the preflight fingerprint drifts, the old
authorization is discarded and a new one minted, which is a fresh grant of
authority, and on a manual policy that mint now succeeds with
authority: configured_policy and no operator anywhere. A registry tag moving was
enough to replace an operator's grant with a policy-authorized one, which is the
precise thing the gate exists to prevent. The race-retry branch had the same
hole.
Both gates now sit in rolloutAuthorized, the single writer every mint path goes
through: the acceptance handoff, the Blueprint dispatch, the startup
reconstruction, the preflight backfill, the operator route, the drift remint and
the race retry. A caller-side check cannot be unbypassable; this can, because
there is nowhere else to mint from. That also closes the separate finding that
the policy was not re-checked at mint time, since the re-check is now the gate
itself.
The caller-side gates stay where they are, because they still give the better
reason for the ordinary case: a missing prerequisite is reported as itself, and
a pause is reported as a pause, before anything reaches the transition.
Two tests assert the property where it now lives, that any policy-authorized
mint reaching the transition on a manual policy is refused, and the same
mid-operation, with the operator path shown to be unaffected. Removing either
gate fails one.
* fix(blueprints): refuse to call unresolvable compose stateless
Stateful service detection is the evidence an automatic placement decision rests
on: a service it does not name is one the system believes carries no data. Four
compose constructs attach a mount to a service whose own block mentions none,
and the detector returned an empty set for all four, which is the permissive
answer.
The merge key is the sharpest case, and worse than a simple blind spot. The
anchored service is found, so a partial answer named it and reported the service
that merges it as stateless, attributing the volume to the wrong service while
the workload that actually inherits it read clean. `include` and `extends` name
another file. `volumes_from` attaches another container's volumes, which is
directly data-bearing.
A document with no services now also returns null. Nothing to place is not the
same as proven stateless, and a caller deciding whether a placement is safe
should not have to tell those two apart.
There is deliberately no partial answer. A set naming only the services the
detector could see reads as permission for the ones it could not, so "cannot
prove" and "is stateless" have to stay different answers. Both callers already
hold on null: the stateful withdrawal guard treats it as unreadable, and the
placement decision treats it as an unknown workload, which is a refusal.
Resolvable content is unchanged, including a tmpfs mount, which is not data the
workload owns, and is still not a stateful service.
* docs(gitops): put two doc comments back on the code they describe
The health-and-rollout policy route's own doc block was left stranded above the
policy-target gate, a hundred lines from the route it describes, so a reader
arriving at either place found the wrong explanation. A helper inserted between
the rollout authorization doc block and its function had the same effect there.
One of the stranded claims was also false by the time it was found. It described
the empty-target-set branch as authorizing against the Blueprint's deploy stack
name; the branch falls back to the application-wide grant, because a stack-scoped
assignment is node-qualified and cannot resolve for a node that does not exist
yet. The explanation now says what the code does, including the consequence an
operator needs to know, which is that configuring a policy on a fresh application
is an administrator action.
No behavior change.
* fix(gitops): let a demoted Blueprint still have its policies set
Demoting a Blueprint back to Inline keeps every policy column, and the placement
decision does not read the target mode, so a `bounded_auto` application keeps
approving its own stateless placement changes after the demotion. Until now no
route could stop that: all three policy writes resolved their target through the
same gate the source and rollout actions use, which requires Git management, and
answered NOT_GIT_MANAGED for a demoted application. The only way back was to
edit the row directly.
Resetting the policy on demotion would be the wrong fix. Where content comes
from and who decides a placement are separate domains, and coupling them here is
the entanglement this model exists to remove. A policy is a statement about who
decides, so a source-domain condition has no business gating it.
The shared resolver is now split by what it actually requires. Actions that act
on Git content, or on a rollout Git produced, keep the Git-management gate: source
acceptance, placement approval, and the rollout lifecycle. The three policy
writes resolve without it, which is the same shape the health policy already had
in spirit and never had in practice.
Widen which applications a policy write reaches, not who may write one, so the
permission behaviour is unchanged and pinned again here: a caller without the
deploy grant is still refused, and a Direct application, which has no policy
domain at all, is still NOT_GIT_MANAGED.
* feat(gitops): project each authority policy and why it declined
The three policies could be configured and enforced but not read. An operator
looking at a placement waiting for review under an automatic policy had no way to
learn that the policy had run, or why it left the decision to them, and a policy
that never fires looks identical to one that is working. That is the same
failure shape as the rest of this branch: a feature that reads as wired up while
reporting nothing.
The projection now carries one read per domain, in stage order, each answering
the two questions an operator actually has: what is configured now, and what the
work in flight was really decided under. The second is read from the snapshot
the generation froze, never from the current setting, because a policy edited
afterwards must not appear to have authorized work that is already running. It is
reported as absent whenever no readable frozen snapshot exists, which includes
every generation predating the policy contract.
The refusal itself had to become readable first, because the decision was
computed, returned to a producer and discarded. It is now recorded on the
application row, in the closed decision vocabulary, and constrained by a check so
a reason no decision can produce cannot be stored or projected. Cleared inside
the two transitions that resolve it, an approval and a policy change, so a reason
that described the old policy cannot be read as the current one.
The refusal is written with the operation's own timestamp, not the wall clock, so
its freshness means the same thing as every other recorded time on the row.
Also fixes a trap this change walked into: the application insert hardcoded its
placeholder count, so adding a column to the list and not the count compiled
cleanly and failed at runtime as a SQLite arity error on the first insert, which
is the first write any create path makes. The count is now derived from the
column list it belongs to.
* feat(gitops): show and set each authority policy where it is decided
The policies could be configured only by editing a row, and the two new ones had
no affordance at all. An operator could not see who was configured to decide a
stage, and could not stop automatic placement once it was on.
Policy reads join the facet cards. The four cards already render source, artifact,
placement and rollout in a fixed order and are shared between the application
view and the rollout preview, so the same evidence cannot read two different ways
on the two surfaces. Each card that governs a decision carries that decision's
policy line; the artifact card carries none, because it reports what was built
rather than who may act. The line says two things and stops: who is configured
now, and what that policy actually did. The second is a separate fact, because a
policy edited after a generation opened does not retroactively claim it, and a
line showing only the configured value would read as though it had. A card with
no decision and no reason shows nothing, so the line is never a restatement of
the card's own state.
Policy edits sit beside the action each one governs: placement policy beside the
placement approval, rollout authorization policy in the rollout overflow where
the health policy already lives. Not on the GitOps Workplace, which is triage and
would gain a second job. One control and one label map serve both, so three
vocabularies cannot appear in three places. The control is absent rather than
disabled for a session that may not write, matching every other gated affordance
here, and it names a value this build does not recognize instead of snapping to a
segment that would overwrite whatever the server is enforcing.
The source policy's Manual mode finally has an option. It was in the contract
since the policy columns landed and the panel could not reach it, because the
panel derived the mode from two booleans that cannot express it: a manual source
and a review source both report `auto_apply_on_webhook: false`. The policy field
was already on the wire and simply undeclared, so the fix is to read it. Manual
is not a flavour of review either. It takes the source out of the unattended
cadence: no polling, no webhook. Both the panel and the create dialog now offer
it and save the automation flags that agree with it, so choosing it does not leave
a source that still applies on its own.
The apply-mode options also state their selection, which was carried only by a
border and a fill and so was invisible to a screen reader.
* fix(gitops): hold a policy write to the authority it names
The placement and rollout authorization policy writes were gated more loosely than
the health policy write beside them: exact deploy per target, and the
application-wide grant only when there was no target to check. A stack-scoped
operator could therefore set either policy to automatic.
That is a real widening rather than a theoretical one. A policy governs every
target of the application, including nodes that do not exist yet, so one
operator's grant on a single stack would authorize placements and rollouts
elsewhere in the fleet. The per-target loop does not close it, because it only
checks the nodes present today.
Both halves are now required, the same requirement the health policy write
already carries and the same one recorded for it in the permission inventory. The
application-wide half is the one a scoped operator can fail, which is exactly why
it belongs on the path. Configuring a policy on a fresh application, before any
target exists, remains an administrator action: there is no per-target check to
make and a stack-scoped assignment is node-qualified so it cannot resolve for a
node that does not exist yet.
The behavior change is pinned from both sides: a stack-scoped operator is refused
on both routes, and a role-based deployer still passes, so tightening the gate
does not lock out the operator it is meant to admit.
Also records the surface and the model. The authority policy model, its single
writer, the bounded automatic placement decision, why the refusal has to be
durable, and what the projection refuses to claim about work in flight are now
written down rather than only living in the code. Alongside it: the two new
service modules and the shared control in the inventories, the two policy routes
in the permission inventory, the new endpoints in the route inventory, and the
architecture map with a flow for the write and the HTML re-inlined to match.
Operator documentation: the source policy's Manual mode, which the panel could not
reach because the two booleans it derives from cannot express it; who decides each
of the three steps, and the whole of what the automatic placement policy may do on
its own; the policy-domains table, which still described placement as a per-rollout
approval rather than a configured policy.
* fix(gitops): do not let a cordon withdraw a workload
Cordoning a node removed it from the Blueprint's desired set, and a single
stateless removal is inside what the bounded automatic policy may approve on its
own. So one cordon silently stopped the workload running on that node, and the
placement policy read as broken rather than as having drained a node.
A cordon says where work may not go. Taking work away from a node is a separate
decision, and the operator who cordoned a node has not necessarily made it. The
cordon is now read for removals as well as additions, and a withdrawal whose only
cause is a cordon falls back to the operator with its own reason.
The reason is separate from the existing cordon override because the two ask
opposite questions of the operator. That one says Sencho would have placed work
on a node you told it to leave alone. This one says the withdrawal you are seeing
follows from the cordon you set. The new reason joins the closed decision
vocabulary, which constrains both the stored column and the wire, so no decision
can produce a value that cannot be stored or projected.
Reading the cordon for removals is not a blanket refusal of removals, and that is
pinned from both sides: a withdrawal of a node that is not cordoned is still
approved. The pure decision test passes the evidence in directly and so cannot
catch a caller that sets it for every removal, which is why the approving case
lives at the level that reads the cordon.
Also corrects two comments that claimed a guarantee their callers defeat.
`recordPlacementShift` said each transition commits on its own, and its
automatic-placement loop said the triggering change had already committed. Both
callers wrap it in a transaction, so neither is true. The behavior is fine and
arguably better, because the decision rolls back atomically with the write that
caused it, and every refusal is caught rather than propagated. What was wrong was
the claim, and a comment that asserts a guarantee the code does not provide is
worse than none.
* test(gitops): record no policy snapshot on a seeded rollout generation
Two test files build a rollout generation row inline, and this branch added a
column to that row, so the merged tree failed the typecheck in both. The fix is
to say what those rows actually are: generations with no recorded policy
snapshot, which is what a generation predating the policy contract looks like.
Null is the honest value rather than a stand-in for something. Both files are
about which generation a target may restore or which one it acknowledged, and
neither is about policy provenance, so recording a snapshot on their rows would
assert a fact the tests have no evidence for and would let a later reader infer
that these rows were minted under a policy.
Test fixtures only. No production code and no behavior change.
* fix(gitops): a migrated Blueprint is not a first placement
The placement baseline falls back to the running target rows when there is no
decomposed placement approval, and reported that it had never been placed. A
migration that carries an approval forward records a pre-decomposition row
instead, which names no target set and so cannot supply a baseline but does prove
the placement was not the first thing that ever happened to that application.
So a migrated Blueprint running on three nodes whose selector narrowed to two was
refused as a first placement across more than one node, for ever, on exactly the
installations the migration was written for. The refusal is the safe direction,
which is why it read as a policy that quietly does nothing.
The two questions are now separated. What the placement was authorized to be is
answered by the decomposed row, and what the running targets are is the baseline
when there is no such row. Whether the application was ever placed under an
operator's authority also accepts the carried-forward row, and only that row, so
a genuinely fresh application is still a first placement. That last half is
pinned directly on the reader, because asserting it through a refusal would have
several other possible causes.
Also moves the rollout authorization backfill's completion marker out of
`global_settings` and into `system_state`. The marker is internal migration
progress, not a setting, and `global_settings` is the allowlisted table the
settings route projects.
The backfill's lifecycle predicate is left exactly as it is, and the reason is
now written down rather than left to be re-derived. A `blueprint` mode row is
never `creating`: a Blueprint application is inserted as `inline_blueprint` and is
`active` from the start, and the transitions that write `creating` are the Direct
create and activate paths, so the predicate excludes nothing today. It is also the
right line if that ever changes, because a Blueprint application still being
created never reached the acceptance handoff and so never auto-authorized.
Granting it `automatic` there would be a privilege grant rather than the behavior
preservation this backfill exists to perform.
* fix(gitops): keep the two compose readings apart, and see an in-flight target
Two release-blocking defects, both found by the pre-merge audit and both mine.
**The stricter compose reading broke automatic source acceptance.** Tightening
stateful detection so an unresolvable compose answers unknown instead of naming
the services it could see was applied to the shared helper. The only other caller
is the stateful withdrawal guard, which treats unknown as unreadable and compares
both the staged candidate and the generation in force, so a compose using a
merge key, an extends, an include, or an override file with no services was held
for ever. Those are ordinary compose idioms, and an install using one stopped
applying updates automatically.
The two consumers need opposite answers to the same document, so they are now
separate entry points rather than one method with a flag. The guard gets the
best-effort reading it has always had, and the placement path gets a strict one
that refuses. A caller about to act on the result must never be handed a partial
set, because a set naming only the services the detector could see reads as
permission for the ones it could not.
**The bounded-auto in-flight check could not see a rollout.** It read only the
application's own operation pointer. A sequential rollout sets its stage on the
target, so a label change during a mid-deploy rollout was approved, cleared the
authorization, and superseded the generation that deploy was running under. The
target-side reader already existed in the rollout handoff, so it is now shared
rather than re-derived.
**Placement content was read from the wrong directory.** Generations are staged
under the application's source stack, not under the Blueprint's name, so a
renamed stack found nothing and every placement on one read as an unknown
workload. And multiple compose files were concatenated into a single document,
which duplicates every service key and does not parse, so a multi-file project
could never be approved either. Each file is now analyzed on its own, which is
what the withdrawal guard has always done.
The end-to-end suite staged its content under the Blueprint's name, which is the
directory the source path never writes to. That is why the wrong path was not
caught: the test agreed with the implementation. It now stages where generations
actually live, and covers a renamed stack, a multi-file project, a multi-file
project whose second file is stateful, and an addition during a target deploy.
* fix(gitops): say what the policies actually did
The projection's claims are the part an operator acts on, and three of them were
wrong in the direction that makes a working feature look broken.
**The default policy reported itself as declining.** Every placement on the
operator policy recorded a `policy_is_operator` refusal, and the projection read a
recorded reason as "declined by policy, waiting on an operator". That policy never
runs, so it has no decision to decline with, and the default application on the
default policy said the system was withholding something. The refusal is no longer
recorded for that reason, and the projection's existing distinction between a
policy that declined and one that has not run now has something to distinguish.
**Source acceptance was always reported as an operator decision**, because the
authority was hardcoded rather than read from the approval. An operator on the
automatic policy was told they had personally accepted a revision nobody showed
them. It is derived now, from the same field every other read uses.
**A generation predating the policy contract reported a frozen policy that never
existed.** The decoder reconstructed the fresh-install defaults for a null
snapshot, so a rollout that ran before these settings were introduced claimed it
had been decided under a policy that did not exist at the time. The module header
already said absence is absence; the decoder was doing the opposite.
**A demoted Blueprint had a live automatic decision and no control.** The
backend accepts the policy write for it, and the placement decision evaluates it,
but the control sat behind a target-mode gate, so an operator whose application
decided its own placement had no way to reach the setting from the interface. Both
Blueprint modes now reach the block, the Git lifecycle actions stay Git-only
because a demoted application has no Git generation to authorize a rollout of, and
the view test that asserted the old behavior is updated with the reason.
**A policy change wrote no before and after.** The application snapshot both
history rows were built from omitted the three policy columns, so the audit trail
recorded that a policy changed without saying which way, and that a recorded
refusal had been cleared without recording the policy that produced it. Both are
in the delta now, pinned on the history row.
**The placement writer did not check its own domain.** `rolloutAuthorized` refuses
a policy-authorized mint unless the rollout authorization policy actually permits
one; `placementApproved` had no equivalent, so a caller could reach the single
writer with a policy-authorized approval and a snapshot that happened to agree.
Agreement by accident is not authority, and the domain rule now cannot be
satisfied by a caller that simply passes a matching snapshot.
**The rollout default after the upgrade is now stated.** The one-time backfill
restores automatic authorization for the live Blueprint applications that already
had it, and everything created afterwards waits for an operator: a newly adopted
Blueprint, one converted back and forward, and any row written by an older binary
during a downgrade. That is a real behavior change for those applications and it
was only explained in a code comment. It is now written where an operator looks.
Also corrects a route docstring that claimed an exact per-target check while
describing the wider gate, and drops the control's dead permission conditionals,
which were unreachable because the control returns nothing without the grant.
One audit finding was checked and rejected: the approval snapshot decoder is used
by the end-to-end suite, so it stays.
* fix(gitops): read the generation in force, not the candidate
The bounded-automatic policy worked only between source acceptance and the first
rollout. Every placement after that read as an unknown workload and went to
operator review.
Promotion removes the applied directory and renames the candidate directory over
it, so a promoted generation has an applied directory and no candidate directory
at all. The placement decision resolved only the candidate, found nothing there,
and treated the absence as unknown evidence. The withdrawal guard already had the
right reader for this: it reads the applied copy first and falls back to the
candidate for a generation that is accepted but not yet promoted. That is the
generation a target would actually receive, so it is the one the policy decision
rests on, and it is now shared rather than duplicated.
The end-to-end fixture staged a candidate directory for every case, which is a
generation that has been accepted and not yet promoted. That is the narrowest
window in the lifecycle and the one an install spends least time in, so the suite
described a state that a working install is almost never in and the reader was
wrong for every generation that mattered. The fixture can now stage a generation
as promoted: content under the applied directory, nothing at the path its row
still names. Three cases cover it, including one where both directories exist and
the applied copy is the stateful one, because the applied copy is what a target
would receive and the unpromoted candidate is what no target is running.
**The strict compose reader missed a merge key below the service.** It checked for
`<<` only directly under a service, so `services: { <<: *shared }` merged whole
services in past the check, and `volumes: [{ <<: *vol }]` attached a mount the
volume walk never visits. Both read as stateless and would have been placed
automatically. A merge key is now looked for at any depth. An alias is not a
merge key and still answers, because the parser resolves it and refusing every
anchored document would hold ordinary compose files for no safety gain.
**The operator-policy regression test could not fail.** It inserted an application
with no intent and no candidate, so the call skipped before any decision ran and
its assertions on the refusal columns held whatever the guard did. The case that
does reach the decision now carries those assertions, and deleting the guard
fails it.
Also reconciles three doc comments left doubled by successive edits, and keeps the
`policy_is_operator` reason text with the reason it stays: a node on an earlier
build still records that reason, and this dashboard reads the reason from whichever
node owns the application, so during an upgrade the raw enum would otherwise reach
an operator.
|
||
|
Anso
|
d87746ce37 |
fix: keep Restore toasts when overlapping silent service gates begin (#1947)
* fix: keep Restore toasts when overlapping silent service gates begin When Deploy Progress is off or dismissed during a service-scoped update, a silent health-gate watch runs in the background. Starting another service update previously stopped that watch, so a later gate failure never offered Restore. List active stack recoveries when a new session starts, re-surface already-failed siblings as Restore toasts, and keep polling siblings that are still observing. * fix(deploy): offer Restore only for the newest recovery per service The /recoveries endpoint returned every active recovery row for a stack, so a superseded failed row could surface a "health gate failed" toast whose Restore rolled a now-healthy service back to a stale snapshot. Return only the newest active row per service, dedup defensively on the client, and guard the sibling-recovery fetch against a newer session starting mid-flight. * fix(deploy): keep a Restore offer alive across every overlapping update The sibling recovery watch belonged to the deploy session, so several ordinary paths still dropped the first service's Restore offer: an overlapping update that failed or returned no health gate, a panel close after the second gate settled, an update on another stack, and a click on a sibling's own Restore action. Sibling watches are now owned by (node, stack) rather than by the session that discovered them, so they end only when every service they owe an offer to has settled, expired, or disappeared. A run that takes the single primary gate slot hands the gate it replaces to its stack watch first, and hands the service back to the watch when that gate gives up, so no observing recovery is left without a poller. A failed recoveries read now reads as a failure rather than as an empty list, and is retried within a bounded budget instead of cancelling every watch on the first blip. A stack-scoped deploy no longer re-surfaces service snapshots, and stops the stack's watches, since it replaced the images those snapshots roll back to; a stack-scoped run that failed leaves them alone. One snapshot is offered once per session whichever path finds it, and a restore that lands without a new gate clears only the record it came from. The stack query also stops falling back to an older recovery row while a newer one for the same service is mid-restore, which would have offered a second, stale rollback for a service already being replaced. * fix(recover): stop a stack runtime change from leaving a stale Restore offer A stack-scoped deploy or update replaced every service's image but left the stack's active service recovery rows offerable, so a later service update re-surfaced a pre-deploy snapshot and offered a rollback behind the deploy the operator had just run. Every runtime change passes through one of two commits, the deploy and update success paths, so a successful one now retires those rows; the stack rollback route runs its own compose up and retires them there. A failed run retires nothing, and retiring never fails the operation that already succeeded. A deliberate container lifecycle change has the same shape of problem from the other end: a gate still observing a container that is being stopped, restarted, or removed has nothing left to report, and left running it fails seconds later with the container missing, which reads as a failed update and offers a Restore for a service the operator just stopped on purpose. Those operations now end the gates they invalidate, recorded as unknown with the operation named. The seam is the compose command itself, so the routes, the scheduler, and the webhook are all covered, and a per-service stop ends only that service's gate. On the client, a stack-scoped run that ends or replaces the stack's containers now also ends that stack's watches, and the retry budget for reading recoveries became a window rather than a count: four strikes was about sixteen seconds, shorter than the rate limiter's window, so one burst of rejections cancelled every pending offer. The budget now restarts only on a successful read, and a run replaced while still in flight hands the gate it returns to its stack watch. * fix(gate): end an invalidated observation before the operation, not after A stop, restart or take down ended the gates it invalidated only once the operation had returned. Compose stops containers one at a time against a ten second Docker timeout, and a gate polls every five, so an observation could still record the deliberate action as a failed update and offer a Restore that undid it. Every seam now ends the gates first: the compose commands, the shared stack container op, and the per-service op, plus the teardown down that removes the containers a stack deletion had under observation. Resolving the compose arguments before ending anything keeps a refused operation from costing a live observation, and the label bulk action now skips a stack with no containers for the same reason. Any deliberate container change compromises every gate on that stack, not just the gate of the service that moved: the sibling gates carried that container in their baseline, so leaving them to fail on a container the operator removed on purpose produced a false failure and a false failed-gate history entry. Ending them is quieter and honest. For the same reason the client no longer offers a service-scoped Restore when a gate failed collaterally: that service is the healthy one, so its snapshot is not the way back. A rollback that replaced the runtime and then failed its recovery probe settled nothing, leaving gates running and snapshots offered; it settles now, like the success path. And a retired recovery row now holds its image from prune until the row's own TTL expires, so the prune that runs right behind an update cannot delete the only known-good image the stack was on. * fix(gate): let a rollback keep the recovery run it arms, and keep offering Restore on a collateral failure A generation rollback ended the gates from earlier runs only after the restore had landed, but the restore arms its own recovery observation. That supersede finalized the run it had just armed as unknown, so a GitOps-managed rollback could never be verified healthy. The gates now end before the restore runs. Hiding the Restore action when a gate failed collaterally removed a recovery path that already existed. The role is assigned by whichever container failed first, not by which one is actually broken, so a collateral failure can be the broken service's only signal, and a service update creates no stack generation for any other screen to offer that rollback from. The offer is back in both the toast and the modal. A single-service stop or restart ends that service's gate and the stack gate, and leaves a sibling's gate observing its own runtime, which is the same rule a service-scoped update or restore uses. Ending every gate on the stack would record an untouched service as unknown and release its auto-heal suppression. * test(rollback): drop the leftover response binding from the gate-ordering test * fix(gate): detach a deliberately operated service from a sibling gate instead of failing it The rule for which gates a container operation ends was never written down, and each attempt to fix it moved the failure: ending every gate recorded a healthy untouched service as unknown, and ending only the operated service's gate let a sibling fail on a container the operator moved on purpose, which offers a Restore for a service that never broke. supersedeForContainerOp now states the rule as a table by operation and gate kind and applies it, with a third outcome: a sibling gate is detached, so the operated service's containers leave its expected, collateral, baseline and role sets and it keeps a verdict on its own runtime. The same detach applies to a prepare that has not begun yet, which is the compose up's own window. A scheduled restart names every service it restarts, so it ends and detaches all of them rather than the first. A refused rollback changes nothing and no longer costs a live gate: the supersede moved inside the compose-up callback compensate hands it, which runs after the eligibility and policy checks and before the restore arms its own recovery observation, so a 409 leaves the gates alone and a permitted rollback still ends the ones it invalidates in time. The deploy path ends the gates after its compose arguments and registry auth have resolved rather than before them, and the two identical gate-ending helpers in ComposeService are now one. * test(deploy): pin that a deploy ends the gates before compose can replace a container * fix(gate): cover the compose and prepare paths the invalidation table left untested A stack-wide operation replaced every service's containers but only ended the armed gates, so a prepare that had not begun yet was still there to arm itself from the pre-operation baseline, and the gate it opened then failed on the services that are now deliberately down. Its tokens are dropped now, like the operated service's token already was. A dropped token is remembered until the token's own TTL, so the begin that follows can say the observation was ended on purpose and the service update can report that instead of reporting success with nothing watching it. A token this process never issued is still reported as unknown. Both stack-scoped commit points now have ordering tests, and a test each for the two ways they can be refused before the compose up, which is the behaviour moving the gate end after argument and auth resolution actually introduced. The refusal tests fail the resolution that belongs to the compose up itself, not the one before it, because an update resolves its pull first: without that they would pass under either placement. |
||
|
Anso
|
40aca5a6ec |
docs(editor): record the mixed-version outcome for Save & Pull Images (#2027)
A control instance on a newer release can send the compose editor's Save & Pull Images action to a node whose Sencho instance has no such route. The node answers 404 with its own unmatched-/api body, the remote proxy passes that through untouched, and the editor reports the image pull as failed after the compose file has already been written. No image is downloaded and no running container changes, but nothing recorded that, so an operator running a mixed-version fleet had to infer it. Document the outcome on the editor page, in Save options and as a troubleshooting entry, and record on the node compatibility page that an action added by a later release is refused rather than capability-gated. The alternative is a capability gate, which is a wire change and belongs in a consistency pass across the other node-scoped actions rather than on its own. The added test pins the exact toast, so the sentence the docs now state cannot drift away from the behavior without a test failing. |
||
|
Anso
|
f0906f86d6 |
feat(editor): add Save & Pull Images to the compose editor (#1948)
* feat(editor): add Save & Pull Images to the compose editor
Image acquisition only happened inside deploy and update, so warming the
image cache before a maintenance window, or staging an update to apply
later, meant deploying first. The editor's save dropdown gains a third
action that saves the authored compose and env edits, acquires the stack's
registry-backed images, and stops. No up, no restart, no recreate, no build.
Backend: POST /api/stacks/:stackName/pull-images, gated on exact stack:edit
and exact stack:deploy, the same pairing Git source apply uses. The route
takes the per-stack operation lock as image_pull, so a pull cannot race a
deploy and a second pull reports "is already pulling images".
ComposeService.pullStackImages passes --ignore-buildable when the stack has
build-backed services, so those are skipped and returned by name rather
than built. The route reconciles nothing, so it opens no recovery
generation, starts no health gate, writes no drift baseline or GitOps
generation, runs no policy gate, and records no activity entry: there is
nothing to roll back to, and the update-guard marker must not move.
Frontend: the item is gated on both permissions on desktop and phone, and
honors the diff-preview setting like its siblings. The phone split-button
gains a chevron menu rather than a third button, which would truncate Save
& Deploy at 390px. A pull that fails after the save succeeded reports only
the pull half, since the save already confirmed itself.
The advanced lifecycle options are not part of this change.
* fix(stacks): sanitize the image pull failure log entry
The pull route's catch block passed the caught error object to the console
unsanitized. A failed pull rejects with the whole compose output, so compose
or registry text could carry line breaks into the server log and forge
entries. Route it through the same sanitizer the stack name already uses,
matching the sibling route catches, and replace line breaks with a separator
first so a multi-service failure stays readable instead of welding together.
No change to the HTTP response: the 500 body still reports the same message.
* fix(editor): condense the image pull failure toast to compose's error lines
A failed pull answers with the whole compose output, layer progress
included, and the error toast quoted all of it. With Deploy Progress off
the toast is the only surface the operator sees, so the cause was buried.
The toast now keeps the last three distinct failure lines compose writes
("<service> Error ..." or "Error ..."), counts any it drops, and falls back
to the last output line when none match. Progress lines that merely contain
the word, such as a service named error-reporter, no longer count as
failures. The progress panel still streams the full output.
* fix(editor): deliver registry credentials, run preflight, and report skipped actions
A pre-merge audit of the Save & Pull Images feature found three defects. All
three are fixed here, plus two non-blocking findings folded in at review time.
Remote nodes were not receiving the main instance's registry credentials for the
new pull route. Two classifiers gate credential delivery independently and both
omitted it: the op classifier, which is the outer gate the proxy consults, had no
pull-images branch at all, so the delivery block never ran. Added a
stack-pull-images stage carrying the stack name from the path, since the route
sends no body, and added the route to the body-budget classifier.
The pull skipped the SOPS and required-env preflight every sibling compose
operation runs, so on a SOPS stack ${VAR} in an image reference would resolve
against still-encrypted files and could fetch an image a deploy would not use.
Both guards now run at the top, and the refusal names the action the operator
chose rather than telling them to deploy.
A pull skipped after a successful save said nothing, leaving one green save
toast as the last word. Both skip paths now report, and only after a save: a
bare call is either blocked by an affordance the UI already disables, or is a
second click while the first action is still starting, where an error toast
would be wrong. Save & Deploy reports the same way, which it previously did not.
The save row and the pull item also disable while a pull is in flight, on
desktop and on the phone, since a pull holds the same per-stack lock a deploy
does.
--ignore-buildable is now passed unconditionally, so a service that builds from
source is skipped whether or not the stack's build declarations were readable.
The skip list remains the report, and a build: declared only in an override file
is skipped but not named.
The op classifier and the body classifier disagreed on the trailing-slash form,
which Express routes to the same handler, so a slashed request was forwarded
with no credentials and failed late with an opaque unauthorized. Both now accept
it, and the rollback discover branch moves with them so a slashed rollback is
still discovered against the restore candidate.
|
||
|
Anso
|
324dd7e1ae |
fix(gitops): restore only the generation a Blueprint target acknowledged (#2024)
* fix(gitops): restore only the generation a target acknowledged A Blueprint's Enforce mode restores state; it does not authorize it. The repair path resolved the identity to restore from the application's current pointers, so accepting a newer generation for any reason silently changed what an already-deployed target was repaired to. On a target acknowledged against an older generation this shipped compose content that no rollout had authorized for it, pinned to an artifact set from a different generation, which is a content and artifact substitution arriving through the one path that is supposed to be safe. The authority for a repair is now resolved in one place from immutable rows: the target's acknowledged generation and the artifact set that generation was qualified under, corroborated for Git-managed content by the live rollout generation that froze the pair. The application's newest accepted generation is no longer read on the repair path at all. Every way the authority can fail to resolve becomes an explicit hold with its own reason rather than a guess or a generic failure: no acknowledged generation, a superseded rollout, a node outside the frozen target set, an unreadable row, or a disagreement between the rollout and the target's acknowledgements. A hold records its own deployment status and history stage, repeats no work on the next tick, and never mutates the workload. The on-node marker now names the generation, artifact set, and rollout that installed it. Without that a repair could not prove what it was about to overwrite, so a marker that does not name one holds rather than assuming. Markers written before this still parse. Tests cover the acknowledged generation winning over a newer accepted one, the superseded-rollout hold, the never-acknowledged hold, and digest pins coming from the acknowledged set. * fix(gitops): hold every runtime repair the policy must not attempt A drift repair that cannot prove what it would overwrite must decline visibly, not fail quietly and not guess. Each way the authority can fail to resolve now becomes a typed hold with its own deployment status, its own history stage, and its own attention reason, so it stays visible instead of reading as converged. The arms are the states the drift model already treats as uncertainty rather than divergence: a target that is unreachable or whose connectivity evidence is stale, a target whose earlier mutation was interrupted, a target an unfinished rollback or an unavailable LKG owns, and a target outside the frozen required set of the rollout it is bound to. Two decisions move ahead of the drift write rather than after it. A stateful or unclassifiable Blueprint is never auto-repaired, and the classification check now runs before the drift is committed, so a held target is not first written as drifted and then rewritten on every tick. The check is repeated at the site that mutates, so the guard holds even when that path is reached directly. A hold is a state, not a latch: a later tick re-checks the target, so clearing the cause lets the next tick repair it. A hold that has not changed is not recorded again, which keeps a permanently held target from appending history forever. The on-node marker gains the generation, artifact set, and rollout that installed it, and the rollout path stamps them, so a drift check on a remote leaf compares against the same authority the hub resolved rather than inferring it. Both transports serialise the same marker, asserted byte for byte in the tests. Tests cover each hold arm, the hold-then-repair transition, replay suppression, survival across a store reset, local and remote reaching the same decision, and Observe and Suggest never mutating. * fix(gitops): re-check held drift targets and report a hold as a hold The hold work that closed the runtime drift policy made two states one-way, which is worse than the behaviour it replaced. A repair that re-resolved its authority and declined was reported as a correction failure and left the row reading "correcting" with nothing in flight, and a held row matched no branch in the decision pass, so an ordinary network flap could strand a target permanently where it had previously self-healed. Both paths now handle a hold as a decision: a declined repair records the hold with its own reason and warning, and a held row keeps its drift check, so clearing the cause lets the next tick repair it. A check that finds the target already matching returns the row to active through a cause that records no history, because nothing was written to a node. Classification holds get their own reason instead of borrowing evidence_incomplete, which sent operators looking for a missing generation when the Blueprint was stateful. Their text drops the drift reason, which was the replay key: a reason changing per tick re-alerted every tick for a hold that never changed. The recovery arm now also covers an in-progress recovery and a rollout awaiting its health verdict, both of which own the target. Docs no longer promise every hold self-clears. Superseding is one-way, so that hold persists until a later rollout covers the target or an operator deploys. * fix(gitops): bound the recovery hold to a recovery in progress The recovery arm tested recovery_phase for non-null, but that column keeps a terminal value for the life of the target and is never reset, so every target that had ever been rolled back would have been held from repair permanently, with only an explicit deploy to clear it. Only capturing, restoring, and compensating own the target now, matching the in-progress set the projection already uses. Clearing a hold without repairing it now advances the observation stage rather than only the deployment row. A row that went back to active while the projection still read the hold, or the drift, that had just cleared left the Drift surface and the attention queue contradicting the deployment table with no event left to correct them. Both carry tests that fail without the fix. The projection case asserts a literal status and requires the target to exist, because the optional chain it replaced made the whole case pass on a missing target. * fix(gitops): let a failed recovery keep holding, and rank both new statuses A failed recovery was the one terminal phase that stopped holding once the arm was narrowed to in-progress phases, so auto-repair would have performed exactly the mutation an explicit deploy is already refused on the same target. Only a recovery that finished successfully releases the target now, and a test pins it. The severity ladder had neither of the new runtime statuses, and a status it does not know does not degrade to something reasonable: it reads as unrecognized and the whole application row becomes unknown. A hold is ranked alongside drift for that reason. Clearing a hold advances the observation stage rather than only the deployment row, so the decision the reconciler made is what the projection reads, instead of depending on the observed-artifact write that happens to follow it. The test asserts the stage, because the derived status is identical either way. A new "converged" runtime status was tried here and reverted: the posture vocabulary has no slot for "checked but no health verdict", so it landed on unknown. A matched drift check projects to the status a Blueprint target already reached by the pointer path, which claims no health verdict either. Each test was checked to fail with its fix reverted. Two earlier attempts at the same tests passed with the bug reintroduced, one through an optional chain and one because a neighbouring check claimed the status first, so both now assert the mechanism rather than the outcome. * test(blueprints): render a held repair in the deployment table The held row is where a declined repair becomes visible, so it is rendered and asserted rather than only typechecked. A status that reaches the table without a label renders as a blank chip, and only the label map being total over the status union would have caught it. * fix(gitops): classify drift for Inline targets and observe held ones The repair authority was corroborated against a rollout generation whenever the target carried a rollout_generation_id, but an Inline target can carry one that was never the authority for its acknowledged pair, because Inline freezes its own generation per acceptance. Every such target was held for a disagreement that could not exist, and a held target reported no drift at all, so a replaced image on an Inline Blueprint was invisible in every drift surface. The corroboration is now keyed on the content mode rather than the pointer. A hold also suppressed the observation, because it returned before the runtime identity was collected. A hold refuses to mutate, not to observe, and what the node is running is exactly the evidence an operator needs when a hold blocks the repair. The identity is now captured before the hold is returned, with the probe and the record each guarded so neither can turn the hold into a failure. Both carry tests verified to fail with the fix reverted. |
||
|
Anso
|
bb7757649b |
fix(mesh): keep stack networks, scope mesh listeners, detect dead tunnels (#2010)
* fix(mesh): keep stack networks, scope mesh listeners, detect dead tunnels Five Mesh defects, all reachable on setups using a reverse proxy, a tunnel or a VPN. The generated override listed only sencho_mesh in each service's networks, which replaced a service's implicit default network. The service lost its links to the rest of its stack and any reverse-proxy route reaching it there. A service that declares no networks now keeps default; one that declares its own gets sencho_mesh unioned in. Services on network_mode (host, none, service:x, container:x) are left out entirely, because Compose rejects extra networks and hosts entries on them and the redeploy failed outright, which is what blocked VPN-sidecar stacks from meshing at all. Because a meshed service is now dual-homed, the dial address had to follow the mesh: containerLookup prefers sencho_mesh and only then falls back. Sencho is attached to sencho_mesh and nothing else, and Docker bridges are isolated, so a default-bridge address is not routable from the mesh hop and every dial would have timed out. The forwarder bound 0.0.0.0, which exposed every meshed service's ports on any network the Sencho container joins, and on all host interfaces under network_mode: host. It now binds Sencho's mesh address, binds nothing until the data plane is up, and the same-node probe dials that address rather than loopback. A half-open tunnel stayed open until the kernel exhausted retransmits, so a tunnel, VPN or NAT restart left the reverse-dialer slot held and every mesh dial timing out. A shared heartbeat terminates a peer after two missed intervals, and the proxy peer now probes the tunnel holding its slot and replaces it only if it does not answer, so two centrals still cannot flap each other. The local socket to tunnel direction had no backpressure, so a bulk upload over a slow link buffered without bound. Both ends now share one high-water mark, and every pause is released on teardown, since a socket paused and never resumed would stall the transfer for good. * fix(mesh): resolve compose merge keys and unstick post-close sockets This closes two defects and four coverage gaps in this branch's first commit. - getDeclaredStackServices parsed compose YAML without resolving merge keys, so a service inheriting network_mode or networks through `<<: *anchor` looked like it declared neither. The override then added networks and extra_hosts to a namespace-sharing service, and Compose rejected the whole redeploy. Checked against docker compose config: with the unfixed parse the project fails validation, with the fix the stack comes up on its own network and the mesh network at the same time. - openCrossNode left the stream marked open after it closed. A container that kept writing got a backpressure signal that could never clear and was paused on a drain that never arrives, stranding the socket. Bytes arriving after the close now tear the local side down instead of parking it. - The WebSocket heartbeat counted only pongs and inbound messages, so a healthy but slow link was terminated every ~90s while the pong queued behind the upload. A shrinking send buffer now counts as proof of life; a flat or growing buffer still terminates. - PilotTunnelBridge.start could install the heartbeat after close() had already run, leaking the interval. - Covered the bridge's reverse-socket backpressure, the switchboard's closed-tunnel drain path, and both not-ready-data-plane guards. The switchboard test this replaces set up state, emitted drain directly instead of driving the poll, and asserted nothing, so it passed with the branch it claimed to cover deleted. - Corrected the network_mode rule in the override comment and the docs: Compose rejects a networks entry alongside every network_mode form and accepts extra_hosts on all of them. A network_mode service never carries a mesh alias, because aliases come from published ports and such a service cannot publish one. |
||
|
Anso
|
bf18c5c8f2 |
test(gitops): scope the preflight backfill count to its own fixtures (#2023)
The preflight backfill test seeds five applications and asserts that the backfill re-evaluated exactly five. It does not clear the shared database first, so the count is really "every authorized application with no stored preflight evidence in this file", which includes whatever earlier cases left behind. This file already has the answer one describe over: `clearGitOpsState`, with a comment saying a count assertion needs only its own application to exist because earlier cases leave their fixtures behind. The new backfill case rests on the opposite assumption, written in its own comment, that applications seeded by earlier cases hold evidence. That is not a property of this file, so the count is only right by luck, and it moves as soon as a case anywhere above holds a live authorization with no stored evidence. Hoisting the helper to module scope and calling it at the top of the case makes the assertion mean what it says. Production behaviour is untouched, and the backfill still walks every authorized application it can see. |
||
|
Anso
|
4380197dd8 |
refactor(gitops): drop the unreachable git-source row rollback in create (#2006)
createStackFromGit tracked a rowInserted flag so its rollback could delete the git-source row, but the flag was only ever set after commitCreate() had already run and flipped gitopsCommitted, and the catch block reports every failure past that boundary instead of compensating. The delete could not run, so the rollback it guarded never happened either way. Removing the flag leaves the post-commit behaviour it implied: past the boundary the stack and its row both stay, and a later failure is reported to the operator rather than undone. A new test injects a failure after the boundary and pins both halves of that, so reintroducing compensation here now fails loudly instead of passing quietly. |