751 Commits

Author	SHA1	Message	Date
Anso	ca90588e4e	fix(gitops): bound GitOps history with a retention window (#2105) ... * fix(gitops): bound GitOps history with a retention window `gitops_history` is insert-only and the reconcile tick appends to it continuously, so it grew for the life of an installation. A new `gitops_history_retention_days` setting (default 30, range 1 to 365) seeds with the other GitOps settings and prunes history from the monitor's cleanup pass. Drained outbox rows are pruned with the history they announced; an undrained row gets a grace period past the window because the boot repair can still drain it from its payload, and a row still undrained after the grace is counted, logged, and dropped so it cannot accumulate for ever. The window is instance-local, like the artifact retry interval beside it: the history being pruned is the one this instance holds, and the monitor that prunes it runs here. * fix(gitops): keep each live application's newest history row through the prune `cleanupOldGitOpsHistory` deleted `gitops_history` purely by age, but `latestTransitionByApplication` reads `MAX(created_at)` per application and feeds the "last activity" fact the portfolio and application sheet render. An application with no history inside the window, because polling is off, a source is suspended, a node is detached, or a Blueprint is idle, lost every row and its activity column silently went empty. The prune now exempts each live application's newest row, so the claim the JSDoc, the PR body and the settings helper text already make is true. Scoped to live applications rather than to every application_id: a tombstone in `gitops_applications` is permanent, so an unscoped exemption would retain one history row per stack ever retired and leave exactly the growth this setting exists to stop. A retired application's rows are now pruned in full. Also folds in three follow-ups from the audit: - The prune is chunked to GITOPS_PRUNE_CHUNK_ROWS rows per table per pass, so a first prune on a long-lived install clears its backlog over several janitor ticks instead of holding SQLite's single writer for the whole delete. The method reports historyRemaining and outboxRemaining so the debug log says whether a pass is still catching up. - The outbox OR-delete is two statements, so the undrained branch uses idx_gitops_settled_outbox_undrained instead of scanning for the drained one. - The retention tests assert exact counts and exact surviving row ids rather than toBeGreaterThanOrEqual over a shared database. New tests pin each guard: a quiet application's newest row survives every window, its older rows are pruned, a tombstoned application clears entirely, an unsettled attempt is still recoverable after a prune both inside the window and outside it, and a backlog clears in bounded passes with the remainder reported. Removing the newest-row exemption turns 10 of the 11 tests red; removing only the liveness clause turns the tombstone test red. Docs: the GitOps portfolio page describes the window and its setting, and now states that the newest entry per application is kept, which is what the code does. * test(gitops): drop the unused db handles from the recovery assertions * fix(gitops): keep a settled history row ahead of its started row A reconcile attempt stamps both rows with the same time, and the prune broke that tie on a random id, so it could delete the settlement and leave startup recovery to reopen the attempt. Recency now follows insertion order, in the exemption and in each chunk. Remaining counts run only while developer mode is on, drained outbox deletes use an age index, and the settings copy says the kept row is the newest one for each live application. * fix(gitops): log the undrained rows a history prune actually deletes A pass is capped, so the warning was reporting the whole backlog. The history-window tests now fail if the permission, the page save, the monitor window, or the getter fallback stops matching the code. * fix(gitops): prune a superseded unsettled attempt past the window An unsettled reservation survives past the retention window only while it is that live application's newest row. Creating applications keep that row; detached applications do not. * fix(gitops): skip the history prune when the retention setting cannot be read A failed settings read used to fall back to 30 days and delete rows the operator had chosen to keep. The getter now throws, and that pass deletes nothing. A missing or unusable stored value still uses 30. * fix(gitops): restore the history window when the retention spec fails The retention spec restored the seeded 30 days as its last statement, not in a finally, so a failed assertion left the shared instance on 14. The CI retry then read 14 and failed the fresh-install assertion, turning one failure into a persistent red. Everything after the first save now runs inside a try whose finally puts the window back, so the spec leaves the instance as it found it whether it passes or fails. * docs(settings): list the GitOps history window in the retention reference The Data Retention table listed every other retention window and not the GitOps one, so an operator checking retention windows did not find it. The portfolio page described the setting in a paragraph, which is not where retention windows are looked up. Adds the row with its default and range, and a line under the table saying the window is set per Sencho instance and where the control lives. * fix(gitops): report an unmeasured prune remainder as unknown The prune reported a remainder of 0 whenever developer mode was off, so the diagnostic log read "none left" for a table it had never counted. It now reports null, and the log line says the remainder was not measured. Three smaller corrections from the same review: - The diagnostics allowlist omitted the history window, so a support bundle could not explain a timeline that had already been pruned. - The GitOps registry entry described every control as belonging to this node, while the retry interval and the history window are instance local. The sentence now names no scope it does not have. * fix(gitops): correct what the history window is documented to govern The copy said the window governs notifications. It does not: the prune deletes history rows and their outbox rows, and the bell entries those outbox rows announce age out under the Notification log window instead. An operator setting 7 days would have expected their bell to follow. It also promised to keep the database small, which nothing delivers. The prune stops old rows from being written; it never reclaims the file. - The helper now names the record behind the GitOps timeline, points bell entries at the Notification log, and says the window stops history from growing. - The settings reference gains the GitOps section it never had, with the window under it rather than under a per-node table that ends in a Save button the control does not use. - The portfolio page stops claiming a shorter window never changes what an application is doing: a webhook redelivery reads this same record, which the same paragraph already said. - A fraction below one day now clamps to a day instead of flooring to zero, which as a cutoff is now and would delete everything not exempt. The only caller passes an integer, so this is a guard, not a reachable fix. * fix(settings): snap a typed number chip to its own step on commit The chip clamped to min and max but never rounded, so typing 1.5 wrote "1.5" and every integer setting rejected it at the route with a bare "Validation failed" toast. Nothing was saved, so no data was at risk, but the control accepted input it could not store. Rounding to the control's step fixes that and leaves a genuinely fractional chip alone: the Docker janitor asks for 0.5 GiB and keeps it. Clamping still runs after rounding, so a fraction below the floor cannot round down under it. The settings reference also listed GitOps last under Infrastructure while the sidebar renders it third. * docs(gitops): replace the unsourced window figure with what the code does The helper and both docs said a window under about 3 days can let a Git host retry a webhook. Nothing in the code picks three days, and the dedupe the sentence describes only applies to the direct webhook path. The text now states the condition instead of a number: a redelivery can run the work again once the record of its first delivery is pruned. A provider endpoint keeps its own delivery record, pruned to a fixed depth, so the window does not affect it, and the sentence says so. * docs(settings): say where a remote instance's window is set The reference explained that the window belongs to an instance rather than a node, but not how to change the window of a remote one, which is on that instance.	2026-10-11 14:02:03 -04:00
Anso	f208a1f9da	fix(gitops): announce one pull result per attempt, and only when it changes (#2104) ... * fix(gitops): announce one pull result per attempt, and only when it changes Two writers announced one source attempt: the fetch path recorded its result immediately, and the settled outbox announced the same outcome after the transition committed. A pull therefore produced two bell entries, and a candidate awaiting review re-notified on every poll. Both writers now carry one dedupe key per attempt and event family, so the second write is a no-op. A ready event that names a staged candidate is keyed to the candidate instead, so the poll that staged it announces it and a later poll for the same commit stays silent. Settled outcomes that carry no news (no source change, candidate already fetched, suspended, superseded, retry scheduled, unknown) write no outbox row, and the boot repair applies the same rule to rows older builds left behind. The category and level each outcome renders are unchanged. * fix(gitops): announce one pull result per attempt, and only when it changes Two defects on the GitOps notification path. Every attempt wrote two bell entries for one event. The fetch path announces its result when it stages or fails, and the settled outbox announces the same attempt's normalized outcome after the transitions commit. Both now carry one dedupe key per attempt, so the second write is a no-op. The key is per attempt rather than per attempt and outcome family: the two writers classify one event independently and disagree by construction (a rolled-back apply records `blocked` while the `applyFailed` settle that confirms it carries `failed_previous_intact`), so a family-scoped key would let both through. A staged candidate re-notified on every poll, because each poll reserved a fresh attempt id. A ready event naming a staged candidate is keyed to the candidate generation instead, so the poll that staged it announces it, later polls of that same candidate stay silent, and a candidate staged afresh announces again. Keying on the commit rather than the generation would keep a commit that is legitimately staged again later silent for the whole notification retention window. A pull whose staged project does not validate now writes no live entry at all: it staged no candidate, so its failed settle is the single entry. Announcing "ready" there both doubled the entries for the attempt and consumed the candidate key a real candidate for that commit needs once the operator fixes the cause and pulls the same commit again. Settled outcomes that carry no news (no source change, candidate already fetched, suspended, superseded, retry scheduled, and the pre-existing unknown) write no outbox row, and boot repair applies the same rule to rows older builds left behind. Category and level per notified outcome are unchanged. The candidate-generation key is used only for a ready outcome, so a blocked or failed attempt stays on the attempt key even though it can also carry a candidate generation. The settled payload gains the candidate generation, decoded tolerantly so rows written before the field existed still drain. New coverage runs real pulls and applies and counts notification rows per attempt rather than by recomputing a key: a ready pull, an invalid-validation pull, a re-pull after the cause is fixed, a commit staged again after a different commit, a rolled-back apply, a converted Blueprint source staging a candidate, and a pre-upgrade outbox row that predates the candidate field. * fix(gitops): announce the failure when a staged result cannot be recorded A pull whose staging transition was rejected announced "ready" and then suppressed its own failure entry. The rejected transition left no durably staged candidate, but the pull carried on to the live write, which took the attempt key; the abandon that closed the operation stamped the attempt failed, so the settle that carried `failed_previous_intact` found the key already taken and wrote nothing. The only bell entry said ready for an attempt that failed. The live write is now skipped whenever the attempt has no staged result to announce, which is the case both when the staged project does not validate and when the staging transition was rejected. The settle is the single entry either way. The notification documentation now claims only what is delivered. It previously promised that a suspended source or an armed retry writes no entry at all, but the live failure write fires on every thrown fetch regardless of the settle, so those attempts do report. It also promised that later polls of a waiting candidate stay silent without saying what makes them silent. The candidate key is enforced by a unique index against rows that still exist, so clearing the bell or letting the retention pass prune the entry frees the key and the next poll announces the candidate once more. That is pinned rather than engineered around: the live write runs on every poll and relies on the key to collapse it, which is what lets a failed write recover on a later poll. Gating it to one write per staged generation would turn a swallowed write error on an automatic source, whose settle carries no news of its own, into a permanently lost notification. * fix(gitops): key a blocked candidate to the candidate, and announce a hold A plan-blocked candidate still notified on every poll. It settles `blocked`, which is news, under a fresh attempt key each interval, so a local conflict left standing filled the bell with the same warning. That is the per-poll repeat this work set out to stop, and the earlier docs sentence claimed it away as out of scope. A blocked candidate is now keyed to the candidate the same way a reviewable one is, and announces once for as long as the conflict stands. The candidate's identity gained its state, read from the live application row at the moment the entry is written: `awaiting_review`, `blocked`, or `held`. The first two are standing states that re-settle every poll. The third is a change, not a state to sit in: a safety refusal on an automatic source turns a candidate the system was going to accept into one needing review, and because that hold settles `pending_review` exactly as the waiting candidate did, it was colliding with the entry the candidate wrote when it was staged and the operator never learned the candidate stopped being automatic. It now announces under its own key, once. The family and the candidate's state must agree before the key is the candidate's, which keeps a blocked attempt on the attempt key: a dispatch refused while a reviewable candidate is live settles `blocked`, and that refusal is about the run, not the candidate. The row-counting test helper matched only rows carrying an attempt id, so a live write that lost its attempt metadata was invisible to every assertion. It now matches by stack name as well, and a second helper asserts no attempt-owned row for the stack is unkeyed. New cases cover a thrown fetch, a pre-mutation apply failure, and a plan that blocks between the pull and the apply; the unattended invocation-drift refusal asserts the same. Each of those sites was checked by dropping its attempt metadata and confirming a test fails. The rejected-staging corner the audit raised is already covered: the staging transition runs inside one transaction and `markSettled` runs inside the same call, so a rejected transition leaves the fetch open, the abandon records the failure, and the settle announces it. The existing test proves the end state.	2026-10-10 13:53:13 -04:00
Anso	0ce9754766	fix(gitops): make automatic placement approvals reach execution (#2103) ... * fix(gitops): make automatic placement approvals reach execution Two defects on the bounded automatic placement path. A removal was never auto-approved: the decision read `gitops_targets.connectivity`, a column seeded to null that no producer ever wrote, so a withdrawal resolved to `unknown_connectivity` even though the target projection already derived reachability from the recorded observation. The decision now reads the projection's own `targetConnectivity`, requires the node registry to have the node online (the same gate an addition passes), and carries an observation that cannot be decoded as a limitation, so the refusal names malformed evidence instead of sending the operator to check a node that answered. An Inline Blueprint's policy placement approval was recorded and then never read: an operational edit clears the combined approval, and the reconciler returned at the legacy gate before consulting the policy's approval. The reconciler now executes a plan a current policy placement approval covers, under seven conditions: the application is inline, the placement policy is still the one that could have written the approval, the approval is a policy's, the rollout authorization policy is automatic, at least one node is already placed at the current revision, every retained node runs the current revision, and the approval covers every action in the plan. An uncovered plan waits for the operator rather than partly executing, a first placement always waits for Apply, and a retained node is observed but never repaired under this approval, because the placement policy decided the node set and not the content. A node-driven roster change no longer hides a policy approval behind a current combined one: a label or cordon change moves the node set without moving the combined approval's fingerprint, so the reconciler tries the policy approval first and falls back to the combined approval when the policy cannot run the whole plan. That fallback narrows the operator's blast to the placement's frozen target set instead of invalidating it, so a removal a policy approved no longer stops drift observation on the nodes that stay; the preview and the lightweight list and detail evaluators read the same narrowing, so no surface reports an approval the executor withholds or withholds one it will run. An operator's approval is never discarded by a policy decision. Demoting an application back to Inline takes the Git-era authority with it: the placement pointer is cleared and the rollout authorization policy returns to operator authorization, because neither was an operator's choice about Inline placement. An Inline Blueprint can reach the automatic rollout authorization the policy path requires from the control beside the placement policy. * fix(gitops): say when automatic placement waits for Apply A selector edit that moves one stateless node runs on the next tick, and Enforce holds a policy-approved plan until Apply. The docs described both as waiting, and the preview showed a pending plan with no reason. * fix(gitops): keep a cleared pin from automatic placement A pin clear left the new intent unpinned, so bounded auto could place the freed node without Confirm Apply. The decision now treats a clear as pin-driven when the standing approval was recorded on a pinned intent, or when a pin was written after an older approval and never confirmed. * fix(gitops): keep a detached snapshot off automatic placement Git-managed deploys stamp the Blueprint revision, and Git content never bumps it, so after Detach a retained node still looked current. Recording a new revision on Detach makes that stamp stale until Apply. * fix(gitops): hold Enforce plans that drop a mid-deploy repair A retained node that is correcting or deploying leaves the plan with no repair, so a policy approval could place an added node in Enforce. The mode itself now holds that plan until Apply, and the preview says so only when Enforce is the reason it is waiting. * fix(gitops): keep a pin chain and a re-armed policy from placing on their own Three authority gaps, all on the automatic placement path, found by the pre-merge audit. A pin that was set and cleared with no Apply between them read as no pin at all once a third intent followed: the intent before the current one was the clear, so the pin that moved the work was out of view and the freed node was placed without Apply. With no decomposed placement approval to anchor on there is no record of any confirmation, so the whole intent history now counts. A Confirm Apply mints the approval the decision anchors on from, and the next change is judged against that, so the wait ends at the next operator decision rather than never. Re-enabling a policy revived a decision the operator had taken back: execution stopped the moment the policy went, but the approval the policy wrote stayed on the row, so setting it back handed the next tick a decision made days earlier on evidence nobody had read since. Arming either policy again now withdraws what that policy decided, with the rollout authorization that only resolves through it. An operator's approval is untouched, the approval rows stay as the record of what was decided, the rollout generation stays so a fresh approval re-drives it, and the history row records the pointers that went with the policy. The refusals a withdrawal produces now read as what the backend checks: a node that has not reported, or that is not a node this Sencho knows about, is a different answer from a node that is not answering, and a withdrawal needs the node to have reported the workload it was running. * fix(gitops): clear the attention a policy placement left behind A Blueprint the placement policy placed a node for sat at Needs attention with a stale acknowledgement and a runtime drift, and neither Confirm Apply nor an API apply cleared it: the retained nodes' plan is observation-only, so nothing ever re-acknowledged them. Two causes, both fixed. The Inline freeze is now keyed on the compose digest rather than the intent revision, so a roster change no longer drops the accepted generation and re-points every placed target at a new generation id for the content it already runs. And the projection's identity test yields when both intent revisions resolve with the same compose digest and the current candidate still requires that node: a node the candidate dropped keeps its signal, because a withdrawal the plan is asking for is not a superseded acknowledgement. Two roster changes inside one tick also left both additions unplaced with no refusal recorded. The reconciler runs one approval and refuses a plan it does not cover whole, so the second approval named only its own addition while the plan still wanted the first. The blast now names every node the fleet does not hold, carrying the earlier decision's unlanded placement forward, while the decision still reads the approved set for its size and safety reasons and the evidence is read for every node the blast names. Removals stay on the approved set, because a withdrawal is judged from the observation of the workload that ran there. The Confirm rollout dialog no longer claims the fleet cannot change until it is confirmed when a configured policy has already approved the plan. * fix(gitops): write the placement blast in canonical order and name a withdrawn node again Two findings from the QA run of this PR, both now pinned by failing-first tests. The first: a roster move that adds a higher node id while withdrawing a lower one left the approval unwritable and the operator looking in the wrong place. The blast effect was assembled places-then-removals, while the decoder requires the node ids strictly increasing, so the write was refused as out of canonical order. The refusal reason then reported that refusal as `conflicting_operation`, so the operator was told an operation was in flight when none was, the plan read `reapproval_required`, and nothing executed. The effect is now sorted by node id, which is the order the decoder wants and carries no meaning of its own: the reconciler reads the effect as a map keyed by node id and the transition re-encodes it to canonical form anyway. Blast decode failures are also recorded as `malformed_evidence` rather than falling through to the conflict default, because a blast that did not decode is a data fault. The roster window that reaches it: the reconciler executes one approval per tick, so an addition approved but not yet landed is carried into the next approval's blast, which then names a placed node alongside the withdrawal. The second: a withdrawn node that the selector wants again produced no row in the plan at all. The tombstone holds deliberately, since only an explicit deploy re-opens a severed placement, but the node stayed online and labelled with nothing to say about why it was skipped, so the plan read as nothing to do. A new informational `skip_withdrawn` row now names what the operator has to do. It is excluded from the confirmable set and from both executor groups, so it authorizes nothing and cannot be confirmed into a mutate, and it yields to the status-precedence pass so an in-flight or guard row for the same node wins rather than leaving two explanations stacked on one node. Follow-up, not addressed here: the replay-guard refusal (`placement approval already recorded for this intent and candidate`) still maps to `conflicting_operation`, which is the same class of untruthful reason on the crash-retry path. It predates this change and is left for its own pass. Tests: a new case drives the add-then-withdraw window end to end and asserts the approval lands with `stateless_removal` and no refusal recorded; a second drives a severed target the selector wants again and asserts the informational row; a third pins that a severed node carrying a `withdrawing` row reports once, as the in-flight withdrawal. Backend: `npx tsc --noEmit`, `npm run lint` (0 errors), `npm test` (11013 passed; the single file-level failure is `git-transport-ssh.integration.test.ts`, which requires loopback port 22 free and fails identically on an unmodified tree because this host runs sshd on 22, freed in CI by `scripts/ci-free-loopback-ssh-port.sh`). Frontend: `npx tsc -b --noEmit`, `npm run lint` (0 errors).	2026-10-09 22:28:41 -04:00
Anso	b8de90d13c	fix(gitops): match webhook deliveries on canonical repository identity (#2102) ... Provider webhook deliveries were matched to the owning source by raw host and pathname equality, so a stored `...repo.git` clone URL never matched the `html_url` a provider sends, and ssh-configured sources, trailing slashes, host case, ssh ports and path case were dropped the same way. The delivery was recorded `source_mismatch` and the source kept polling as if no webhook existed, with nothing surfacing the mismatch. Compare canonical repository keys instead, folding path case before the key is built so a differently-cased `.git` suffix normalizes too. The fold is local to this signature-verified check: an endpoint binds exactly one source, so it reads a signed statement rather than looking a repository up, and the pull that follows uses the configured URL, never the delivered one. Folding in the Blueprint claim guard would add refusals, so the guard keeps its case-sensitive path: a case-sensitive host can hold two repositories whose paths differ only in case.	2026-10-08 22:09:47 -04:00
Hambrook	a6eaea0f54	feat(editor): create the stack .env from the editor on first save (#2097) ... * feat(composeEditor): create .env file on demand when saved instead of relying on doing so via Files * fix(editor): scope .env creation to the editor and adopt the resolved path - PUT /:stackName/env creates the default .env only when the request carries create=1 and the stack exists; every other caller keeps the handled 404, so a Fleet Secrets push to a stack with no env file never creates one. - The create response returns the resolved path; the editor adopts it, so the next save targets the file directly instead of failing the allowed-file check. - Deep links to /env open the env tab on a stack with no env file. - Mobile always shows the compose/.env toggle and the creation notice. - Backend and frontend tests pin the create path, the canonical second save, the no-flag 404, the deep-link behavior, and the notice; docs, the guardrails tutorial, and the mobile screenshot are updated. * fix(editor): make the first env save exclusive and pin the confirmed overwrite Audit follow-up on the create-on-first-save path: - PUT /:stackName/env?create=1 now creates exclusively with open(..., 'wx'). An existing target answers the same 412 conflict the optimistic-concurrency path uses, with its current content and path, so a stale editor that still believes no env file exists can never overwrite one it has not read. - The editor adopts the conflict path, and its confirmed overwrite retries with create=1&force=1&file=<path>, pinned to the file the dialog named and re-validated against the allowlist. - With project env files configured, the create targets the first configured file (the one Compose reads), derived from the shared env resolver instead of a second copy of the fallback; a configured path that resolves outside the stack directory is refused with 409 rather than written. - The conflict read repeats the containment check and refuses to follow a link, so the conflict payload cannot be used to read outside the compose root. - A directory target maps to 409 instead of an opaque 500. - Tests cover the conflict instead of an overwrite, the forced retry, the pinned retry, the configured target with a stray .env untouched, the escaping config refused, force ignored on the non-create path, and the frontend conflict, decline, and fallback flows. - Docs restore the compose row's "Always available." note, describe the creation target and the no-overwrite guarantee, and make the notices generic. * fix(editor): guard the env create against older nodes and unknown state Audit follow-up on the create-on-first-save path: - The non-forced create request now carries If-Match: W/"0". This build's exclusive create ignores it; a node that predates create-on-save treats it as a normal conditional write, so an existing env file answers 412 with its content instead of being overwritten, and a missing file keeps the handled 404. - A failed stack load clears the previous stack's env state instead of leaving a stale "no env file" assertion that the next save could act on. - A failed env inventory or content read is tracked as unknown, shows a failure notice, locks the buffer, and refuses the save; it is no longer presented as an empty file. - Declining the create conflict on a node whose 412 names no path keeps the conflicted content and refreshes the inventory under the current load's signal; a failed refresh reports the failure instead of a false reload. - Save & Deploy and Save & Pull on an untouched empty env tab no longer create an empty file; the deploy proceeds. - The exclusive create opens with O_NOFOLLOW so a planted symlink cannot be followed, and a failed write's leftover is removed only when the path still names that same file (inode and size match). - Tests cover the older-node precondition, concurrent creates, a directory target, escaping and dangling symlinks, the write-failure cleanups, the unknown-state refusal, the conflict-retry copy, and the deploy skip. * fix(editor): leave create mode on older nodes and return 409 for path conflicts Audit follow-up: - A create-mode 200 without envPath is the normal response of a node that predates the envPath field. The editor now refreshes the env inventory in that case, so it leaves create mode and later saves are normal conditional writes with a fresh ETag instead of prompting for an overwrite every time. - The recovery refresh uses the load signal captured before the PUT, so a stack switch during the request aborts it and the new load's state is not repainted. - PUT /:stackName/env maps INVALID_PATH and SYMLINK_ESCAPE from the create path to 409, so a symlink at the target is a recoverable conflict instead of an opaque 500. - Docs: deep-links no longer says the created file is always the default .env. - Tests: the older-node create, confirm and overwrite sequence asserts the editor leaves create mode and the next save is a conditional write; a route test covers an in-root symlinked .env returning 409 without leaking content. - Comment cleanups from the audit. * fix(editor): do not paint a create response that lands after a stack switch Audit follow-up: - saveFile checks the load signal captured before the PUT (opLoadSignal) in the 412 and success paths: a response landing after the editor switched stacks now returns without adopting the old stack's envPath, ETag, or buffer state and without prompting, so the new stack's editor state stays intact. - Tests: two hook tests switch stacks mid-PUT, for both the success and the conflict response, and assert no adoption and no prompt. - Docs: node-compatibility no longer claims every older node detects the conflict (nodes before 0.87 cannot) and notes that the confirmed-overwrite prompt does not repeat. - Comment: createTarget wording matches the resolver's containment filter. * fix(editor): create missing parent directories for a configured env file Audit follow-up: - A configured project env file can point into a subdirectory that does not exist yet; the exclusive open failed with ENOENT and the route answered an opaque 500. The create now makes the missing parents inside the checked, stack-bounded path and retries the exclusive open once, so the first save creates the file Compose reads at its configured path. - The retry re-runs the containment check and still loses to EEXIST if another actor created the file meanwhile. - Tests: a service test for a nested missing-parent target and a route test that configures config/prod.env with no config directory, creates it, and saves again through the returned path. * fix(editor): report a vanished stack on env create instead of resurrecting it Audit follow-up: - Drop openExclusiveCreatingParents and its recursive mkdir. The state it served (a configured project env file in a missing subdirectory) cannot be produced by the API: PUT /project-env-files rejects names with "/" or "\" and requires the file to exist, so no product path reaches that branch. The only reachable ENOENT is a stack directory deleted between requireStackExists and the exclusive create; recreating it left the deleted stack visible only as a hidden folder holding the env content and blocked the stack name. - Restore the direct exclusive open and map ENOENT in the route catch to the handled 404 Stack not found, so the race is reported and nothing is written or recreated. - Replace the two tests for the unreachable nested target with regression tests: the service refuses a target under a missing stack directory and creates nothing; the route answers 404 Stack not found when the directory vanishes before the create (real service ENOENT path) and does not recreate it. * fix(editor): recover when the create conflict target is deleted mid-dialog QA follow-up: - The confirmed create retry pins the file the conflict named. When that file is deleted before the operator confirms, the route rejected the pinned path with a 400 because it was no longer a present env source, and the editor (which had already adopted the path) failed every later save the same way. The create branch now accepts the pinned path when it is the resolved create target, so the retry recreates the file; any other pinned path keeps the scoped rejection. - Declining a create conflict now reloads the env state instead of trusting the 412 payload: if the file was deleted meanwhile, the editor returns to create mode instead of adopting a path that no longer exists. - Save failures toast the server's error text (404 vanished stack, 409 unusable path, 413 too large) instead of the raw JSON envelope. - The save-refusal toast for an unreadable env state uses the same wording as the editor notice; the file list may have loaded while the content did not. - A viewer with no env file sees "This stack has no environment file yet." instead of an unexplained empty read-only tab; editors keep the create promise. - The route test for a vanished stack restores its spy in finally. --------- Co-authored-by: Anso <dev@anso.codes>	2026-10-08 12:12:26 -04:00
Anso	cee2ec83a6	fix(gitops): capture Pilot recovery and stop refused rollbacks from parking the application (#2087) ... * fix(blueprints): capture recovery on Pilot apply-local hops The leaf's apply-local route honored allowGitManagedContent and captureRecovery only under the node_proxy machine scope, but a request forwarded through the Pilot tunnel arrives with the agent's pilot_tunnel loopback token. The auth middleware treats the two scopes as the same hub-to-leaf credential and every sibling route already accepts both, so the gate silently dropped the capture and a later rollout rollback on a Pilot target failed with no recovery point to restore. Accept either machine scope for both flags. Browser sessions and API tokens hold neither and are still refused. * fix(gitops): keep a refused rollback from parking the application A rollback a node refused before its restore touched anything was recorded application-wide as a failed recovery. The source facet then read recovery_failed, offered no action, withheld deploys, and stopped polling, while the only exit, a successful restore, was impossible on the refused target. Recovering it took manual row edits. Classify each failed restore by whether it can have mutated the target. The restoring node's own pre/post verdict, attached to the compensation error and carried over the rollback response, wins; otherwise only codes that prove the restore moved nothing are pre-mutation. A pre-mutation refusal is recorded on the target and settles the application restoring stamp in the same transaction, keeping a same-ref completion receipt when another target came back. A 2xx that does not echo the requested generation is its own unconfirmed code and still holds, because the node did restore something unproven. A later successful deploy acknowledgement now retires a stale recovery failure on the acknowledging target and, when no target still reports one, on the application. * docs(blueprints): describe how a refused rollout rollback resolves A target refused before its restore changes anything keeps its visible failure, but it no longer holds the application. The source keeps converging and the next successful deploy to that target retires the stale failure; only a rollback that may have changed a target holds automated work. * fix(gitops): keep rollback holds conservative across retries and refusals A later pre-mutation refusal could erase the application hold an earlier target still needed when that target may have been half-restored: the settle cleared the stamp without checking the other targets, and a retry refused during an unresolved failure overwrote the target's moved class. The settle now restores the hold from a surviving mutated failure, keeps the stamp while any restore is still moving, excludes the refusing targets themselves, and only a pre-mutation refusal defers to an earlier failure's class. A refusal whose own record write throws is settled by the route for every missed target, not just the first. Also carried through the same path: the node's own pre/post classification rides the compensation result and every uncoded refusal (a restore-not-sent outcome for a request that never left the hub, a coded stack-not-found, the self-stack refusal), stack-busy answers hold instead of settling because another operation is moving that stack, a failed mid-write restore whose own revert failed is classified post-mutation, and Enforce runtime repair follows the same pre-mutation release rule. * fix(gitops): derive rollback holds from targets and separate recovery claims The settle no longer keys on one operation's recovery ref: it derives the application phase from the live targets, so two overlapping rollbacks can no longer pin the application in restoring, and the other operation's refusal settles it. The open writes the target's active-operation marker, so a crashed open is selected by boot and reclassified instead of staying restoring, and a missed refusal record is retried for its target rather than only settling the application. The health executor settles a proven refusal whose record throws, the same as the manual route. A recovery hold is now claimed by one shared predicate, targetRecoveryFailureMoved: a failure stage that has moved on no longer claims a recovery, and a pre-mutation refusal never did. The settle, the acknowledgement clearing, and the repair binding use it, and the failure writers clear a superseded recovery phase, so a deploy failure on a refused target cannot re-park the application or re-hold repair. * fix(gitops): retire replaced recovery claims and keep repair behind a retry A deploy failure that replaced a moved recovery failure cleared the target's claim but left the application hold with no claimant, and the acknowledgement's clearing branch then never ran. Claims are now retired through one helper that re-derives the application hold in the same transaction; a successful bind, an acknowledgement and a revived placement retire the claim too, and every rollback terminal retires the recovery_started interruption a crashed open left. Repair checks a restore in flight before the failure claim, so a retry over a refused target holds repair again. A terminal rollback whose recovery ref no longer matches the target changes nothing, a newer rollback open takes over the active marker, and a failed record is retried with the class the attempt was proving for either class. * fix(gitops): keep a moved recovery claim through later failures A recovery claim that may have moved a target was retired whenever a deploy or withdraw failure replaced it, releasing the application hold with nothing proving the target consistent. The claim now keeps its own class in a dedicated column, so a later failure records itself beside the claim instead of overwriting it. Only success evidence (an acknowledgement or a bind) retires a moved claim; a claim that moved nothing is still retired by a later failure. A restore in flight is never rewritten: retirement returns early on capturing, restoring and compensating, and a deploy, withdrawal or acknowledgement terminal that clears its own marker hands it back to the restore so a crash remains boot-recoverable. Superseded terminals are scoped by class: a refusal that moved nothing is a true no-op with no row write, history row or application stamp, while a failure that may have moved the target records its class without touching the newer attempt's phase, ref or marker. rollbackCompleted carries the same ref guard, and the settle re-reads each row after its retry and drops one a newer rollback re-opened. The application hold counts a restoring sibling as a claimant, and a tombstoned target stops answering for the hold it claimed. * fix(gitops): serialize rollbacks per target A second rollback could open over a target whose restore was still running, take over its recovery slot and ref, and leave the application hold released while the first restore was still moving files. One restore per target is now the rule: rollbackInProgress and recoveryStarted refuse a target that is already restoring, the route answers with 409 ROLLBACK_IN_PROGRESS before any restore starts, and the health executor skips its rollback on a target that is already restoring. With overlap impossible, the machinery that arbitrated it is gone: the ref and marker takeover, the skipped-terminal plumbing and the ref guards. A refusal that defers to a moved claim leaves the shared failure slot alone, so a later deploy failure is not relabelled a recovery failure, and the claim keeps its own class and time in dedicated columns. rollbackCompleted, recoverySucceeded and a moved rollbackPartialFailed derive the application stamp from the live targets instead of writing it outright, so one target's outcome cannot release or overwrite a sibling's hold. A restore marker is handed back on every terminal that clears it, including through a tombstone, and both revival paths share one helper. * test(gitops): restore the boot recovery selection test The superseded-terminal tests removed with the rollback serialization also took the boot test that drove the real startup selection, leaving its import unused. Restore it against the current semantics: the open's marker makes the application selectable, boot reclassifies the restore as interrupted, and a later terminal rollback retires the interruption. * fix(gitops): record a terminal when a health rollback hold throws The health executor opened a rollback and then ran the hold and two store reads outside the block that records a terminal. A throw there left the target restoring, and with rollbacks now serialized per target that state refused every later rollback in-process. Everything after the open now lives inside the recording try, so every exit records a terminal; a regression test forces the hold to throw and checks that a later manual rollback is accepted. Boot also selects applications whose own or any target's recovery phase is in flight without an active-operation marker, and records the interruption as recovery_started, so a row an older build left restoring is reclassified instead of being refused for ever. The docs now say every Git-managed rollout deploy captures the node's previous project on every target type, and that a target can have one rollback at a time. Tests removed with the rollback serialization were restored where the behavior still exists. * fix(gitops): stop stamping an application interruption for a recovery Boot reclassification wrote recovery_started into the application's interruption_stage for a marker-less in-flight restore. No recovery terminal clears an application interruption stage, so after any crashed rollback the value stayed for ever, and the drift collectors read a set application stage as a standing suppression while a Direct source deferred its retry. The application row records the interrupted recovery in its recovery phase; the target keeps the operation stage. A regression test reclassifies a crashed rollback and asserts the application stays clearable through a later successful rollback. The route's 409 test now asserts the live rollout authorization and generation are untouched, so a regression that supersedes the rollout before refusing fails the test. The repair test for a refused recovery replaced by a deploy failure now drives the real writers, and the rollback docs no longer claim the source keeps converging. * fix(gitops): keep the poll cadence through a recovery hold A held application settles every evaluation into recovery_required, which armed neither a retry cursor nor a fresh poll. The fetch that ran while the hold stood consumed the poll cursor, so once success evidence cleared the hold the source sat idle: a pushed commit was never fetched and next_poll_at stayed null until the poll interval was saved again. The controller now re-arms the poll for a recovery wait, an in-flight restore or a hold that success evidence retires. Automatic acceptance stays gated on success-shaped outcomes, so the hold still withholds progression. A regression test holds an application through the real rollback writers, consumes the cursor through the real fetch transitions, and asserts the cursor is re-armed. The rollback section of the Blueprint docs again states that every Git-managed rollout deploy captures the node's previous project on every target type.	2026-10-07 20:05:41 -04:00
Anso	a54ebcc7e2	feat(fleet): separate the Fleet tabs into two sections, with a layout setting (#2092) ... * feat(fleet): group the Fleet tabs into Observe and Operate, with a layout setting The Fleet tab strip was ten tabs in one row, wider than a 1280px window. The strip now sets the tabs under two labelled groups on one strip: Observe (Overview, Readiness, Map, Docker Labels) and Operate (Snapshots, Blueprints, Routing, Federation, Actions, Secrets). It scrolls sideways when the window is too narrow and keeps the tab you are on in view. Every tab still opens with one click and every tab keeps its value, so deep links and URL sync are unchanged. A new account preference, Settings, Appearance, Fleet, Fleet tab layout, picks how the strip is laid out: Grouped (default), Flat (the previous single row), or Compact (Overview, Readiness, Map, Blueprints and Actions, with the rest under a More menu that names the active hidden tab). It is saved to the account and synced like the other appearance preferences, defaults to Grouped for a document that lacks it, and is cleared with the rest of the preference cache when another account signs in. * refactor(fleet): drop the Observe and Operate labels and the old flat strip The group labels only lengthened the tab row, and the previous single-row layout is no longer needed. The Fleet tab strip is now one row: Overview, Readiness, Map and Docker Labels, a vertical separator, then Snapshots, Blueprints, Routing, Federation, Actions and Secrets. It still scrolls sideways when the window is too narrow and keeps the active tab in view. The Fleet tab layout setting now offers Flat (the default, this row) and Compact (five tabs with the rest under More). The retired grouped value reads as Flat. * test(fleet): raise the cross-tab storage event without the StorageEvent constructor The code quality scan flagged the two-argument StorageEvent calls as superfluous trailing arguments. The tests now build the event from a plain Event carrying the key and the new value, which is all the hook reads.	2026-10-07 09:14:25 -04:00
Anso	d7e0ebc4e5	feat(fleet): draw the Map one node at a time as each answers (#2091) ... * feat(fleet): draw the Map one node at a time as each answers The Map tab asked for the whole fleet in one request, so every node waited on the slowest one and a single unreachable node held the tab until its timeout. GET /api/fleet/dependency-map now takes an optional nodeId (positive integer; 400 if malformed, 404 if unknown) and answers for that one node in the same shape. The Map tab reads each node on its own, four at a time, and draws each slice as soon as it arrives, with "n of N nodes" beside Refresh while the rest are pending. A node that fails keeps its last slice and is named in the banner with its reason, and Retry in the banner reloads only the failed nodes. A node that never answers is given up on after 20 seconds so Refresh and Retry never stay locked behind it, and an answer that is not a map is reported as a failure instead of drawing nothing. When no node could be read the tab says so instead of calling the fleet empty. * test(fleet): compare the request origin, not a URL prefix, in the map filter tests CodeQL flagged two startsWith checks on a base URL as incomplete URL substring sanitization. The assertions now parse the request URL and compare its whole origin.	2026-10-07 08:56:14 -04:00
Anso	fd5824c142	feat(fleet): add Cordon and Delete to the node details sheet (#2090) ... * refactor(fleet): give each node one state and one action, and redesign the node details sheet The Fleet node card carried up to six chips (version, update, critical, cordoned, GitOps, integration image) and said "Node unreachable" three ways with no reason and no action. One model now reads a node's state for both surfaces: the loudest condition is the Answer, the rest are counted. The card shows the version as quiet text, one state chip with a +N for the others, and at most one action button, shown only when the session may run it. An offline node says why in one line (Pilot disconnected, last seen 12m ago, or Proxy unreachable, last contact 1h ago) with a Test connection button that runs in place, instead of a dimmed card. Stale update, GitOps and networking readings are no longer shown for an offline node, but a failed or running update stays visible with its Retry and Dismiss. The details sheet opens on an Overview that answers first, with a quiet row of the stages that apply and a collapsed Details section, plus Resources and Settings tabs. Its toolbar holds the object-level actions (Edit, Cordon or Uncordon, Delete) and the action for every other state the node is in, so no action is lost when a louder state is the Answer. Cordon confirmation and the connection test are shared by the card and the sheet. * refactor(fleet): keep the original node card and one-page sheet, add Cordon and Delete to the sheet The one-state-chip card and the tabbed details sheet are withdrawn. The node card is back to its original layout, where the only action button is the node update, and the details sheet is back to its single page. What stays: the details sheet toolbar now carries Edit node, Cordon node or Uncordon node, and Delete node (with View networking when the node has a networking signal), under the same permission rules as the card menu. The cordon confirmation is one shared component used by the card and the sheet, so the reason field, busy lock and failure handling are the same in both. The phone screen is unchanged.	2026-10-07 08:33:33 -04:00
Anso	c925c7b887	refactor(fleet): put the Fleet masthead on the shared pattern and list nodes attention first (#2089) ... The Fleet masthead was hand-built: its state word always read "The fleet", it showed a healthy tone with zero nodes while loading, and its "alerts" tile restated the critical-node count. It is now a PageMasthead hero with a FLEET kicker and a real verdict (Healthy, Degraded, Critical, plus Checking before the first overview, Unavailable if that first overview failed, and No nodes for an empty fleet). One detail line names the node count, offline and critical counts and sync age, and CPU, memory and containers sit in the metadata. The peak-CPU sub-line and the alerts tile are gone. Fleet nodes can now be sorted by Attention (offline, then critical, then name), which is the default for new browsers. The Local node stays pinned first in the grid.	2026-10-07 08:33:12 -04:00
Anso	612c25a670	feat(gitops): complete the Git-managed Blueprint rollout path (#2084) ... * fix(gitops): materialize frozen rollout targets on dispatch A Git-managed Blueprint rollout dispatched against its authorized frozen target set failed on every node except the source stack's own node with "Could not open deploy for node N: target not found". Only the Inline producer created a target row on first contact, and the Git adapter sent to nodes that had never received a Direct deploy. Create an empty target row for each frozen node that has none before the deploy queue runs, mirroring the Inline producer's deploy_start write. A node that no longer exists is skipped and still refused by name. An existing row is left untouched, so a tombstone, fence, or spent retry budget survives row creation. * feat(gitops): materialize and freeze the accepted generation for Git-managed rollouts A converted source keeps polling, but the Direct pipeline is refused for a claimed source, so nothing promoted a new generation into its applied directory and nothing resolved its artifact set. The rollout dispatch reads the applied materialization and authorization refuses an unresolved set, so a second commit could never be authorized. Promote the accepted generation's candidate into its applied directory on the hub (no writes to the live source stack) and resolve the artifact set from that materialization through the shared approved-content parser. The reconciler's artifact retry now re-runs both halves for a Git-managed Blueprint, so a transient failure heals. A generation that materializes more than one compose file is refused rather than resolved from a subset, and candidate and applied paths get the same shape validation the Direct promotion uses. * fix(gitops): rebind the rollout candidate to a newly accepted generation After a source-only change, sourceAccepted cleared the rollout authorization but kept the placement approval, while the candidate still named the previous generation and artifact. authorizationIngredients refuses a candidate whose binding disagrees with the application, so a second commit could never be authorized. Move the candidate's generation binding to the newly accepted generation and clear its artifact binding in the same transaction. The artifact binding is cleared rather than pointed at the acceptance's seed set because the freeze advances the application's set to the resolved one; authorization re-stamps both from that resolved set. Both acceptance entry points share the rebind, and the store method checks ownership and is idempotent. * fix(gitops): hold an authorized rollout whose dispatch is refused A refused dispatch after a successful authorization returned only a note on a 200 and left the projection reading rollout_queued with nothing an operator could act on. Hold the rollout through the same application pause the health executor uses, so the Answer carries the reason and Resume is the resolving verb. The hold is a no-op when no authorization is live and never overwrites an existing hold. It is placed by the authorize route, the automatic acceptance path, the resume path when the retry is refused again, and the startup reconstruction, with a fresh operation id each time so a repeated hold still writes history and announces itself. A hold that cannot be recorded is logged, never thrown into a dispatch that already answered. * test(gitops): cover a second source commit end to end Drive the audit's repro at the service level: a Git-managed Blueprint rollout to two frozen nodes, a second source commit accepted and prepared through the real materialize and freeze producers, re-authorization through the configured policy, and a second dispatch that deploys the new compose to both nodes. The test asserts the promoted materialization, the resolved artifact set, the minted authorization, the rebound candidate, the deployed compose bytes, and the per-node applied generation. * docs: document the Git-managed source update path Describe what a new commit does on a Git-managed Blueprint: acceptance materializes the generation and attempts to resolve its artifact identity (the reconciler retries an incomplete resolve), placement stays approved across a content-only change, and authorization starts the sequential rollout again, on its own under the automatic policy or held with the reason when preflight or artifact evidence is not ready. The Resume bullet records that a refused retry re-holds. The What's New blurb now describes the working chain instead of a blocked state. * fix(gitops): retry Git-managed preparation and classify dispatch holds A newly accepted generation whose preparation failed (a registry outage, or a crash between the acceptance commit and the materialize call) was stranded: re-accepting is refused, and the per-target artifact retry only looks at the generation a target already holds. The reconciler now runs an application-driven retry for every live Git-managed application, re-materializing and re-resolving the accepted generation and completing the automatic handoff once it resolves. A compose shape the authored-text parser cannot model is refused and remembered, and its reason is persisted as an evidence limitation the projection can show. Blocked dispatches now carry whether they are holdable. Lock contention, an already-recorded pause, a per-target pause, a superseded generation, and a suspended source are not: pausing on them stopped a healthy rollout another dispatch was advancing, and the per-target case needed two Resumes to undo one decision. The health rollout executor shares the classification. A converged Git-managed Blueprint no longer projects the Direct manifest cache as managed-project drift. * fix(gitops): clear system holds on acceptance and record their origin A system hold (a refused dispatch, a health fence) belonged to the rollout it was holding, but acceptance never cleared it: the next generation dispatched into "The rollout is paused." with a reason describing the superseded one, and before this change the next commit dispatched on its own. Holds now carry an origin. A system hold is cleared when a source acceptance supersedes its rollout, for every target mode, with a rollout_unpaused history row; an operator pause survives until the operator resumes. Health-executor and fence holds record the system origin too, so the same rule covers them. The docs and the accept-route note now describe the retry, the automatic handoff, and the unmodellable-compose refusal the way the code behaves. * fix(gitops): close the re-audit findings on holds and the retry The health executor now forwards the pause-already-on-record marker and the adapter marks the paused refusals with it, so a per-target pause is not escalated to an application hold on that path. Every blocked result declares whether it is holdable, and the Direct pipeline's exact-shape tests follow. The reconciler's preparation pass runs before the enabled check for a disabled-only fleet, uses the same durable retry gate as the per-target path, and is detached from the tick; the tests prove the isolation, the permanent-failure gate, and the non-blocking tick. The accept route's dispatch wait is bounded, the resolved-artifact gate lives in the one handoff, and the unmodellable-compose reason now reaches the operator in the caveat and the accept note. * test(gitops): use the fixture in the non-blocking tick test The test declared a fixture without reading it, which failed the lint gate on CI. The fixture is now authorized and the test asserts that the released durable refusal held the application, which also proves the detached preparation pass reached the dispatch. * fix(gitops): re-drive the automatic handoff and keep hold origins honest The reconciler's content pass now has a handoff arm: a resolved generation under the automatic policy with no live authorization (a slow accept, a re-enabled Blueprint) or with unacked targets (a transiently refused dispatch) is dispatched on the next pass. The warranted predicate keeps it off a settled, progressing, withdrawn, or not-yet-placed rollout, and an in-memory floor bounds a failure that records no evidence row to the artifact retry interval. The pause origin now comes from the refusal's kind: executor dispatch refusals are system holds that a new commit clears, and only real health decisions (a stop, pause, or rollback) carry the health origin. The accept route gates its dispatch half on stack:deploy while the acceptance stays stack:create, the inventory and the UI match, the unmodellable-compose copy no longer prescribes a fix for the wrong refusal, and its cause is capped. * fix(gitops): tighten the redrive guards and report both accept halves The warranted predicate now leaves a disabled Blueprint, a target with an operation in flight, a fenced target, and a stale open run under an observe policy alone, so the reconciler does not race a progressing rollout or re-drive a no-op. A refusal that did not hold the rollout is diagnostic rather than a per-tick warning. The accept route reports the preparation note and the withheld-deploy note together, the UI fails closed when the rollout policy is unreadable, and the rollback-fence clear is scoped to the live rollout generation. Per-application memory is pruned. * fix(gitops): detach a superseded rollout on acceptance and bound the redrive A source acceptance now always detaches the named rollout generation, marking it superseded only when it was not already, so an earlier Supersede or Rollback cannot leave the frozen strategy reading as moved on and disable the automatic handoff for every later commit. The warranted predicate is generation-aware and skips per-target pauses, and the handoff arm has an in-memory floor at the artifact retry interval and a concurrency cap, so a steady blocked preflight or a persistent refusal is not re-evaluated on every tick. A rollback fence blocks the hold clear whatever generation wrote it, matching the dispatch queue's behaviour. The accept route drops the deploy split: the configured Automatic policy is the authority for starting the rollout, as it is for the automatic source acceptance, so the acceptance stays `stack:create`, the UI gate returns to `stack:create`, and the inventory row says so. * fix(gitops): guard the preflight-drift supersede and pin the review tests The preflight-drift invalidation only records a supersede when the generation is not already marked, so a repeat drift does not append a duplicate history row. The accept-route test seeds the automatic policy it names, and the paused-target test asserts the predicate's decision rather than only that it ran. * chore: re-trigger CI after the Actions runner delay * fix(gitops): make the withdrawal check durable and refuse over an unfinished rollback The automatic re-drive predicate no longer trusts the rollout generation pointer for a withdrawal: it refuses while the latest rollout authorization recorded for the accepted source is superseded, so a placement re-approval cannot revive a rollout the operator withdrew. It also refuses to mint when a live target still carries a rollback_pending fence, whichever generation wrote it. The supersede recorder is idempotent, so repeat invalidations and re-approvals stop appending duplicate history rows; the hold clear and the predicate both ignore retired rows, which can never finish a rollback. The accept route joins the preparation note and the dispatch refusal again. * chore: refresh CI after the Actions outage * fix(gitops): mark an operator withdrawal explicitly and share the rollback gate A dedicated withdrawn_at on the rollout generation is written only by the operator's Supersede and Rollback, so the automatic policy can tell a deliberate withdrawal from a system supersede (a preflight drift, a placement invalidation, a re-authorization) and still re-drives after those. Legacy superseded rows backfill as withdrawn, the safe direction. The unfinished-rollback refusal moved into the shared handoff, so the accept route and the SourceController honour it too, with the reason logged; the predicate no longer carries its own copy. The accept note joins its parts as sentences. * fix(gitops): withdraw the latest authorization and hold over an unfinished rollback The operator's Supersede and Rollback now mark the latest rollout authorization recorded for the accepted source as withdrawn, not only the generation the pointer names, so a placement re-approval after a system supersede cannot leave the automatic policy free to redeploy what was rolled back. The unfinished-rollback refusal moved into the shared handoff as a holdable blocked refusal: it writes the reason on the application and the accept note shows it, instead of a silent skip. The rollback target set falls back to the fenced target's own generation, so the rollback can still be finished after an acceptance detached the pointer. The migration backfill is flag-tracked and transactional, the marker lookup is ordered by rowid, and the accept note joins its parts as sentences. * test(gitops): pin the rollback route, the withdrawal migration and the tick tests The rollback route gained its own tests: a placement pointer with an older authorization is withdrawn, and a fenced target keeps a resolvable target set after the pointer detached. The withdrawal backfill has a migration test covering the four states (fresh boot, column without flag, later system supersede, dropped column), and the latest-authorization lookup has a skewed-clock ordering test. The re-drive tests now use the settled tick-until helper instead of a single tick, so the handoff concurrency cap cannot defer the application under test. The rollback hold reason, the feature docs and the Resume note now say "finish it, then resume". * chore(deps): bump proxy-addr to 2.0.8 and source-map-js to 1.2.2 New advisories (GHSA-jqcg-44mw-7w3h, critical, and GHSA-68fv-2mgg-jv7q, high) require either a patched version or a VEX statement. Both patches are inside the existing dependency ranges, so the lockfiles move without any application change. * test(gitops): pin every withdrawal-migration state, including the flag gate The migration test now boots one database state per case: the column dropped without the flag, the column present without the flag (an earlier head, or a crash between the ALTER and the backfill), a fresh boot with no rows, and a later system supersede left untouched on the next boot. Reverting the flag logic to a column-presence check now fails three of the four, which is what the flag exists for. * fix(gitops): report the handoff outcome on accept and scope the docs claims The accept route now returns a skipped handoff's reason, so the operator can tell a started rollout from one waiting for authorization, paused, or held, and the UI prefixes the acceptance to the warning toast. The Resume reply under the Automatic policy no longer tells the operator to authorize when the reconciler will start the rollout itself. The feature docs scope the recovery-capture claim to Local and proxy nodes (Pilot capture is a tracked follow-up) and state that a Git-managed Blueprint skips the managed-project drift comparison by design. * fix(gitops): stop the accept note repeating the artifact reason When the preparation did not resolve the artifact identity, the accept note carried the preparation note and the handoff's unresolved-artifact skip, which say the same thing. The skip reason is now a shared constant and is suppressed when the preparation note already covers it; every other skip reason still reaches the operator.	2026-10-06 13:35:29 -04:00
Anso	661412a7cb	fix(blueprints): announce a Suggest-mode drift once per episode (#2086) ... * fix(blueprints): announce a Suggest-mode drift once per episode A Blueprint in Suggest drift mode raised one identical blueprint_drift_detected notification per reconciler tick for as long as the drift lasted. A drifted row is re-checked every tick so a drift that resolves is noticed, and every check re-observes the same drift, so the suggest branch dispatched on every pass and filled the bell and any routed channel. The check now passes whether it observed a new episode into the mode's response. An episode starts when the row first reads a drift state and ends when a check finds the workload converged, so a repeat is silent while a drift that clears and returns announces again. A held target belongs to the same episode: the hold is a decision about the repair, not a resolution of the drift, and the episode keeps its original start time through it. Enforce and Observe behavior is unchanged. Verified: 10687 backend tests pass; the only failure is the SSH transport integration test, which needs loopback port 22 and fails the same way on main because this host runs an SSH daemon. Typecheck and lint are clean. The new tests drive the real reconciler tick on both the Inline and Git-managed paths: three ticks on a persistent drift produce one alert, a drift that clears and returns produces two, and a held target produces none while keeping its drift age. * fix(blueprints): stamp the drift start when a hold began the episode An episode that began as an Enforce hold carried no drift start: the hold write records no last_drift_at, and the drift write that followed the hold treated the row as already drifted, so it skipped the stamp. The row then read drifted with a null start time, and the deployment table fell back to last_deployed_at for "Last activity", showing a stale deploy time for a current drift. The drift write now stamps the episode start when the row has none and keeps the existing start otherwise. Both the Inline and Git-managed observation paths read the episode through one helper, so the episode definition and its start time have a single home. Verified: 108 tests pass in the touched files, including a new test that drives a real Enforce tick from active into a hold, switches the Blueprint to Suggest with a fresh approval, clears the hold, and asserts the drift start is recorded with no new alert. Typecheck and lint are clean.	2026-10-05 19:53:41 -04:00
Anso	fc96137788	fix(gitops): authenticate SOPS values with their key path (#2081) ... * fix(gitops): authenticate SOPS values with their key path SOPS authenticates every encrypted value with its key path as AES-GCM additional authenticated data. The decoder never passed it, so values in a real sops file failed authentication and the pull surfaced a generic error while the poller retried. Thread the key path through YAML/JSON decryption (colon-joined with a trailing colon, array elements keep the parent path) and derive it for dotenv and INI stores, where a [sops] section marks INI and values outside a named section belong to DEFAULT. Authentication failures now classify as decrypt_failed instead of being reported as a wrong identity. The regression test decrypts files generated by sops 3.13.3 with a test-only age key, so fixtures no longer share the decoder's assumptions. * fix(gitops): keep dotenv newlines and nested sops keys in SOPS output A dotenv value containing a newline was emitted across two lines, which truncated the secret and turned the tail into a separate env_file line. Escape newlines back to \n for dotenv output, matching sops. A nested map key named sops was dropped because the metadata skip applied at every depth. Strip sops from the root record only; nested keys are data. Add real-sops fixtures and exact assertions for both, and pin the full INI output instead of substring checks. * fix(gitops): quote dotenv output Compose would otherwise rewrite sops escapes newlines as \n, but Compose expands escapes only inside double-quoted env_file values. Unquoted output delivered a literal \n, an interpolated $, or a comment-truncated value to the container. Quote and escape exactly the values Compose rewrites: newline, carriage return, tab, backslash, double quote, $, #, a leading single quote, and edge whitespace. Add a real-sops metacharacter fixture, an exact-output unit test, and a docker-integration test that feeds the decrypted file to docker compose and asserts the container receives the original bytes. * fix(gitops): format decrypted dotenv for the consumer role Compose-style quoting applies only to inputs Compose parses as env files. A dotenv-format file consumed verbatim as a file-backed secret kept different bytes than the plaintext: $ was escaped, # quoted, and edge whitespace wrapped. Pass the consumer role through the overlay and default the decoder to the sops-faithful output, so only env_file, include-env, and interpolation-env inputs get Compose formatting. Add role-split coverage at the overlay level and a fixture test pinning the sops-faithful output. * test(gitops): cover the secret role in the dotenv consumer split The overlay role test covered env and config; add a secret input so a broadened Compose-formatting condition fails the suite. * fix(gitops): validate and deploy SOPS stacks from the decrypt overlay Two pre-existing defects kept a SOPS-managed stack from ever validating or deploying: - The compose runner's working-directory guard allowed only the managed area and the temp root, so candidate validation rejected the decrypt overlay under $DATA_DIR/git-secrets with "Invalid working directory". Allow that root, shared with the overlay's own path resolution. - A single-file stack carries no project flag in its live or captured prefix, because the Compose project name normally comes from the working directory basename. Running from the overlay made Compose create a stray project named after the operation id and leave the real stack untouched, while the API reported success. Pin -p <stack> whenever Compose runs from an override directory that is not the canonical stack directory. Add regression tests for the guard and for live and captured prefixes, and cover the no-double-pin case. * fix(gitops): correct SOPS readiness, JSON output, and failure reporting Three QA follow-ups in one pass: - Multi-recipient readiness: a file encrypted to several age recipients is decryptable by any one of them. Readiness, LKG, recovery, rollback readiness, rotation impact, and the readiness route now evaluate per input instead of requiring every recipient on the node. Legacy capabilities without per-input recipients keep the flat-list check. require_encrypted now governs managed repository content only, so a missing unmanaged project .env no longer refuses every stack. - JSON output: a JSON SOPS document is detected and re-emitted as JSON with sops's tab indentation instead of being converted to YAML. - Decrypt failure reporting: prepareOverlay preserves SopsDecryptError, a shared asGitSourceFailure maps it to a new SOPS_DECRYPT_FAILED code for fetch, apply, and reconcile, the route returns 400 with the message, and backoff treats it as operator action instead of a transient retry. * fix(gitops): enforce the encrypted-source policy on a synced .env A plaintext repository .env with sync_env on was accepted under require_encrypted because discovery records it as unmanaged and the policy judged only managed inputs. The synthesized sync entry now carries its encryption kind and is merged into the capability inputs, so the file Sencho writes to the stack root is judged by the policy. The SOPS project-name pin overrode a compose-declared name:, starting a second project beside the running stack. The pin now stands down when the compose layers declare a top-level name. * fix(gitops): contain the compose name probe to the stack tree The compose-name check read every -f layer, including generated overlay files outside the stack tree, with no containment barrier and a stat-then-read pair. Resolve the stack directory once, skip any layer that resolves outside it with an inline barrier at the read sink, and read directly with a content-length bound so the path-injection and file-system-race findings are cleared. Generated overlay layers cannot declare a project name, so behavior is unchanged.	2026-10-05 09:18:05 -04:00
Anso	143f5a2005	feat(blueprints): open a Blueprint and the create dialog over the GitOps workplace (#2083) ... From the GitOps workplace, opening a Blueprint or declaring a new one sent you to Fleet first and only then opened the sheet, so the list you came from, with its filters and scroll, was gone. The application sheet's Open Blueprint did the same, and Escape could not take you back. The Blueprint detail sheet and the create dialog now open over the workplace, the way a Direct application's Git source already does. They are the same sheets the Fleet Blueprints tab hosts, driven by one shared hook, so permissions, the create-then-review flow and label handling are identical in both places. A row's Open Blueprint, the application sheet's Open Blueprint (which hands its sheet over) and New Blueprint all go through it, and fall back to the Fleet Blueprints tab where the workplace cannot host them. The link back to the GitOps portfolio is left out of a sheet that already sits over it. Reading node labels is sequenced so an older answer never replaces a newer one, and a failed read says so instead of leaving label targets silently empty.	2026-10-04 18:20:13 -04:00
Anso	8a9115f48c	feat(blueprints): call the tab Blueprints, add a Needs attention filter and keep the catalog on screen (#2082) ... The Fleet tab was called Deployments while every surface inside it said Blueprints, its empty state repeated a heading and offered two create buttons, and the populated catalog opened with a third naming of the same thing. The filter row had a Drifted chip that missed failed, conflicting, blocked and awaiting-confirmation Blueprints, and a tile showed only a coloured dot, never the state it stood for. Every refresh blanked the whole tab (closing an open sheet's context) behind a plain "Loading blueprints" line. The tab is now Blueprints. One header row carries the count and New Blueprint; the empty state is just its card with one create action. Needs attention replaces Drifted and covers failed, name conflict, evict blocked, awaiting confirmation, held repair, drift and a rollout that needs reconfirming; a tile names its strongest non-active state, and an empty filter offers Show all. The first load shows a skeleton shaped like the catalog, and only if it is slow. Later refreshes keep the catalog and any open sheet on screen; a failed refresh over a catalog shows a notice with Retry and the last list, and the error card is only for a tab with nothing to show. Overlapping refreshes are sequenced, and a failed refresh after creating a Blueprint no longer opens its sheet later. Docs and the e2e tab lookup follow the new name.	2026-10-04 17:35:40 -04:00
Anso	d3fb532dfa	feat(blueprints): give the Blueprint sheet one Answer and a shorter toolbar (#2080) ... The Blueprint detail sheet opened on GitOps rollout and runtime stages that say nothing true about a Blueprint that is applied rather than rolled out: a healthy, fully deployed one showed an amber "rollout not executable" and "runtime never applied". A failed deploy was told five times, its Docker log pushed Re-apply and Withdraw off the right edge of the table, and the toolbar carried five buttons, three of them rare. An Inline Blueprint now leads with one status block built from its own deployments: in sync, failed, drifted, needs confirmation, reconciler off and so on, naming the affected nodes and offering the verb that resolves the loudest problem (Re-apply, Review rollout, Review state, Evict, Enable) only when this session may run it and the server would accept it. The GitOps status takes over when it holds a louder warning, and the source that did not speak is named in a quiet marker. Rollout "not executable" and runtime "never applied" are left out for a Blueprint that has no Git rollout; a paused rollout or a failed runtime still show. The toolbar is Apply now, Edit, Enable or Disable, and Delete. Convert to Git, Detach Git and Retire to Direct sit in the Content row next to the source they change, and a pinned Blueprint gets an Unpin button. A failed row shows the one line that says why, with the full output behind a control, and its action is named Re-apply to match the status block.	2026-10-04 00:46:50 -04:00
Anso	7051673553	feat(blueprints): pin the create actions and show form errors beside the field (#2079) ... The Blueprint create dialog kept Cancel and the primary action at the end of the scrolling body, so at common window heights the button was off screen, and a missing name or target only surfaced as a toast while the field it named was scrolled out of view. The create actions now live in the dialog's own footer and stay in view, with a short line saying whether the action adds node labels or deploys nothing yet. Name, compose and target problems show below their field, the name field takes focus, and each message clears as soon as that field is fixed. Reconciler enabled is a switch with a one-line consequence, the three stateful notices are one line, a failed compose analysis is logged and labelled instead of silently leaving a stale classification, and the default compose and drift card captions read in plain words. Deleting a Blueprint uses the shared destructive confirm with a busy state and keeps the typed-name check.	2026-10-03 21:45:45 -04:00
Anso	03e2636e6a	feat(blueprints): create node labels and see matched nodes from the Blueprint form (#2077) ... The Blueprint form sent operators to Settings > Nodes whenever no node label existed, and closing it to do so discarded everything typed. It also showed the same labels in two rows (any / also all) with no list of the nodes a selector actually reaches. Targets now has one row of node labels with a Match any / Match all switch and a live "Matches N nodes" list. "+ Label" stages a new label on chosen nodes inside the form; the pending tags are listed and removable, and they count toward the match list immediately. Nothing is written to a node until the form is committed: Review rollout (or Create blueprint / Save changes where a rollout cannot be reviewed) saves the Blueprint, adds the staged labels through the existing node label endpoint, then opens the rollout preview. Save as draft stores the Blueprint only. Staged labels that the final selector does not use are never written. A Blueprint saved with both an any and an all list keeps showing two rows. Docs and the create-and-approve tutorial follow the new flow; two screenshots that no longer match any step are removed.	2026-10-03 21:12:28 -04:00
Anso	966cf7255b	fix(security): make ignored forwarded headers loud (#2069) ... * fix(security): make ignored forwarded headers loud Behind a reverse proxy, forwarding headers are accepted only from peers listed in SENCHO_TRUSTED_PROXY_CIDRS. When the list is unset or invalid, that failure was silent: client addresses, secure-cookie detection, SSO callback URLs, Pilot enrollment URLs, and rate-limit keys all fell back to the proxy, and an invalid list (for example a bare IP) was ignored without a word. Log the effective policy at boot, naming every rejected entry and the reason. Warn once per untrusted peer when a request carries forwarding headers the policy is ignoring, with the exact CIDR to set. Document the behavior in the reverse-proxy and troubleshooting pages. The fail-closed semantics of the allowlist are unchanged. The login limiter test now drives the real createApp pipeline instead of setting trust proxy by hand, which is what hid this regression. * fix(security): keep forwarding headers out of the pilot tunnel The tunnel bridge framed the hub's incoming request headers as-is, so a reverse proxy in front of the hub had its X-Forwarded-* headers relayed to the agent. The agent replays them to its loopback listener, which saw them from 127.0.0.1 and warned about an untrusted proxy, suggesting the operator trust loopback. Strip forwarding headers when framing http_req and ws_open so the tunnel carries no unverifiable client address; the hub's own pipeline has already applied its trusted-proxy policy to the original request. Forwarded (RFC 7239) is stripped defensively. Also narrow the reverse-proxy docs to the limiters that actually share a bucket (sign-in, SSO, and other unauthenticated requests; signed-in requests stay per user or per token), assert the boot log wiring, and word the runtime warning as "add" when the list is configured, or point at the rejected entries when it is invalid. * docs(security): qualify trusted-proxy attribution and warning scope QA follow-ups on the previous commit. The reverse-proxy paragraph now says the proxy transport is the one where a remote can name the operator's address if it trusts the forwarding instance; Pilot agents receive tunneled requests without forwarding headers, so their local audit trail records the tunnel ingress and the hub's audit log names the operator. The runtime warning is documented as HTTP-only, since WebSocket upgrades bypass Express middleware. The warning itself now says "a reverse proxy or another Sencho instance" because a proxy-mode remote's peer is the hub.	2026-10-03 10:35:43 -04:00
Anso	84f335097c	refactor(dashboard): give the Home stack health table's GitOps state its own column (#2068) ... The GitOps source-state tag (accepted, review required, pending update, and so on) sat beside the stack name and crowded it. It now has a GITOPS column after SOURCE, the name cell keeps just the name and the update icon, and clicking the tag still opens that stack in the GitOps portfolio. The column only appears while at least one stack in the table has a state the tag can show, so an instance with no Git-managed stacks keeps the table as it was. Stacks without a state read "--" in it, and a state this build cannot name no longer leaves an empty clickable button behind.	2026-10-02 20:55:00 -04:00
Anso	67889e168f	refactor(gitops): put the workplace masthead on the shared hero and drop the duplicate attention column (#2066) ... The GitOps masthead is now the shared hero masthead with a GITOPS kicker. Its meta line carries how many nodes are reporting and when the data was updated, and reads "refreshing" while a refresh is in flight, so the floating refresh pill over the page is gone. A failed refresh keeps the last-known data with the same qualifier as before. Attention is one of the masthead counts. The application table no longer repeats attention in its own column: the attention list above it, the row tint and the state dot already carry it. The dot now has an accessible name (and a tooltip) giving the most urgent reason, so a row beyond the list's cap is still explained and screen readers get the reason too. Facet chips read the shared tone classes, and the portfolio load-error Retry is a regular button.	2026-10-02 20:23:19 -04:00
Anso	2d89f13ef2	feat(gitops): resolve attention in place (#2065) ... The attention queue now lists each application once, with its most urgent reason inline and a count of the rest, and every entry carries the step that resolves it where the operator already is. Retry and Resume on a Direct source run from the queue on the application's own node; Review update opens that stack's Git source sheet already pulled to its diff, so applying an update is two clicks from the GitOps page. A step the session may not run, or the server does not offer, opens the Git source sheet instead of a button that would be refused. The full-page application view is replaced on desktop by a sheet over the portfolio list, which stays mounted underneath, so closing it lands back on the same filters and page. A Direct application is its stack's Git source, so a row or a link that names one opens that sheet and never a second read-only copy. Blueprint applications keep their authority and rollout controls in their own sheet. The whole table row is now clickable; the actions menu is unchanged. The Retry, Resume and Suspend request now lives in one shared handler used by the sheet and the queue, and names the stack in its toasts. The phone screen keeps its application page.	2026-10-02 20:05:05 -04:00
Anso	ee62a449b2	refactor(gitops): organize the Git source sheet into tabs and make editing deliberate (#2064) ... The sheet was one long scroll with the repository form always open and a toolbar of five buttons. A linked source now has tabs: - Overview: the status block, with Review update, Retry, or Resume in it. - Source: a read-only summary of the configuration, an Edit button into the form (with Cancel), Adopt onto Blueprint, and the manifest. - Automation: the apply behavior, Suspend or Resume, and the provider hooks. - Secrets. - Drift, only while the application has drift. An unlinked stack gets no tabs, only the form. The toolbar keeps Pull now and Detach (named the same as its confirmation); Update appears only while something differs from what is stored, so it never sits there doing nothing. Reading and refreshing are more careful. A failed read now says so with a Retry instead of presenting an empty "link a source" form; a change that was saved but could not be re-read keeps the linked source on screen; a pull, apply, suspend, or resume refreshes the status without discarding a change that is still being made; and a slow answer for a stack the sheet has left can no longer overwrite the one it shows. Detaching empties the form instead of leaving the removed source prefilled. Docs, the Git sources page, and the two end-to-end specs that drive the sheet are updated.	2026-10-02 18:58:27 -04:00
Anso	b689733447	refactor(gitops): one layered status across the Drift tab, Git source sheet, and application view (#2063) ... * refactor(gitops): one layered status across the Drift tab, Git source sheet, and application view The GitOps state used to be a stack of tinted cards on five surfaces: a posture card, approval chips, a card per facet, a card per target, and a caveats box, restating the same state several times. Each surface now shows one status: a single answer (with its resolving button, such as Review update or Retry), a quiet row naming each stage that applies, and an Evidence control holding recorded authority, policy lines, targets, and anything Sencho could not prove. - Stages that say nothing about an application (unbound placement, an absent rollout) are left out of the row instead of printed as "n/a". - Evidence is never silently lost: the answer carries a marker for partial evidence, unreached nodes, and unproven items, and every attention reason is kept, the first in the answer and the rest in the evidence. - The Git source sheet's Status section drops from about 450px to about 170px, and its separate source-state and last-applied rows go; the header now says "never applied" until something is. - The application view no longer shows the posture twice. Adds a domain-free StatusPath primitive, a pure status model, and a GitOpsStatus composer; retires the facet-card and approval-chip components and keeps the policy lines. Docs, the tutorial, and the two affected screenshots are updated. * test(gitops): open the status evidence in the digest drift spec The per-service digests now sit under the Evidence control of the GitOps status, so the spec opens it before looking for them.	2026-10-02 18:30:16 -04:00
Anso	bba9d30204	feat(gitops): open the Git source in place and animate Pull (#2062) ... Opening a Direct application's Git source from the GitOps portfolio (attention queue, row menu, application header) now opens the Git Source sheet over the portfolio instead of switching to the stack editor. The sheet targets the application's own node, so reads, pulls, applies, provider hooks, repository secrets, and the manifest all go to that node even when another node is active. Changes made in the sheet refresh the portfolio and the open application view. Adopt onto Blueprint is offered only when that node is the hub, where adoption runs. Sheet toolbar actions gain a pending state: the button locks at once and, once the work outlasts the busy delay, its own icon spins. Pull now and Save use it, so the refresh circle turns while a pull runs.	2026-10-02 17:39:37 -04:00
Anso	d2d2a5d347	fix(gitops): match Blueprint repo claims on repository identity (#2054) ... * fix(gitops): match Blueprint repo claims on repository identity The guard that keeps one live Blueprint-mode application per repository compared configured_repo_url as an exact string. A repository has an unbounded number of valid spellings, so the HTTPS URL and the SSH URL for the same clone each bound their own Blueprint, with or without a .git suffix, with a trailing slash, and with host casing either way. Compare a canonical repository key instead: host lowercased, trailing slashes dropped, and one trailing .git stripped from the last path segment, with the path left case-sensitive because Git hosts treat it that way. The key is derived through the same RepoIdentity shape already stored per application, so the stored identity and a recomputed key agree by construction. Fail-closed. A URL naming no repository cannot be compared against anything, so it refuses rather than passes, which means one unidentifiable live claimant blocks every later conversion until it is retired, detached, or tombstoned. Ingress now refuses such a URL outright so the state cannot be created, and the refusal names the Blueprint that holds the claim so the operator can act on it. GitOpsTransitionError carries an optional code, because the binding service was telling two refusals apart by testing a message for the word "repo". * fix(gitops): drop the port from the repo claim key and align ingress Two ways a Blueprint could still bind a repository another Blueprint already claimed, both found reviewing the claim guard. RepoIdentity folds an SSH port into the host it serializes, so a forge serving HTTPS on 443 and SSH on 2222 gave the HTTPS and SSH spellings of one repository two different keys, and both converted. The port is now not part of the key. Two repositories on two ports of one host therefore collide, which costs a refusal rather than a second claim, which is the only direction this key is allowed to be wrong in. Ingress judged the submitted URL while the guard judges the stored one, and those can disagree. An scp-style URL carries `?` and `#` through verbatim, so it keyed fine as submitted while the ssh:// form actually persisted could not be parsed back at all, and a host holding a slash reparsed as a different repository. Those rows were accepted on the way in and refused at conversion for a URL the operator never sees. Ingress now requires the stored form to name the same repository the submitted one did, so a source is never accepted and then unclaimable. The transition fixture now stores the derived ssh:// form, which is what every production writer persists and what the guard reads, so it no longer asserts against a row shape the code cannot create.	2026-10-02 17:18:43 -04:00
Anso	c0b5708dd7	fix(gitops): date a recorded placement refusal instead of presenting it as current (#2042) ... A placement decision that refuses records its reason once, and the reason is cleared only when an operator approves the placement, edits the policy, or a later change re-decides. A decision runs only when a placement change is pending, so a refusal recorded because an operation was in flight outlives that operation and nothing corrects it on its own. The facet card rendered the reason in the present tense. An operator arriving after a deploy finished was told the system was withholding a placement because a conflict was in flight, and had to work out that the conflict was over. The record is kept and the moment it was made is shown. The timestamp was already stored and already projected as the decision's own date; the frontend carried the field and rendered nothing. Re-deciding instead would need a trigger that does not exist today, and would let an automatic approval land at a moment the operator is not watching, which is a larger claim than the evidence supports. Both tests fail if the behavior is reverted. The backend case drives the real transitions: an operation in flight, a refusal recorded for it, the operation finishing, and the projection still reporting the decline carrying the moment it was recorded rather than the moment it was read. Verified against a running instance at desktop and phone width: the card carries the date in both, and the only desktop delta is the one added line.	2026-10-02 09:47:41 -04:00
Anso	1e1bea0af5	fix(gitops): read a redeployed failed generation as work in progress (#2045) ... * fix(gitops): read a redeployed failed generation as work in progress A recorded health failure keeps being reported until a newer verdict lands, which is right, but nothing knew that a redeploy of that same generation was in flight. The attention reason carrying failure tone outranks the portfolio's in-flight rule, so redeploying a failed generation reported the application as failed for the whole duration of the redeploy, naming a settled state nobody was working on. The projection now carries whether a live operation is running against the generation a recorded failure is about, and the attention classifier withholds the failure reason when it is. The health facet still reports failed, which stays true: the last verdict did fail, and its promotion was withdrawn. The runtime facet already reported the same target as deploying, so the portfolio reads the application as work in progress instead, and a redeploy that fails again reports its own verdict. The generation match is the rule, and it is compared against the failure's own record rather than the running pointer, so a redeploy of a different generation cannot silence a failure that belongs to what is still running. The scope is a target whose operation recorded a generation, which only a Direct deploy writes, so a redeployed Blueprint target keeps reporting its failure as before. * fix(gitops): keep a redeployed failed generation in progress through its health window The in-progress reading ended at deploy-bind, which clears the live operation. The health gate reserves its run before the deploy and leaves it open while it observes, so the application dropped back to failed for the whole observation window: failed, in progress, failed, then the new verdict. An operator watching it, or a UI that polls, sees a flip-flop and can miss the progress entirely. A second moment now qualifies alongside the live operation: a stack health run for the same generation that is still observing. That reads a run that already exists and changes no gate behavior, window, or decision. The two arms cover different populations, which is why neither is redundant. A live operation on the failed generation needs an active_generation_id, which only a Direct deploy writes, so a Blueprint retry is visible only through its reserved run. The generation match is the failure's own record on both arms, so an observation of some other generation cannot supersede it either. Two corrections to how this was described. The automatic retry the health rollout policy performs dispatches through the Blueprint adapter, which refuses a Direct application, so the Direct deploy step covers an operator's own redeploy rather than the automatic one. And the mixed-version contract is now enforced rather than described: the remote boundary accepts only an absent field or a boolean, and the classifier reads it as true, so a peer cannot suppress a real failure by sending a truthy non-boolean. Test pins cover both arms independently, including the case that distinguishes judging an observation against the failure's generation from judging it against the running pointer, an observation of another generation, an observation that has already settled, one that names no generation, the Blueprint path, and a projection arriving with the field absent. * fix(gitops): stop the health supersede rule outliving the facts it rests on Four corrections to the rule that keeps a redeployed failed generation reading as work in progress. The flag was computed for every target, including ones whose own last verdict passed and ones a recovery is restoring. It now answers false from the row before any lookup, so a healthy target costs nothing and the flag cannot be true where the comment on it says it is meaningless. A Direct redeploy has a handover gap the previous change papered over. A Direct deploy binds the generation, then does its baseline, drift and exposure bookkeeping, and only then does the route open the observation, so for that window neither moment has anything to read and the recorded failure shows again. Blueprint rollouts reserve the run before the apply and never have that gap. The gap is now stated and pinned by a test rather than described as covered, because closing it would mean opening the observation before the bind, which reorders the deploy path. The catch around blueprintDeployStarted returned blocked without abandoning the run it had reserved, unlike its refusal, ack and outcome siblings. Left open, that row reads as observing for a deploy that never started, and the supersede rule would read it as this failure's successor. It is abandoned now, for the reason the ack catch already gives. Two claims were wrong. The mixed-version protection is the shape check at the remote boundary, not the comparison in the classifier: for the values that can reach it, a bare negation behaves identically, so the comments now say what actually holds. And a redeploy that settles unknown does restore the recorded failure, because the rule only ever stood in for a verdict that was coming, so that is now pinned rather than assumed. Posture is now asserted wherever the flag is, not just the reason, since the suppression only reaches in_progress because the runtime facet already reads as in flight. * test(gitops): pin the abandoned reservation when a deploy cannot be opened The abandon in the deploy-start catch was the only thing standing between a refused dispatch and a run row left observing for a deploy that never opened, and deleting it failed no test. That path now has one. A dispatch whose deploy-start transition throws leaves nothing pending, because the throw happens inside the transaction before the target row is written, so what is left behind is the run row itself. The test makes the transition throw, and pins that the reservation ends with a terminal status and an end time rather than reading as an open observation of the accepted generation. The comment above the catch said the run would make a later dispatch skip the target, which is the ack catch's mechanism and not this one. It now names the hazard that is actually here: an unarmed observing row that the health projection would read as work about to produce a replacement verdict. Also pins the tombstone guard, which was the other untested branch, and settles the run under the same id the verdict reports rather than a second one.	2026-10-02 09:23:47 -04:00
Anso	50a9450abb	refactor(fleet): flatten the Readiness tab chrome and start its check when Fleet opens (#2060) ... The tab stacked its own title block and a hero summary card under the Fleet masthead. Replace both with one flat status row and put the findings table first. The readiness check now runs from the Fleet shell, so it is usually done by the time the tab is opened and survives tab switches. The skeleton only appears when the check is actually slow, and a failed or stale result is re-checked when the tab is reopened.	2026-10-02 09:23:31 -04:00
Anso	8eaa954927	feat(updates): table view as the default for the update page (#2059) ... The Update page now lists pending updates as a sortable table, one row per stack across nodes, with the node-grouped card board kept behind a Table / Cards switch. The choice is remembered per browser. Row details hold the changelog, blocked reason, and per-service Apply.	2026-10-02 08:44:11 -04:00
Anso	3667651a2b	refactor(fleet): trim node card badges to what matters (#2058) ... Reachable nodes no longer carry an Online badge; Offline shows only when a node is unreachable. The local/remote type badge is gone, since the local card already has its pin. Remote cards now show a Proxy or Pilot marker in the same corner slot so the connection type is clear at a glance. The persistent Networking exposure badge is removed from the card. It fired on most nodes without detail; exposure and drift stay available in the node details sheet and the Networking filter.	2026-10-02 00:43:59 -04:00
Anso	e4ad9029a5	fix(gitops): retry an unresolved Blueprint artifact freeze from the drift check (#2039) ... * fix(gitops): retry an unresolved Blueprint artifact freeze from the drift check A Blueprint freeze resolves exact image digests for the generation a target was approved at. When that resolve cannot reach the registry, the expected artifact set stays unresolved and every drift check reports unverified until an unrelated redeploy of the same revision happens to re-run it. That made a redeploy the price of proving what a node is running, on a condition that is often a transient registry blip. The drift check now re-resolves the frozen generation itself, throttled by a new operator-set interval so a registry that is down is not asked once per target per tick. The expectation can only advance through the existing transition, so a retry cannot redefine an identity that was ever approved. Throttling reads the target's latest evidence row, which advances on every attempt including a failed one, rather than the expected set, which does not; the retry takes the target's deploy lock and skips itself when a deploy is already resolving the same generation. A target whose expected identity is unresolved now also records a caveat saying why nothing is being compared, and clears it when the identity becomes provable. The retry interval lives in a new GitOps settings section, alongside the Git source polling control it now sits next to. It is instance-scoped rather than node-scoped, because the reconciler that reads it runs the checks for every node the instance manages. * fix(gitops): resolve the approved intent on a freeze retry, not the node disk The retry introduced to recover an artifact freeze resolved whatever compose was in the stack directory at the time it ran, and recorded that as the approved identity. A reconcile tick can run days after the deploy, by which point that directory may have been hand-edited or restored from a backup, so a local change could become the approved identity for every target on the generation, and a target restarted onto the edit read as matched against content no approval ever saw. The retry now parses the compose text the intent revision hashed, after checking that hash against the live Blueprint row so an edit since approval refuses the retry outright, and reads only the node's Docker platform from the node. Which manifest child is correct is a property of the machine; which images were approved is not. The deploy-time freeze still reads the directory it just wrote, and both now share one implementation so the two paths cannot drift. Also from review: - A permanent failure is no longer retried. `unsupported_registry` and `platform_ambiguity` record `unavailable`, so gating on the qualification alone retried them forever, spending registry traffic and appending a row to a table nothing prunes. The decision is now made on the recorded failure classes. - The tick that produces an expectation does not judge by it. Enforce would otherwise redeploy a node onto a digest resolved from a tag that may have moved while the identity was unproven, which is the automatic advancement a retry must not cause. The next tick compares instead. - The lost-write-race rethrow is reverted. The race it claimed to handle cannot occur, the two swallow sites it needed were unjustified, and nothing tested them. - A remote leaf is asked for its platform over its own route rather than for a rendered compose context, so the retry never pulls a remote node's directory into the answer. Registered as a static path so it is not read as a stack named `platform`. The evidence advances in the retry tests now go through the real transition rather than by moving pointers, so the advance rule under test is the one production applies, and the deploy-time freeze is asserted to still read the node it just wrote. * test(e2e): match the role the settings sidebar actually renders The section-discovery test looked for a link named GitOps, but SettingsSidebar renders every settings section as a button that calls onSectionChange, with no anchor behind it, so the locator could never resolve and the test failed in CI before reaching its URL or viewport assertions. The section itself is fine: the registry entry, the URL scheme built in senchoRoute, and the Drift verification and Retry an unresolved image identity copy were all checked against the source while tracing this. Only the role in the locator was wrong. Locator only. No product code changes. * test(e2e): drop settings-nav assertions that were guesses, keep the behaviour Two CI runs failed on this spec's section-discovery test, and both failures were the test's fault rather than the section's. It asserted the GitOps entry is in the settings navigation and that the retry control renders at desktop and phone width, written from the settings registry rather than from the rendered DOM. The first locator matched nothing. The corrected one matched nothing either, because reading that the sidebar renders a button was never the same as checking that the entry renders at all. Two attempts and two guesses cost a CI cycle each and proved nothing, so the assertions come out rather than getting a third guess. What they were standing in for needs a browser, and the reason is recorded in the spec so the next person does not reinstate them as though they had passed. The remaining test is settings API traffic with no deploy in it, so the Docker gate and the six-minute timeout that came with the dropped deploy assertions go too. Left in place they would have skipped this spec on any machine without Docker despite not needing it. Kept: the seeded default, an in-range round trip, and the refused bounds. The resolve behaviour is covered by the backend suite against the real transition. * fix(gitops): refuse compose shapes the retry cannot faithfully resolve The retry parsed the authored compose and treated every key under `services:` as a service, but the freeze resolves the *rendered* model. Anything Compose expands can make the two disagree, and a service the authored text declares but the rendered model omits enters the expected set, is absent from the observation, and `observationMatchesExpected` reads a missing expected service as drift. That is a healthy stack reported as drifted, and under Enforce the repair cannot converge it, so it would redeploy on every tick. Four constructs now refuse the retry instead, each logged with the service that carries it: `include`, `profiles`, `extends`, and `<<` merge keys. Merge keys are included because the YAML parser does not resolve them at all -- it leaves a literal `<<`, so merged fields vanish silently, and a service can carry its own `image` while a merge key imports `profiles` invisibly. Refusing leaves the target unverified, which is where it stood before this retry existed. Also from review: - The gate and the throttle now read the latest recorded row, not the expected one. The expected pointer only advances for exact/qualified, so a failed resolve wrote a row the gate never saw: a build-only stack stayed `unresolved` there while every recorded row said `local_build_unverified`, and it retried and appended a row every interval. Both now read one row, fetched once. - A throwing retry is attempted once. It previously logged the failure and fell through to a second attempt after `finally` had released the lock, doubling registry traffic during an outage and running the repeat concurrently with a deploy. - The caveat fires when the expectation is not comparable, and the latest row only suppresses the permanent build case. Keying it purely on the latest row would have claimed drift was unchecked whenever newer non-advancing evidence followed a comparable expectation, which is a state a stale probe produces. - The removed categories are honest now: the tick hold is one reconciler interval rather than a review window, and the two GitSourceService catches for a race that cannot occur are gone, along with the comment claiming a rethrow that no longer exists. - A null platform records `platform_ambiguity` rather than a guessed child, so a hub running ahead of its leaves costs one row instead of one per interval. - Retry evidence carries its own history trigger, so an audit trail can tell a deploy-time pin from a later recovery. * fix(gitops): stop the retry looping on refusal, and unify the failure-class list Three findings from the audit of the previous commit, plus one the review of this one turned up. An interpolated image reference is now refused rather than resolved. The intuition that `${TAG:-1.25}` fails safely is wrong and the code comment said so: `parseImageRef` accepts the literal text, splits it on the last colon, and the registry is asked about a reference that cannot exist. That comes back as an ordinary retryable error, so the retry ran forever instead of never. `$` cannot appear in a legal image reference, so the check has no false positives, and a mixed compose is refused as a unit because the set's qualification is the weakest across services: resolving the other services would leave the set uncomparable while writing a row that looks like progress. A node that could not report its platform is no longer treated as a node that never can. Both situations used to record `platform_ambiguity`, which is permanent, so a single timeout on the hub-to-leaf platform read switched the retry off for that generation until an unrelated redeploy, which is the exact failure this path exists to remove. A 404 is now `route_missing` and stays permanent, because a leaf older than this hub will not grow the route on its own; a timeout, a 5xx, an unreachable daemon, or a missing proxy target records the new `platform_unavailable` and is retried. A refused retry is remembered per generation. It writes no evidence row, so the latest row's stamp never moves, the time-based throttle never closes again, and the refusal was re-derived and re-logged every reconcile tick forever rather than once per interval. The retry now reports a three-state outcome and the caller keeps a process-local map of refused generations, checked before anything else. Nothing is persisted on purpose: a refusal is a stable property of a generation, an edit clears the freeze pointers and a redeploy re-resolves from the rendered model without consulting the map, and a restart costs one redundant refusal per affected target. From the review of this delta: the failure-class allowlist existed twice. `portfolioAggregator.ts` had its own copy, so adding a class to `json.ts` alone left the fleet projection rejecting any record that carried it and degrading a remote row to unknown evidence. Both call sites now use one exported predicate that reads the same set the type is declared beside. GitSourceService.ts is reverted to match main: it had accumulated comment-only lines here, and the file is not this PR's concern.	2026-10-01 17:45:06 -04:00
Anso	b6b059a3fb	fix(gitops): surface withdrawal residue for a tombstoned target (#2053) ... A tombstoned target reports the tombstone before the runtime facet reads any recovery field, so a target withdrawn while its recovery had failed reported nothing anywhere. Both retirement transitions clear the failure and LKG columns but leave recovery_phase and recovery_ref on the row, and no projection read them back, so the failed recovery was gone from the runtime status, the rollout status, and the attention queue in one step. Mint a withdrawal_residue limitation in deriveTarget for a tombstoned row carrying a failed recovery, on the same condition deriveRuntime reads, so the caveat and the status cannot disagree about a row. It rides the limitations array deriveTarget already threads, so it reaches the caveats surface with no wire change, no new route, and no new broadcast. Tombstone precedence is unchanged: the runtime status stays tombstoned and the rollout status never sees the recovery. Only a retirement that carried a failed recovery is qualified. After any withdrawal Sencho cannot prove the compose project came down, because the teardown is best effort and swallows its own error, but qualifying every retirement would train a reader to skip the block.	2026-09-30 18:29:41 -04:00
Anso	3623ce2d89	feat(gitops): persist the observed apply invocation and classify invocation drift (#2032) ... The invocation drift class had no producer because the observed compose invocation was never persisted anywhere the deriver could read it. Emitting the authored invocation against a permanent unknown would have been constant fabricated drift, so the class was absent rather than wrong. This records the observation and adds the producer that compares it. What Compose was invoked with is read back off the running project's container labels at apply time, not off the argv Sencho built, because reading the argv back would only ever confirm what Sencho already believes. It lands on a new nullable per-target column, and the column stays null when the node cannot be reached, which is the same absence as never having looked. A new producer compares that record against the invocation the accepted generation was authored with, and reports nothing while either side is missing, unreadable, or not comparable, naming each case in a caveat rather than reporting agreement. The single-file case is what the design turns on. Compose omits every flag for a single-file selection, so a generation's argv is empty while the observed label always names at least one file. The authored file order therefore falls back to the application's configured compose selection, which is what makes a true negative reachable on the majority of stacks instead of a standing false positive. The comparison is Direct-only: a Blueprint target is deployed under its own stack name from a generation whose argv was authored for the source stack, so there is no correct expected side to compare against and the class stays silent for it. The read requires the project's containers to agree. An ordinary up recreates only the services whose configuration changed, so a half-recreated project carries labels from two invocations; a disagreement records nothing rather than sampling whichever container the daemon listed first, which would flap across applies and could report agreement while a sibling container disagrees. The history column for this observation already existed in the schema without a writer, so the audit row now records the evidence rather than only the fact that a column moved. A new target column is added additively for existing installs, and a migrated target stays unobserved rather than being backfilled with a value no apply recorded. Validated over the live route: a differing invocation produces one item with both identities real, a matching one produces nothing, and a missing observation produces nothing plus a caveat. Unit coverage spans the canonicalization, the label read, the three acceptance states, the additive migration, and the drift row rendering.	2026-09-29 08:50:51 -04:00
Anso	b833e554b4	fix(security): resolve the WebSocket client IP through the trusted-proxy predicate (#2031) ... The host console WebSocket handler took the audit row's client IP from the first X-Forwarded-For hop ahead of the socket address, without checking that the peer was a trusted reverse proxy. Express's `trust proxy` setting does not apply on an upgrade, so the header was taken at face value and any principal able to open a host shell could write an arbitrary IP into the host-console audit trail. CWE-348, audit-log integrity only: the address is not used for an access decision in that handler. Client-IP resolution on the upgrade path now goes through a single helper that applies the same trusted-proxy predicate as the HTTP path and mirrors proxy-addr: a forwarded hop counts only when the direct peer matches SENCHO_TRUSTED_PROXY_CIDRS, and the recorded address is the leftmost hop that no trusted proxy supplied. Unset, malformed, or duplicate proxy configuration fails closed to the socket address, as it already does for `req.ip`. The host console audit row still records the real client address behind a properly configured reverse proxy, and now matches what the HTTP audit path records for the same request. A remote node records the forwarding instance's address unless that instance is allowlisted, which the configuration reference now states.	2026-09-29 04:20:59 +00:00
Anso	c92004dc6a	fix(blueprints): retry a refused drift repair instead of stranding the row (#2028) ... * fix(blueprints): retry a refused drift repair instead of stranding the row A drift repair that could not finish left the deployment row reading `correcting` forever, so nothing ever retried it. `drift_enforce_start` writes `correcting` before the repair runs, and a repair that then answered without ever reaching a deploy wrote no status of its own. That state is in-flight as far as the decision is concerned, so `computeDecision` skipped it and the preview projection rendered it as the informational `in_flight_correct`. No later tick re-checked the node: the workload kept running the moved image, one correction-failed alert fired, and then nothing, for as long as the drift lasted. Four ticks on a refused repair recorded one drift check, one repair attempt and one alert, and did nothing on the rest. A refusal, or a repair that throws, now settles the row back to `drifted` with the reason in `last_error`, so the next tick checks and retries it. A throw counts because resolving a remote node's platform label is a network call made before any deploy, so it can reject with nothing written and used to escape to the per-blueprint catch with the row still `correcting`. The row deliberately does not become `failed` or `pending`. Both are retry buckets for an ordinary deploy, so the next tick would repair by tag and adopt whatever the tag points at now, which is the one thing a digest-pinned repair exists to prevent. The failure alert fires when the row changed across the attempt, compared against a snapshot read before the repair ran. It cannot be decided from the row afterwards: a repair that reaches a deploy writes its own `last_error` and returns the very same string, so asking the row afterwards finds the failure already sitting there and mistakes a new incident for a repeat, dropping the first failure of every kind silently. Losing the deploy lock is a deferral rather than a failure, so the row still leaves `correcting` with a reason and the operator is not paged about a race they cannot act on. A converged row now forgets its last failure, as the deploy acknowledgement already did, so a failure that returns after a repair counts as a new incident. computeDecision also treats a `drifted` row at the current revision as a drift-check target. The projection already reached it through its own status fallback with the same action, severity and confirmable membership, so nothing an operator sees changes; it is here so the decision a tick acts on is not two files away in the preview projection. Verified: 10333 backend tests pass with no failures, typecheck and lint clean, and an e2e case drives drift, repair and reconvergence on a real Inline Blueprint, asserting the row returns to Active with no drift summary and the workload runs the approved digest again. * fix(blueprints): correct the drift repair settle found in review Four corrections to the refused-repair settle, all found by reviewing the change rather than by a failing test. The deferral note was the serious one. Losing the deploy lock is silent, and it wrote a note about the race into `last_error`, which is the replay key. One lost-lock tick between two identical failures therefore displaced the failure being replayed, and the next tick reported that same known failure again as though it were new. A deferral now leaves `last_error` alone: the row keeps saying what it last actually failed with, which is worth more to an operator than a note about a race that is already over. An alert now quotes the row rather than the outcome where the outcome carries only a shorthand. The name-conflict paths return the bare word `name_conflict` and then park the row for good, so that one alert is the only signal the operator will ever get, and it has to name the directory in the way. The row's own text is only preferred when the repair actually wrote a status: when the row is still `correcting`, nothing was written and the row is still carrying the previous failure, which must not be reported as the current one. A target that converges while its repair is in flight is no longer reported. The drift the failure belonged to no longer exists, so paging the operator about it is stale news. A repair that throws is now logged with its stack as well as alerted. This boundary cannot tell an operational failure from a bug in the repair and both arrive identically; an alert alone would repeat a bug every tick and never reach a developer, and re-throwing would strand the row again. Separately, the drift observation restamped `last_drift_at` every time it re-observed the same drift, so a drift outliving one tick reported itself as just found, forever. It is stamped only when the row was not already drifted. Both observation sites did this, the Inline one and the Git-managed one. Coverage added for the Git-managed repair path, which is the third dispatch and had no case, for the timestamp, and for the three alert rules above. 19 cases now, 5 of which fail with the whole change reverted.	2026-09-28 19:54:31 -04:00
Anso	709c7e0269	fix(gitops): report a Blueprint reconciler drift in the canonical drift list (#2025) ... * fix(gitops): report a Blueprint reconciler drift in the canonical list A Blueprint digest drift was detected, recorded, and visible everywhere that reads the deployment row or the per-target digest comparison, while the canonical GitOps drift list stayed empty for the same application. The portfolio's drift summary by class never counted it and the attention queue never named it, so every surface that read the model disagreed with the two that read the deployment. The deriver emits a runtime-family item from two statuses, and a Blueprint observation is neither. The reconciler records its observation as a third status, which outranks the pointer and artifact comparisons in deriveRuntime precisely so it cannot be masked by them, and that ordering meant the artifact comparison was never reached. The item is now built from that status as well. The class stays a fact about the target rather than a spelling of the status, read from the same binding deriveRuntime uses, so a rollout target is reported once as rollout instead of also as runtime. The comparison against the expected set is what still decides whether anything is reported, so a stale stage, an unverified observation, and a digest Enforce is repairing on this pass all report nothing. The item's observed identity now carries the per-service evidence, because the identity string is a fingerprint over the whole set and cannot name which service moved to which digest on its own. The reconciler records the same status for a revision mismatch and for a container that is not running, and refreshes the observation only on the digest path, so that identity is the last one observed rather than necessarily a current reading. The cause is not held on the target row, so the reason states the comparison that was made and leaves the cause to the facet that does report it, with the evidence's own age on the item. * fix(gitops): require a comparable approved set before reporting Blueprint drift A Blueprint observation reaches the drift list with the runtime status the reconciler recorded, and that status is assigned before deriveRuntime checks the approved set's qualification. The two artifact-identity statuses are assigned after it, so those two arrived at the collector already known to be comparable and the collector never checked. The observation path has no such guarantee, and the collector's only test was that the approved identity is not null. That is not enough. A set Sencho could not resolve, or resolved against a stale resolution, still carries a real fingerprint identity, so a Blueprint whose stack builds locally, or whose approved set has since gone stale, could report a confirmed runtime divergence against an expectation nothing verified, while the same target with no recorded observation reads the comparison as pending. The facet and the canonical list disagreed. The collector now requires the approved set to be exact or qualified, which is the same test deriveRuntime applies, so the two agree on every case. Also spell the rollout-binding test the way deriveRuntime spells it, so the facet and the class cannot answer differently about the same target, and correct the comment, which had claimed the guards were complete when they covered only the observed side. * fix(gitops): anchor a drift comparison to the generation being served A recorded acknowledgement rebinds a target to the accepted generation's approved set and leaves the stored observation as the previous generation recorded it. The two are then facts about different generations, and the observation says nothing about the set it is being compared against. Reading an acknowledgement rebinds the target to the new generation's set while still deploying the previous one, and a drift recorded on the next tick through a cause that returns before re-observing, a stopped container or a marker or revision mismatch, leaves that observation untouched. The comparison then reports a digest divergence that was never measured, and it reports it as confirmed: a class, a reason, and per-service digests that all read as real evidence. The two artifact-identity statuses cannot reach that state, because deriveRuntime returns applied_not_deployed before it compares identities whenever desired and deployed disagree. The reconciler observation is assigned before that pointer check, so it is the only path that can, and the collector now refuses to compare across generations. The approved set names the generation it belongs to, so that is what anchors the comparison, and it fails closed when the set cannot be read or the target has never recorded a deployed generation, which is the direction that reports nothing. The same anchor applies to the convergence gate, which was reaching the same comparison with the same rows, so a target part-way between generations can no longer read as exactly converged. * Revert "fix(gitops): anchor a drift comparison to the generation being served" This reverts commit 4383c3c3, and the reason is that the guard it added cannot work for the targets it was written for. It required the approved artifact set to belong to the target's deployed_generation_id before any drift item is emitted. Nothing binds a deploy for a Blueprint-managed stack: the only production caller of deployBound resolves its application through the Direct-mode lookup, so a real Blueprint target's deployed pointer is always null and the guard was false for every one of them. That made the whole drift report dead, because the collector skipped every real target, and it made a Blueprint rollout report a false runtime_artifact_divergence on a healthy fleet, because the convergence gate reads the same rows. The tests did not catch it because the fixture set that pointer. A Blueprint target fixture leaves it null, which is the shape production produces, and the fixture comment now says so. With the realistic shape, the reverted guard was verified to fail: the drift tests return an empty list, and exact convergence is refused. Two gaps that this branch does not close are now pinned instead of being implicit, both of which the unrealistic fixture had been hiding: - On a Blueprint target neither artifact drift status is reachable at all, because the facet reads the applied and deployed pointers before it compares identities and the deployed pointer is always null. A Blueprint whose digests genuinely disagree contributes nothing to the canonical list while the per-target comparison still shows the divergence. The reconciler's observation stage does not go through that pointer check, which is the only reason a recorded Blueprint drift reaches the list. Closing this means deciding what the running pointer is for Blueprint mode and changing the facet's status surface with it. - A drift recorded through a cause that returns before re-observing still compares the superseded generation's digests against the new generation's set, and reports a confirmed item that misdirects the operator. The test pins the misleading fields so the case is diagnosable, and the comment records the two code-only routes that remain open rather than claiming a schema change is the only way.	2026-09-28 18:39:47 -04:00
Anso	e686f55145	fix(gitops): address the runtime drift repair audit (#2026) ... * fix(gitops): detect drift even when the repair is blocked, and hold only in Enforce A pre-merge audit of the drift-repair work found that the hold was evaluated before any drift, in every mode. Two consequences, both fleet-wide on upgrade: every existing Git-managed target read "Repair held" because its marker predates the generation fields, including targets with no drift, and a genuinely stopped container was never reported as drifted at all. Separately, the stateful classification hold ran in Observe and Suggest, so Suggest stopped alerting, which is the one thing Suggest exists to do. Detection and the repair decision are now separate. The check always completes and carries the block with it, so an unprovable repair authority no longer blinds the drift surfaces. A block is consulted only in Enforce, because it is a decision about auto-repair and means nothing to someone who chose not to have one. A held target takes exactly one write per tick. Writing the drift and then the hold flipped the row between the two states, so neither write was ever suppressed and the hold appended history every minute. Keeping the hold's stage on the row is also what stops the runtime observation from overwriting it and making the hold vanish from the projection on the next tick. * fix(gitops): a drift-mode change no longer supersedes the rollout The audit found the edit that turns Enforce on was the one edit guaranteed to stop Enforce working. drift_mode was classified as an operational key, so changing it opened a rollout candidate, which invalidated the placement approval and superseded the live rollout generation, which held every target with nothing left to advance it. The issue this work belongs to required a runtime policy-only change not to reopen placement approval, so the requirement was unmet rather than merely inconvenient. The drift mode chooses what Sencho does about divergence it already observes and changes nothing any node runs, which is the same test every non-operational key already met. It is now metadata_only, so a mode edit writes the Blueprint and stops. The end-to-end test needed the fixture to persist the application's live rollout pointer, which the pointer writer deliberately does not carry and the authorization transition normally writes. Without it the test could not see a supersede production really performs. * fix(blueprints): carry a held repair through every surface that lists drift The audit found repair_held missing from the three places that decide what an operator is shown and what they are allowed to do. A Blueprint could be disabled while its containers kept running, because the disable guard did not count a held deployment. A held row offered no Withdraw, so the documented way out of that state did not exist. And the catalog left the status out of its priority list, its tile colour and the Drifted filter, so a Blueprint whose only problem was a declined repair rendered as a grey dot that the Drifted chip could not find. A hold is a decision not to overwrite a running workload, so it belongs with drift everywhere drift is counted, and the workload is still there everywhere lifecycle actions are gated. * docs: correct the drift-repair claims the audit found untrue Three things the previous change documented were not what the code did. It said Enforce "restores the older one" when a newer generation has been rolled out. Accepting and rolling out a newer generation supersedes the rollout that owned the target, so the honest outcome is a declined repair, not a quiet restore of the generation the fleet has moved past. It said a superseded hold clears "or until you deploy to it". A Git-managed Blueprint has no manual redeploy to fall back on, so the rollout is the only way out, and saying otherwise sends an operator looking for a button that is not there. It never documented the repair-held notification at all, in the category table or the message list. * fix(gitops): carry the repair block on every drifted return, and keep the hold in the projection A re-audit of this work found two blocking problems. The block was dropped on three early drifted returns: a missing marker, a marker belonging to another Blueprint, and a revision mismatch. Each returned a verdict with no block, so the reconciler treated the target as repairable and Enforce overwrote exactly the targets the hold exists to protect: an Inline target a recovery owned was redeployed by a path that uses the binding only to stamp the marker and never refuses, and a Git-managed node on a legacy marker was overwritten by a reapply that had a valid binding. All drifted returns now go through one helper that attaches the block, so a fourth path cannot forget it. The hold still dropped out of the GitOps projection on the second tick. The runtime observation stamps a stage on every check, and the observation stage maps to no runtime status, so it silently replaced the hold's stage: the deployment row read held while the Drift panel and the attention queue said nothing was wrong. Evidence does not supersede a decision, so recording the observed artifact now leaves a Blueprint observation stage standing until a decision changes it. * fix(gitops): report the drift mode the Blueprint is configured with Making drift_mode non-operational meant a mode-only edit mints no intent, so the intent's runtime_drift_policy snapshot went stale at exactly the moment an operator changed it. The reconciler reads the live Blueprint, so behaviour was right, but the drift surfaces and the next rollout's frozen record reported the old mode, and an explanation of what Sencho will do here that names the wrong mode is worse than none. Intents are immutable revisions, so this is not fixed by rewriting the snapshot. The projection and the freeze read the live Blueprint and fall back to the snapshot only when the application is not a Blueprint. * test(blueprints): cover the held row's status, dot and Withdraw control The catalog's priority order, its dot colour and the deployment table's Withdraw control were all changed for the held status without a render test, so a later edit could drop the status from any of the three and only the types would notice. The tile's status dot is asserted specifically rather than by "any warning class": the tile carries other dots, and an assertion loose enough to match the wrong one would have passed on a tile that showed nothing useful. * chore: drop the leftovers this work left behind, and correct its docs A dead best-effort observation helper from the earlier hold-first design, a count computed and never rendered, a typo, and a doc block that had drifted onto the wrong method and still claimed the drift was recorded when it is not. The docs also made four claims the code does not meet. A held row shows the reason the repair was declined, not the drift, so the Drift panel is where the drift is read. A row held under Enforce keeps reading held after a switch to Observe or Suggest while the drift cannot be classified. And on an Inline Blueprint a mode change is still an edit to the Blueprint, so it re-enters Awaiting confirmation before the new mode takes effect. * fix: settle cleared drift states and keep holds on unclassifiable checks Two ways a hold or a cleared state could be lost on a later tick. The observation write preserved every Blueprint decision stage, because the projection reads the target's latest stage ahead of the pointer path. That correctly protected a hold, but it also protected a cleared drift, which records that the drift is gone and projects to deployed-health-pending. Nothing moves that stage on, so every recovered target stayed health-pending for good and the portfolio reported a healthy application as in progress until the next rollout. The preserved set is now the stages that record a decision someone still has to act on. Evidence may supersede a cleared drift; it may not supersede a hold. Separately, the returns that cannot classify the target dropped the repair block, while the reconciler records its hold off an unverified result too. A held target therefore lost its hold for exactly as long as the node stayed unobservable, which is the state a hold most needs to cover. Both result builders are hoisted above the try so the catch reaches them, and every return made after the binding is resolved now routes through them. The two returns before it carry no block, because there is nothing to carry yet. Also corrects two assertions that proved less than they claimed. The Inline hold was pinned to the digest repair path, which an Inline Blueprint never takes; it now asserts on the binding the repair dispatch actually consults. The no-new-decisions check counted every history row under a loose bound, where the evidence written each tick drowned out the thing being measured; it now counts the decision stages and expects none. Each change has a test that fails with it reverted. * test: cover the live drift mode reported on a placement drift item The reported drift policy is read from the Blueprint row rather than the mode frozen in the intent, because a mode-only edit mints no intent by design, so the snapshot goes stale while the operator's choice does not. That was already fixed and had no test: the derive suite never seeded a Blueprint row, so the live read always fell back to the intent and any test of it would have passed with the stale read restored. This leaves the intent snapshot on one mode and the Blueprint on another, so reading the wrong one is visible. The neighbouring placement-drift fixture is now shared between the two tests, which leaves the difference between them as the one thing each has to say. * fix: keep a marker that cannot prove the repair authority out of unverified holds The previous commit attached the repair block to every return the drift check makes, which is right for the returns that lead to a write and wrong for the ones that do not. Two different things suppress a repair, and only one of them is a decision. A binding hold says Sencho must not write to this target at all, whatever the check can or cannot classify. A marker that names no generation, or a different one, says only that a repair could not prove what it would overwrite, and nothing about whether the workload is drifted. The reconciler's only use of the block on a result it could not classify is recording a hold and raising an alert, so attaching the marker case there reported every target whose marker predates the generation fields as Repair held on any tick the check could not classify, and announced a declined auto-fix that was never due. It flapped too: a transient unreachable node wrote a hold and an alert, the next healthy tick cleared it, and the next blip alerted again. They are now separate. A drifted result carries whichever applies, because a repair is about to be attempted and the mutation sites trust that result. An unverified result carries the binding hold alone. Two tests pin it, one for each way a check fails to classify on a legacy marker: an unreachable node, and a running image that is a local build with no comparable digest. The second never recovers on its own, so without this it would hold the target for good. Both assert the deployment stays active, that nothing is written to the node, and that no repair-held alert fires. Also converts one test's direct assignment on the service singleton into a spy. A direct assignment outlives the test and would have left every later test in the file looking at an unreachable node.	2026-09-28 17:03:41 -04:00
Anso	ab34e85b86	feat(gitops): independent source, placement, and rollout authority policies (#2020) ... * feat(gitops): add the versioned authority policy contract Source acceptance, placement approval, and rollout authorization each get their own value in one closed, versioned snapshot, so no reader can reach a single value that looks like it governs all three. Placement policy and rollout authorization policy are per application, each with its own CHECK. Both default to the safe answer on a new install: an operator approves placement, and rollout authorization waits. Existing installs are a different case and the migration is the delicate part. Blueprint applications already authorized their own rollouts on the acceptance handoff, so leaving rollout authorization at its manual default would strip that behavior from installs that never asked for the change. The backfill restores it, scoped to live Blueprint applications, and runs only on the boot that created the column. Every additive column here runs on every start, so an ungated backfill would re-apply on every restart and quietly re-grant authority an operator had turned off. A rollout generation freezes the snapshot it executes, in a nullable column with no default. NULL is what makes absence readable: a row predating the column reconstructs as the behavior it actually had, which for a generation that authorized itself was automatic, not the fresh-install default. An empty-object default would make absence impossible and turn every existing generation into a decode failure. Decoding fails closed on anything present but unreadable, and refuses an unrecognized key rather than dropping it, because a fourth key means the writer and the reader disagree about what governs. * feat(gitops): bound automatic placement to one stateless change Placement policy decides whether a change in Blueprint target intent needs an operator. Under bounded auto the only approved shapes are one stateless addition or one stateless removal, and every other case falls back to explicit operator review with a named reason rather than a silent skip. The baseline is the last approved target set, never the previous candidate. An intent revision clears the placement approval before a new candidate opens, so the only set still in hand at decision time is a superseded one nobody approved. Diffing against it lets a pending candidate absorb a later edit: a candidate asking for one set sits awaiting review, a selector change mints a larger one, the difference reads as a single addition, and the approval authorizes a node on the strength of a decision no operator made. A test pins that case in the refusing direction, and the mutation that would reintroduce the bug is shown to return the approval. Statelessness is derived from the compose content that would be placed rather than from the Blueprint's stored classification. Compose edits are refused for a Git-managed Blueprint, so that column says what the editor last wrote and a repository push that adds a stateful service never updates it. An absent reading is treated as stateful, because a missing field defaulting to the one answer that approves is the failure this policy exists to prevent. The decision is pure and total. Node lists are normalized on the way in, since they arrive from a JSON column the compiler never sees, so damaged evidence produces a refusal rather than a TypeError thrown from inside a judgement made inside somebody else's write. Directory-name ownership is left out of the reason union on purpose. Whether a stack directory on some node carries this Blueprint's marker is a question about who owns a name, not about whether moving a stateless workload is safe, and the deploy path already refuses an ownership conflict by name. * feat(gitops): apply bounded placement decisions outside the producer write The decision is applied after the producer's transaction commits, in a transaction of its own. Evaluating inside the producer would nest the single-writer transition inside the caller's write, and a refusal or a defect in the decision code would roll back the operator's own Blueprint create, update, or pin. Nothing in the path throws at the caller for the same reason: the Blueprint edit already succeeded, and an unapproved candidate waits for an operator either way, so the failure direction is safe without a rollback. Atomicity is therefore not the mechanism. Idempotency is. An approval now refuses when one already exists against the same intent revision, because the currency checks only catch a stale approval: replaying the same current one passed all of them, minted a second approval row, and superseded the generation the first had just opened. The guard is keyed on the intent revision, which is the precise key, since a placement approval records no candidate and an intent opens exactly one. Authority and the snapshot that produced it are now required together rather than trusted separately. A policy-authorized approval that records no snapshot cannot be reviewed later, and an operator approval claiming a policy decided it is a mislabelled authority record; both are refused at the write. The two policy edits are configuration, not work. Each refuses while an operation is in flight and refuses a write to the value already set, mints no intent or candidate, and clears no source acceptance, placement approval, or rollout authorization. An edit decides what the next decision may do; an approval already standing keeps running under the snapshot that authorized it. That is why a policy change never appears as drift, and it is pinned in the direction that matters: a fully approved application keeps every reference and its generation byte for byte. * fix(gitops): refuse rollout authorization while an operation is in flight The pause check only knows about a deliberate pause, so an authorization could be minted while a fetch, apply, deploy, or recovery was still running for the application. The two then disagree about what the next operation acts on, and the authorization is the record everything downstream treats as settled. Two refusals, both with a named reason and both failing closed. The application guard reads the application's own active operation. The target guard reads the target rows, because a target can be mid-apply while the application has nothing running at all: they are separate machines and only the target knows about its own apply. Reading only the application pointer would have missed exactly that case, which is why the second refusal exists separately rather than folded into the first. * feat(gitops): add the placement and rollout authorization policy writes Two endpoints, each naming in advance what the system may do, so each is authorized by exact deploy on every frozen target before anything is written. A policy that half-applied would leave the fleet split across two policies, half of which nobody chose, so a caller entitled for one target and not another is refused outright rather than applied where it happens to be allowed. Both require only the authority a placement approval or a rollout authorization already needs, and neither requires a global create grant. That grant is held only by roles that hold a global deploy grant as well, so demanding it would silently upgrade the gate to application-wide and make the per-target loop unreachable for every caller that got past it. The create authority belongs to the approval the policy authorizes, which already requires it. Before the first placement there is no target to scope an exact grant to, and a stack-scoped assignment is node-qualified, so it cannot be resolved for a node that does not exist yet. The application-wide grant is the only gate evaluable in that state, and the limitation is pinned as a test rather than worked around: a scoped operator configures the policy once a target exists. Each write refuses a value outside its own vocabulary, a write to the value already set, and a Direct or Inline application, which has no placement or rollout domain to configure. * refactor(gitops): drop the unreachable marker conflict from placement The reason could never mean what it said. The query behind it only ever looked at this application's own intent revisions, so it could not detect a conflict with any other Blueprint, and the one thing it did detect, a pinned node inside the candidate set, is already reported as pin-driven placement. A reason in a closed union that either never fires or duplicates another advertises a check that does not exist, which is worse than not having it. There is no database signal to replace it with. Blueprint names are unique and deployments are keyed by Blueprint, so whether a stack directory on some node carries this Blueprint's marker can only be answered by probing that node, which the deploy path already does and refuses with a named error. Content-binding contention is refused on the convert and adopt paths instead, and placement cannot create the contention it would have reported. The permission tests are reframed to state what the gate is for. Before anything is placed, a bounded_auto policy authorizes work on whatever nodes the selector matches later and that set is not known, so the application-wide grant is the conservative answer and a narrowly-scoped operator is correctly refused. Once targets exist the per-target loop applies, and an operator scoped to exactly those targets is authorized, which is now asserted in the passing direction. * fix(gitops): address the policy review findings Seven findings, four of them load-bearing. The rollout authorization policy was never read. The column was stored, the route wrote it, the migration backfilled it, and generations froze it, but no enforcement path consulted it, so setting it to manual returned success and the next dispatch minted anyway. A policy-authorized mint now requires a policy that says so; an operator authorizing by hand is unaffected, because an operator is themselves the authority. The placement baseline was unreachable. It resolved through the application's placement approval pointer, which an intent revision clears before a new candidate opens, so the pointer branch never ran. Every application therefore looked un-approved and every multi-node candidate was refused as a first placement, which is the opposite of the stated bound of one stateless change. It now resolves the most recent approval row, which survives the pointer and is still the record of the last set an operator accepted. A cordon was never consulted. The refusal existed and the evidence gathering passed a literal false, so an automatic approval could place a workload onto a node an operator had cordoned. The flag is now read for the nodes being added. Node evidence is also read for removals, not only for the candidate set, and a stale target counts as unreachable rather than as reachable, which had made the stale-node refusal unreachable for the targets it exists for. The migration backfill could be stranded. Gating it on the boot that created the column meant a process dying between the ALTER and the UPDATE skipped it for ever, because every later boot sees a duplicate column. A completion marker now decides, so the backfill is offered on every boot and runs only once. A row an operator moved back to manual is indistinguishable from a never-backfilled row, which is exactly why re-running it on its own predicate would have been wrong. An approval's snapshot and the generation it opens are now compared inside the write transaction. The authorizing snapshot is read by the caller beforehand, so a policy edit landing in between could leave the two records disagreeing about one decision, under a policy the operator had already revoked. The replay guard is keyed on the intent and candidate together, since nothing enforces one candidate per intent, and a placement approval now records the candidate it was made against. A refusal keeps its own cause. Every approval failure was reported as a conflicting operation, which sent an operator looking for work that was not running and turned the replay guard's precise signal into a lie on the retry path. Also: an unreadable candidate target set is logged rather than swallowed, and the empty-target-set permission comment no longer claims to be stricter than the per-target path when it is the same shape as the health policy write beside it. The GitOps suite passes apart from two files that fail identically on an unmodified checkout, one on a filesystem exhaustion this host is hitting and one on a cold-start timeout. Typecheck and lint are clean. * fix(gitops): report the unmet rollout prerequisite, not the policy The rollout authorization policy gate ran before the authorization ingredients, so it short-circuited with its own reason before the ingredient block could say what was actually missing. An application with no placement approval was reported as needing an operator rather than as needing a placement approval. The gate answers "may policy authorize this, or must a human", which has no meaning until there is something to authorize, so it now runs after the ingredients pass and before the artifact qualification check. `authorizationIngredients` is a pure read with no side effects, so nothing mutates between the two positions and the only behaviour that changes is which reason is reported. This was the same class of misdiagnosis the previous review round flagged, where every approval failure was relabelled with one fixed reason. The prerequisite that is missing is the more specific and more actionable fact, and reporting anything else sends an operator looking in the wrong place. The operator path is unaffected either way: an operator asking to authorize is themselves the authority, and no policy stands in the way of one. Verified: the previously failing assertion returns to the placement reason, the two suites covering the gate pass, and removing the gate from its new position still fails the suite, so the move did not make it unreachable. * fix(gitops): make bounded automatic placement reachable Automatic placement could never approve anything, and never had. Two defects compounded, and the first one hid the second. Node evidence was read from the target row. An added node has no target row and never will until the placement lands, so every addition resolved to unknown and was refused. Nothing writes connectivity on a target row either: the only constructor seeds it to null, so the column is null for every node for ever. The pure decision was exhaustively correct and completely unreachable. The effect was also hand-assembled as a JSON object where the schema wants an array, so the transition's own decode rejected every approval. Because that refusal is the safe direction it surfaced only as an operator review, which is how a feature can look wired up end to end while never once approving anything. Each side of a change is now judged by the evidence that actually exists for it. A node being added is known from the node registry, which is maintained rather than seeded, and is refused when it is unregistered or not answering. A node being left is judged by the observation of the workload that ran there, where a stale observation does not count as reachable. Cordoned nodes are refused for additions, which the previous code could detect but never supplied. The eight new tests drive the real path rather than a hand-built input: a stateless addition reaches auto_approved and records the snapshot that decided it, and each refusal is reachable from real state. Two of them fail if the node evidence regresses to the target row, and two fail if the blast encoding regresses, so neither defect can come back silently. * test(gitops): drop an unused binding from the placement end-to-end suite * fix(gitops): do not re-decide an authorization that already exists An operator authorized a rollout, got a success response, and nothing deployed. The mint succeeded because an operator is the authority. The dispatch that follows re-enters the same function on the automatic path, where the policy gate saw the fresh-install default of manual and refused, so the policy governed the execution of authority that had already been granted. The policy governs whether authority may be created, not what may be done with it. A grant is not withdrawn by a later configuration change, and it should not be re-litigated on every dispatch. The same reasoning applies to the in-flight guard: it exists so an authorization is not minted while a fetch, apply, deploy, or recovery is running, and a routine background source fetch must not pause a rollout that is already authorized and running. Both gates now stand down when a live binding already exists, using the existing live-binding helper rather than a new notion of what counts as authorized. The preflight evaluation and the drift remint still run either way, since that is this function's other job and it is about evidence rather than authority. The gates keep their teeth for the case they exist for: with nothing granted, the policy is the only thing that could grant authority, and on manual it refuses. Both directions are asserted, for the application-level and the target-level in-flight case, and each fails if the live-authorization bypass is removed. * fix(gitops): let an operator re-approve a placement on the same intent The replay guard refused any second placement approval for an intent and candidate that already had one. That is correct for the automatic path, where a duplicate is a replay, and wrong for an operator. The Inline Apply path reuses the application's current intent rather than minting a new one, so an operator pressing Apply again lands on exactly the same intent and candidate. The guard refused it as a replay and regressed a flow that had worked for ever, reporting it as a duplicate decision when it was a deliberate second one. The guard is now scoped to policy-authorized approvals, which is the only caller that can produce a duplicate by being invoked twice. The operator path is unrestricted, and the automatic path keeps its protection. The test that previously asserted the refusal now asserts the operator's second approval succeeds, and a separate policy-authorized case keeps the guard load-bearing. Widening the guard back to all authorities fails two tests, and removing it fails one. * fix(gitops): enforce the mint-time gates in the single writer The previous fix let a live authorization bypass the policy and in-flight gates, which was right for executing authority that already exists and wrong for creating it. The distinction I drew was between a binding existing and a mint happening, and I enforced it on the former. So the remint path was ungated. When the preflight fingerprint drifts, the old authorization is discarded and a new one minted, which is a fresh grant of authority, and on a manual policy that mint now succeeds with authority: configured_policy and no operator anywhere. A registry tag moving was enough to replace an operator's grant with a policy-authorized one, which is the precise thing the gate exists to prevent. The race-retry branch had the same hole. Both gates now sit in rolloutAuthorized, the single writer every mint path goes through: the acceptance handoff, the Blueprint dispatch, the startup reconstruction, the preflight backfill, the operator route, the drift remint and the race retry. A caller-side check cannot be unbypassable; this can, because there is nowhere else to mint from. That also closes the separate finding that the policy was not re-checked at mint time, since the re-check is now the gate itself. The caller-side gates stay where they are, because they still give the better reason for the ordinary case: a missing prerequisite is reported as itself, and a pause is reported as a pause, before anything reaches the transition. Two tests assert the property where it now lives, that any policy-authorized mint reaching the transition on a manual policy is refused, and the same mid-operation, with the operator path shown to be unaffected. Removing either gate fails one. * fix(blueprints): refuse to call unresolvable compose stateless Stateful service detection is the evidence an automatic placement decision rests on: a service it does not name is one the system believes carries no data. Four compose constructs attach a mount to a service whose own block mentions none, and the detector returned an empty set for all four, which is the permissive answer. The merge key is the sharpest case, and worse than a simple blind spot. The anchored service is found, so a partial answer named it and reported the service that merges it as stateless, attributing the volume to the wrong service while the workload that actually inherits it read clean. `include` and `extends` name another file. `volumes_from` attaches another container's volumes, which is directly data-bearing. A document with no services now also returns null. Nothing to place is not the same as proven stateless, and a caller deciding whether a placement is safe should not have to tell those two apart. There is deliberately no partial answer. A set naming only the services the detector could see reads as permission for the ones it could not, so "cannot prove" and "is stateless" have to stay different answers. Both callers already hold on null: the stateful withdrawal guard treats it as unreadable, and the placement decision treats it as an unknown workload, which is a refusal. Resolvable content is unchanged, including a tmpfs mount, which is not data the workload owns, and is still not a stateful service. * docs(gitops): put two doc comments back on the code they describe The health-and-rollout policy route's own doc block was left stranded above the policy-target gate, a hundred lines from the route it describes, so a reader arriving at either place found the wrong explanation. A helper inserted between the rollout authorization doc block and its function had the same effect there. One of the stranded claims was also false by the time it was found. It described the empty-target-set branch as authorizing against the Blueprint's deploy stack name; the branch falls back to the application-wide grant, because a stack-scoped assignment is node-qualified and cannot resolve for a node that does not exist yet. The explanation now says what the code does, including the consequence an operator needs to know, which is that configuring a policy on a fresh application is an administrator action. No behavior change. * fix(gitops): let a demoted Blueprint still have its policies set Demoting a Blueprint back to Inline keeps every policy column, and the placement decision does not read the target mode, so a `bounded_auto` application keeps approving its own stateless placement changes after the demotion. Until now no route could stop that: all three policy writes resolved their target through the same gate the source and rollout actions use, which requires Git management, and answered NOT_GIT_MANAGED for a demoted application. The only way back was to edit the row directly. Resetting the policy on demotion would be the wrong fix. Where content comes from and who decides a placement are separate domains, and coupling them here is the entanglement this model exists to remove. A policy is a statement about who decides, so a source-domain condition has no business gating it. The shared resolver is now split by what it actually requires. Actions that act on Git content, or on a rollout Git produced, keep the Git-management gate: source acceptance, placement approval, and the rollout lifecycle. The three policy writes resolve without it, which is the same shape the health policy already had in spirit and never had in practice. Widen which applications a policy write reaches, not who may write one, so the permission behaviour is unchanged and pinned again here: a caller without the deploy grant is still refused, and a Direct application, which has no policy domain at all, is still NOT_GIT_MANAGED. * feat(gitops): project each authority policy and why it declined The three policies could be configured and enforced but not read. An operator looking at a placement waiting for review under an automatic policy had no way to learn that the policy had run, or why it left the decision to them, and a policy that never fires looks identical to one that is working. That is the same failure shape as the rest of this branch: a feature that reads as wired up while reporting nothing. The projection now carries one read per domain, in stage order, each answering the two questions an operator actually has: what is configured now, and what the work in flight was really decided under. The second is read from the snapshot the generation froze, never from the current setting, because a policy edited afterwards must not appear to have authorized work that is already running. It is reported as absent whenever no readable frozen snapshot exists, which includes every generation predating the policy contract. The refusal itself had to become readable first, because the decision was computed, returned to a producer and discarded. It is now recorded on the application row, in the closed decision vocabulary, and constrained by a check so a reason no decision can produce cannot be stored or projected. Cleared inside the two transitions that resolve it, an approval and a policy change, so a reason that described the old policy cannot be read as the current one. The refusal is written with the operation's own timestamp, not the wall clock, so its freshness means the same thing as every other recorded time on the row. Also fixes a trap this change walked into: the application insert hardcoded its placeholder count, so adding a column to the list and not the count compiled cleanly and failed at runtime as a SQLite arity error on the first insert, which is the first write any create path makes. The count is now derived from the column list it belongs to. * feat(gitops): show and set each authority policy where it is decided The policies could be configured only by editing a row, and the two new ones had no affordance at all. An operator could not see who was configured to decide a stage, and could not stop automatic placement once it was on. Policy reads join the facet cards. The four cards already render source, artifact, placement and rollout in a fixed order and are shared between the application view and the rollout preview, so the same evidence cannot read two different ways on the two surfaces. Each card that governs a decision carries that decision's policy line; the artifact card carries none, because it reports what was built rather than who may act. The line says two things and stops: who is configured now, and what that policy actually did. The second is a separate fact, because a policy edited after a generation opened does not retroactively claim it, and a line showing only the configured value would read as though it had. A card with no decision and no reason shows nothing, so the line is never a restatement of the card's own state. Policy edits sit beside the action each one governs: placement policy beside the placement approval, rollout authorization policy in the rollout overflow where the health policy already lives. Not on the GitOps Workplace, which is triage and would gain a second job. One control and one label map serve both, so three vocabularies cannot appear in three places. The control is absent rather than disabled for a session that may not write, matching every other gated affordance here, and it names a value this build does not recognize instead of snapping to a segment that would overwrite whatever the server is enforcing. The source policy's Manual mode finally has an option. It was in the contract since the policy columns landed and the panel could not reach it, because the panel derived the mode from two booleans that cannot express it: a manual source and a review source both report `auto_apply_on_webhook: false`. The policy field was already on the wire and simply undeclared, so the fix is to read it. Manual is not a flavour of review either. It takes the source out of the unattended cadence: no polling, no webhook. Both the panel and the create dialog now offer it and save the automation flags that agree with it, so choosing it does not leave a source that still applies on its own. The apply-mode options also state their selection, which was carried only by a border and a fill and so was invisible to a screen reader. * fix(gitops): hold a policy write to the authority it names The placement and rollout authorization policy writes were gated more loosely than the health policy write beside them: exact deploy per target, and the application-wide grant only when there was no target to check. A stack-scoped operator could therefore set either policy to automatic. That is a real widening rather than a theoretical one. A policy governs every target of the application, including nodes that do not exist yet, so one operator's grant on a single stack would authorize placements and rollouts elsewhere in the fleet. The per-target loop does not close it, because it only checks the nodes present today. Both halves are now required, the same requirement the health policy write already carries and the same one recorded for it in the permission inventory. The application-wide half is the one a scoped operator can fail, which is exactly why it belongs on the path. Configuring a policy on a fresh application, before any target exists, remains an administrator action: there is no per-target check to make and a stack-scoped assignment is node-qualified so it cannot resolve for a node that does not exist yet. The behavior change is pinned from both sides: a stack-scoped operator is refused on both routes, and a role-based deployer still passes, so tightening the gate does not lock out the operator it is meant to admit. Also records the surface and the model. The authority policy model, its single writer, the bounded automatic placement decision, why the refusal has to be durable, and what the projection refuses to claim about work in flight are now written down rather than only living in the code. Alongside it: the two new service modules and the shared control in the inventories, the two policy routes in the permission inventory, the new endpoints in the route inventory, and the architecture map with a flow for the write and the HTML re-inlined to match. Operator documentation: the source policy's Manual mode, which the panel could not reach because the two booleans it derives from cannot express it; who decides each of the three steps, and the whole of what the automatic placement policy may do on its own; the policy-domains table, which still described placement as a per-rollout approval rather than a configured policy. * fix(gitops): do not let a cordon withdraw a workload Cordoning a node removed it from the Blueprint's desired set, and a single stateless removal is inside what the bounded automatic policy may approve on its own. So one cordon silently stopped the workload running on that node, and the placement policy read as broken rather than as having drained a node. A cordon says where work may not go. Taking work away from a node is a separate decision, and the operator who cordoned a node has not necessarily made it. The cordon is now read for removals as well as additions, and a withdrawal whose only cause is a cordon falls back to the operator with its own reason. The reason is separate from the existing cordon override because the two ask opposite questions of the operator. That one says Sencho would have placed work on a node you told it to leave alone. This one says the withdrawal you are seeing follows from the cordon you set. The new reason joins the closed decision vocabulary, which constrains both the stored column and the wire, so no decision can produce a value that cannot be stored or projected. Reading the cordon for removals is not a blanket refusal of removals, and that is pinned from both sides: a withdrawal of a node that is not cordoned is still approved. The pure decision test passes the evidence in directly and so cannot catch a caller that sets it for every removal, which is why the approving case lives at the level that reads the cordon. Also corrects two comments that claimed a guarantee their callers defeat. `recordPlacementShift` said each transition commits on its own, and its automatic-placement loop said the triggering change had already committed. Both callers wrap it in a transaction, so neither is true. The behavior is fine and arguably better, because the decision rolls back atomically with the write that caused it, and every refusal is caught rather than propagated. What was wrong was the claim, and a comment that asserts a guarantee the code does not provide is worse than none. * test(gitops): record no policy snapshot on a seeded rollout generation Two test files build a rollout generation row inline, and this branch added a column to that row, so the merged tree failed the typecheck in both. The fix is to say what those rows actually are: generations with no recorded policy snapshot, which is what a generation predating the policy contract looks like. Null is the honest value rather than a stand-in for something. Both files are about which generation a target may restore or which one it acknowledged, and neither is about policy provenance, so recording a snapshot on their rows would assert a fact the tests have no evidence for and would let a later reader infer that these rows were minted under a policy. Test fixtures only. No production code and no behavior change. * fix(gitops): a migrated Blueprint is not a first placement The placement baseline falls back to the running target rows when there is no decomposed placement approval, and reported that it had never been placed. A migration that carries an approval forward records a pre-decomposition row instead, which names no target set and so cannot supply a baseline but does prove the placement was not the first thing that ever happened to that application. So a migrated Blueprint running on three nodes whose selector narrowed to two was refused as a first placement across more than one node, for ever, on exactly the installations the migration was written for. The refusal is the safe direction, which is why it read as a policy that quietly does nothing. The two questions are now separated. What the placement was authorized to be is answered by the decomposed row, and what the running targets are is the baseline when there is no such row. Whether the application was ever placed under an operator's authority also accepts the carried-forward row, and only that row, so a genuinely fresh application is still a first placement. That last half is pinned directly on the reader, because asserting it through a refusal would have several other possible causes. Also moves the rollout authorization backfill's completion marker out of `global_settings` and into `system_state`. The marker is internal migration progress, not a setting, and `global_settings` is the allowlisted table the settings route projects. The backfill's lifecycle predicate is left exactly as it is, and the reason is now written down rather than left to be re-derived. A `blueprint` mode row is never `creating`: a Blueprint application is inserted as `inline_blueprint` and is `active` from the start, and the transitions that write `creating` are the Direct create and activate paths, so the predicate excludes nothing today. It is also the right line if that ever changes, because a Blueprint application still being created never reached the acceptance handoff and so never auto-authorized. Granting it `automatic` there would be a privilege grant rather than the behavior preservation this backfill exists to perform. * fix(gitops): keep the two compose readings apart, and see an in-flight target Two release-blocking defects, both found by the pre-merge audit and both mine. **The stricter compose reading broke automatic source acceptance.** Tightening stateful detection so an unresolvable compose answers unknown instead of naming the services it could see was applied to the shared helper. The only other caller is the stateful withdrawal guard, which treats unknown as unreadable and compares both the staged candidate and the generation in force, so a compose using a merge key, an extends, an include, or an override file with no services was held for ever. Those are ordinary compose idioms, and an install using one stopped applying updates automatically. The two consumers need opposite answers to the same document, so they are now separate entry points rather than one method with a flag. The guard gets the best-effort reading it has always had, and the placement path gets a strict one that refuses. A caller about to act on the result must never be handed a partial set, because a set naming only the services the detector could see reads as permission for the ones it could not. **The bounded-auto in-flight check could not see a rollout.** It read only the application's own operation pointer. A sequential rollout sets its stage on the target, so a label change during a mid-deploy rollout was approved, cleared the authorization, and superseded the generation that deploy was running under. The target-side reader already existed in the rollout handoff, so it is now shared rather than re-derived. **Placement content was read from the wrong directory.** Generations are staged under the application's source stack, not under the Blueprint's name, so a renamed stack found nothing and every placement on one read as an unknown workload. And multiple compose files were concatenated into a single document, which duplicates every service key and does not parse, so a multi-file project could never be approved either. Each file is now analyzed on its own, which is what the withdrawal guard has always done. The end-to-end suite staged its content under the Blueprint's name, which is the directory the source path never writes to. That is why the wrong path was not caught: the test agreed with the implementation. It now stages where generations actually live, and covers a renamed stack, a multi-file project, a multi-file project whose second file is stateful, and an addition during a target deploy. * fix(gitops): say what the policies actually did The projection's claims are the part an operator acts on, and three of them were wrong in the direction that makes a working feature look broken. **The default policy reported itself as declining.** Every placement on the operator policy recorded a `policy_is_operator` refusal, and the projection read a recorded reason as "declined by policy, waiting on an operator". That policy never runs, so it has no decision to decline with, and the default application on the default policy said the system was withholding something. The refusal is no longer recorded for that reason, and the projection's existing distinction between a policy that declined and one that has not run now has something to distinguish. **Source acceptance was always reported as an operator decision**, because the authority was hardcoded rather than read from the approval. An operator on the automatic policy was told they had personally accepted a revision nobody showed them. It is derived now, from the same field every other read uses. **A generation predating the policy contract reported a frozen policy that never existed.** The decoder reconstructed the fresh-install defaults for a null snapshot, so a rollout that ran before these settings were introduced claimed it had been decided under a policy that did not exist at the time. The module header already said absence is absence; the decoder was doing the opposite. **A demoted Blueprint had a live automatic decision and no control.** The backend accepts the policy write for it, and the placement decision evaluates it, but the control sat behind a target-mode gate, so an operator whose application decided its own placement had no way to reach the setting from the interface. Both Blueprint modes now reach the block, the Git lifecycle actions stay Git-only because a demoted application has no Git generation to authorize a rollout of, and the view test that asserted the old behavior is updated with the reason. **A policy change wrote no before and after.** The application snapshot both history rows were built from omitted the three policy columns, so the audit trail recorded that a policy changed without saying which way, and that a recorded refusal had been cleared without recording the policy that produced it. Both are in the delta now, pinned on the history row. **The placement writer did not check its own domain.** `rolloutAuthorized` refuses a policy-authorized mint unless the rollout authorization policy actually permits one; `placementApproved` had no equivalent, so a caller could reach the single writer with a policy-authorized approval and a snapshot that happened to agree. Agreement by accident is not authority, and the domain rule now cannot be satisfied by a caller that simply passes a matching snapshot. **The rollout default after the upgrade is now stated.** The one-time backfill restores automatic authorization for the live Blueprint applications that already had it, and everything created afterwards waits for an operator: a newly adopted Blueprint, one converted back and forward, and any row written by an older binary during a downgrade. That is a real behavior change for those applications and it was only explained in a code comment. It is now written where an operator looks. Also corrects a route docstring that claimed an exact per-target check while describing the wider gate, and drops the control's dead permission conditionals, which were unreachable because the control returns nothing without the grant. One audit finding was checked and rejected: the approval snapshot decoder is used by the end-to-end suite, so it stays. * fix(gitops): read the generation in force, not the candidate The bounded-automatic policy worked only between source acceptance and the first rollout. Every placement after that read as an unknown workload and went to operator review. Promotion removes the applied directory and renames the candidate directory over it, so a promoted generation has an applied directory and no candidate directory at all. The placement decision resolved only the candidate, found nothing there, and treated the absence as unknown evidence. The withdrawal guard already had the right reader for this: it reads the applied copy first and falls back to the candidate for a generation that is accepted but not yet promoted. That is the generation a target would actually receive, so it is the one the policy decision rests on, and it is now shared rather than duplicated. The end-to-end fixture staged a candidate directory for every case, which is a generation that has been accepted and not yet promoted. That is the narrowest window in the lifecycle and the one an install spends least time in, so the suite described a state that a working install is almost never in and the reader was wrong for every generation that mattered. The fixture can now stage a generation as promoted: content under the applied directory, nothing at the path its row still names. Three cases cover it, including one where both directories exist and the applied copy is the stateful one, because the applied copy is what a target would receive and the unpromoted candidate is what no target is running. **The strict compose reader missed a merge key below the service.** It checked for `<<` only directly under a service, so `services: { <<: *shared }` merged whole services in past the check, and `volumes: [{ <<: *vol }]` attached a mount the volume walk never visits. Both read as stateless and would have been placed automatically. A merge key is now looked for at any depth. An alias is not a merge key and still answers, because the parser resolves it and refusing every anchored document would hold ordinary compose files for no safety gain. **The operator-policy regression test could not fail.** It inserted an application with no intent and no candidate, so the call skipped before any decision ran and its assertions on the refusal columns held whatever the guard did. The case that does reach the decision now carries those assertions, and deleting the guard fails it. Also reconciles three doc comments left doubled by successive edits, and keeps the `policy_is_operator` reason text with the reason it stays: a node on an earlier build still records that reason, and this dashboard reads the reason from whichever node owns the application, so during an upgrade the raw enum would otherwise reach an operator.	2026-09-28 15:56:44 -04:00
Anso	40aca5a6ec	docs(editor): record the mixed-version outcome for Save & Pull Images (#2027) ... A control instance on a newer release can send the compose editor's Save & Pull Images action to a node whose Sencho instance has no such route. The node answers 404 with its own unmatched-/api body, the remote proxy passes that through untouched, and the editor reports the image pull as failed after the compose file has already been written. No image is downloaded and no running container changes, but nothing recorded that, so an operator running a mixed-version fleet had to infer it. Document the outcome on the editor page, in Save options and as a troubleshooting entry, and record on the node compatibility page that an action added by a later release is refused rather than capability-gated. The alternative is a capability gate, which is a wire change and belongs in a consistency pass across the other node-scoped actions rather than on its own. The added test pins the exact toast, so the sentence the docs now state cannot drift away from the behavior without a test failing.	2026-09-28 08:27:48 -04:00
Anso	f0906f86d6	feat(editor): add Save & Pull Images to the compose editor (#1948) ... * feat(editor): add Save & Pull Images to the compose editor Image acquisition only happened inside deploy and update, so warming the image cache before a maintenance window, or staging an update to apply later, meant deploying first. The editor's save dropdown gains a third action that saves the authored compose and env edits, acquires the stack's registry-backed images, and stops. No up, no restart, no recreate, no build. Backend: POST /api/stacks/:stackName/pull-images, gated on exact stack:edit and exact stack:deploy, the same pairing Git source apply uses. The route takes the per-stack operation lock as image_pull, so a pull cannot race a deploy and a second pull reports "is already pulling images". ComposeService.pullStackImages passes --ignore-buildable when the stack has build-backed services, so those are skipped and returned by name rather than built. The route reconciles nothing, so it opens no recovery generation, starts no health gate, writes no drift baseline or GitOps generation, runs no policy gate, and records no activity entry: there is nothing to roll back to, and the update-guard marker must not move. Frontend: the item is gated on both permissions on desktop and phone, and honors the diff-preview setting like its siblings. The phone split-button gains a chevron menu rather than a third button, which would truncate Save & Deploy at 390px. A pull that fails after the save succeeded reports only the pull half, since the save already confirmed itself. The advanced lifecycle options are not part of this change. * fix(stacks): sanitize the image pull failure log entry The pull route's catch block passed the caught error object to the console unsanitized. A failed pull rejects with the whole compose output, so compose or registry text could carry line breaks into the server log and forge entries. Route it through the same sanitizer the stack name already uses, matching the sibling route catches, and replace line breaks with a separator first so a multi-service failure stays readable instead of welding together. No change to the HTTP response: the 500 body still reports the same message. * fix(editor): condense the image pull failure toast to compose's error lines A failed pull answers with the whole compose output, layer progress included, and the error toast quoted all of it. With Deploy Progress off the toast is the only surface the operator sees, so the cause was buried. The toast now keeps the last three distinct failure lines compose writes ("<service> Error ..." or "Error ..."), counts any it drops, and falls back to the last output line when none match. Progress lines that merely contain the word, such as a service named error-reporter, no longer count as failures. The progress panel still streams the full output. * fix(editor): deliver registry credentials, run preflight, and report skipped actions A pre-merge audit of the Save & Pull Images feature found three defects. All three are fixed here, plus two non-blocking findings folded in at review time. Remote nodes were not receiving the main instance's registry credentials for the new pull route. Two classifiers gate credential delivery independently and both omitted it: the op classifier, which is the outer gate the proxy consults, had no pull-images branch at all, so the delivery block never ran. Added a stack-pull-images stage carrying the stack name from the path, since the route sends no body, and added the route to the body-budget classifier. The pull skipped the SOPS and required-env preflight every sibling compose operation runs, so on a SOPS stack ${VAR} in an image reference would resolve against still-encrypted files and could fetch an image a deploy would not use. Both guards now run at the top, and the refusal names the action the operator chose rather than telling them to deploy. A pull skipped after a successful save said nothing, leaving one green save toast as the last word. Both skip paths now report, and only after a save: a bare call is either blocked by an affordance the UI already disables, or is a second click while the first action is still starting, where an error toast would be wrong. Save & Deploy reports the same way, which it previously did not. The save row and the pull item also disable while a pull is in flight, on desktop and on the phone, since a pull holds the same per-stack lock a deploy does. --ignore-buildable is now passed unconditionally, so a service that builds from source is skipped whether or not the stack's build declarations were readable. The skip list remains the report, and a build: declared only in an override file is skipped but not named. The op classifier and the body classifier disagreed on the trailing-slash form, which Express routes to the same handler, so a slashed request was forwarded with no credentials and failed late with an opaque unauthorized. Both now accept it, and the rollback discover branch moves with them so a slashed rollback is still discovered against the restore candidate.	2026-09-27 19:24:45 -04:00
Anso	324dd7e1ae	fix(gitops): restore only the generation a Blueprint target acknowledged (#2024) ... * fix(gitops): restore only the generation a target acknowledged A Blueprint's Enforce mode restores state; it does not authorize it. The repair path resolved the identity to restore from the application's current pointers, so accepting a newer generation for any reason silently changed what an already-deployed target was repaired to. On a target acknowledged against an older generation this shipped compose content that no rollout had authorized for it, pinned to an artifact set from a different generation, which is a content and artifact substitution arriving through the one path that is supposed to be safe. The authority for a repair is now resolved in one place from immutable rows: the target's acknowledged generation and the artifact set that generation was qualified under, corroborated for Git-managed content by the live rollout generation that froze the pair. The application's newest accepted generation is no longer read on the repair path at all. Every way the authority can fail to resolve becomes an explicit hold with its own reason rather than a guess or a generic failure: no acknowledged generation, a superseded rollout, a node outside the frozen target set, an unreadable row, or a disagreement between the rollout and the target's acknowledgements. A hold records its own deployment status and history stage, repeats no work on the next tick, and never mutates the workload. The on-node marker now names the generation, artifact set, and rollout that installed it. Without that a repair could not prove what it was about to overwrite, so a marker that does not name one holds rather than assuming. Markers written before this still parse. Tests cover the acknowledged generation winning over a newer accepted one, the superseded-rollout hold, the never-acknowledged hold, and digest pins coming from the acknowledged set. * fix(gitops): hold every runtime repair the policy must not attempt A drift repair that cannot prove what it would overwrite must decline visibly, not fail quietly and not guess. Each way the authority can fail to resolve now becomes a typed hold with its own deployment status, its own history stage, and its own attention reason, so it stays visible instead of reading as converged. The arms are the states the drift model already treats as uncertainty rather than divergence: a target that is unreachable or whose connectivity evidence is stale, a target whose earlier mutation was interrupted, a target an unfinished rollback or an unavailable LKG owns, and a target outside the frozen required set of the rollout it is bound to. Two decisions move ahead of the drift write rather than after it. A stateful or unclassifiable Blueprint is never auto-repaired, and the classification check now runs before the drift is committed, so a held target is not first written as drifted and then rewritten on every tick. The check is repeated at the site that mutates, so the guard holds even when that path is reached directly. A hold is a state, not a latch: a later tick re-checks the target, so clearing the cause lets the next tick repair it. A hold that has not changed is not recorded again, which keeps a permanently held target from appending history forever. The on-node marker gains the generation, artifact set, and rollout that installed it, and the rollout path stamps them, so a drift check on a remote leaf compares against the same authority the hub resolved rather than inferring it. Both transports serialise the same marker, asserted byte for byte in the tests. Tests cover each hold arm, the hold-then-repair transition, replay suppression, survival across a store reset, local and remote reaching the same decision, and Observe and Suggest never mutating. * fix(gitops): re-check held drift targets and report a hold as a hold The hold work that closed the runtime drift policy made two states one-way, which is worse than the behaviour it replaced. A repair that re-resolved its authority and declined was reported as a correction failure and left the row reading "correcting" with nothing in flight, and a held row matched no branch in the decision pass, so an ordinary network flap could strand a target permanently where it had previously self-healed. Both paths now handle a hold as a decision: a declined repair records the hold with its own reason and warning, and a held row keeps its drift check, so clearing the cause lets the next tick repair it. A check that finds the target already matching returns the row to active through a cause that records no history, because nothing was written to a node. Classification holds get their own reason instead of borrowing evidence_incomplete, which sent operators looking for a missing generation when the Blueprint was stateful. Their text drops the drift reason, which was the replay key: a reason changing per tick re-alerted every tick for a hold that never changed. The recovery arm now also covers an in-progress recovery and a rollout awaiting its health verdict, both of which own the target. Docs no longer promise every hold self-clears. Superseding is one-way, so that hold persists until a later rollout covers the target or an operator deploys. * fix(gitops): bound the recovery hold to a recovery in progress The recovery arm tested recovery_phase for non-null, but that column keeps a terminal value for the life of the target and is never reset, so every target that had ever been rolled back would have been held from repair permanently, with only an explicit deploy to clear it. Only capturing, restoring, and compensating own the target now, matching the in-progress set the projection already uses. Clearing a hold without repairing it now advances the observation stage rather than only the deployment row. A row that went back to active while the projection still read the hold, or the drift, that had just cleared left the Drift surface and the attention queue contradicting the deployment table with no event left to correct them. Both carry tests that fail without the fix. The projection case asserts a literal status and requires the target to exist, because the optional chain it replaced made the whole case pass on a missing target. * fix(gitops): let a failed recovery keep holding, and rank both new statuses A failed recovery was the one terminal phase that stopped holding once the arm was narrowed to in-progress phases, so auto-repair would have performed exactly the mutation an explicit deploy is already refused on the same target. Only a recovery that finished successfully releases the target now, and a test pins it. The severity ladder had neither of the new runtime statuses, and a status it does not know does not degrade to something reasonable: it reads as unrecognized and the whole application row becomes unknown. A hold is ranked alongside drift for that reason. Clearing a hold advances the observation stage rather than only the deployment row, so the decision the reconciler made is what the projection reads, instead of depending on the observed-artifact write that happens to follow it. The test asserts the stage, because the derived status is identical either way. A new "converged" runtime status was tried here and reverted: the posture vocabulary has no slot for "checked but no health verdict", so it landed on unknown. A matched drift check projects to the status a Blueprint target already reached by the pointer path, which claims no health verdict either. Each test was checked to fail with its fix reverted. Two earlier attempts at the same tests passed with the bug reintroduced, one through an optional chain and one because a neighbouring check claimed the status first, so both now assert the mechanism rather than the outcome. * test(blueprints): render a held repair in the deployment table The held row is where a declined repair becomes visible, so it is rendered and asserted rather than only typechecked. A status that reaches the table without a label renders as a blank chip, and only the label map being total over the status union would have caught it. * fix(gitops): classify drift for Inline targets and observe held ones The repair authority was corroborated against a rollout generation whenever the target carried a rollout_generation_id, but an Inline target can carry one that was never the authority for its acknowledged pair, because Inline freezes its own generation per acceptance. Every such target was held for a disagreement that could not exist, and a held target reported no drift at all, so a replaced image on an Inline Blueprint was invisible in every drift surface. The corroboration is now keyed on the content mode rather than the pointer. A hold also suppressed the observation, because it returned before the runtime identity was collected. A hold refuses to mutate, not to observe, and what the node is running is exactly the evidence an operator needs when a hold blocks the repair. The identity is now captured before the hold is returned, with the probe and the record each guarded so neither can turn the hold into a failure. Both carry tests verified to fail with the fix reverted.	2026-09-27 18:49:20 -04:00
Anso	bb7757649b	fix(mesh): keep stack networks, scope mesh listeners, detect dead tunnels (#2010) ... * fix(mesh): keep stack networks, scope mesh listeners, detect dead tunnels Five Mesh defects, all reachable on setups using a reverse proxy, a tunnel or a VPN. The generated override listed only sencho_mesh in each service's networks, which replaced a service's implicit default network. The service lost its links to the rest of its stack and any reverse-proxy route reaching it there. A service that declares no networks now keeps default; one that declares its own gets sencho_mesh unioned in. Services on network_mode (host, none, service:x, container:x) are left out entirely, because Compose rejects extra networks and hosts entries on them and the redeploy failed outright, which is what blocked VPN-sidecar stacks from meshing at all. Because a meshed service is now dual-homed, the dial address had to follow the mesh: containerLookup prefers sencho_mesh and only then falls back. Sencho is attached to sencho_mesh and nothing else, and Docker bridges are isolated, so a default-bridge address is not routable from the mesh hop and every dial would have timed out. The forwarder bound 0.0.0.0, which exposed every meshed service's ports on any network the Sencho container joins, and on all host interfaces under network_mode: host. It now binds Sencho's mesh address, binds nothing until the data plane is up, and the same-node probe dials that address rather than loopback. A half-open tunnel stayed open until the kernel exhausted retransmits, so a tunnel, VPN or NAT restart left the reverse-dialer slot held and every mesh dial timing out. A shared heartbeat terminates a peer after two missed intervals, and the proxy peer now probes the tunnel holding its slot and replaces it only if it does not answer, so two centrals still cannot flap each other. The local socket to tunnel direction had no backpressure, so a bulk upload over a slow link buffered without bound. Both ends now share one high-water mark, and every pause is released on teardown, since a socket paused and never resumed would stall the transfer for good. * fix(mesh): resolve compose merge keys and unstick post-close sockets This closes two defects and four coverage gaps in this branch's first commit. - getDeclaredStackServices parsed compose YAML without resolving merge keys, so a service inheriting network_mode or networks through `<<: *anchor` looked like it declared neither. The override then added networks and extra_hosts to a namespace-sharing service, and Compose rejected the whole redeploy. Checked against docker compose config: with the unfixed parse the project fails validation, with the fix the stack comes up on its own network and the mesh network at the same time. - openCrossNode left the stream marked open after it closed. A container that kept writing got a backpressure signal that could never clear and was paused on a drain that never arrives, stranding the socket. Bytes arriving after the close now tear the local side down instead of parking it. - The WebSocket heartbeat counted only pongs and inbound messages, so a healthy but slow link was terminated every ~90s while the pong queued behind the upload. A shrinking send buffer now counts as proof of life; a flat or growing buffer still terminates. - PilotTunnelBridge.start could install the heartbeat after close() had already run, leaking the interval. - Covered the bridge's reverse-socket backpressure, the switchboard's closed-tunnel drain path, and both not-ready-data-plane guards. The switchboard test this replaces set up state, emitted drain directly instead of driving the poll, and asserted nothing, so it passed with the branch it claimed to cover deleted. - Corrected the network_mode rule in the override comment and the docs: Compose rejects a networks entry alongside every network_mode form and accepts extra_hosts on all of them. A network_mode service never carries a mesh alias, because aliases come from published ports and such a service cannot publish one.	2026-09-27 15:15:09 -04:00
Anso	2e0bf078c4	docs: retire the standalone known-limitations file (#2022) ... Limitations are disclosed on the feature page they qualify and fixed forward. Removes KNOWN_LIMITATIONS.md, the pre-1.0 disclaimer notes that linked to it, and names the Compose v2 plugin and Linux production host in the quickstart prerequisites.	2026-09-27 13:40:40 -04:00
Anso	acfcb5567b	fix(gitops): keep an application addressable when its evidence cannot be read (#2000) ... * fix(gitops): keep an application addressable when its evidence cannot be read A remote application whose evidence this hub cannot walk answered 404, the same as one that does not exist, so an operator's only signal was an application that had vanished. It now answers 503 with an `evidence_unavailable` code when the row names the application but the projection cannot be read, and the view says evidence is unavailable and offers a retry. A row that never named the application still answers 404, and an unreadable row stays 404 so the two cannot be told apart. Validating that answer meant walking the projection properly. The detail endpoint and the portfolio now check the same evidence: an artifact qualification the facet and its latest evidence disagree on, a blank generation or identity standing in for a real one, and a status from a newer node all read as unknown evidence rather than as a convergence claim. Convergence itself is now proven from the target evidence rather than read off the rollout status, an Inline Blueprint no longer inherits a Blueprint's claim, and a target the intent withdrew no longer keeps an application in the attention queue. Reading a remote row also authorizes it against the node that owns it rather than the node the caller is viewing, and the phone portfolio says partial when the merge cap dropped rows, not only when a node failed. * fix(gitops): grade unfamiliar evidence as unknown instead of unavailable Two shape validators rejected a vocabulary this build does not recognize, which made a whole projection unusable rather than merely unfamiliar. An unusable projection costs the operator its row: the portfolio shows an evidence unavailable placeholder and the detail route answers 503. That claims the evidence is unavailable, which is a stronger statement than an unfamiliar word supports. A last-known-good unavailable reason is now accepted on its structure and graded as unknown evidence instead, on both sides of the wire. The browser refusing the reason the server had already graded would only have replaced the honest answer with a total read failure, and the two validators would have disagreed about the same payload. A drift identity kind is accepted the same way, so one unfamiliar kind costs that row its label instead of the whole application. The label names the kind it cannot read instead of rendering nothing. The kind list is a total record over the identity union, so adding a kind without wording for it is a compile error, and the lookup is own-property, so an inherited key reads as unfamiliar. A blank reason is still rejected, and the mirror consistency rules still reject an incoherent last-known-good record. Verified: backend and frontend tsc clean; seven backend GitOps suites pass (446 tests); full frontend suite passes (3597 tests); frontend lint reports 0 errors. * fix(gitops): restore the recovery-required rollback scope and drop a dead import The conflict resolution carried this branch's three-member failed-runtime set over the four-member set on the main branch, because this branch had reordered the same three entries and the two edits merged without a conflict marker. A target whose recovery is restoring or compensating reports recovery_required, so it stopped counting as a failed target and the rollback dialog stopped offering the failed-targets scope for exactly the stacks that need recovery. The set now matches the main branch, with the type annotation kept. The existing test covered recovery_failed only, so the scope is now asserted for both recovery states. Also removes a type import the same resolution orphaned. The type checker does not report unused imports, so this surfaced only in the backend lint job.	2026-09-27 04:06:03 +00:00
Anso	248154a2d3	fix(blueprints): gate digest-pinned remote apply on leaf capability (#2005) ... Enforce repairs a frozen image-digest disagreement by redeploying the stack at the approved digest with the pull suppressed, so a drifted image cannot come back under the same tag. For a remote target the hub carried those pins in the apply request, but it never checked that the node receiving the request understood them. A node running a release that predates digest pins drops the field and redeploys by tag anyway, which is the one outcome the pin exists to prevent: the drifted image is pulled again, and the row is marked active while the drift is still there. The existing mixed-version guard on that request only fires when the node lacks the atomic apply endpoint entirely, so a node in between the two behaviours was never caught. The hub now probes the target for a new capability and refuses to send a digest-pinned apply to a node that does not advertise it, or whose metadata it could not read. The refusal is the same upgrade-required error the missing-endpoint case already raises, and it lands before the delivery contract is negotiated, so nothing about the apply leaves the hub. Applies that carry no pins are untouched, so nodes that never take part in a digest repair need no upgrade for them. The capability is advertised through node metadata, so an upgraded node starts serving pinned applies with no further configuration.	2026-09-26 17:28:44 -04:00
Anso	32fd95dd57	feat: health-gated Blueprint rollout policy (#2018) ... * feat: health-gated Blueprint rollout policy A rollout policy that reacts to each target's health outcome instead of deploying the whole fleet blind. Five modes, applied one target at a time under any non-observe policy: observe record the outcome and keep deploying the rest pause hold the rollout before the next target retry_once deploy the same generation one more time stop hold, and leave the failed target where it is rollback put the failed target back on the generation it ran before The policy is frozen into the rollout generation at authorization, so a fleet is never halfway between two policies, and it is carried across Blueprint edits. Observe is the default and is exactly the old behaviour. How it works: - A health run is reserved, with the whole rollout binding, before the apply goes out, so a lost response still leaves a durable record. - The next target is the first required node not settled by health, not the first unacked one, so a fleet cannot run ahead of a check. - The verdict drives a pure decision function; the executor carries the result out afterwards, so the state is durable before anything is applied or restored. - `unknown` never retries, stops, or rolls back. Missing evidence is held for a human, not acted on. - Remote targets are refused for non-observe policies before the first apply, because this instance cannot produce a health verdict for a remote node's Docker. Permissions: the policy write and the controls take `stack:deploy` at the application, and every target the policy can act on is checked exactly. * fix: hold before a policy restore and never lose its record Five review findings on the health-gated rollout policy, all on the paths the policy exists to make safe. A completion that could not be recorded was swallowed. `holdRollout` ran only after `restoreTargetToGeneration` returned, so a `rollbackCompleted` that threw left the outer catch with nothing to do: no hold, no partial record, and the target stuck in `restoring` while the fleet carried on as if the policy had not stopped it. The restore and the record of it are now one unit, and any failure records a partial failure on the target *and* the application, since `rollbackInProgress` opened the application into `restoring`. The hold now precedes the restore rather than following it. A restore is external and slow, and until it returns the rollout is still authorized, so an operator dispatch or an auto-accept could deploy the next target, or this one again, while it was in flight. A decided rollback was indistinguishable from a pause, so a crash between deciding and doing lost it: the fence read as a question, a resume answered it, and the rollout moved on past a target still on the failed generation. It now has its own reason, which no resume clears. A resume re-paused on every restart. A kept fence is also what reconstruction holds on, so a rollout the operator had already answered for was held again each time. A resume now records the answer as `stop_acknowledged`, which still keeps the target out of the queue but no longer holds the rollout. A policy rollback passed null captured refs, which cleared the source acceptance the restored generation was bound to and recorded it unprovable. That is source-authority state, and a health verdict has none over it. A blocked follow-up dispatch now holds the rollout with the reason it was refused. There is no scheduler, so that dispatch is the only thing that brings the next target. Also: the policy route required the fleet-wide deploy grant on top of the per-target checks, unlike the pause and resume routes beside it, so an operator whose grants are scoped to specific stacks could pause and resume but not configure the policy. The fleet-wide grant is now required only when there is no target to check. The rollback happy-path test asserted only that the restore had been *called*, and passed while `rollbackCompleted` threw inside the executor because the fixture's generation row was fake. It now seeds a real one and asserts the completion, plus the completion-failure, restore-throws, and source-acceptance cases. * fix: a pending roll back stops the rollout, not just its own target Excluding a `rollback_pending` target from the queue is not enough. The queue steps over it and deploys the next one, so a fleet advances past a target still on the generation that failed, with no restore done and nothing telling the operator. Resuming did not help either: it left the fence in place and unpaused the application, so the next dispatch moved on and the following restart paused the rollout again, repeating for as long as the operator alternated the two. A gated dispatch now refuses outright while any target of the current rollout generation is `rollback_pending`, and the refusal names the manual restore route as the way out. The fence stays where it was, so the target card still reports it. The test now uses two nodes and dispatches after the resume, so the step-over is visible; it is verified to fail without the change. Also corrects the schema comment, which claimed a restart mid-rollback "reconciles the rollback". It fences the unfinished restore and does not replay it, which is what the comment says now.	2026-09-26 15:34:14 -04:00
Anso	91648b049d	fix(gitops): keep Stack Drift, notifications, and metrics on the canonical posture (#2011) ... * fix(gitops): keep Stack Drift, notifications, and metrics on the canonical posture Three surfaces consumed GitOps application status without going through the posture read model, so each could contradict the portfolio about the same application. The portfolio, the application detail panel, and the Blueprint surfaces were already on it; these three were not. Stack Drift rendered only the node-local revision, which has no posture and no evidence freshness. Every facet card could read satisfied with no drift listed while the portfolio reported the application unsettled, because the evidence was stale, a node had gone dark, or a target was unreachable. The tab now reads the hub-owned portfolio row by application id and renders the posture, the evidence notices, and per-target evidence freshness, and each drift item shows the age of the observation behind it. The read is keyed on the application id the projection already carries rather than on a stack name, because a Blueprint application has no stack name of its own and two applications can share a deploy stack. A settled source attempt whose outcome is unproven notified as a pull failure at error level, while the portfolio reported the same application as in-progress. Unproven is now its own category at info level, saying the attempt settled without proving a result. Genuine failures are unchanged. A recorded health-check failure produced no notification at all, because the finalize stage was not notifiable, so a known-bad workload left the portfolio reporting a failure with nothing behind it in the bell. A recorded failure now notifies, and only a recorded failure: a passing check, a check that could not reach a verdict, a verdict that could not be tied to the running stack, and a failure recorded against a generation a rollout has already superseded are all silent. The health transition now records the verdict it actually persisted, which is what the notification keys on, so the bell and the projection cannot disagree about which failures count. Metrics was audited rather than changed. Its counters key only on transition stage and outcome, both closed unions carrying no application identity, so no count derived from them can be read as a claim about an application. That is now pinned by a compile-time exhaustiveness assertion and a behavioural test, because a settled word reaching that union is the only way it could start lying. * fix(gitops): stop the posture consumers contradicting the portfolio Follows review of the first attempt at this change. Four defects, three of them introduced by that attempt and one pre-existing. The health notification was a duplicate. The only producer of a recorded GitOps health verdict is the health gate, which announces the failure as it records it, so a failed run produced two bell entries saying the same thing. The GitOps outbox now stays out of it, and a test pins that a recorded failure yields exactly one notification and names where it comes from. The unproven reconcile outcome now does not notify at all, rather than notifying at a lower level. A settled attempt that proved nothing is evidence the portfolio reports as in progress or unknown, and the criterion is that the bell must not report on such evidence. What is unproven is the portfolio's to show, not the bell's, so the row is suppressed at the point the outbox decision is made. The insert and the drain carry the same transition record, so the two cannot disagree about which rows exist. The posture card warned falsely in two cases. A detached stack still projects a revision, so an identity was built for it and the portfolio route answered 404, producing a permanent warning about an application the portfolio does not list. A Blueprint stack read by a caller without the fleet-wide node grant answered 403 and produced the same permanent warning for a stack that caller is entitled to read. Both now say nothing: the route answers 404 for missing and not-granted alike on purpose, so neither status can honestly be reported as a fault, and the tab keeps the node-local view it always had. The unreadable warning is now reserved for the one case that is a real fault, a permitted read the server could not answer. The metrics invariant test claimed to be an exhaustiveness tripwire while only checking that its listed values were allowed. It now also asserts no unlisted member exists, verified by adding a value and watching it fail. Tests for the new hook cover the answers per outcome, the hub-only read, a slow answer for a previous identity being discarded, and the announced refetch with its debounce. Backend 1025 GitOps tests, 479 neighbouring tests, and 3506 frontend tests pass; both typechecks, both builds, and both lints are clean. * fix(gitops): scope the posture read and drop the report that no longer fits Follows the non-blocking findings on the posture work. Six items; four changed code, two are recorded as needing no change. The change announcement names the application it came from, so the posture read filters on it rather than treating every GitOps event in the fleet as this application changing. That was wasted work and a card that flickered on activity it had nothing to do with. An announcement that names no application is not filtered, because it cannot be attributed and dropping it could miss a change to this very application. A node change left the previous node's report in state for the length of the new request, so the new node was composed with the previous node's application and one identity that cannot exist was probed. Clearing the report was not sufficient on its own: the render that follows the node change already sees the old report, because the effect that clears it runs after that render. The report now records the node it was read from, and the posture read is skipped unless that is the node now selected. A test drives the switch and asserts the mismatched request is never sent. Work in flight and cannot-prove shared a glyph. They are the two postures most easily read as one state from a distance, and they are exactly the two that must not look alike on the surface an operator opens to find out which it is, so each tone now resolves its own icon. Boot repair drains rows an earlier process inserted, so a build predating the unproven suppression can leave one behind. The repair now declines it too, rather than announcing a failed pull the current insert would refuse to write: the one path meant to reproduce the original outcome must not be the path that diverges from it. The row is still drained, and the suppression is logged so a silent entry is a recorded decision rather than something a reader later has to explain. The health gate being disabled mid-run, which leaves a failed verdict announced while the projection reports health as not applicable, needs no change here. The entry it describes is the health gate's own, and the GitOps notification that would have doubled it is gone. The reclassification of a stored row is no longer intended but is no longer reachable either, since the category it was reclassified into does not exist.	2026-09-26 15:04:57 -04:00
Anso	61b25f48d2	feat(gitops): show per-target image digests and document the drift model (#1984) ... * feat(gitops): show per-target image digests and document the drift model The observed per-service digests already reached the projection, but the approved side did not, so a reader could see that two identities differed without ever seeing which digest each side held. The frozen artifact set's per-service evidence now rides along with the expected identity, and each target's card compares them per service: the approved digest beside the running one, the approved child for the platform that node actually runs, and matched, drifted, unverified, or not compared as the honest answer. The comparison follows the model's own rules, so the view cannot claim a match the projection does not, a stale or unqualified identity stays unverified, a built service is labelled not compared rather than matched, and a value that is not a content digest is ignored instead of compared. Adds the drift model reference the model was missing: the seven classes and what each is measured against, how to read an item, where each surface shows it, the states that are progress rather than drift, what Sencho cannot prove, and the four policy domains. The Blueprint page gains the digest-drift section it never had, trimmed to the Enforce mechanics. The end-to-end spec pins an approved digest, replaces the running workload with different content under the same project, and asserts the deployment reports the drift, the projection carries both digests, and the Drift tab renders the comparison without adopting what the tag now points at. * test(gitops): pin the digest drift spec to a resolvable repository The spec pinned a locally invented repository, e2e-digest-probe:<stamp>@sha256:..., which is not a reference any registry knows. That works or fails depending on what the local Docker version decides to do with a tag-and-digest reference that has no matching repo digest, and the observation filters the running image's repo digests by the authored repository, so an invented repository can never produce an observed digest at all. On a runner whose Docker resolves it differently, the deploy never lands and the spec waits out its poll. Both sides are now real upstream tags of one repository, so the approved reference resolves from the local content store without a registry call and the moved side is a real digest difference. The digests are read from the daemon per repository rather than trusting Docker's ordering, and the two sets are asserted disjoint before anything runs, so two tags that resolve to one manifest fail by name instead of by timeout. The spec also stopped assuming things it never checked. The baseline is proven from the daemon, because the container that was actually created from the approved reference is the ground truth for what is running, and reading it costs one call where waiting for the projection's observation costs a reconciler tick. Every poll now asserts its status, so an API error reports itself instead of surfacing as a timeout, the apply retry is limited to the stale-preview case the UI asks the operator to confirm again, Docker calls are bounded, teardown clears its handle only after a confirmed delete and then checks the containers are actually gone, and a Docker that is installed but broken fails instead of skipping. Verified against local dev servers across four runs at 16 to 55 seconds each, with no leftover images, containers, or override files, and type-clean under a strict invocation. * test(gitops): prove Enforce repairs to the approved digest, not the moved tag The digest drift spec proves detection. It does not prove repair, because it runs the Blueprint in observe mode, and the Inline path is not the one that repairs. The Git-managed loop re-observes a drifted target and applies the configured drift mode, so that is where the repair mechanic lives, and it now has a test of its own. The test starts from a genuinely drifted target rather than an assumed one. The fixture seeds a newer candidate artifact set carrying a moved digest, points the latest pointer at it while the approved and last-known-good pointers stay on the original, and records a runtime observation carrying the moved digest. That starting state is asserted before repair runs, so a fixture that quietly stopped seeding it fails instead of passing on a correct-looking end state. Repair then has to do the one thing that matters: hand the deploy the approved platform child digest, take it from the accepted artifact set rather than the most recent resolution or the running image, and leave every pointer that says what was approved exactly where it was. The observation still reads as drifted afterwards, because repair changes what runs, not what was agreed. The test is mutation-checked. Pointing repair at the latest artifact set instead of the expected one makes it fail with the moved digest, which is the bug this exists to catch. The taxonomy page also records the settings-only reconcile decision properly: source drift compares commits, so a change of compose path, env path, or context directory creates no expected-versus-observed pair and no drift item. It surfaces as reconcile required, and for a Direct application that is where you fetch again. The page says plainly that this is a choice about what the word drift means, not proof that the materialized content was unchanged, because changing where Compose is read from can change the content. The portfolio page no longer keeps its own copy of the class list. It names the classes by reference to the drift model page, so the two cannot disagree. Verified: backend typecheck and lint clean, 9515 backend tests passing, frontend typecheck clean with zero lint errors and 3457 tests across 347 files, docs config parses. * test(gitops): pin a settings-only reconcile as attention, not drift The decision is recorded on the taxonomy page: a change to the compose path, env path, or context directory, with the accepted commit still matching the ref, creates no source drift item. Source drift compares commits, and that shape has no expected-versus-observed pair of commits. That decision needed a test to hold it, and the coverage that existed asserted the facet status and the offered action without ever asserting that no source item is emitted. The behavior was therefore unpinned in both directions: nothing would have failed if the producer started reporting it, and nothing would have failed if it stopped. This adds the missing half. The fixture is the decided case exactly: same commit, different materialization fingerprint. No source drift item, the facet reads reconcile required, and fetch is offered. It is mutation-checked, and dropping the commit comparison from the producer makes it fail, so it holds the line rather than merely restating it.	2026-09-24 19:19:19 -04:00
Anso	e1c044d05b	fix(exec): fall back to /bin/sh when a container has no bash (#1997) ... Opening a shell on a container without bash (Alpine, BusyBox) failed with "exec: /bin/bash: no such file or directory" instead of falling back to /bin/sh. With a TTY attached, Docker accepts exec creation and start for a missing binary and only reports the failure as stream output, so the try/catch fallback never ran. Probe /bin/bash then /bin/sh with a short non-interactive exec and pick the first that exits 0. Exit 126/127 means the shell is absent; any other probe failure (timeout, daemon error) is reported with its real message instead of a misleading "no shell", and the socket is closed on failure. Also replace the empty catch on the Host Console Copy button with a log and an error toast.	2026-09-24 19:09:05 -04:00