11 Commits

Author	SHA1	Message	Date
Anso	f5178889eb	feat(recovery): complete authored-project atomic rollback generations (#1819) ... * feat(recovery): capture complete authored Compose project for atomic rollback Replace the root-compose-only backup slot with staged recovery generations that record the managed inventory, exact Compose invocation, and prior image identity, and wire the same engine through deploy, update, manual rollback, and Git apply. * fix(recovery): satisfy CodeQL path barriers and update-guard mock Inline resolve+startsWith checks at generation/inventory fs sinks and stub getCurrentStackUpdateRecovery in UpdateGuardService tests. * fix(recovery): drop unused FileSystemService import in generation store test * fix(recovery): harden authored-project rollback for upgrade and restore safety Preserve legacy UUID backup rows, restore Git deploy state with files, make multi-file restore recoverable, evaluate policy on the restored target, and fail closed when Git capture cannot cover an apply. * fix(recovery): unblock Git apply unit tests and CodeQL pre-restore TOCTOU Mock recovery capture in git-source-service tests after fail-closed apply capture, and re-resolve live paths immediately before pre-restore snapshot reads. * fix(recovery): fall back to authored inventory when Git manifesto is missing First Git apply captures before promote, so a missing managed-project manifesto must not block rollback capture when the live stack already has authored files. * fix(recovery): make authored-project rollback atomic across Git state Restore the managed-project manifesto with files, keep nullable Git identity on first-apply captures, persist Git side-state in restore intents for startup reconcile, compensate legacy materialize failures, and refuse directory collisions before mutation. * fix(recovery): satisfy CodeQL path and TOCTOU barriers on manifesto restore Add inline resolve barriers for manifesto read/clear sinks and remove the access-then-read race when restoring a generation manifesto snapshot. * fix(recovery): close third-audit rollback generation blockers Fail closed on incomplete Git inventory fallbacks, execute captured Compose invocation during recovery, refuse startup and mutations while restore intents remain unresolved, propagate legacy stale-delete failures, and add Docker-level exact prior-image coverage plus regression tests. * fix(recovery): mark acquired before handoff in prior-image Docker test Match the production updateStack CAS sequence so the exact prior-image integration test does not fail handoff from the captured phase. * fix(recovery): close fourth-audit rollback safety blockers Evaluate policy against held images, use index-based pre-restore snapshots, hold the shared stack lock across Git apply, replay Mesh and empty captured invocations exactly, restore POSIX modes with fail-closed sensitive permissions, keep case-sensitive paths, and link Git auto-deploy health gates. Add regression coverage for these cases. * test(recovery): fix mocks for health-gate link and authored compose args Add linkGateOrRetain to the Git apply recovery mock, and mock authoredComposeArgs so the case-collision inventory test is not masked by a missing getComposeDir stub. * fix(recovery): close fifth-audit rollback safety blockers Share git_apply locking for webhook auto-apply, fail closed on malformed recovery service records, refuse mixed-image capture, and require exact probe counts with hold-tag eligibility checks. * fix(recovery): close sixth-audit rollback safety blockers Preserve the legacy backup slot during generation capture, encrypt sensitive pre-restore snapshots, revert files on a failed health probe without committing Git, fail closed when an absent-file revert would delete a directory, skip Compose one-offs, route manual and scheduled backup through the current generation, and persist runtime image platform identity. * fix(recovery): close seventh-audit rollback safety blockers Fleet snapshot restore and restore-all now capture a recovery generation under the stack lock before any authored file write, including on remote nodes. * fix(recovery): keep pre-deploy generations during health-gate observe Link deploy recovery generations to the observing gate so backup cannot replace them mid-observe. Distinguish missing hold tags from probe failures, refuse generation release when services metadata is corrupt, classify mixed-replica and coverage refusals, and toast the backend rollback message. * fix(recovery): wrap webhook deploy case for eslint const bindings in an unbraced switch case trip no-case-declarations. Match the pull case block.	2026-08-13 03:48:09 -04:00
Anso	578ce7684d	feat(git): complete-project materialization with a managed-project manifest (#1786) ... * feat(git): add managed-project manifest types and DB cache columns Introduces the canonical managed-project manifest contract types (schema v1) and the stack_git_sources cache columns manifest_version / manifest_state / manifest_generation. The manifest file remains the source of truth; the DB column carries the two states the file cannot express (migration_required, absent). * feat(git): add vendored Docker .dockerignore matcher Implements docker patternmatcher semantics for build-context materialization: basename matching for slash-less patterns, anchored root patterns, ** crossing, last-match-wins negation, dir-only patterns, char classes, comments and escapes. Table-driven tests cover the full rule set. * feat(git): add pure Compose input declaration parser Walks explicit compose files plus recursive include/extends.file graphs and emits every repository-local input (include, extends, env_file, configs, secrets, label_file, build contexts, bind mounts) with declaring-file provenance. Side-effect free: file contents are injected via a read callback. Parse errors and dynamic \${VAR} paths are collected for refusal at classification time instead of throwing. * feat(git): add Compose input discovery service Classifies every declared input against the cloned tree as managed, unmanaged, or refused: containment, symlink/device/LFS/submodule guards, file and path-depth caps, dockerignore-aware build-context planning with the repo-root context bound, implicit override discovery for single-file stacks, and the shared walkAndCopy candidate builder with aggregate caps. * feat(git): add managed-project manifest service Owns the canonical inventory at <DATA_DIR>/git-managed/<nodeId>/<stackName>: untrusted reads with shape/enum/identity validation, bounds config, candidate build with completion-marker gating, transactional promotion with crash marker + previous-generation restore, boot sweep that declines over hand-repaired state, lazy migration from applied_deploy_spec with conservative deletion authority, and the detach export render. * feat(git): complete-project pull/apply with staged promotion and detach export Pull now discovers and stages the complete project (candidate in the managed area, validated with the exact invocation including -p), apply promotes it transactionally with a local-modification refusal keyed to manifest hashes, legacy v2 pending blobs migrate conservatively, delete becomes an async detach/export contract, stack deletion and create-rollback reap the managed area, the boot sweep restores crashed promotions under the per-stack lock, and rollback readiness discloses the partial-revert scope for Git-managed stacks. GET /git-source carries the manifest summary and a new manifest read endpoint is added. * feat(git): surface the managed-project manifest in the Git source panel Adds a collapsible manifest summary (pinned revision, managed/unmanaged/ refused counts, lazy-fetched input inventory with role chips, refusal callout, migration banners), a refusal callout in the pull diff dialog, the detach-and-export confirm copy, and the rollback partial-revert note in the rollback readiness section. * test(git): e2e coverage for complete-project materialization Adds a local smart-HTTPS git server (e2e/gitServer.helper.ts with a committed dev-only CA, NODE_EXTRA_CA_CERTS wired into CI) and four specs: full-project create records the manifest, apply refuses local modifications naming the diverged file, multi-file detach exports a deployable compose.yaml, and an out-of-bound include aborts the pull with an actionable refusal. * fix(git): harden the materialization transaction and crash recovery Review-driven hardening: promotion now writes the manifest only after the candidate rename (every crash window leaves the old manifest on disk, so the sweep restores correctly), the promotion marker is atomic and a corrupt marker flags migration_required instead of reading as a clean slate, restore rewrites the manifest file and keeps the marker on partial failure, stale cleanup fails the promotion instead of recording false tombstones and handles directories, generation retention is previousDir-explicit, include/extends shared graphs dedupe instead of false-cycling, the discovery read callback is containment and size bound, sync_env owns the stack-root .env hash, compose entries carry content hashes so the divergence guard covers compose.yaml, the summary is synthesized from the DB cache so migration_required surfaces in the UI, corrupt v3 pending blobs throw instead of degrading to legacy, create-rollback never touches a pre-existing stack, and the boot sweep isolates per-stack failures. * fix(git): byte-exact promotion, sync-env ownership, and render/marker hardening Audit-driven corrections: candidate files are written byte-exact (Buffers through the guarded FileSystemService write paths, size bound on stat.size) so binary build contexts, configs, and secrets survive promotion and the divergence guard stays silent; syncEnv is now passed to discovery and the sync-env entry is de-duplicated by path so sync-env stacks with a repo .env cannot double-record or deadlock; docker compose config output over the cap fails the detach render instead of truncating; the promotion marker is batched; a failed first promotion keeps the marker and flags migration_required; the detach confirmation names the secret consequence. Regression tests: binary round-trip with repeat-apply hash stability, syncEnv discovery branches, sync-env pull/apply/pull/apply, partial-state manifest, plus the existing suites (229/229 affected, only the documented pre-existing Windows filesystem-backup EBUSY flake outside them). * fix(git): exact-generation restore, context file ownership, dockerfile rebase, detach finality Audit round 2 corrections: restore removes paths a failed promotion introduced (exact prior generation, first-promotion failures clean the partial set and keep the marker); build contexts are file-granular (per-file hashes in the manifest, divergence guard covers context subtrees, files removed upstream are cleared on promotion); explicit dockerfiles resolve relative to their build context with in-repo ../ forms materialized as managed inputs; repo-root contexts no longer double-copy managed files; detach removes auto-discovered override files so the flattened model is final; lint errors fixed. Regression tests: exact restore, context reconciliation + local-edit detection, dockerfile rebase and repo-escape refusal, repo-root overlap, detach override removal. 213/213 affected backend tests. * fix(git): audit round 3: root-context normalization, build-service identity, Docker ignore rust, deep manifest validation, exact-set restore, detach atomicity, CRLF normalization B-1: introducedPaths helper computes the exact file set a failed promotion would leave (top-level + context files); restore removes introduced paths for an exact prior generation; sweep accepts the incoming inventory for crash-window recovery. B-2: repo-root context (build: .) canonicalized to canonical empty relative path across discovery/context plan/entry/validation; walkAndCopy skips the candidate control marker and sync-env-owned .env so root contexts never copy Sencho metadata into the live stack dir. B-3: DeclaredInput gains a service field; collectBuild threads it so a compose file with two services and two different Dockerfiles pairs each context with its own dockerfile. Additional contexts never inherit the service dockerfile. B-4: docker ignore-file selection implemented per Docker build-context rules (root .dockerignore, with Dockerfile-specific <name>.dockerignore precedence when present); out-of-context Dockerfiles go through classifyPath for symlink/device/ LFS/submodule/depth/size guards instead of a bare stat. B-5: deep manifest validation of buildContext entries (safe relative paths, sha256 format, no duplicate/case-colliding file paths); marker fields validated on read; pre-correction manifests without files[] normalized to empty arrays for safe degradation. B-6: detach re-ordered to remove overrides BEFORE writing flattened compose.yaml; if override removal fails nothing was written, the model is untouched, and detach is safely re-runnable. S-1: ComposeService.ts LFs normalized to repository convention. All 213 affected backend tests pass; tsc + lint clean both sides. * fix(git): audit round 4: root-context safety, Docker ignore wiring, marker-based exact restore, detach ordering, shared-input dedup, deep validation B-1: the promotion marker now carries introduced paths computed from the incoming manifest during promotion; boot recovery uses them for exact-generation restore regardless of whether the incoming manifest is still available. `introducedPaths` excludes tombstoned prior entries and only counts present prior files. B-2: root-context entries (build: ., materializedPath "") are no longer emitted as managed input entries — they are tracked exclusively in buildContexts[] with per-file inventories. `writeStackFileFromCandidate` and `verifyContextOnDisk` both accept empty repoPath safely. B-3: Dockerfile-specific .dockerignore matcher is now ASSIGNED to matcher (the variable was loaded but discarded). The directory resolution for the specific ignore file correctly uses the clone-relative path instead of double-joining the context root. B-4: detach now writes the flattened compose.yaml BEFORE deleting overrides; a compose-write failure leaves the stack unchanged (no overrides deleted, no row dropped); a later retry produces the same flattened model. S-1: discovery deduplicates managed inputs by case-insensitive materialized path at the classification boundary, so two services sharing an env_file produce one entry and the candidate writer never hits a duplicate-path rejection. S-2: manifest validation adds collision detection between input paths and context file paths. All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): audit round 5: root-context collision fix, marker-based recovery, Docker ignore root case, context divergence, detach rollback B-1: root-context files that collided with managed input paths (compose.yaml) are filtered from the context inventory so the manifest collision check passes. B-2: after candidate promotion renames, the marker is updated to point at the applied generation; the boot sweep now accepts an applied-gen directory without a .candidate-complete marker as valid for recovery. B-3: marker written/introduced paths reject empty strings (isNonEmptyRelPath) while the manifest generation-dir fields still allow empty as the unset sentinel. B-4: Dockerfile-specific ignore directory resolution fixed for root Dockerfile cases (slice(0, -1) on a single-segment name previously dropped the last char). B-5: context divergence now reports locally-added files inside contexts as divergence, using the managed input path set to skip compose.yaml/.env/files that have a non-context owner. B-6: detach now snapshots the prior compose.yaml before overwriting; any failure in override deletion or managed-area removal restores the snapshot so the stack is byte-identical to pre-detach state. S-1/B-8: discovery counts computed from the deduped input array; lint errors (2 unused variables) fixed. All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): audit round 6: root-context promotion, recovery ordering, context-file merge, divergence precision, detach rollback B-1: root-context files now explicitly promoted from the candidate via a context-file loop after the managed-input promotion step. Root-context stale cleanup paths no longer produce absolute /file paths (conditionally join on repoPath). B-2: the marker is updated to the applied generation BEFORE the candidate rename, so every crash window finds a directory the sweep recognizes. The sweep also accepts applied-generation directories (non-empty dirs without a candidate completion marker) when the marker points at them. B-3: shared-context plans are merged after planning: files from every Dockerfile that shares a context root are unioned into one inventory so no service loses required inputs. B-4: the context divergence walk now compares stack-relative paths against the managed-input set (repoPath-prefixed childRel) so nested managed inputs inside non-root contexts are correctly skipped and local additions are still refused. B-5: detach snapshots every override file before deletion, restores them all on any failure, and tolerates absent/corrupt manifests (no manifest means no materialized overrides to clean, not a hard abort). All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): audit round 7: root Dockerfile containment, inventory-driven context copy, file-only marker recovery, detach transaction B-1: root-context Dockerfile containment check fixed for root contexts ("" or "."). Any repo-relative Dockerfile without ../ is in-context. B-2: context copy now reads from the plan inventory (plan.context.files) instead of re-walking the source with the first matcher. Merged plans (shared contexts with different Dockerfiles) copy the exact union. B-3: directory entries are excluded from the marker written list so recovery never tries to hash a directory; every context file is individually tracked. Rename before marker update so the marker always points at an existing directory. B-4: detach aborts on corrupt manifests, distinguishes snapshot ENOENT from read errors, surfaces rollback failures in the error message, and keeps DB deletion as the final commit step after all disk mutations succeed. All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): sanitize log messages and fix CodeQL log-injection finding The one genuine CodeQL alert (log-injection + format-string at line 1002) is resolved by wrapping stackName with sanitizeForLog(), matching existing precedent in ComposeService.ts and routes/stacks.ts. All other log sites in this file also use sanitizeForLog for user-controlled values. * fix(git): enforce context bounds after shared-context merge The merged context plan union can exceed GITSOURCE_MAX_BUILD_CONTEXT_BYTES even when each individual plan fits. Recheck the cap against the unionized inventory after merging. * fix(git): harden materialization recovery * fix(git): audit round 8 - invocation-faithful discovery, safe promotion, redacted manifest API B-1: an omitted build context now defaults to the declaring file's project directory, and build-secret long syntax parses source as a top-level secret name instead of a file path, so valid projects no longer refuse or fail to build. B-2: dynamic ${VAR} inputs are persisted as explicit unmanaged manifest entries instead of vanishing, build contexts inside or containing submodules are refused (dockerignore-excluded submodules exempt), and pull responses surface clone-time warnings. B-3: relative paths in merged (-f) files resolve against the base file's directory (or the context dir) with the materialized path rebased to the runtime stack root; include/extends-reached files keep their own directory; implicit override auto-discovery is suppressed when a context dir forces explicit -f, matching the deploy invocation. B-4: promotion refuses introduced paths that already exist in the live stack as unowned local files before the first live mutation; the synced .env and fresh-stack creation stay exempt. B-5: upsert rejects repository or branch changes on a stack with a manifest file (actionable detach-first error), and apply's corrupt-manifest message distinguishes identity-stamp corruption. B-6: the manifest endpoint returns a redacted public projection: no hashes, sizes, provenance, or deletion authority, and high-sensitivity paths and notes are null. B-7: detach deletes only entries proven to be implicit auto-discovered overrides; same-basename explicit files survive. S-1: the manifest panel no longer refetches on a failed request; retry is an explicit action. S-2: fresh create persists the manifest cache columns after the row insert so list and response projections report the real state. S-3: git-sources.mdx matches the corrected detach, submodule, and dynamic-path behavior. * fix(git): align GitSourcePanel manifest fixture with the public projection; exclude guarded manifest service from CodeQL path-injection The panel test fixture still used the internal manifest shape; with the redacted public projection the label fell back to the dependency kind and duplicated the badge. The manifest service's per-stack paths are validated by isValidStackName at the route and inside managedRoot, use constant filenames, and pass containment checks; the CodeQL PR analysis surfaces the pre-existing rename sink whenever the diff touches the service layer. * fix(git): restore ComposeService.ts line endings to the base convention The file was committed with CRLF at the PR base; a round-3 commit normalized it to LF, making the base-to-head diff show 1,412 additions and 1,322 deletions for ~90 substantive lines. Restoring CRLF collapses the diff to the functional changes only. * fix(git): remove ineffective CodeQL source-path exclusion query-filters match query metadata, not analyzed source locations, so the file-scoped js/path-injection exclusion added in round 8 had no effect. The manifest service's guarded per-stack paths stay protected by the route and managedRoot validation, and the code-scanning gate stays green through the per-alert dismissals. * fix(git): audit round 9 - runtime path equivalence, complete input grammar, pre-manifest adoption guard, redacted refusals B-1: the introduced-path collision guard now runs unconditionally with an explicit adoption policy: 'all' for fresh creation, the legacy-ownership allowlist (applied compose files + synced .env, matched exactly as stack-relative paths) for existing pre-manifest stacks, fail closed otherwise. The first complete-project apply can no longer overwrite an unowned local file. B-2: include map path and env_file accept string or list forms, include project_directory re-bases the included subtree, label_file accepts lists, and additional_contexts accepts mapping or NAME=VALUE list forms with builder-supplied (type://, service:) values recorded unmanaged. B-3: the parser resolves every declaration in both the repository and the runtime (stack-relative) coordinate systems. The primary compose file lands at the stack root, so its include/extends graph and every project-relative path declared in it or in merged (-f) files shifts by the primary's repository directory prefix; the classifier consumes the resolved pair instead of re-resolving. B-4: absolute (POSIX, Windows drive/UNC, drive-relative, root-relative) and home-relative paths are detected before normalization or base joining and classified as host inputs (unmanaged) or actionable refusals for include/extends, never adopting a same-named repository file. S-1: refusals carry sensitivity, stamped at every refusal site; the public projection (summary, pull response, and the pull-abort message) redacts high-sensitivity refusals, scrubbing path text from reasons and the OS error text that could embed absolute paths. Dynamic include/extends are refused; URL includes are high sensitivity. S-3: ComposeService.ts line endings restored (separate commit). S-2: invalid CodeQL source-path filter removed (separate commit). * chore: bump nanoid to 3.3.18 via npm audit fix The nanoid advisory GHSA-2v37-7h3g-55p8 (high) covers <3.3.17 and was published after the last green CI run; both lockfiles pinned 3.3.16. npm audit fix bumps the transitive dependency to 3.3.18. * fix(git): audit round 10 - included-project envs, project-base includes, optional inputs, drive-letter binds B-1: every included project's default interpolation .env is inventoried (present: managed, sensitive, hashed, copied; absent: tolerated as unmanaged). interpolation: false and same-base includes skip the entry. B-2: include, include-env, and extends.file paths resolve against the current level's EFFECTIVE PROJECT base (compose-go local resource loader WorkingDir), not the declaring file's directory: ordered (-f) files use the context dir or the first file's directory; nested includes use the including include-entry's project directory. Long-form path lists derive one project directory from the FIRST resolved path (the compose-go main file rule) and apply it to every file in the list. Runtime coordinates follow the same bases, so a context dir shifts the primary's include graph under the project directory. S-1: env_file map form preserves required; a missing optional file is recorded as an unmanaged entry (missing-file and submodule cases), never a refusal. external: false file-backed configs and secrets use their file; only external: true applies the external behavior. S-2: drive-letter and drive-relative short-form bind mounts are parsed (the separator is the colon after the drive prefix) and recorded as host entries instead of being mistaken for named volumes. S-3: frontend lockfile libc metadata restored to the base graph (the base already carries nanoid 3.3.18). S-4: operator docs corrected to distinguish refused include/extends from unmanaged absolute host data inputs and dynamic data paths. * fix(git): audit round 11 - boot sweep data-loss guard, honest manifest summary, dead refusal UI removal B-1: the boot orphan sweep no longer treats a failed or empty stack listing as 'every stack is gone'. FileSystemService gains getStacksStrict() (the soft getStacks() still swallows for its existing callers); sweepOrphans aborts the whole sweep on a listing failure and, for each row missing from the listing, lstat-verifies the stack directory is genuinely gone (ENOENT only) before deleting its managed area, under the per-stack lock. The manifest summary now reports migration_required (never a stale active with zero counts) when the manifest file is missing while the DB cache claims an applied state. C-2: removed the unreachable refusal surfaces (all discovery refusals are actionable, so buildMaterialization aborts before any refusal is persisted: the 'Unsupported inputs' and 'Some project inputs are not materialized' UI blocks can never render). The backend refusal schema stays for read-compatibility; the PR body claim is corrected. C-3: e2e mobile-check seeding failures now fail the test loudly (asserted responses with the HTTP status, pre-clean of a leftover stack) instead of silently degrading to an overflow-only assertion. * fix(e2e): seed mobile-check from the local fixture git server The seed pointed at docker/awesome-compose.git with compose_paths ['compose.yaml'], but that repository has no root compose.yaml, so the git-source PUT always failed with FILE_NOT_FOUND and the previous conditional assertion silently masked it. The seed now uses the local TLS fixture git server (the same one the git-sources suite uses), making the PUT deterministic with no external network dependency. * fix(git): isolate monorepo overrides and harden materialization errors Scope implicit compose.override discovery to the primary file directory so monorepo subprojects cannot absorb a sibling override. Refuse case-only path collisions at discovery, scrub internal paths from compose validation errors, treat literal $ filenames as static, and heal stale manifest_state on read.	2026-08-10 17:12:55 -04:00
Anso	78475d96ef	fix(compose-doctor): resolve effective healthcheck coverage (#1713) ... * fix(compose-doctor): resolve effective healthcheck coverage Compose Doctor now classifies healthcheck coverage from the Compose model, running containers, and local images so image-provided HEALTHCHECKs are not false positives. Update Guard shares the same presence helper so test NONE is not treated as active. * fix(compose-doctor): fix healthcheck project label and empty compose HC Use the Compose project name for runtime container listing so stacks whose name: differs from the directory still get runtime evidence. Treat empty or timing-only healthcheck objects as absent rather than active. * fix(compose-doctor): treat inherited healthcheck as All Clear note Inherited image healthchecks no longer block All Clear; they surface under a notes section and cannot be acknowledged.	2026-07-28 14:26:00 -04:00
Anso	6688da97b1	fix(image-updates): match any local RepoDigest against the remote tag (#1695) ... * fix(image-updates): match any local RepoDigest against the remote tag Docker can list a stale multi-arch index digest ahead of the current one on the same image. Selecting only the first RepoDigest caused false same-tag rebuilds (for example redis:8.8.0) even when another digest equaled the registry primary. Compare every matching candidate and keep fail-closed behavior for empty, unknown-platform, and classification errors. Fixes #1684 * fix(image-updates): surface digest verification failures to operators Carry comparator errors into update-preview as check_error / verification_failed, prefer failed checks over sticky has_update in Fleet and the sidebar, and keep Update Guard from claiming no pending update when verification failed. * fix(e2e): align sidebar truncation spec with check-failed precedence StackRow now shows the check-failed icon over a stale update dot, but this spec still asserted the old precedence and failed deterministically in CI on every attempt. * fix(fleet): treat verification-only previews as non-actionable Fresh update-preview wins over sticky fleet booleans: disable Apply, exclude from ready counts, and move verification-only stacks into the check-failures advisory (including remote-labeled names). * fix(fleet): move preview actionability helpers out of the view Exporting non-components from AutoUpdateReadinessView tripped react-refresh lint in CI. Keep the helpers in a shared lib module and drop an unused mock arg. * fix(fleet): drop sticky cards when fresh preview clears the update A successful no-update preview now removes the pending Fleet card instead of leaving Apply enabled. Verification-only stacks still go to the advisory, and empty-state copy no longer claims all-clear while checks remain unresolved. * fix(image-updates): hold full-stack apply for review when another image fails verification A confirmed update or rebuild on one image previously left the whole stack fully actionable even when a different image in the same stack failed digest verification: Anatomy claimed "safe to apply", Update Guard reported ready, and Fleet's full-stack Apply stayed enabled, all while showing the verification-failure text right next to those claims. isActionableUpdatePreview now requires no verification failure anywhere in the stack; a new isReviewRequiredUpdatePreview flags the mixed state so Fleet still surfaces the card (not silently cleared) with Apply now disabled and a "Review · unverified" badge. Anatomy's banner says "review required" instead of a bump-based safety claim and withholds its Apply button. Update Guard's pending-update signal downgrades from ok to attention. Per-service apply (Fleet's per-image row) is deliberately left enabled since a service-scoped update to the confirmed image does not touch the unverified one. * fix(image-updates): treat rebuild_available symmetrically with has_update in Update Guard updatePreviewSignal only downgraded to 'attention' inside the has_update branch, so a rebuild-only stack (has_update false, rebuild_available true) with a sibling verification failure fell through to the plain verification-only 'unknown' branch and never mentioned the pending rebuild, inconsistent with isReviewRequiredUpdatePreview on the frontend which treats has_update and rebuild_available the same way. Also adds desktop-card coverage for the mixed state (previously only the mobile card was exercised) and locks in blocked/major-bump precedence over the new review-required badge/banner in both Fleet and Anatomy. * fix(image-updates): derive the mixed-verification review-hold from per-image detail, not the stack aggregate has_update and check_error are independent per image: a tag-based update can be confirmed via the registry's tag list even when that same image's own digest comparison against the current tag errored (already covered by an existing update-preview-service test). The stack-level verification_failed and has_update flags can therefore both be true for the SAME single image, which the previous review-hold treated identically to a genuinely different image failing verification: Update Guard said "another image failed digest verification" and Fleet told the user to "apply the confirmed service individually" on a single-service stack where no such affordance exists. isReviewRequiredUpdatePreview (and isActionableUpdatePreview) now walk the preview's images to require a pure failure image (check_error, no has_update of its own) alongside a genuinely different confirmed image or rebuild, falling back to the old aggregate-only judgment when per-image detail is unavailable. StackAnatomyPanel now imports the shared helper instead of hand-rolling the same predicate, so Fleet and Anatomy cannot drift apart. Backend updatePreviewSignal gets the same per-image treatment via a new optional images parameter, threaded through from UpdateGuardService. * fix(image-updates): fail closed on platform-unavailable indexes and legacy previews, allow anonymous tag listing Four independent gaps from the same QA pass, all in the digest/tag verification path this PR introduced or touches: - compareLocalToRemoteTag now distinguishes a remote index with no descriptor at all for the local platform (including an empty or fully-filtered index) from a genuine mismatch: the former returns an error instead of reporting a speculative update. A node cannot pull a platform the index does not offer. - selectLocalRepoDigests no longer falls back to a sole unrelated-repository RepoDigest when nothing matches the configured repo; comparing against a registry state that has nothing to do with the declared image risks a false update. Returns unresolved instead of guessing. - isClearedUpdatePreview no longer treats a preview with verification_failed missing entirely (not merely false) as proof the stack is clean. The current backend always includes this field, so its absence identifies an older remote node's response, which cannot vouch for a clean result the way an explicit false can. - listRegistryTags (and the underlying listRegistryTagsResult) no longer short-circuits to an empty list whenever no registry credentials are configured. getAuthToken already resolves anonymous tokens for public repositories; skipping it meant tag-based update detection silently never fired for any public image without a stored credential. * fix(image-updates): correct platform-check overreach, add cache and advisory for prior fixes Addresses code-review findings on the previous commit: - The platform-unavailable check fired too eagerly: an index whose runnable descriptors legally omit platform (OCI-permitted, routed to exactDigests) has real pullable content, so it must not be confused with a genuinely empty or fully-filtered index. Now only errors when both platform-labeled descriptors and exactDigests are empty. - listRegistryTags is now cached (15 min TTL): anonymous listing has no other rate limiting, and Fleet fans this out across every image on every reload. - A legacy preview (kept rather than cleared) now also pushes a check-failure advisory entry explaining why, instead of rendering as an unexplained pending card. - Corrected docstrings that described the old sole-unmatched-digest fallback and inverted how anonymous registry auth actually resolves. Adds coverage for: nested-index and attestation-only-filtered platform unavailability, a platform-less-but-populated index staying a match, the unrelated-repo digest rejection wired through the real preview-computation path (not just the registry-api unit), and legacy-preview interaction with an actionable has_update:true. * fix(image-updates): fail closed on mixed platform indexes, stop caching tag-list failures Addresses a second review round on the previous commit, including an empirically-verified regression: - The exactDigests fallback was unconditional: an index mixing a platform-labeled descriptor for a DIFFERENT platform with an unlabeled leaf let that leaf stand in as this platform's content, reporting a speculative update for a genuinely incompatible platform. Now an unlabeled leaf is only trusted when it is the ONLY kind of descriptor in the index (nothing else claims a different platform); a mixed index errors instead. - listRegistryTags was caching failed lookups for the full 15-minute TTL (a 429, an unreachable registry, or credentials not yet configured all looked identical to a real empty tag list). The fetcher now throws on failure so only a success is ever cached; CacheService's existing stale-on-error fallback still serves the last good list when one exists. - The manual "Recheck" action now also drops the tag-list cache, so a newly published tag is visible immediately instead of waiting out the TTL. - The legacy-preview advisory no longer fires when the same preview is already actionable on its own terms (a remote's own confirmed has_update/rebuild_available): pairing a "could not be checked" banner with an enabled Apply button next to it contradicted itself. - Corrected docstrings and a self-contradictory inline comment left over from the prior fix. New coverage: the exact mixed-index regression this round found and fixed, cache hit/no-repeat-fetch and failure-not-cached behavior, and the legacy-preview-plus-already-actionable non-contradiction. * fix(image-updates): add digest_error unmasked field, reorder RiskBadge, fix actionability gates Add digest_error to UpdatePreviewImage as an always-populated field that is independent of check_status masking: a confirmed tag-based update on the same image resolves check_status to 'ok' and nulls check_error, but digest_error stays set since the image's current tag content was never verified. Switch hasUnverifiedOtherImage to read digest_error. Reorder RiskBadge precedence so reviewRequired is checked before uncertain (both derive from check_error, so uncertain was unreachable). Fix self-contradiction in test fixtures (digest_update:true + check_error). Add masked-tag regression test with two-image fixture. Simplify hasUnverifiedOtherImage per code review feedback.	2026-07-25 21:57:30 -04:00
Anso	0daddfde00	fix: reconcile sticky update indicators with Anatomy preview (#1698) ... * fix: reconcile sticky update indicators with Anatomy preview Sidebar, Updates filter, and Fleet treated retained partial/failed scanner has_update as confirmed. Keep raw state for retention/notifications, project confirmed-only to APIs, show distinct incomplete indicators, and clear sticky rows only after an authoritative-negative preview. Closes #1685 * test: align sidebar truncate E2E with failed-over-retained precedence Purple update indicators are confirmed-only; hasUpdate with a failed check correctly shows the failed trailing icon. * fix: clear confirmed update rows on authoritative-negative preview Address audit SF-1/SF-2/SF-3: observation-watermark clears for older ok+has_update rows (DB + memory gens), Fleet checkability parity with backend not_checkable, and Updates chip confirmed-only regressions. * fix: tombstone equal-generation writers on preview clear Advance the per-stack write generation when clearing at the observation watermark so a scanner reserved before preview cannot recreate the row after an authoritative-negative reconcile. * fix: clear sticky updates with digest and tag preview parity Share detection across scanner and preview, keep GET read-only with POST reconcile, gate Apply to digest and rebuild updates, and invalidate the hub fleet cache on clear. * test: set digestUpdate on auto-update checkImage mocks Scheduler and execute routes now gate Compose on digest drift; fixtures that expect an apply need digestUpdate so they exercise the update path. * fix: clear unused lint errors on sticky update branch Drop unused partial helper and fleet invalidate import; keep the CacheService inflight self-ref as let with an eslint exception so tsc stays green. * fix: use inflight holder for CacheService prefer-const Keep generation-aware ownership without a let self-reference that fights ESLint and tsc.	2026-07-25 15:42:19 -04:00
Anso	3f1f15a6f4	fix: keep running containers until stack pull/build succeeds (#1657) ... * fix: keep running containers until stack pull/build succeeds Acquire images before reconcile, capture a recovery generation for compensation, and only remove classified orphans after handoff. * fix: address recovery audit blockers for safe stack updates Retire abandoned and expired recovery artifacts, probe compensated runtimes before reporting rollback success, preserve local Docker when deleting a node, validate the exact Compose invocation before capture, and repair updateStack return-contract fixtures. * fix: resolve ESLint errors blocking CI on this branch Unused-import and unused-variable errors left over from the stack deletion refactor: MeshService in stacks.ts (its opt-out cascade moved into DeployedStackDeletionService), a redundant pruneVolumes destructure in deleteDeployedStack (the real one is re-derived from the same input object inside runDeletionBody), and an unused beforeAll import in a Docker-integration test stub. Also scopes the webhook pull-action case body in a block to satisfy no-case-declarations; purely syntactic, no behavior change. * fix: harden recovery probe, cleanup retry, and failed-pull Docker test Reject absent or unhealthy expected replicas before reporting rollback success, keep cleanup records until artifacts are actually removed, fail closed when a mesh override cannot be generated, and assert a real failed pull leaves the original container running. * fix: verify recovery probe image identity and stack-scoped override paths Reject recovered runtimes that use the wrong image or leave scale-zero services running, and confine tombstone override deletion to the intent stack directory so forged cross-stack paths cannot be swept. * test: batch notification cap fixtures in a SQLite transaction Unbatched 1200-row inserts were timing out at the default 30s under CI load even though the same assertions pass in under 2s when green.	2026-07-21 12:18:01 -04:00
Anso	63213c0960	feat: add service-scoped Compose update and restore (#1648) ... * feat: add service-scoped Compose update and restore Allow updating or rebuilding one declared Compose service on multi-service stacks without recreating siblings, with recovery snapshots, health-gate observation, and prune holds for rollback images. Full-stack update paths and single-service UX stay unchanged. * fix: sanitize service-scoped update log messages for CodeQL * fix: address service-scoped update audit findings B-01 through B-07 * fix: complete service-scoped update audit metadata and surfaces * test: wrap Updates readiness tests for deploy-feedback context * fix: keep service recovery reachable without Deploy Progress Make failed service-gate recovery discoverable when Deploy Progress is disabled or dismissed, suppress stale image-scan notification side effects, normalize ComposeService line endings, and add focused regression coverage. * fix: resurface ContainersHealth density and expand on multi-service stacks Service grouping hid the summary strip and Compact/Detailed/Expand controls that still applied to multi-container stacks.	2026-07-19 02:42:29 -04:00
Anso	f2b5c68d84	feat: add build-aware compose stack updates (#1561) ... Detect services with build: in the update preview and run compose build --pull plus pull --ignore-buildable when Update is triggered on those stacks, while keeping the existing pull-only path for image-only stacks.	2026-07-05 04:45:45 -04:00
Anso	4077546492	feat: acknowledge Compose Doctor preflight findings per stack (#1560) ... * feat: acknowledge Compose Doctor preflight findings per stack Add node-scoped preflight acknowledgements with read-time filtering. Supports four expiry modes and activeStatus for banner, tab dot, and readiness. * fix: align preflight acknowledge UI with design system Use Combobox, modal chrome, mono fields, and non-destructive clear confirm. * fix: update test mocks to match new preflight field names The preflight-acknowledgements feature renamed status-\>activeStatus and highestSeverity-\>activeHighestSeverity in the preflight report shape. The corresponding test mocks in three files still used the old field names, causing 6 test failures across backend and frontend. - backend: update-guard-service mock now passes activeStatus - frontend PreflightPanel: Report interface and report() helper now include activeStatus, activeHighestSeverity, activeCount, acknowledgedCount - frontend StackAnatomyPanel doctor: mock API response now includes activeHighestSeverity and activeStatus	2026-07-05 04:12:03 -04:00
Anso	5f1baa7522	fix: harden deploy/update concurrency and node-targeting safety (#1390) ... * fix: harden deploy/update concurrency and node-targeting safety Release stabilization for deploy/update operational safety. Per-stack operation locking is now global. Background lifecycle paths (scheduler auto stop/down/start/backup/update, webhook execute, Git source auto-deploy, image auto-update, label bulk actions, fleet snapshot redeploy, and mesh redeploy) acquire the per-node, per-stack lock through a new StackOpLockService.runExclusive helper and skip rather than race a manual deploy/update/rollback/backup on the same stack and node. Skips surface honestly (a failed scheduled run, a recorded webhook failure, a per-stack batch result, or a thrown error) instead of a silent no-op. Update readiness and policy-bypass now run against the node captured when the dialog opened, not the live active node, so switching nodes while a dialog is open cannot retarget the update or the bypass retry. Rollback readiness no longer presents a moving-tag or unpinned image as a ready image revert. Restoring files does not revert a moving tag, so those stacks read as partial, and the rollback success message states that the compose and env files were restored. * fix: lock blueprint reconcile against manual ops and correct rollback wording Follow-up to the deploy/update safety hardening, closing two more gaps from a verification pass. BlueprintService.deployLocal and withdrawLocal called ComposeService directly, so blueprint reconciliation could race a manual deploy/update/rollback/backup on an owned stack. Both now run their compose lifecycle call through StackOpLockService.runExclusive and skip (recorded as a failed reconcile, retried on the next cycle) on conflict. The withdraw holds the lock across both the compose down and the directory delete so neither races a manual operation. The runtime rollback messages overstated recovery: a rollback restores the compose and env files and recreates containers, but does not revert an image behind a moving tag. The auto-rollback deploy-progress output, the recovery panel and chip, the failure toasts, and the manual rollback route message now state that the compose and env files were restored, with the matching OpenAPI example and atomic-deployments doc updated. * fix: acquire stack lock before blueprint deploy mutates compose and marker files Local blueprint deploy wrote the compose and marker files and ran the policy assert before acquiring the per-stack lock; the lock only wrapped the deploy itself. A reconcile could therefore rewrite an owned stack's files while a manual deploy/update/rollback/backup was running. The lock now wraps the whole critical section (create, write compose, write marker, policy assert, deploy), so on conflict nothing is written and the reconcile records a failed outcome. Adds a test asserting a deploy under a held lock records failed, writes no marker file, and leaves the manual lock untouched. * fix: make remote blueprint apply atomic under the receiving node's stack lock Remote blueprint deploy wrote the compose and marker files to the target node via separate HTTP calls and only locked on the final deploy, so the file writes could race a manual operation on that node. A node's operation lock is process-local and cannot be held by the hub across HTTP calls, so the locked create/write/deploy now runs on the receiving node. The locked critical section is extracted into BlueprintService.applyLocalUnderLock and exposed via POST /api/blueprints/apply-local. The hub posts the blueprint to that endpoint in one call; the receiving node runs create + write compose+marker + deploy under its own per-stack lock. Older nodes without the route answer 404 and fall back to the legacy multi-call flow. The endpoint is gated by paid tier and the same per-stack stack:edit and stack:deploy permissions as the PUT-compose + deploy it bundles, validates the stack name, compose size, and marker structure, and returns 409 on a lock conflict without writing anything. Adds tests for the atomic single-call path, the 404 legacy fallback, the 409 lock-conflict mapping, the route validation and permission paths, and the write-compose-then-marker-then-deploy ordering of the shared locked apply. * fix(deps): bump undici to 7.28.0 to clear high-severity advisory The frontend CI npm audit gate (--audit-level=high) failed on a transitive undici 7.25.0 (a dev-only dependency via jsdom): TLS certificate validation bypass (GHSA-vmh5-mc38-953g) and cross-user cache information disclosure (GHSA-pr7r-676h-xcf6). Bumping undici within jsdom's existing ^7.25.0 range to 7.28.0 clears the high-severity advisory and unblocks the frontend job. Lockfile only; no direct dependency or source change.	2026-06-18 13:38:19 -04:00
Anso	38aabe7064	feat: health-gated updates and rollback readiness (#1354) ... * feat: classify stack deploy and update failures with suggested next actions Failed deploy and update responses now carry a failure classification (cause category, headline, and suggested next step) derived from the compose error output. The recovery panel and chip render the classification and include it in copied diagnostics, and gateway-style failures surface as a node-unreachable cause. * feat: add update and rollback readiness reports for stacks Before a manual update, Sencho now shows an advisory readiness verdict computed from the stored preflight result, open drift findings, live container health, the pending image change, the rollback backup slot, and node disk headroom. The Stack Dossier gains a rollback readiness section that states what a rollback can restore and explicitly discloses that volume and bind-mounted data are not covered. Toolbar and sidebar updates now share one update path, and admins can create a fleet snapshot from the readiness dialog before updating. Nodes that do not advertise the capability keep the direct update flow. * feat: observe stack health after updates with a post-deploy health gate After a deploy or update succeeds, Sencho now watches the stack for a configurable observation window and records a passed, failed, or unknown verdict: containers must stay running, healthchecks must report healthy, and restart loops or disappearing containers fail the gate. The deploy panel shows the observation live and holds off auto-closing until the verdict lands, a failed gate surfaces the existing recovery actions including rollback, and the stack timeline records update started and gate verdict events. Scheduled, webhook, bulk, and git-source updates are gated the same way; rollbacks and installs are deliberately not. The gate is observational only and can be tuned or disabled per node under host alert settings. * docs: document health-gated updates and rollback readiness New operator page covering the update readiness dialog, the post-update health gate and its settings, the rollback readiness disclosure, and classified failures, with cross-links from the atomic deployments and deploy progress pages. The API reference gains the readiness and health-gate endpoints, the healthGateId success field, and the failure classification schema on deploy and update error responses. * feat: withhold the success verdict while the health gate observes An update used to show a green Succeeded that a failed health gate then contradicted moments later. The deploy modal now reports Verifying health while the gate observes, shows success only when the gate passes, and makes a failed or unknown gate the headline result; success toasts soften to a verifying message while a gate runs. The mobile recovery card groups its actions behind one bottom-right Take action menu so it stays compact on a phone, with the classified cause still visible on the card. A successful image update now also counts as the last known-good marker in rollback readiness, and the docs gain screenshots of the readiness dialog, gate states, dossier section, and settings. * fix: harden log format strings and the env existence path check Log calls that interpolated the stack name into the console format string now use constant format strings with placeholder arguments, and envExists validates path containment inline at its filesystem access, matching the established patterns used elsewhere in the same files. * test: adapt deploy modal success specs to the post-deploy health gate The deploy feedback modal now withholds its success verdict while the health gate observes the new containers, showing "Verifying health" until the gate passes. The two success-path E2E tests waited for "Succeeded" within the gate's 90s default window and timed out. Shorten the observation window to the 15s minimum for these tests via the settings API, assert the verify-then-succeed sequence the modal actually renders, and restore the default window afterward so the test value does not leak into later runs. * fix: serialize health gate polling and harden gate observation Address race conditions in the post-update health gate found in review. Backend: the gate poller used setInterval, so a Docker observe slower than the 5s tick could overlap the next poll and corrupt the restart and missing-container accounting, and a wedged socket could leave a poll pending forever. Polling is now single-flight: each cycle self-schedules the next only after it settles, and the observe is bounded by an 8s timeout so a hung probe counts as a poll error and resolves the gate unknown after three in a row. Frontend: the gate poller could overlap requests, letting a slow earlier "observing" response overwrite an already-applied terminal verdict. It is now single-flight with a terminal latch, so a late response can never roll the UI back from passed or failed. Also reject a non-digit nodeId on the snapshot coverage route instead of letting parseInt coerce it, document that turning off the deploy progress panel opts out of the live gate UI while the gate still runs server-side, and add gate-coverage tests for the webhook, git source, and auto-update apply paths plus the new single-flight, observe-timeout, and recovery cases.	2026-06-11 00:26:26 -04:00