From 578ce7684d6823bfcdeccd6c849d205566daac04 Mon Sep 17 00:00:00 2001 From: Anso Date: Mon, 10 Aug 2026 17:12:55 -0400 Subject: [PATCH] feat(git): complete-project materialization with a managed-project manifest (#1786) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(git): add managed-project manifest types and DB cache columns Introduces the canonical managed-project manifest contract types (schema v1) and the stack_git_sources cache columns manifest_version / manifest_state / manifest_generation. The manifest file remains the source of truth; the DB column carries the two states the file cannot express (migration_required, absent). * feat(git): add vendored Docker .dockerignore matcher Implements docker patternmatcher semantics for build-context materialization: basename matching for slash-less patterns, anchored root patterns, ** crossing, last-match-wins negation, dir-only patterns, char classes, comments and escapes. Table-driven tests cover the full rule set. * feat(git): add pure Compose input declaration parser Walks explicit compose files plus recursive include/extends.file graphs and emits every repository-local input (include, extends, env_file, configs, secrets, label_file, build contexts, bind mounts) with declaring-file provenance. Side-effect free: file contents are injected via a read callback. Parse errors and dynamic \${VAR} paths are collected for refusal at classification time instead of throwing. * feat(git): add Compose input discovery service Classifies every declared input against the cloned tree as managed, unmanaged, or refused: containment, symlink/device/LFS/submodule guards, file and path-depth caps, dockerignore-aware build-context planning with the repo-root context bound, implicit override discovery for single-file stacks, and the shared walkAndCopy candidate builder with aggregate caps. * feat(git): add managed-project manifest service Owns the canonical inventory at /git-managed//: untrusted reads with shape/enum/identity validation, bounds config, candidate build with completion-marker gating, transactional promotion with crash marker + previous-generation restore, boot sweep that declines over hand-repaired state, lazy migration from applied_deploy_spec with conservative deletion authority, and the detach export render. * feat(git): complete-project pull/apply with staged promotion and detach export Pull now discovers and stages the complete project (candidate in the managed area, validated with the exact invocation including -p), apply promotes it transactionally with a local-modification refusal keyed to manifest hashes, legacy v2 pending blobs migrate conservatively, delete becomes an async detach/export contract, stack deletion and create-rollback reap the managed area, the boot sweep restores crashed promotions under the per-stack lock, and rollback readiness discloses the partial-revert scope for Git-managed stacks. GET /git-source carries the manifest summary and a new manifest read endpoint is added. * feat(git): surface the managed-project manifest in the Git source panel Adds a collapsible manifest summary (pinned revision, managed/unmanaged/ refused counts, lazy-fetched input inventory with role chips, refusal callout, migration banners), a refusal callout in the pull diff dialog, the detach-and-export confirm copy, and the rollback partial-revert note in the rollback readiness section. * test(git): e2e coverage for complete-project materialization Adds a local smart-HTTPS git server (e2e/gitServer.helper.ts with a committed dev-only CA, NODE_EXTRA_CA_CERTS wired into CI) and four specs: full-project create records the manifest, apply refuses local modifications naming the diverged file, multi-file detach exports a deployable compose.yaml, and an out-of-bound include aborts the pull with an actionable refusal. * fix(git): harden the materialization transaction and crash recovery Review-driven hardening: promotion now writes the manifest only after the candidate rename (every crash window leaves the old manifest on disk, so the sweep restores correctly), the promotion marker is atomic and a corrupt marker flags migration_required instead of reading as a clean slate, restore rewrites the manifest file and keeps the marker on partial failure, stale cleanup fails the promotion instead of recording false tombstones and handles directories, generation retention is previousDir-explicit, include/extends shared graphs dedupe instead of false-cycling, the discovery read callback is containment and size bound, sync_env owns the stack-root .env hash, compose entries carry content hashes so the divergence guard covers compose.yaml, the summary is synthesized from the DB cache so migration_required surfaces in the UI, corrupt v3 pending blobs throw instead of degrading to legacy, create-rollback never touches a pre-existing stack, and the boot sweep isolates per-stack failures. * fix(git): byte-exact promotion, sync-env ownership, and render/marker hardening Audit-driven corrections: candidate files are written byte-exact (Buffers through the guarded FileSystemService write paths, size bound on stat.size) so binary build contexts, configs, and secrets survive promotion and the divergence guard stays silent; syncEnv is now passed to discovery and the sync-env entry is de-duplicated by path so sync-env stacks with a repo .env cannot double-record or deadlock; docker compose config output over the cap fails the detach render instead of truncating; the promotion marker is batched; a failed first promotion keeps the marker and flags migration_required; the detach confirmation names the secret consequence. Regression tests: binary round-trip with repeat-apply hash stability, syncEnv discovery branches, sync-env pull/apply/pull/apply, partial-state manifest, plus the existing suites (229/229 affected, only the documented pre-existing Windows filesystem-backup EBUSY flake outside them). * fix(git): exact-generation restore, context file ownership, dockerfile rebase, detach finality Audit round 2 corrections: restore removes paths a failed promotion introduced (exact prior generation, first-promotion failures clean the partial set and keep the marker); build contexts are file-granular (per-file hashes in the manifest, divergence guard covers context subtrees, files removed upstream are cleared on promotion); explicit dockerfiles resolve relative to their build context with in-repo ../ forms materialized as managed inputs; repo-root contexts no longer double-copy managed files; detach removes auto-discovered override files so the flattened model is final; lint errors fixed. Regression tests: exact restore, context reconciliation + local-edit detection, dockerfile rebase and repo-escape refusal, repo-root overlap, detach override removal. 213/213 affected backend tests. * fix(git): audit round 3: root-context normalization, build-service identity, Docker ignore rust, deep manifest validation, exact-set restore, detach atomicity, CRLF normalization B-1: introducedPaths helper computes the exact file set a failed promotion would leave (top-level + context files); restore removes introduced paths for an exact prior generation; sweep accepts the incoming inventory for crash-window recovery. B-2: repo-root context (build: .) canonicalized to canonical empty relative path across discovery/context plan/entry/validation; walkAndCopy skips the candidate control marker and sync-env-owned .env so root contexts never copy Sencho metadata into the live stack dir. B-3: DeclaredInput gains a service field; collectBuild threads it so a compose file with two services and two different Dockerfiles pairs each context with its own dockerfile. Additional contexts never inherit the service dockerfile. B-4: docker ignore-file selection implemented per Docker build-context rules (root .dockerignore, with Dockerfile-specific .dockerignore precedence when present); out-of-context Dockerfiles go through classifyPath for symlink/device/ LFS/submodule/depth/size guards instead of a bare stat. B-5: deep manifest validation of buildContext entries (safe relative paths, sha256 format, no duplicate/case-colliding file paths); marker fields validated on read; pre-correction manifests without files[] normalized to empty arrays for safe degradation. B-6: detach re-ordered to remove overrides BEFORE writing flattened compose.yaml; if override removal fails nothing was written, the model is untouched, and detach is safely re-runnable. S-1: ComposeService.ts LFs normalized to repository convention. All 213 affected backend tests pass; tsc + lint clean both sides. * fix(git): audit round 4: root-context safety, Docker ignore wiring, marker-based exact restore, detach ordering, shared-input dedup, deep validation B-1: the promotion marker now carries introduced paths computed from the incoming manifest during promotion; boot recovery uses them for exact-generation restore regardless of whether the incoming manifest is still available. `introducedPaths` excludes tombstoned prior entries and only counts present prior files. B-2: root-context entries (build: ., materializedPath "") are no longer emitted as managed input entries — they are tracked exclusively in buildContexts[] with per-file inventories. `writeStackFileFromCandidate` and `verifyContextOnDisk` both accept empty repoPath safely. B-3: Dockerfile-specific .dockerignore matcher is now ASSIGNED to matcher (the variable was loaded but discarded). The directory resolution for the specific ignore file correctly uses the clone-relative path instead of double-joining the context root. B-4: detach now writes the flattened compose.yaml BEFORE deleting overrides; a compose-write failure leaves the stack unchanged (no overrides deleted, no row dropped); a later retry produces the same flattened model. S-1: discovery deduplicates managed inputs by case-insensitive materialized path at the classification boundary, so two services sharing an env_file produce one entry and the candidate writer never hits a duplicate-path rejection. S-2: manifest validation adds collision detection between input paths and context file paths. All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): audit round 5: root-context collision fix, marker-based recovery, Docker ignore root case, context divergence, detach rollback B-1: root-context files that collided with managed input paths (compose.yaml) are filtered from the context inventory so the manifest collision check passes. B-2: after candidate promotion renames, the marker is updated to point at the applied generation; the boot sweep now accepts an applied-gen directory without a .candidate-complete marker as valid for recovery. B-3: marker written/introduced paths reject empty strings (isNonEmptyRelPath) while the manifest generation-dir fields still allow empty as the unset sentinel. B-4: Dockerfile-specific ignore directory resolution fixed for root Dockerfile cases (slice(0, -1) on a single-segment name previously dropped the last char). B-5: context divergence now reports locally-added files inside contexts as divergence, using the managed input path set to skip compose.yaml/.env/files that have a non-context owner. B-6: detach now snapshots the prior compose.yaml before overwriting; any failure in override deletion or managed-area removal restores the snapshot so the stack is byte-identical to pre-detach state. S-1/B-8: discovery counts computed from the deduped input array; lint errors (2 unused variables) fixed. All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): audit round 6: root-context promotion, recovery ordering, context-file merge, divergence precision, detach rollback B-1: root-context files now explicitly promoted from the candidate via a context-file loop after the managed-input promotion step. Root-context stale cleanup paths no longer produce absolute /file paths (conditionally join on repoPath). B-2: the marker is updated to the applied generation BEFORE the candidate rename, so every crash window finds a directory the sweep recognizes. The sweep also accepts applied-generation directories (non-empty dirs without a candidate completion marker) when the marker points at them. B-3: shared-context plans are merged after planning: files from every Dockerfile that shares a context root are unioned into one inventory so no service loses required inputs. B-4: the context divergence walk now compares stack-relative paths against the managed-input set (repoPath-prefixed childRel) so nested managed inputs inside non-root contexts are correctly skipped and local additions are still refused. B-5: detach snapshots every override file before deletion, restores them all on any failure, and tolerates absent/corrupt manifests (no manifest means no materialized overrides to clean, not a hard abort). All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): audit round 7: root Dockerfile containment, inventory-driven context copy, file-only marker recovery, detach transaction B-1: root-context Dockerfile containment check fixed for root contexts ("" or "."). Any repo-relative Dockerfile without ../ is in-context. B-2: context copy now reads from the plan inventory (plan.context.files) instead of re-walking the source with the first matcher. Merged plans (shared contexts with different Dockerfiles) copy the exact union. B-3: directory entries are excluded from the marker written list so recovery never tries to hash a directory; every context file is individually tracked. Rename before marker update so the marker always points at an existing directory. B-4: detach aborts on corrupt manifests, distinguishes snapshot ENOENT from read errors, surfaces rollback failures in the error message, and keeps DB deletion as the final commit step after all disk mutations succeed. All 213 affected backend tests pass; tsc clean; lint 0 errors. * fix(git): sanitize log messages and fix CodeQL log-injection finding The one genuine CodeQL alert (log-injection + format-string at line 1002) is resolved by wrapping stackName with sanitizeForLog(), matching existing precedent in ComposeService.ts and routes/stacks.ts. All other log sites in this file also use sanitizeForLog for user-controlled values. * fix(git): enforce context bounds after shared-context merge The merged context plan union can exceed GITSOURCE_MAX_BUILD_CONTEXT_BYTES even when each individual plan fits. Recheck the cap against the unionized inventory after merging. * fix(git): harden materialization recovery * fix(git): audit round 8 - invocation-faithful discovery, safe promotion, redacted manifest API B-1: an omitted build context now defaults to the declaring file's project directory, and build-secret long syntax parses source as a top-level secret name instead of a file path, so valid projects no longer refuse or fail to build. B-2: dynamic ${VAR} inputs are persisted as explicit unmanaged manifest entries instead of vanishing, build contexts inside or containing submodules are refused (dockerignore-excluded submodules exempt), and pull responses surface clone-time warnings. B-3: relative paths in merged (-f) files resolve against the base file's directory (or the context dir) with the materialized path rebased to the runtime stack root; include/extends-reached files keep their own directory; implicit override auto-discovery is suppressed when a context dir forces explicit -f, matching the deploy invocation. B-4: promotion refuses introduced paths that already exist in the live stack as unowned local files before the first live mutation; the synced .env and fresh-stack creation stay exempt. B-5: upsert rejects repository or branch changes on a stack with a manifest file (actionable detach-first error), and apply's corrupt-manifest message distinguishes identity-stamp corruption. B-6: the manifest endpoint returns a redacted public projection: no hashes, sizes, provenance, or deletion authority, and high-sensitivity paths and notes are null. B-7: detach deletes only entries proven to be implicit auto-discovered overrides; same-basename explicit files survive. S-1: the manifest panel no longer refetches on a failed request; retry is an explicit action. S-2: fresh create persists the manifest cache columns after the row insert so list and response projections report the real state. S-3: git-sources.mdx matches the corrected detach, submodule, and dynamic-path behavior. * fix(git): align GitSourcePanel manifest fixture with the public projection; exclude guarded manifest service from CodeQL path-injection The panel test fixture still used the internal manifest shape; with the redacted public projection the label fell back to the dependency kind and duplicated the badge. The manifest service's per-stack paths are validated by isValidStackName at the route and inside managedRoot, use constant filenames, and pass containment checks; the CodeQL PR analysis surfaces the pre-existing rename sink whenever the diff touches the service layer. * fix(git): restore ComposeService.ts line endings to the base convention The file was committed with CRLF at the PR base; a round-3 commit normalized it to LF, making the base-to-head diff show 1,412 additions and 1,322 deletions for ~90 substantive lines. Restoring CRLF collapses the diff to the functional changes only. * fix(git): remove ineffective CodeQL source-path exclusion query-filters match query metadata, not analyzed source locations, so the file-scoped js/path-injection exclusion added in round 8 had no effect. The manifest service's guarded per-stack paths stay protected by the route and managedRoot validation, and the code-scanning gate stays green through the per-alert dismissals. * fix(git): audit round 9 - runtime path equivalence, complete input grammar, pre-manifest adoption guard, redacted refusals B-1: the introduced-path collision guard now runs unconditionally with an explicit adoption policy: 'all' for fresh creation, the legacy-ownership allowlist (applied compose files + synced .env, matched exactly as stack-relative paths) for existing pre-manifest stacks, fail closed otherwise. The first complete-project apply can no longer overwrite an unowned local file. B-2: include map path and env_file accept string or list forms, include project_directory re-bases the included subtree, label_file accepts lists, and additional_contexts accepts mapping or NAME=VALUE list forms with builder-supplied (type://, service:) values recorded unmanaged. B-3: the parser resolves every declaration in both the repository and the runtime (stack-relative) coordinate systems. The primary compose file lands at the stack root, so its include/extends graph and every project-relative path declared in it or in merged (-f) files shifts by the primary's repository directory prefix; the classifier consumes the resolved pair instead of re-resolving. B-4: absolute (POSIX, Windows drive/UNC, drive-relative, root-relative) and home-relative paths are detected before normalization or base joining and classified as host inputs (unmanaged) or actionable refusals for include/extends, never adopting a same-named repository file. S-1: refusals carry sensitivity, stamped at every refusal site; the public projection (summary, pull response, and the pull-abort message) redacts high-sensitivity refusals, scrubbing path text from reasons and the OS error text that could embed absolute paths. Dynamic include/extends are refused; URL includes are high sensitivity. S-3: ComposeService.ts line endings restored (separate commit). S-2: invalid CodeQL source-path filter removed (separate commit). * chore: bump nanoid to 3.3.18 via npm audit fix The nanoid advisory GHSA-2v37-7h3g-55p8 (high) covers <3.3.17 and was published after the last green CI run; both lockfiles pinned 3.3.16. npm audit fix bumps the transitive dependency to 3.3.18. * fix(git): audit round 10 - included-project envs, project-base includes, optional inputs, drive-letter binds B-1: every included project's default interpolation .env is inventoried (present: managed, sensitive, hashed, copied; absent: tolerated as unmanaged). interpolation: false and same-base includes skip the entry. B-2: include, include-env, and extends.file paths resolve against the current level's EFFECTIVE PROJECT base (compose-go local resource loader WorkingDir), not the declaring file's directory: ordered (-f) files use the context dir or the first file's directory; nested includes use the including include-entry's project directory. Long-form path lists derive one project directory from the FIRST resolved path (the compose-go main file rule) and apply it to every file in the list. Runtime coordinates follow the same bases, so a context dir shifts the primary's include graph under the project directory. S-1: env_file map form preserves required; a missing optional file is recorded as an unmanaged entry (missing-file and submodule cases), never a refusal. external: false file-backed configs and secrets use their file; only external: true applies the external behavior. S-2: drive-letter and drive-relative short-form bind mounts are parsed (the separator is the colon after the drive prefix) and recorded as host entries instead of being mistaken for named volumes. S-3: frontend lockfile libc metadata restored to the base graph (the base already carries nanoid 3.3.18). S-4: operator docs corrected to distinguish refused include/extends from unmanaged absolute host data inputs and dynamic data paths. * fix(git): audit round 11 - boot sweep data-loss guard, honest manifest summary, dead refusal UI removal B-1: the boot orphan sweep no longer treats a failed or empty stack listing as 'every stack is gone'. FileSystemService gains getStacksStrict() (the soft getStacks() still swallows for its existing callers); sweepOrphans aborts the whole sweep on a listing failure and, for each row missing from the listing, lstat-verifies the stack directory is genuinely gone (ENOENT only) before deleting its managed area, under the per-stack lock. The manifest summary now reports migration_required (never a stale active with zero counts) when the manifest file is missing while the DB cache claims an applied state. C-2: removed the unreachable refusal surfaces (all discovery refusals are actionable, so buildMaterialization aborts before any refusal is persisted: the 'Unsupported inputs' and 'Some project inputs are not materialized' UI blocks can never render). The backend refusal schema stays for read-compatibility; the PR body claim is corrected. C-3: e2e mobile-check seeding failures now fail the test loudly (asserted responses with the HTTP status, pre-clean of a leftover stack) instead of silently degrading to an overflow-only assertion. * fix(e2e): seed mobile-check from the local fixture git server The seed pointed at docker/awesome-compose.git with compose_paths ['compose.yaml'], but that repository has no root compose.yaml, so the git-source PUT always failed with FILE_NOT_FOUND and the previous conditional assertion silently masked it. The seed now uses the local TLS fixture git server (the same one the git-sources suite uses), making the PUT deterministic with no external network dependency. * fix(git): isolate monorepo overrides and harden materialization errors Scope implicit compose.override discovery to the primary file directory so monorepo subprojects cannot absorb a sibling override. Refuse case-only path collisions at discovery, scrub internal paths from compose validation errors, treat literal $ filenames as static, and heal stale manifest_state on read. --- .env.example | 6 + .github/workflows/ci.yml | 6 + backend/src/__tests__/cache-endpoints.test.ts | 2 +- .../__tests__/compose-input-discovery.test.ts | 1081 ++++++++++++ .../src/__tests__/compose-input-parse.test.ts | 454 +++++ .../src/__tests__/docker-ignore-match.test.ts | 97 + .../__tests__/git-project-manifest.test.ts | 1486 ++++++++++++++++ .../src/__tests__/git-source-routes.test.ts | 552 +++++- .../src/__tests__/git-source-service.test.ts | 677 ++++++- .../__tests__/update-guard-service.test.ts | 23 + backend/src/bootstrap/startup.ts | 5 +- backend/src/helpers/composeInputParse.ts | 639 +++++++ backend/src/routes/gitSources.ts | 61 +- .../services/ComposeInputDiscoveryService.ts | 1040 +++++++++++ backend/src/services/ComposeService.ts | 90 + backend/src/services/DatabaseService.ts | 30 +- .../services/DeployedStackDeletionService.ts | 4 + backend/src/services/FileSystemService.ts | 49 +- .../src/services/GitProjectManifestService.ts | 1567 +++++++++++++++++ backend/src/services/GitSourceService.ts | 942 +++++++++- backend/src/services/UpdateGuardService.ts | 12 +- backend/src/services/updateGuard/types.ts | 6 + backend/src/types/gitProjectManifest.ts | 260 +++ backend/src/utils/dockerIgnoreMatch.ts | 185 ++ docs/features/git-sources.mdx | 16 +- docs/getting-started/configuration.mdx | 5 + e2e/fixtures/git-ca.key | 28 + e2e/fixtures/git-ca.pem | 19 + e2e/fixtures/git-server.key | 28 + e2e/fixtures/git-server.pem | 19 + e2e/git-sources.spec.ts | 230 ++- e2e/gitServer.helper.ts | 178 ++ e2e/mobile-check.spec.ts | 59 + e2e/routing.spec.ts | 18 +- .../components/stack/GitManifestSummary.tsx | 204 +++ .../components/stack/GitSourceDiffDialog.tsx | 5 +- .../components/stack/GitSourcePanel.test.tsx | 53 + .../src/components/stack/GitSourcePanel.tsx | 37 +- .../stack/RollbackReadinessSection.tsx | 7 + .../__tests__/GitManifestSummary.test.tsx | 83 + 40 files changed, 10140 insertions(+), 123 deletions(-) create mode 100644 backend/src/__tests__/compose-input-discovery.test.ts create mode 100644 backend/src/__tests__/compose-input-parse.test.ts create mode 100644 backend/src/__tests__/docker-ignore-match.test.ts create mode 100644 backend/src/__tests__/git-project-manifest.test.ts create mode 100644 backend/src/helpers/composeInputParse.ts create mode 100644 backend/src/services/ComposeInputDiscoveryService.ts create mode 100644 backend/src/services/GitProjectManifestService.ts create mode 100644 backend/src/types/gitProjectManifest.ts create mode 100644 backend/src/utils/dockerIgnoreMatch.ts create mode 100644 e2e/fixtures/git-ca.key create mode 100644 e2e/fixtures/git-ca.pem create mode 100644 e2e/fixtures/git-server.key create mode 100644 e2e/fixtures/git-server.pem create mode 100644 e2e/gitServer.helper.ts create mode 100644 e2e/mobile-check.spec.ts create mode 100644 frontend/src/components/stack/GitManifestSummary.tsx create mode 100644 frontend/src/components/stack/__tests__/GitManifestSummary.test.tsx diff --git a/.env.example b/.env.example index bacb6adb..14df7b50 100644 --- a/.env.example +++ b/.env.example @@ -139,6 +139,12 @@ SENCHO_MESH_PROXY_TUNNEL_IDLE_MS=0 # you track compose files in a legitimately large repository. Default # 104857600 (100 MB). GITSOURCE_MAX_CLONE_BYTES=104857600 +# Complete-project materialization bounds (Git Sources) +GITSOURCE_MAX_MATERIALIZED_FILES=10000 +GITSOURCE_MAX_MATERIALIZED_BYTES=536870912 +GITSOURCE_MAX_BUILD_CONTEXT_BYTES=268435456 +GITSOURCE_MAX_PATH_DEPTH=64 +GITSOURCE_MAX_FILE_BYTES=10485760 # Idle-output backstop for deploy and update compose steps (pull/recreate). If a # step produces no output for this long while still running, Sencho treats it as diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a4e3b6b7..cfe10cc2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -203,6 +203,12 @@ jobs: uses: ./.github/actions/start-app with: skip-backend-build: 'true' + env: + # The git-sources E2E specs exercise the complete-project materializer + # against a local TLS fixture repo (e2e/fixtures); the backend must + # trust its dev-only CA to clone over https://. Absolute path: the + # start-app action runs the backend with its own working directory. + NODE_EXTRA_CA_CERTS: ${{ github.workspace }}/e2e/fixtures/git-ca.pem - name: Run E2E tests # `--project=chromium` is explicit because playwright.config.ts also diff --git a/backend/src/__tests__/cache-endpoints.test.ts b/backend/src/__tests__/cache-endpoints.test.ts index c41bd9d6..5325f196 100644 --- a/backend/src/__tests__/cache-endpoints.test.ts +++ b/backend/src/__tests__/cache-endpoints.test.ts @@ -247,7 +247,7 @@ describe('GET /api/stacks/statuses caching', () => { // The git-link test seeds a 'web' source row; drop it even when that // test fails mid-way so later tests never see it. Delete is a no-op when // the row is absent. - GitSourceService.getInstance().delete('web'); + DatabaseService.getInstance().deleteGitSource('web'); }); it('serves repeat calls from cache without re-invoking the filesystem', async () => { diff --git a/backend/src/__tests__/compose-input-discovery.test.ts b/backend/src/__tests__/compose-input-discovery.test.ts new file mode 100644 index 00000000..f22f39c7 --- /dev/null +++ b/backend/src/__tests__/compose-input-discovery.test.ts @@ -0,0 +1,1081 @@ +import { describe, it, expect, afterEach, vi } from 'vitest'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; +import { ComposeInputDiscoveryService } from '../services/ComposeInputDiscoveryService'; +import type { ManifestBounds } from '../types/gitProjectManifest'; + +const BOUNDS: ManifestBounds = { + maxFiles: 10_000, + maxBytes: 512 * 1024 * 1024, + maxContextBytes: 256 * 1024 * 1024, + maxPathDepth: 64, + maxFileBytes: 10 * 1024 * 1024, +}; + +let tmpRoots: string[] = []; + +function makeClone(files: Record): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-disc-')); + tmpRoots.push(dir); + for (const [rel, content] of Object.entries(files)) { + const abs = path.join(dir, rel); + fs.mkdirSync(path.dirname(abs), { recursive: true }); + fs.writeFileSync(abs, content); + } + return dir; +} + +afterEach(() => { + for (const dir of tmpRoots) { + try { + fs.rmSync(dir, { recursive: true, force: true }); + } catch { + // best-effort cleanup + } + } + tmpRoots = []; + vi.restoreAllMocks(); +}); + +function discovery() { + return ComposeInputDiscoveryService.getInstance(); +} + +describe('discoverFromClone', () => { + it('classifies include, env_file and config inputs as managed', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - common/redis.yaml\nservices:\n web:\n image: nginx\n env_file: web.env\n configs: [cfg]\nconfigs:\n cfg:\n file: nginx/nginx.conf\n', + // Included files resolve their own relative paths against their + // own directory (each included file is its own project). + 'common/redis.yaml': 'services:\n redis:\n image: redis\n env_file: redis.env\n', + 'web.env': 'FOO=bar\n', + 'common/redis.env': 'REDIS=1\n', + 'nginx/nginx.conf': 'server {}\n', + '.env': 'PROJECT=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const byKind = (k: string) => result.inputs.filter((i) => i.dependencyKind === k); + expect(byKind('explicit').map((i) => i.sourcePath)).toEqual(['compose.yaml']); + expect(byKind('explicit')[0].materializedPath).toBe('compose.yaml'); + expect(byKind('include')[0].sourcePath).toBe('common/redis.yaml'); + const envFiles = byKind('env_file'); + expect(envFiles.some((i) => i.sourcePath === 'web.env')).toBe(true); + // The included file's env_file resolves against its own directory. + expect(envFiles.some((i) => i.sourcePath === 'common/redis.env')).toBe(true); + expect(byKind('config')[0].sourcePath).toBe('nginx/nginx.conf'); + // The included project's default interpolation .env (common/.env) is + // absent here, so it is recorded unmanaged and tolerated; every file + // actually present in the repository is managed. + const missingDefaultEnv = result.inputs.find((i) => i.dependencyKind === 'interpolation-env' && i.sourcePath === 'common/.env'); + expect(missingDefaultEnv?.ownership).toBe('unmanaged'); + expect(missingDefaultEnv?.note).toContain('No project .env'); + expect(result.inputs.filter((i) => i.ownership === 'unmanaged')).toHaveLength(1); + expect(result.inputs.filter((i) => i.ownership === 'managed').every((i) => i.state === 'present')).toBe(true); + // content hashes are computed for file-backed inputs. + expect(envFiles[0].contentSha256).toBeTruthy(); + }); + + it('refuses out-of-bound ../ include targets', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - ../outside.yaml\nservices: {}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'out-of-bounds' && r.actionable)).toBe(true); + }); + + it('refuses URL includes', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - https://example.com/remote.yaml\nservices: {}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'url-include')).toBe(true); + }); + + it('refuses symlink inputs as unsafe-symlink', async () => { + const clone = makeClone({ 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: linked.env\n' }); + // Create a real target and a symlink pointing at it. + fs.writeFileSync(path.join(clone, 'target.env'), 'X=1\n'); + try { + fs.symlinkSync('target.env', path.join(clone, 'linked.env')); + } catch { + // Symlinks unavailable (Windows perms); skip. + return; + } + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'unsafe-symlink')).toBe(true); + }); + + it('refuses special files as special-file', async () => { + const clone = makeClone({ 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: fifo.env\n' }); + // FIFOs cannot be created on Windows; mock the lstat result instead. + fs.writeFileSync(path.join(clone, 'fifo.env'), 'X=1\n'); + const originalLstat = fs.promises.lstat.bind(fs.promises); + vi.spyOn(fs.promises, 'lstat').mockImplementation(async (p) => { + if (String(p).endsWith('fifo.env')) { + return { + isSymbolicLink: () => false, + isDirectory: () => false, + isFile: () => true, + isCharacterDevice: () => true, + isBlockDevice: () => false, + isSocket: () => false, + isFIFO: () => false, + size: 4, + } as unknown as fs.Stats; + } + return originalLstat(p); + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'special-file')).toBe(true); + }); + + it('refuses paths inside submodules', async () => { + const clone = makeClone({ + '.gitmodules': '[submodule "vendor"]\n\tpath = vendor/lib\n', + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: vendor/lib/settings.env\n', + 'vendor/lib/settings.env': 'X=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'submodule' && r.sourcePath === 'vendor/lib/settings.env')).toBe(true); + }); + + it('refuses LFS pointer content for file-backed inputs', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: lfs.env\n', + 'lfs.env': 'version https://git-lfs.github.com/spec/v1\noid sha256:abcd\ntype file\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'lfs-pointer')).toBe(true); + }); + + it('records host binds and external resources as unmanaged', async () => { + const clone = makeClone({ + 'compose.yaml': `services: + web: + image: nginx + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ./data:/data +configs: + ext: + external: true +`, + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + const unmanaged = result.inputs.filter((i) => i.ownership === 'unmanaged'); + expect(unmanaged.some((i) => i.dependencyKind === 'bind-mount')).toBe(true); + expect(unmanaged.some((i) => i.dependencyKind === 'config' && i.sourcePath === null)).toBe(true); + expect(unmanaged.every((i) => i.deletionAuthority === 'none')).toBe(true); + }); + + it('materializes build contexts with dockerignore filtering and stable byte accounting', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n', + 'web/.dockerignore': 'node_modules\n*.log\n', + 'web/Dockerfile': 'FROM node\n', + 'web/index.js': 'console.log(1)\n', + 'web/node_modules/pkg/index.js': 'big\n', + 'web/debug.log': 'trace\n', + }); + const first = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(first.refusals).toEqual([]); + expect(first.buildContexts).toHaveLength(1); + const ctx = first.buildContexts[0]; + expect(ctx.repoPath).toBe('web'); + expect(ctx.dockerignoreApplied).toBe(true); + // node_modules + *.log ignored: only Dockerfile + index.js + .dockerignore counted. + expect(ctx.ignoredCount).toBeGreaterThanOrEqual(2); + expect(ctx.contextBytes).toBeGreaterThan(0); + + // Context byte accounting is stable across pulls of the same revision. + const second = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(second.buildContexts[0].contextBytes).toBe(first.buildContexts[0].contextBytes); + }); + + it('refuses a repo-root build context that exceeds the context cap', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build: .\n', + 'a.bin': 'a'.repeat(500), + 'b.bin': 'b'.repeat(500), + 'c.bin': 'c'.repeat(500), + }); + const bounds = { ...BOUNDS, maxContextBytes: 1000 }; + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds }); + expect(result.refusals.some((r) => r.kind === 'context-unbounded')).toBe(true); + }); + + it('counts a shared-context union once per unique file', async () => { + const clone = makeClone({ + 'compose.yaml': `services: + one: + build: + context: web + dockerfile: Dockerfile.one + two: + build: + context: web + dockerfile: Dockerfile.two +`, + 'web/Dockerfile.one': 'FROM scratch\n', + 'web/Dockerfile.two': 'FROM scratch\n', + 'web/Dockerfile.one.dockerignore': 'b.bin\n', + 'web/Dockerfile.two.dockerignore': 'a.bin\n', + 'web/common.bin': 'c'.repeat(100), + 'web/a.bin': 'a'.repeat(100), + 'web/b.bin': 'b'.repeat(100), + }); + const result = await discovery().discoverFromClone({ + cloneDir: clone, + composePaths: ['compose.yaml'], + contextDir: null, + bounds: { ...BOUNDS, maxContextBytes: 350 }, + }); + + expect(result.refusals.some((r) => r.kind === 'context-unbounded')).toBe(false); + expect(result.buildContexts).toHaveLength(1); + expect(result.buildContexts[0].contextBytes).toBe(338); + expect(result.buildContexts[0].files).toHaveLength(7); + + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-context-union-')); + tmpRoots.push(dest); + const managedFiles = result.inputs + .filter((input) => input.ownership === 'managed' + && input.materializedPath !== null + && input.dependencyKind !== 'build-context' + && input.dependencyKind !== 'build-additional-context') + .map((input) => ({ srcRel: input.sourcePath!, destRel: input.materializedPath! })); + await discovery().walkAndCopy(clone, dest, managedFiles, result.contextCopyPlans, BOUNDS); + expect(fs.existsSync(path.join(dest, 'web/a.bin'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'web/b.bin'))).toBe(true); + }); + + it('refuses LFS pointers inside a build context', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build: web\n', + 'web/Dockerfile': 'FROM node\n', + 'web/model.bin': 'version https://git-lfs.github.com/spec/v1\noid sha256:abcd\ntype file\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'lfs-in-context')).toBe(true); + }); + + it('discovers the implicit compose override for single-file stacks only', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n', + 'compose.override.yaml': 'services:\n web:\n environment:\n A: b\n', + }); + const single = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(single.inputs.some((i) => i.dependencyKind === 'implicit-override' && i.sourcePath === 'compose.override.yaml')).toBe(true); + + const multi = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml', 'compose.override.yaml'], contextDir: null, bounds: BOUNDS }); + expect(multi.inputs.some((i) => i.dependencyKind === 'implicit-override')).toBe(false); + }); + + it('scopes implicit override discovery to the primary file directory (monorepo isolation)', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n app:\n image: nginx\n ports: ["8099:80"]\n', + 'compose.override.yaml': 'services:\n app:\n environment:\n FROM: root\n', + 'monorepo/project-a/compose.yaml': 'services:\n a:\n image: alpine\n', + 'monorepo/project-a/compose.override.yaml': 'services:\n a:\n environment:\n FROM: nested\n', + 'monorepo/project-b/compose.yaml': 'services:\n b:\n image: busybox\n', + }); + + const nestedWithLocal = await discovery().discoverFromClone({ + cloneDir: clone, + composePaths: ['monorepo/project-a/compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }); + const overrideA = nestedWithLocal.inputs.find((i) => i.dependencyKind === 'implicit-override'); + expect(overrideA?.sourcePath).toBe('monorepo/project-a/compose.override.yaml'); + expect(overrideA?.materializedPath).toBe('compose.override.yaml'); + expect(nestedWithLocal.inputs.some((i) => i.sourcePath === 'compose.override.yaml')).toBe(false); + + const nestedWithoutLocal = await discovery().discoverFromClone({ + cloneDir: clone, + composePaths: ['monorepo/project-b/compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }); + expect(nestedWithoutLocal.inputs.some((i) => i.dependencyKind === 'implicit-override')).toBe(false); + }); + + it('refuses case-only materialized path collisions instead of dropping one file', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - case/Config.yml\n - case/config.yml\nservices: {}\n', + 'case/Config.yml': 'services:\n upper:\n image: alpine\n', + 'case/config.yml': 'services:\n lower:\n image: busybox\n', + }); + const result = await discovery().discoverFromClone({ + cloneDir: clone, + composePaths: ['compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }); + expect(result.refusals.some((r) => r.kind === 'case-collision' && r.actionable)).toBe(true); + const collision = result.refusals.find((r) => r.kind === 'case-collision'); + expect(collision?.reason).toMatch(/Config\.yml/i); + expect(collision?.reason).toMatch(/config\.yml/i); + const caseInputs = result.inputs.filter( + (i) => i.materializedPath !== null && /^case\/config\.yml$/i.test(i.materializedPath), + ); + expect(caseInputs).toHaveLength(1); + }); + + it('materializes env_file paths that contain a literal $ but no Compose variable', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: config$.env\n', + 'config$.env': 'A=1\n', + }); + const result = await discovery().discoverFromClone({ + cloneDir: clone, + composePaths: ['compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file' && i.sourcePath === 'config$.env'); + expect(env?.ownership).toBe('managed'); + expect(env?.materializedPath).toBe('config$.env'); + expect(result.dynamic.some((d) => d.sourcePath === 'config$.env')).toBe(false); + }); + + it('enforces the aggregate file and byte caps during classification', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file:\n - a.env\n - b.env\n - c.env\n', + 'a.env': 'A=1\n', + 'b.env': 'B=1\n', + 'c.env': 'C=1\n', + }); + const fileBounds = { ...BOUNDS, maxFiles: 3 }; // compose.yaml + a.env + b.env fit; c.env crosses + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: fileBounds }); + expect(result.refusals.some((r) => r.kind === 'too-many-files')).toBe(true); + }); +}); + +describe('walkAndCopy', () => { + it('copies managed files preserving the nested layout, skipping .git', async () => { + const clone = makeClone({ + 'compose.yaml': 'services: {}\n', + 'deploy/prod.yaml': 'services: {}\n', + '.git/config': '[remote]\n', + }); + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-cand-')); + tmpRoots.push(dest); + const result = await discovery().walkAndCopy(clone, dest, [ + { srcRel: 'compose.yaml', destRel: 'compose.yaml' }, + { srcRel: 'deploy/prod.yaml', destRel: 'deploy/prod.yaml' }, + ], [], BOUNDS); + expect(result.copiedFiles).toBe(2); + expect(fs.existsSync(path.join(dest, 'compose.yaml'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'deploy/prod.yaml'))).toBe(true); + expect(fs.existsSync(path.join(dest, '.git'))).toBe(false); + }); + + it('copies build contexts with dockerignore filtering', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n', + 'web/.dockerignore': 'node_modules\n', + 'web/Dockerfile': 'FROM node\n', + 'web/index.js': 'console.log(1)\n', + 'web/node_modules/pkg/index.js': 'big\n', + }); + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-cand-')); + tmpRoots.push(dest); + const discovery = ComposeInputDiscoveryService.getInstance(); + const inventory = await discovery.discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + await discovery.walkAndCopy(clone, dest, [], inventory.contextCopyPlans, BOUNDS); + expect(fs.existsSync(path.join(dest, 'web/Dockerfile'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'web/index.js'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'web/node_modules'))).toBe(false); + }); + + it('throws with running counts when the byte cap is crossed mid-copy', async () => { + const clone = makeClone({ + 'a.bin': 'a'.repeat(1024), + 'b.bin': 'b'.repeat(1024), + }); + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-cand-')); + tmpRoots.push(dest); + const bounds = { ...BOUNDS, maxBytes: 1500 }; + await expect( + discovery().walkAndCopy(clone, dest, [ + { srcRel: 'a.bin', destRel: 'a.bin' }, + { srcRel: 'b.bin', destRel: 'b.bin' }, + ], [], bounds), + ).rejects.toThrow(/exceeds 1500 bytes/); + }); + + it('rejects case-colliding destination paths', async () => { + const clone = makeClone({ 'A.TXT': '1\n', 'a.txt': '2\n' }); + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-cand-')); + tmpRoots.push(dest); + await expect( + discovery().walkAndCopy(clone, dest, [ + { srcRel: 'A.TXT', destRel: 'A.TXT' }, + { srcRel: 'a.txt', destRel: 'a.txt' }, + ], [], BOUNDS), + ).rejects.toThrow(/case-insensitive/); + }); +}); + +describe('syncEnv ownership (audit C-2)', () => { + it('marks the repo-root .env unmanaged and unhashed when syncEnv owns the path', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n', + '.env': 'REPO=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, syncEnv: true, bounds: BOUNDS }); + const envEntries = result.inputs.filter((i) => i.materializedPath === '.env' || i.dependencyKind === 'interpolation-env'); + expect(envEntries).toHaveLength(1); + expect(envEntries[0].ownership).toBe('unmanaged'); + expect(envEntries[0].contentSha256).toBeNull(); + expect(envEntries[0].deletionAuthority).toBe('none'); + }); + + it('keeps the repo .env managed when syncEnv is off', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n', + '.env': 'REPO=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, syncEnv: false, bounds: BOUNDS }); + const envEntries = result.inputs.filter((i) => i.dependencyKind === 'interpolation-env'); + expect(envEntries).toHaveLength(1); + expect(envEntries[0].ownership).toBe('managed'); + expect(envEntries[0].contentSha256).toBeTruthy(); + }); +}); + +describe('explicit dockerfile resolution (audit round 2 C-3)', () => { + it('rebases the dockerfile against its build context', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n dockerfile: Dockerfile.dev\n', + 'web/Dockerfile.dev': 'FROM nginx\n', + 'web/app.js': 'console.log(1)\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + expect(result.buildContexts).toHaveLength(1); + expect(result.buildContexts[0].repoPath).toBe('web'); + expect(result.buildContexts[0].dockerfile).toBe('web/Dockerfile.dev'); + // The dockerfile's files are part of the context inventory. + expect(result.buildContexts[0].files.some((f) => f.path === 'Dockerfile.dev')).toBe(true); + }); + + it('allows a ../ dockerfile inside the repository and materializes it separately', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n dockerfile: ../shared/Dockerfile\n', + 'web/app.js': 'x\n', + 'shared/Dockerfile': 'FROM nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const dockerfileEntry = result.inputs.find((i) => i.dependencyKind === 'dockerfile' && i.materializedPath === 'shared/Dockerfile'); + expect(dockerfileEntry).toBeTruthy(); + expect(dockerfileEntry?.ownership).toBe('managed'); + expect(dockerfileEntry?.contentSha256).toBeTruthy(); + }); + + it('refuses a dockerfile that escapes the repository', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n dockerfile: ../../outside/Dockerfile\n', + 'web/app.js': 'x\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'out-of-bounds' && String(r.sourcePath).includes('Dockerfile'))).toBe(true); + }); +}); + +describe('dynamic and submodule-backed inputs (audit round 8 B-2)', () => { + it('persists dynamic ${VAR} paths as explicit unmanaged entries', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: ${ENV_FILE:-default.env}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const dyn = result.inputs.filter((i) => i.dependencyKind === 'env_file' && i.sourcePath?.includes('${ENV_FILE')); + expect(dyn).toHaveLength(1); + expect(dyn[0].ownership).toBe('unmanaged'); + expect(dyn[0].materializedPath).toBeNull(); + expect(dyn[0].deletionAuthority).toBe('none'); + expect(dyn[0].note).toContain('resolved by Compose at deploy time'); + // The dynamic path is never searched for as a file in the clone. + expect(result.refusals.some((r) => r.kind === 'missing-file')).toBe(false); + }); + + it('refuses a build context rooted inside a submodule', async () => { + const clone = makeClone({ + '.gitmodules': '[submodule "vendor"]\n\tpath = vendor/lib\n', + 'compose.yaml': 'services:\n web:\n build: vendor/lib\n', + 'vendor/lib/Dockerfile': 'FROM nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'submodule' && String(r.sourcePath).includes('vendor/lib'))).toBe(true); + }); + + it('refuses a repo-root context containing a submodule directory', async () => { + const clone = makeClone({ + '.gitmodules': '[submodule "vendor"]\n\tpath = vendor/lib\n', + 'compose.yaml': 'services:\n web:\n build: .\n', + 'Dockerfile': 'FROM nginx\n', + 'vendor/lib/Dockerfile': 'FROM nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'submodule' && String(r.reason).includes('vendor/lib'))).toBe(true); + }); + + it('allows a submodule directory excluded by the context dockerignore', async () => { + const clone = makeClone({ + '.gitmodules': '[submodule "vendor"]\n\tpath = vendor/lib\n', + 'compose.yaml': 'services:\n web:\n build: .\n', + 'Dockerfile': 'FROM nginx\n', + '.dockerignore': 'vendor\n', + 'vendor/lib/Dockerfile': 'FROM nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + }); +}); + +describe('effective invocation path resolution (audit round 8 B-3)', () => { + it('resolves merged-file relative paths against the base file directory, never the declaring file', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services:\n web:\n image: nginx\n configs: [cfg]\nconfigs:\n cfg:\n file: nginx.conf\n', + // The second -f file lives in a DIFFERENT directory: its relative + // paths still resolve against the base file's directory (deploy/), + // not its own. + 'configs/prod.yaml': 'services:\n web:\n env_file: prod.env\n', + 'deploy/nginx.conf': 'server {}\n', + 'deploy/prod.env': 'A=1\n', + // Same-named files at the repo root must NOT be picked up. + 'nginx.conf': 'WRONG server {}\n', + 'prod.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml', 'configs/prod.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const prodEnv = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(prodEnv?.sourcePath).toBe('deploy/prod.env'); + // Materialized at the stack root (the runtime project dir). + expect(prodEnv?.materializedPath).toBe('prod.env'); + const cfg = result.inputs.find((i) => i.dependencyKind === 'config'); + expect(cfg?.sourcePath).toBe('deploy/nginx.conf'); + expect(cfg?.materializedPath).toBe('nginx.conf'); + }); + + it('refuses a base-dir-relative file that is missing instead of falling back to a same-named root file', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services:\n web:\n image: nginx\n env_file: app.env\n', + // No deploy/app.env: the old declaring-dir-first + repo-root + // fallback would silently materialize the root decoy. + 'app.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.kind === 'missing-file' && r.sourcePath === 'deploy/app.env')).toBe(true); + expect(result.inputs.some((i) => i.sourcePath === 'app.env' && i.ownership === 'managed')).toBe(false); + }); + + it('resolves the primary\'s includes against the context dir but keeps the included file\'s own paths on its own project directory', async () => { + const clone = makeClone({ + // With --project-directory deploy, the primary's include path + // resolves against deploy/ (the top-level project base). + 'compose.yaml': 'include:\n - common/redis.yaml\nservices: {}\n', + 'deploy/common/redis.yaml': 'services:\n redis:\n image: redis\n env_file: redis.env\n', + // The included file's own env_file resolves against ITS project + // directory (deploy/common), not the context dir. + 'deploy/common/redis.env': 'REDIS=1\n', + 'deploy/redis.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: 'deploy', bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const include = result.inputs.find((i) => i.dependencyKind === 'include'); + expect(include?.sourcePath).toBe('deploy/common/redis.yaml'); + expect(include?.materializedPath).toBe('deploy/common/redis.yaml'); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.sourcePath).toBe('deploy/common/redis.env'); + expect(env?.materializedPath).toBe('deploy/common/redis.env'); + }); + + it('materializes base-file-relative paths at the stack root, stripping the base directory prefix', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services:\n web:\n image: nginx\n env_file: prod.env\n', + 'deploy/prod.env': 'A=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + // The primary file lands at the stack root, so the runtime project dir + // is the stack root: prod.env must live at the stack root, not under + // deploy/. + expect(env?.sourcePath).toBe('deploy/prod.env'); + expect(env?.materializedPath).toBe('prod.env'); + }); + + it('resolves project-relative paths against the configured project directory', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file: app.env\n', + 'deploy/app.env': 'A=1\n', + 'app.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: 'deploy', bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.sourcePath).toBe('deploy/app.env'); + expect(env?.materializedPath).toBe('deploy/app.env'); + }); + + it('does not auto-discover an override when the project directory makes the invocation explicit', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n', + 'compose.override.yaml': 'services:\n web:\n environment:\n A: b\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: 'deploy', bounds: BOUNDS }); + // A context dir forces explicit -f at runtime, which suppresses + // auto-discovery; the override must not enter the inventory. + expect(result.inputs.some((i) => i.dependencyKind === 'implicit-override')).toBe(false); + }); + + it('rebases a subdir build context to the stack root for the materialized layout', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services:\n web:\n build:\n context: web\n dockerfile: Dockerfile\n', + 'deploy/web/Dockerfile': 'FROM nginx\n', + 'deploy/web/app.js': 'console.log(1)\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + expect(result.buildContexts).toHaveLength(1); + const ctx = result.buildContexts[0]; + expect(ctx.repoPath).toBe('web'); + expect(ctx.dockerfile).toBe('deploy/web/Dockerfile'); + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-b3-ctx-')); + tmpRoots.push(dest); + const managed = result.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null && i.dependencyKind !== 'build-context') + .map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })); + await discovery().walkAndCopy(clone, dest, managed, result.contextCopyPlans, BOUNDS); + expect(fs.existsSync(path.join(dest, 'web/Dockerfile'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'web/app.js'))).toBe(true); + }); +}); + +describe('nested-primary runtime path equivalence (audit round 9 B-3)', () => { + it('materializes a nested primary\'s include graph at the stack root', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'include:\n - common.yaml\nservices: {}\n', + 'deploy/common.yaml': 'services:\n web:\n image: nginx\n env_file: web.env\n', + 'deploy/web.env': 'A=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const include = result.inputs.find((i) => i.dependencyKind === 'include'); + expect(include?.sourcePath).toBe('deploy/common.yaml'); + expect(include?.materializedPath).toBe('common.yaml'); + // The included file's own project-relative input also moves to the + // stack root: at runtime compose.yaml includes ./common.yaml, whose + // env_file resolves against its own (relocated) directory. + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.sourcePath).toBe('deploy/web.env'); + expect(env?.materializedPath).toBe('web.env'); + + // The candidate mirrors the runtime layout. + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-b3-graph-')); + tmpRoots.push(dest); + const managed = result.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null && i.dependencyKind !== 'build-context') + .map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })); + await discovery().walkAndCopy(clone, dest, managed, result.contextCopyPlans, BOUNDS); + expect(fs.existsSync(path.join(dest, 'common.yaml'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'web.env'))).toBe(true); + expect(fs.existsSync(path.join(dest, 'deploy/common.yaml'))).toBe(false); + }); + + it('recurses through a nested primary\'s include graph, stripping the prefix at every level', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'include:\n - common.yaml\nservices: {}\n', + 'deploy/common.yaml': 'include:\n - nested/inner.yaml\nservices: {}\n', + 'deploy/nested/inner.yaml': 'services:\n web:\n image: nginx\n env_file: inner.env\n', + 'deploy/nested/inner.env': 'A=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const inner = result.inputs.find((i) => i.dependencyKind === 'include' && i.sourcePath === 'deploy/nested/inner.yaml'); + expect(inner?.materializedPath).toBe('nested/inner.yaml'); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.sourcePath).toBe('deploy/nested/inner.env'); + expect(env?.materializedPath).toBe('nested/inner.env'); + }); + + it('resolves an additional -f file\'s includes against the base file\'s directory', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services: {}\n', + // The merged project's base is deploy/ (the first -f file); an + // include declared in the second file resolves against it. + 'infra/prod.yaml': 'include:\n - prod-common.yaml\nservices: {}\n', + 'deploy/prod-common.yaml': 'services:\n web:\n image: nginx\n', + 'infra/prod-common.yaml': 'services:\n WRONG:\n image: nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml', 'infra/prod.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const include = result.inputs.find((i) => i.dependencyKind === 'include'); + expect(include?.sourcePath).toBe('deploy/prod-common.yaml'); + // At runtime the project dir is the stack root, so the included file + // moves there too. + expect(include?.materializedPath).toBe('prod-common.yaml'); + }); + + it('applies a divergent include project_directory to the subtree project base', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - path: app/compose.yaml\n project_directory: other\nservices: {}\n', + 'app/compose.yaml': 'services:\n web:\n image: nginx\n env_file: x.env\n', + 'other/x.env': 'A=1\n', + 'app/x.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + // project_directory re-bases the subtree: the env file lives under + // other/, not next to the included file. + expect(env?.sourcePath).toBe('other/x.env'); + expect(env?.materializedPath).toBe('other/x.env'); + }); + + it('materializes a nested primary\'s extends.file target at the stack root with its own inputs', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services:\n web:\n extends:\n file: web-base.yaml\n service: web-base\n', + 'deploy/web-base.yaml': 'services:\n web-base:\n image: nginx\n label_file: labels.txt\n', + 'deploy/labels.txt': 'a=b\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const ext = result.inputs.find((i) => i.dependencyKind === 'extends'); + expect(ext?.sourcePath).toBe('deploy/web-base.yaml'); + expect(ext?.materializedPath).toBe('web-base.yaml'); + const label = result.inputs.find((i) => i.dependencyKind === 'label_file'); + expect(label?.sourcePath).toBe('deploy/labels.txt'); + expect(label?.materializedPath).toBe('labels.txt'); + }); + + it('refuses a nested-primary include that escapes the stack root', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'include:\n - ../shared.yaml\nservices: {}\n', + 'shared.yaml': 'services: {}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + // ../shared.yaml stays inside the repository (deploy/../shared.yaml) + // but escapes the stack root at runtime; it must be refused, never + // silently dropped or adopted from the repo root. + expect(result.refusals.some((r) => r.actionable && r.kind === 'out-of-bounds' && String(r.sourcePath).includes('../shared.yaml'))).toBe(true); + expect(result.inputs.some((i) => i.sourcePath === 'shared.yaml' && i.ownership === 'managed')).toBe(false); + }); +}); + +describe('included-project default env and project-base includes (audit round 10 B-1/B-2)', () => { + it('materializes an included project\'s default .env as a managed sensitive input', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - modules/app/compose.yaml\nservices: {}\n', + 'modules/app/compose.yaml': 'services:\n app:\n image: app:${TAG:-latest}\n', + 'modules/app/.env': 'TAG=production\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const defaultEnv = result.inputs.find((i) => i.dependencyKind === 'interpolation-env' && i.sourcePath === 'modules/app/.env'); + expect(defaultEnv?.ownership).toBe('managed'); + expect(defaultEnv?.materializedPath).toBe('modules/app/.env'); + expect(defaultEnv?.contentSha256).toBeTruthy(); + expect(defaultEnv?.sensitivity).toBe('high'); + }); + + it('tolerates a missing included-project default .env without refusing', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - modules/app/compose.yaml\nservices: {}\n', + 'modules/app/compose.yaml': 'services:\n app:\n image: app:${TAG:-latest}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const defaultEnv = result.inputs.find((i) => i.dependencyKind === 'interpolation-env' && i.sourcePath === 'modules/app/.env'); + expect(defaultEnv?.ownership).toBe('unmanaged'); + expect(defaultEnv?.note).toContain('No project .env'); + }); + + it('resolves extends.file in an additional -f file against the base file\'s directory', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'services: {}\n', + 'infra/prod.yaml': 'services:\n web:\n extends:\n file: web-base.yaml\n service: web-base\n', + 'deploy/web-base.yaml': 'services:\n web-base:\n image: nginx\n', + 'infra/web-base.yaml': 'services:\n WRONG:\n image: nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml', 'infra/prod.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const ext = result.inputs.find((i) => i.dependencyKind === 'extends'); + expect(ext?.sourcePath).toBe('deploy/web-base.yaml'); + expect(ext?.materializedPath).toBe('web-base.yaml'); + }); + + it('resolves extends.file against the context dir with the subtree on its own project directory', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n extends:\n file: web-base.yaml\n service: web-base\n', + 'deploy/web-base.yaml': 'services:\n web-base:\n image: nginx\n label_file: labels.txt\n', + 'deploy/labels.txt': 'a=b\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: 'deploy', bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const ext = result.inputs.find((i) => i.dependencyKind === 'extends'); + expect(ext?.sourcePath).toBe('deploy/web-base.yaml'); + const label = result.inputs.find((i) => i.dependencyKind === 'label_file'); + expect(label?.sourcePath).toBe('deploy/labels.txt'); + }); + + it('uses the first path as the included project\'s main file for a multi-directory path list', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - path:\n - modules/base/compose.yaml\n - overrides/compose.yaml\nservices: {}\n', + 'modules/base/compose.yaml': 'services:\n base:\n image: nginx\n', + 'overrides/compose.yaml': 'services:\n base:\n env_file: shared.env\n', + // The included project's base is modules/base (the first path); + // the override's relative inputs resolve against it, never the + // override's own directory. + 'modules/base/shared.env': 'A=1\n', + 'overrides/shared.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.sourcePath).toBe('modules/base/shared.env'); + expect(env?.materializedPath).toBe('modules/base/shared.env'); + }); +}); + +describe('optional env_file and external: false semantics (audit round 10 S-1)', () => { + it('does not refuse a missing optional env_file', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file:\n - path: optional.env\n required: false\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.ownership).toBe('unmanaged'); + expect(env?.note).toContain('Optional env file'); + }); + + it('materializes a present optional env_file as managed', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n env_file:\n - path: optional.env\n required: false\n', + 'optional.env': 'A=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.ownership).toBe('managed'); + expect(env?.contentSha256).toBeTruthy(); + }); + + it('treats only external: true as external for file-backed configs', async () => { + const clone = makeClone({ + 'compose.yaml': 'configs:\n app:\n file: configs/app.conf\n external: false\nservices:\n web:\n image: nginx\n configs: [app]\n', + 'configs/app.conf': 'server {}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const cfg = result.inputs.find((i) => i.dependencyKind === 'config'); + expect(cfg?.sourcePath).toBe('configs/app.conf'); + expect(cfg?.ownership).toBe('managed'); + expect(cfg?.contentSha256).toBeTruthy(); + }); + + it('treats only external: true as external for file-backed secrets', async () => { + const clone = makeClone({ + 'compose.yaml': 'secrets:\n app-key:\n file: secrets/app.key\n external: false\nservices:\n web:\n image: nginx\n secrets: [app-key]\n', + 'secrets/app.key': 'secret\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const sec = result.inputs.find((i) => i.dependencyKind === 'secret'); + expect(sec?.sourcePath).toBe('secrets/app.key'); + expect(sec?.ownership).toBe('managed'); + expect(sec?.sensitivity).toBe('high'); + }); +}); + +describe('absolute and home-relative path classification (audit round 9 B-4)', () => { + it('records absolute env/config paths as unmanaged host entries, never adopting repo decoys', async () => { + const clone = makeClone({ + 'compose.yaml': `services: + web: + image: nginx + env_file: /etc/secrets/web.env + configs: [cfg] +configs: + cfg: + file: /etc/nginx/app.conf +`, + // Repo decoys with the same basenames must never be adopted. + 'etc/secrets/web.env': 'WRONG=1\n', + 'etc/nginx/app.conf': 'WRONG server {}\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const env = result.inputs.find((i) => i.dependencyKind === 'env_file'); + expect(env?.ownership).toBe('unmanaged'); + expect(env?.materializedPath).toBeNull(); + expect(env?.note).toContain('Host path'); + const cfg = result.inputs.find((i) => i.dependencyKind === 'config'); + expect(cfg?.ownership).toBe('unmanaged'); + expect(result.inputs.some((i) => i.ownership === 'managed' && i.materializedPath?.includes('etc/'))).toBe(false); + }); + + it('refuses absolute includes without adopting a repo decoy', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - /etc/compose/extra.yaml\nservices: {}\n', + 'etc/compose/extra.yaml': 'services:\n web:\n image: nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals.some((r) => r.actionable && r.kind === 'out-of-bounds' && String(r.sourcePath).includes('/etc/compose/extra.yaml'))).toBe(true); + expect(result.inputs.some((i) => i.materializedPath === 'etc/compose/extra.yaml')).toBe(false); + }); + + it('records absolute build contexts and dockerfiles as unmanaged host entries', async () => { + const clone = makeClone({ + 'compose.yaml': `services: + web: + build: + context: /opt/build/web + dockerfile: /opt/build/Dockerfile +`, + 'opt/build/web/Dockerfile': 'FROM nginx\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + expect(result.buildContexts).toHaveLength(0); + const ctx = result.inputs.find((i) => i.dependencyKind === 'build-context'); + expect(ctx?.ownership).toBe('unmanaged'); + expect(ctx?.note).toContain('Host path'); + const df = result.inputs.find((i) => i.dependencyKind === 'dockerfile'); + expect(df?.ownership).toBe('unmanaged'); + }); + + it('records Windows drive, UNC, and home-relative paths as host inputs', async () => { + const clone = makeClone({ + 'compose.yaml': `services: + web: + image: nginx + env_file: + - C:\\\\config\\\\web.env + - \\\\\\\\server\\\\share\\\\x.env + - ~/web.env +`, + 'config/web.env': 'WRONG=1\n', + 'web.env': 'WRONG=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const envFiles = result.inputs.filter((i) => i.dependencyKind === 'env_file'); + expect(envFiles).toHaveLength(3); + expect(envFiles.every((e) => e.ownership === 'unmanaged' && e.materializedPath === null)).toBe(true); + }); +}); + +describe('default build context and build-secret grammar (audit round 8 B-1)', () => { + it('materializes an omitted build context at the repo root', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n dockerfile: Dockerfile\n', + 'Dockerfile': 'FROM nginx\n', + 'src/app.js': 'console.log(1)\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + expect(result.buildContexts).toHaveLength(1); + const ctx = result.buildContexts[0]; + expect(ctx.repoPath).toBe(''); + expect(ctx.dockerfile).toBe('Dockerfile'); + expect(ctx.files.some((f) => f.path === 'Dockerfile')).toBe(true); + expect(ctx.files.some((f) => f.path === 'src/app.js')).toBe(true); + }); + + it('resolves an omitted build context against the configured project directory', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n dockerfile: Dockerfile\n', + 'deploy/Dockerfile': 'FROM nginx\n', + 'deploy/app.js': 'console.log(1)\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: 'deploy', bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + expect(result.buildContexts).toHaveLength(1); + const ctx = result.buildContexts[0]; + expect(ctx.repoPath).toBe('deploy'); + expect(ctx.dockerfile).toBe('deploy/Dockerfile'); + expect(ctx.files.some((f) => f.path === 'app.js')).toBe(true); + }); + + it('resolves an omitted build context in an included file against the included file\'s directory', async () => { + const clone = makeClone({ + 'compose.yaml': 'include:\n - services/app/compose.yaml\nservices: {}\n', + 'services/app/compose.yaml': 'services:\n app:\n build:\n dockerfile: Dockerfile\n', + 'services/app/Dockerfile': 'FROM nginx\n', + 'services/app/app.js': 'console.log(1)\n', + 'root-decoy.txt': 'must not be in the context\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + expect(result.buildContexts).toHaveLength(1); + const ctx = result.buildContexts[0]; + expect(ctx.repoPath).toBe('services/app'); + expect(ctx.dockerfile).toBe('services/app/Dockerfile'); + expect(ctx.files.some((f) => f.path === 'app.js')).toBe(true); + expect(ctx.files.some((f) => f.path === 'root-decoy.txt')).toBe(false); + }); + + it('does not double-prefix an include map-form env_file from a subdirectory base file', async () => { + const clone = makeClone({ + 'deploy/base.yaml': 'include:\n - path: web.yaml\n env_file: e.env\nservices: {}\n', + 'deploy/web.yaml': 'services:\n web:\n image: nginx\n', + 'deploy/e.env': 'A=1\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['deploy/base.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const includeEnv = result.inputs.find((i) => i.dependencyKind === 'include-env'); + // Repo-side the env file sits next to the base file; at runtime the + // primary lands at the stack root, so the env file lives there too. + expect(includeEnv?.sourcePath).toBe('deploy/e.env'); + expect(includeEnv?.materializedPath).toBe('e.env'); + }); + + it('does not search for a file named after a build-secret source', async () => { + const clone = makeClone({ + 'compose.yaml': `secrets: + build-key: + file: keys/build.env +services: + web: + build: + context: web + secrets: + - id: build-key + source: build-key +`, + 'web/Dockerfile': 'FROM nginx\n', + 'keys/build.env': 'TOKEN=x\n', + }); + const result = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + expect(result.refusals).toEqual([]); + const secretEntry = result.inputs.find((i) => i.dependencyKind === 'secret' && i.sourcePath === 'keys/build.env'); + expect(secretEntry).toBeTruthy(); + expect(secretEntry?.ownership).toBe('managed'); + const buildSecret = result.inputs.find((i) => i.dependencyKind === 'build-secret'); + expect(buildSecret?.sourcePath).toBeNull(); + expect(buildSecret?.ownership).toBe('unmanaged'); + }); +}); + +describe('repo-root build context overlap (audit round 2)', () => { + it('copies a repo-root context without duplicating managed files', async () => { + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build: .\n', + 'src/main.go': 'package main\n', + }); + const inv = await discovery().discoverFromClone({ cloneDir: clone, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + const managed = inv.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null && i.dependencyKind !== 'build-context'); + const dest = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-rootctx-')); + tmpRoots.push(dest); + const result = await discovery().walkAndCopy(clone, dest, managed.map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), inv.contextCopyPlans, BOUNDS); + // compose.yaml + src/main.go: no duplicate-path failure, no double copy. + expect(result.copiedFiles).toBe(2); + expect(fs.readFileSync(path.join(dest, 'src/main.go'), 'utf8')).toBe('package main\n'); + }); +}); diff --git a/backend/src/__tests__/compose-input-parse.test.ts b/backend/src/__tests__/compose-input-parse.test.ts new file mode 100644 index 00000000..a07f1633 --- /dev/null +++ b/backend/src/__tests__/compose-input-parse.test.ts @@ -0,0 +1,454 @@ +import { describe, it, expect } from 'vitest'; +import { isDynamicPath, parseDeclaredInputs } from '../helpers/composeInputParse'; +import type { DeclaredInput, DynamicInput } from '../types/gitProjectManifest'; + +/** In-memory repo fixture: maps repo paths to contents. */ +function repo(files: Record) { + return { + read: (p: string): string | null => files[p] ?? null, + files, + }; +} + +function parse(files: Record, projectRoot: string | null = null) { + const r = repo(files); + const primary = Object.keys(files).find((p) => p.endsWith('compose.yaml') || p === 'compose.yml') ?? 'compose.yaml'; + return { + result: parseDeclaredInputs([{ path: primary, content: files[primary] ?? '' }], { + projectRoot, + read: r.read, + }), + files: r.files, + }; +} + +function byKind(result: ReturnType['result'], kind: string): DeclaredInput[] { + return result.inputs.filter((i) => i.kind === kind); +} + +describe('parseDeclaredInputs', () => { + it('emits the interpolation env at the project root', () => { + const { result } = parse({ 'compose.yaml': 'services:\n web:\n image: nginx\n' }); + const interp = byKind(result, 'interpolation-env'); + expect(interp).toHaveLength(1); + expect(interp[0].sourcePath).toBe('.env'); + expect(interp[0].baseDir).toBe('project-root'); + }); + + it('uses the project root for the interpolation env path', () => { + const { result } = parse({ 'deploy/compose.yaml': 'services: {}\n' }, 'deploy'); + const interp = byKind(result, 'interpolation-env'); + expect(interp[0].sourcePath).toBe('deploy/.env'); + }); + + it('walks include list form and recurses into included files', () => { + const { result } = parse({ + 'compose.yaml': 'include:\n - common/redis.yaml\nservices:\n web:\n image: nginx\n', + // Included files resolve their relative paths against their own + // directory (each included file is its own project). + 'common/redis.yaml': 'services:\n redis:\n image: redis\n env_file: redis.env\n', + }); + const includes = byKind(result, 'include'); + expect(includes.map((i) => i.sourcePath)).toEqual(['common/redis.yaml']); + const envFiles = byKind(result, 'env_file'); + expect(envFiles.map((e) => e.sourcePath)).toContain('common/redis.env'); + expect(envFiles.find((e) => e.sourcePath === 'common/redis.env')?.baseDir).toBe('compose-file-dir'); + }); + + it('walks include map form with include-specific env_file', () => { + const { result } = parse({ + 'compose.yaml': 'include:\n - path: apps/api.yaml\n env_file: apps/api.env\n', + 'apps/api.yaml': 'services:\n api:\n image: api\n', + }); + expect(byKind(result, 'include')[0].sourcePath).toBe('apps/api.yaml'); + const includeEnv = byKind(result, 'include-env'); + expect(includeEnv).toHaveLength(1); + expect(includeEnv[0].sourcePath).toBe('apps/api.env'); + // Already resolved by the parser; the classifier must not re-resolve it. + expect(includeEnv[0].baseDir).toBe('repo-root'); + }); + + it('detects include cycles', () => { + const { result } = parse({ + 'compose.yaml': 'include:\n - a.yaml\n', + 'a.yaml': 'include:\n - compose.yaml\nservices: {}\n', + }); + expect(result.parseErrors.some((e) => e.includes('cycle'))).toBe(true); + }); + + it('reports unreadable include targets as parse errors', () => { + const { result } = parse({ + 'compose.yaml': 'include:\n - missing.yaml\n', + }); + expect(result.parseErrors.some((e) => e.includes('missing.yaml'))).toBe(true); + }); + + it('walks extends.file recursion and reports cycles', () => { + const { result } = parse({ + 'compose.yaml': 'services:\n web:\n extends:\n file: base/web.yaml\n service: web-base\n', + // Included files resolve their own relative paths against their + // own directory (each included file is its own project). + 'base/web.yaml': 'services:\n web-base:\n image: nginx\n label_file: labels.txt\n', + }); + const extendsRefs = byKind(result, 'extends'); + expect(extendsRefs.map((e) => e.sourcePath)).toEqual(['base/web.yaml']); + expect(byKind(result, 'label_file')[0].sourcePath).toBe('base/labels.txt'); + }); + + it('does not recurse into same-file string-form extends', () => { + const { result } = parse({ + 'compose.yaml': 'services:\n web:\n extends: base\n base:\n image: nginx\n', + }); + expect(byKind(result, 'extends')).toHaveLength(0); + }); + + it('walks service env_file in string, list and map forms', () => { + const { result } = parse({ + 'compose.yaml': `services: + a: + image: a + env_file: a.env + b: + image: b + env_file: + - shared.env + - b.env + c: + image: c + env_file: + path: c.env + required: true +`, + }); + const envFiles = byKind(result, 'env_file'); + expect(envFiles.map((e) => e.sourcePath).sort()).toEqual(['a.env', 'b.env', 'c.env', 'shared.env'].sort()); + }); + + it('classifies top-level configs and secrets file forms', () => { + const { result } = parse({ + 'compose.yaml': `configs: + nginx-conf: + file: nginx/nginx.conf + ext: + external: true + env-injected: + environment: NGNIX_CONFIG +secrets: + db-password: + file: secrets/db.env + sops-secret: + environment: DB_PASSWORD +services: + web: + image: nginx + configs: [nginx-conf] + secrets: [db-password] +`, + }); + const configs = byKind(result, 'config'); + expect(configs.map((c) => c.sourcePath)).toContain('nginx/nginx.conf'); + const secrets = byKind(result, 'secret'); + expect(secrets.map((s) => s.sourcePath)).toContain('secrets/db.env'); + // external/env forms become unmanaged placeholders (null source). + expect(configs.filter((c) => c.sourcePath === null)).toHaveLength(2); + expect(secrets.filter((s) => s.sourcePath === null)).toHaveLength(1); + }); + + it('walks build context, dockerfile, secrets and additional contexts', () => { + const { result } = parse({ + 'compose.yaml': `secrets: + ssh-key: + file: web/ssh-key +services: + web: + build: + context: web + dockerfile: Dockerfile.dev + secrets: + - npm-token + - id: ssh-key + source: ssh-key + additional_contexts: + certs: web/certs +`, + }); + const contexts = byKind(result, 'build-context'); + expect(contexts.map((c) => c.sourcePath)).toEqual(['web']); + expect(byKind(result, 'dockerfile')[0].sourcePath).toBe('Dockerfile.dev'); + // String-form and long-syntax (source = top-level secret name) build + // secrets are both recorded as unmanaged references; the top-level + // secrets walk emits the actual file. + const buildSecrets = byKind(result, 'build-secret'); + expect(buildSecrets.map((s) => s.sourcePath)).toEqual([null, null]); + expect(byKind(result, 'secret').map((s) => s.sourcePath)).toContain('web/ssh-key'); + expect(byKind(result, 'build-additional-context')[0].sourcePath).toBe('web/certs'); + }); + + it('defaults an omitted build context to the project directory', () => { + const { result } = parse({ + 'compose.yaml': 'services:\n web:\n build:\n dockerfile: Dockerfile\n', + }); + const contexts = byKind(result, 'build-context'); + expect(contexts.map((c) => c.sourcePath)).toEqual(['.']); + // The classifier resolves '.' against the per-file project directory + // (context dir, base file dir, or the declaring file's own dir). + expect(contexts[0].baseDir).toBe('compose-file-dir'); + }); + + it('never treats a build-secret source as a file path', () => { + const { result } = parse({ + 'compose.yaml': `secrets: + build-key: + external: true +services: + web: + build: + context: web + secrets: + - id: build-key + source: build-key +`, + }); + // The secret name must not be searched as a file in the repository. + const buildSecrets = byKind(result, 'build-secret'); + expect(buildSecrets.map((s) => s.sourcePath)).toEqual([null]); + // The referenced (external) top-level secret is recorded unmanaged. + expect(byKind(result, 'secret').some((s) => s.sourcePath === null)).toBe(true); + }); + + it('walks string-form build context', () => { + const { result } = parse({ + 'compose.yaml': 'services:\n web:\n build: web\n', + }); + expect(byKind(result, 'build-context')[0].sourcePath).toBe('web'); + }); + + it('classifies bind mounts: relative, absolute and named volumes', () => { + const { result } = parse({ + 'compose.yaml': `services: + web: + image: nginx + volumes: + - ./data:/data + - ../shared:/shared + - /etc/hosts:/etc/hosts:ro + - ~/cache:/cache + - named-vol:/vol + - type: bind + source: ./config + target: /config + - type: volume + source: other-vol + target: /other +`, + }); + const binds = byKind(result, 'bind-mount'); + // Relative binds resolve against the project base; absolute and + // ../-escaping binds are host paths (never adopted from the repo). + expect(binds.map((b) => b.sourcePath)).toEqual(['data', '../shared', null, null, 'config']); + expect(binds.map((b) => b.baseDir)).toEqual(['project-root', 'host', 'host', 'host', 'project-root']); + expect(binds.map((b) => b.materializedPath)).toEqual(['data', null, null, null, 'config']); + }); + + it('routes dynamic paths into the dynamic list, not inputs', () => { + const { result } = parse({ + 'compose.yaml': `services: + web: + image: nginx + env_file: \${ENV_FILE:-default.env} +`, + }); + expect(byKind(result, 'env_file')).toHaveLength(0); + const dynamic = result.dynamic as DynamicInput[]; + expect(dynamic.some((d) => d.kind === 'env_file' && d.sourcePath.includes('${ENV_FILE'))).toBe(true); + }); + + it('records out-of-bound and URL include targets for the classifier', () => { + const { result } = parse({ + 'compose.yaml': `include: + - ../outside.yaml + - https://example.com/remote.yaml +`, + }); + const includes = byKind(result, 'include'); + expect(includes.map((i) => i.sourcePath)).toEqual(['../outside.yaml', 'https://example.com/remote.yaml']); + }); + + it('enforces the include depth cap', () => { + const files: Record = {}; + files['compose.yaml'] = 'include:\n - f1.yaml\nservices: {}\n'; + for (let i = 1; i < 19; i++) { + files[`f${i}.yaml`] = `include:\n - f${i + 1}.yaml\nservices: {}\n`; + } + files['f19.yaml'] = 'services: {}\n'; + const { result } = parse(files); + expect(result.parseErrors.some((e) => e.includes('depth 16'))).toBe(true); + }); + + it('reports unparseable compose files as parse errors', () => { + const { result } = parse({ + 'compose.yaml': 'services: [unclosed\n', + }); + expect(result.parseErrors.some((e) => e.includes('Cannot parse'))).toBe(true); + }); + + it('walks label_file in string and list forms', () => { + const { result } = parse({ + 'compose.yaml': `services: + a: + image: a + label_file: a.labels + b: + image: b + label_file: + - b1.labels + - b2.labels +`, + }); + const labelFiles = byKind(result, 'label_file'); + expect(labelFiles.map((l) => l.sourcePath).sort()).toEqual(['a.labels', 'b1.labels', 'b2.labels'].sort()); + }); + + it('walks additional_contexts in mapping and NAME=VALUE list forms', () => { + const { result } = parse({ + 'compose.yaml': `services: + a: + build: + context: web + additional_contexts: + certs: web/certs + b: + build: + context: web + additional_contexts: + - resources=web/resources + - app=docker-image://my-app:latest + - base=service:base +`, + }); + const contexts = byKind(result, 'build-additional-context'); + // Path values resolve project-relative; builder-supplied values are + // recorded unmanaged (host) and never searched for in the repo. + const pathValues = contexts.filter((c) => c.baseDir !== 'host'); + expect(pathValues.map((c) => c.sourcePath).sort()).toEqual(['web/certs', 'web/resources'].sort()); + const builderValues = contexts.filter((c) => c.baseDir === 'host'); + expect(builderValues.map((c) => c.sourcePath).sort()).toEqual(['docker-image://my-app:latest', 'service:base'].sort()); + }); + + it('walks include map path and env_file in list forms', () => { + const { result } = parse({ + 'compose.yaml': `include: + - path: + - a.yaml + - b.yaml + env_file: + - a.env + - b.env +services: {} +`, + 'a.yaml': 'services: {}\n', + 'b.yaml': 'services: {}\n', + }); + const includes = byKind(result, 'include'); + expect(includes.map((i) => i.sourcePath).sort()).toEqual(['a.yaml', 'b.yaml'].sort()); + const includeEnv = byKind(result, 'include-env'); + expect(includeEnv.map((e) => e.sourcePath).sort()).toEqual(['a.env', 'b.env'].sort()); + }); + + it('applies include project_directory to the included subtree', () => { + const { result } = parse({ + 'compose.yaml': 'include:\n - path: app/compose.yaml\n project_directory: app\nservices: {}\n', + 'app/compose.yaml': 'services:\n web:\n image: nginx\n env_file: x.env\n', + }); + // The included file's project-relative paths resolve against the + // included project_directory ('app'), not its own directory. + const envFiles = byKind(result, 'env_file'); + expect(envFiles.map((e) => e.sourcePath)).toEqual(['app/x.env']); + }); + + it('parses Windows drive-letter short-form bind mounts (audit round 10 S-2)', () => { + // Plain YAML scalars keep backslashes literally (no escape processing). + const { result } = parse({ + 'compose.yaml': `services: + web: + image: nginx + volumes: + - C:\\data:/data + - C:creds:/creds + - \\\\server\\share:/share + - ./data:/data + - named-vol:/vol + - C:\\data:/data:ro +`, + }); + const binds = byKind(result, 'bind-mount'); + // Drive-letter, drive-relative, and UNC short binds are recorded as + // host entries (sourcePath null per the host-bind convention) instead + // of being mistaken for named volumes and dropped; ./data resolves + // project-relative; named-vol is not a bind. + expect(binds).toHaveLength(5); + expect(binds.map((b) => b.baseDir)).toEqual(['host', 'host', 'host', 'project-root', 'host']); + expect(binds.find((b) => b.kind === 'bind-mount' && b.materializedPath === 'data')?.sourcePath).toBe('data'); + }); + + it('preserves env_file required: false and required: true (audit round 10 S-1)', () => { + const { result } = parse({ + 'compose.yaml': `services: + web: + image: nginx + env_file: + - path: optional.env + required: false + - required.env +`, + }); + const envFiles = byKind(result, 'env_file'); + expect(envFiles.find((e) => e.sourcePath === 'optional.env')?.required).toBe(false); + expect(envFiles.find((e) => e.sourcePath === 'required.env')?.required).toBe(true); + }); + + it('classifies absolute and home-relative paths as host paths', () => { + const { result } = parse({ + 'compose.yaml': `include: + - /etc/compose/extra.yaml +services: + web: + image: nginx + env_file: /etc/secrets/web.env + configs: [cfg] + label_file: ~/labels.txt +configs: + cfg: + file: C:\\\\config\\\\app.conf +`, + }); + const includes = byKind(result, 'include'); + expect(includes[0].sourcePath).toBe('/etc/compose/extra.yaml'); + expect(includes[0].baseDir).toBe('host'); + const envFile = byKind(result, 'env_file')[0]; + expect(envFile.baseDir).toBe('host'); + expect(envFile.materializedPath).toBeNull(); + const cfg = byKind(result, 'config')[0]; + // Plain YAML scalars do not process escapes, so the parsed value + // carries the double backslashes as written. + expect(cfg.sourcePath).toBe('C:\\\\config\\\\app.conf'); + expect(cfg.baseDir).toBe('host'); + const labelFile = byKind(result, 'label_file')[0]; + expect(labelFile.baseDir).toBe('host'); + }); +}); + +describe('isDynamicPath', () => { + it('treats Compose variable forms as dynamic', () => { + expect(isDynamicPath('${ENV_FILE}')).toBe(true); + expect(isDynamicPath('${ENV_FILE:-default.env}')).toBe(true); + expect(isDynamicPath('$ENV_FILE')).toBe(true); + expect(isDynamicPath('prefix-$ENV_FILE')).toBe(true); + }); + + it('treats a literal $ that is not a variable start as static', () => { + expect(isDynamicPath('config$.env')).toBe(false); + expect(isDynamicPath('file$.yaml')).toBe(false); + expect(isDynamicPath('plain.env')).toBe(false); + }); +}); diff --git a/backend/src/__tests__/docker-ignore-match.test.ts b/backend/src/__tests__/docker-ignore-match.test.ts new file mode 100644 index 00000000..2a436b89 --- /dev/null +++ b/backend/src/__tests__/docker-ignore-match.test.ts @@ -0,0 +1,97 @@ +import { describe, it, expect } from 'vitest'; +import { compileDockerIgnore } from '../utils/dockerIgnoreMatch'; + +function matches(lines: string[], path: string, isDir = false): boolean { + return compileDockerIgnore(lines).matches(path, isDir); +} + +describe('compileDockerIgnore', () => { + it('ignores empty lines and comment lines', () => { + const m = compileDockerIgnore(['', ' ', '# comment', '*']); + expect(m.matches('anything.txt')).toBe(true); + }); + + it('star matches everything including dotfiles', () => { + expect(matches(['*'], 'compose.yaml')).toBe(true); + expect(matches(['*'], '.env')).toBe(true); + expect(matches(['*'], 'a/b/c.txt')).toBe(true); + }); + + it('slash-less patterns match basenames at any depth', () => { + expect(matches(['*.md'], 'README.md')).toBe(true); + expect(matches(['*.md'], 'docs/README.md')).toBe(true); + expect(matches(['*.md'], 'docs/nested/README.md')).toBe(true); + expect(matches(['*.md'], 'notes.txt')).toBe(false); + expect(matches(['foo'], 'foo')).toBe(true); + expect(matches(['foo'], 'a/foo')).toBe(true); + expect(matches(['foo'], 'foobar')).toBe(false); + }); + + it('slash-bearing patterns are anchored to the context root', () => { + expect(matches(['/foo'], 'foo')).toBe(true); + expect(matches(['/foo'], 'sub/foo')).toBe(false); + expect(matches(['a/b'], 'a/b')).toBe(true); + expect(matches(['a/b'], 'x/a/b')).toBe(false); + }); + + it('trailing slash restricts to directories', () => { + expect(matches(['build/'], 'build', true)).toBe(true); + expect(matches(['build/'], 'build')).toBe(false); + expect(matches(['build/'], 'sub/build', true)).toBe(true); + expect(matches(['build/'], 'build/output.txt')).toBe(false); + }); + + it('double-star crosses directories', () => { + expect(matches(['**/logs'], 'logs')).toBe(true); + expect(matches(['**/logs'], 'a/logs')).toBe(true); + expect(matches(['**/logs'], 'a/b/logs')).toBe(true); + expect(matches(['**/logs'], 'a/b/log.txt')).toBe(false); + expect(matches(['a/**/z'], 'a/z')).toBe(true); + expect(matches(['a/**/z'], 'a/b/c/z')).toBe(true); + }); + + it('single star does not cross directory boundaries', () => { + expect(matches(['a/*.log'], 'a/x.log')).toBe(true); + expect(matches(['a/*.log'], 'a/b/x.log')).toBe(false); + }); + + it('question mark matches exactly one character', () => { + expect(matches(['temp?'], 'temp1')).toBe(true); + expect(matches(['temp?'], 'temp10')).toBe(false); + }); + + it('character classes match within a segment', () => { + expect(matches(['[ab].txt'], 'a.txt')).toBe(true); + expect(matches(['[ab].txt'], 'b.txt')).toBe(true); + expect(matches(['[ab].txt'], 'c.txt')).toBe(false); + }); + + it('last matching pattern wins, negation re-includes at any depth', () => { + expect(matches(['*.md', '!README.md'], 'README.md')).toBe(false); + expect(matches(['*.md', '!README.md'], 'docs/README.md')).toBe(false); + expect(matches(['!README.md', '*.md'], 'README.md')).toBe(true); + expect(matches(['*', '!keep'], 'keep')).toBe(false); + expect(matches(['*', '!keep'], 'other')).toBe(true); + }); + + it('escaped hash starts a literal pattern, bare hash is a comment', () => { + expect(matches(['\\#file'], '#file')).toBe(true); + expect(matches(['\\#file'], 'other')).toBe(false); + expect(matches(['#file'], '#file')).toBe(false); + }); + + it('bare negation is a no-op', () => { + expect(matches(['!'], 'anything')).toBe(false); + }); + + it('directory matches prune via the caller: dir check on the dir path', () => { + // The matcher reports a matched directory as ignored (isDir is only + // consulted for dir-only patterns); the CALLER prunes the subtree, so + // children are never queried individually. + expect(matches(['node_modules'], 'node_modules', true)).toBe(true); + expect(matches(['node_modules'], 'node_modules')).toBe(true); + // A file that merely lives under a matched directory is not itself + // matched by the basename pattern; the caller's prune handles it. + expect(matches(['node_modules'], 'pkg/index.js')).toBe(false); + }); +}); diff --git a/backend/src/__tests__/git-project-manifest.test.ts b/backend/src/__tests__/git-project-manifest.test.ts new file mode 100644 index 00000000..86f62609 --- /dev/null +++ b/backend/src/__tests__/git-project-manifest.test.ts @@ -0,0 +1,1486 @@ +import { describe, it, expect, vi, beforeAll, afterAll } from 'vitest'; +import fs from 'fs'; +import os from 'os'; +import path from 'path'; +import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb'; +import { DatabaseService } from '../services/DatabaseService'; +import { FileSystemService } from '../services/FileSystemService'; +import { GitProjectManifestService, PROMOTION_MARKER, CANDIDATE_COMPLETE_MARKER } from '../services/GitProjectManifestService'; +import type { ComposeInputEntry, GitProjectManifest, ManifestBounds } from '../types/gitProjectManifest'; + +const BOUNDS: ManifestBounds = { + maxFiles: 10_000, + maxBytes: 512 * 1024 * 1024, + maxContextBytes: 256 * 1024 * 1024, + maxPathDepth: 64, + maxFileBytes: 10 * 1024 * 1024, +}; + +let tmpDir: string; + +beforeAll(async () => { + tmpDir = await setupTestDb(); +}); + +afterAll(() => { + cleanupTestDb(tmpDir); +}); + +function stackDir(stackName: string): string { + const dir = path.join(process.env.COMPOSE_DIR!, stackName); + fs.mkdirSync(dir, { recursive: true }); + return dir; +} + +function writeStackFile(stackName: string, rel: string, content: string): void { + const abs = path.join(stackDir(stackName), rel); + fs.mkdirSync(path.dirname(abs), { recursive: true }); + fs.writeFileSync(abs, content); +} + +function readStackFile(stackName: string, rel: string): string { + return fs.readFileSync(path.join(stackDir(stackName), rel), 'utf8'); +} + +function managedEntry(partial: Partial & { materializedPath: string }): ComposeInputEntry { + return { + sourcePath: partial.materializedPath, + materializedPath: partial.materializedPath, + role: partial.role ?? 'compose-primary', + dependencyKind: partial.dependencyKind ?? 'explicit', + ownership: partial.ownership ?? 'managed', + provenance: partial.provenance ?? 'fetch', + sensitivity: partial.sensitivity ?? 'medium', + contentSha256: partial.contentSha256 ?? null, + sizeBytes: partial.sizeBytes ?? 10, + state: partial.state ?? 'present', + deletionAuthority: partial.deletionAuthority ?? 'sencho', + note: partial.note ?? null, + }; +} + +function buildManifest(stackName: string, inputs: ComposeInputEntry[], prior: GitProjectManifest | null = null, contexts: import('../types/gitProjectManifest').BuildContextPlan[] = []): GitProjectManifest { + return GitProjectManifestService.getInstance().buildManifest({ + stackName, + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc123', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: stackName, + invocation: ['-f', 'compose.yaml', '-p', stackName], + inputs, + refusals: [], + buildContexts: contexts, + bounds: BOUNDS, + priorManifest: prior, + state: prior ? 'active' : 'active', + }); +} + +function makeClone(files: Record): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-clone-')); + for (const [rel, content] of Object.entries(files)) { + const abs = path.join(dir, rel); + fs.mkdirSync(path.dirname(abs), { recursive: true }); + fs.writeFileSync(abs, content); + } + return dir; +} + +const REPO = { repo_url: 'https://github.com/example/repo.git', branch: 'main' }; + +function seedGitSource(stackName: string): void { + DatabaseService.getInstance().upsertGitSource({ + stack_name: stackName, + repo_url: REPO.repo_url, + branch: REPO.branch, + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); +} + +function writePromotionMarker(stackName: string, marker: { + phase?: 'applying' | 'committing'; + sha: string; + manifestVersion: number; + candidateRelPath: string; + appliedRelPath: string; + affected: string[]; + introduced?: string[]; +}): void { + fs.writeFileSync( + path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER), + JSON.stringify({ schemaVersion: 2, phase: marker.phase ?? 'applying', introduced: [], ...marker }), + 'utf8', + ); +} + +describe('toPublicRefusals (audit round 9 S-1)', () => { + it('redacts high-sensitivity refusals and leaves others untouched', () => { + const svc = GitProjectManifestService.getInstance(); + const projected = svc.toPublicRefusals([ + { sourcePath: 'secrets/db.env', kind: 'missing-file', reason: 'File not found in repository: secrets/db.env', actionable: true, sensitivity: 'high' }, + { sourcePath: 'configs/app.conf', kind: 'missing-file', reason: 'File not found in repository: configs/app.conf', actionable: true, sensitivity: 'high' }, + { sourcePath: 'compose.yaml', kind: 'missing-file', reason: 'File not found in repository: compose.yaml', actionable: true, sensitivity: 'medium' }, + { sourcePath: 'web', kind: 'unsafe-context', reason: 'Build context web contains a symlink', actionable: true }, + ]); + expect(projected[0].sourcePath).toBeNull(); + expect(projected[0].reason).toBe('File not found in repository: [redacted]'); + expect(projected[0].reason).not.toContain('secrets/db.env'); + expect(projected[1].sourcePath).toBeNull(); + expect(projected[1].reason).not.toContain('configs/app.conf'); + // Medium and unspecified sensitivity pass through unchanged. + expect(projected[2]).toEqual({ sourcePath: 'compose.yaml', kind: 'missing-file', reason: 'File not found in repository: compose.yaml', actionable: true, sensitivity: 'medium' }); + expect(projected[3].sourcePath).toBe('web'); + expect(projected[3].reason).toBe('Build context web contains a symlink'); + }); +}); + +describe('readManifest / writeManifest', () => { + it('round-trips and bumps the manifest version', async () => { + const svc = GitProjectManifestService.getInstance(); + const m1 = buildManifest('roundtrip', [managedEntry({ materializedPath: 'compose.yaml' })]); + await svc.writeManifest('roundtrip', m1); + const read = await svc.readManifest('roundtrip', REPO.repo_url, REPO.branch); + expect(read).not.toBeNull(); + if (read === null || 'corrupt' in read) throw new Error('expected a manifest'); + expect(read.manifestVersion).toBe(1); + expect(read.identity.stackName).toBe('roundtrip'); + + const m2 = buildManifest('roundtrip', [managedEntry({ materializedPath: 'compose.yaml' })], read); + await svc.writeManifest('roundtrip', m2); + const read2 = await svc.readManifest('roundtrip', REPO.repo_url, REPO.branch); + if (read2 === null || 'corrupt' in read2) throw new Error('expected a manifest'); + expect(read2.manifestVersion).toBe(2); + }); + + it('rejects non-JSON manifests as corrupt', async () => { + const svc = GitProjectManifestService.getInstance(); + fs.mkdirSync(path.join(tmpDir, 'git-managed', '1', 'corrupt-json'), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, 'git-managed', '1', 'corrupt-json', 'manifest.v1.json'), 'not json', 'utf8'); + const read = await svc.readManifest('corrupt-json', REPO.repo_url, REPO.branch); + expect(read).not.toBeNull(); + expect(read && 'corrupt' in read).toBe(true); + }); + + it('rejects a hand-tampered deletionAuthority as corrupt', async () => { + const svc = GitProjectManifestService.getInstance(); + const m = buildManifest('tampered', [managedEntry({ materializedPath: 'compose.yaml' })]); + await svc.writeManifest('tampered', m); + const manifestPath = path.join(tmpDir, 'git-managed', '1', 'tampered', 'manifest.v1.json'); + const raw = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); + raw.inputs[0].deletionAuthority = 'attacker'; + fs.writeFileSync(manifestPath, JSON.stringify(raw), 'utf8'); + const read = await svc.readManifest('tampered', REPO.repo_url, REPO.branch); + expect(read && 'corrupt' in read).toBe(true); + }); + + it('rejects an identity mismatch (orphan adoption) as corrupt', async () => { + const svc = GitProjectManifestService.getInstance(); + const m = buildManifest('identity-a', [managedEntry({ materializedPath: 'compose.yaml' })]); + await svc.writeManifest('identity-a', m); + // A same-named successor pointing at a different repository must not adopt it. + const read = await svc.readManifest('identity-a', 'https://github.com/other/repo.git', 'main'); + expect(read && 'corrupt' in read).toBe(true); + }); + + it('degrades legacy context inventories without file sizes to directory granularity', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'legacy-context-sizes'; + const manifest = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })], null, [{ + repoPath: 'app', + dockerfile: null, + contextBytes: 12, + ignoredCount: 0, + dockerignoreApplied: false, + excludedFromCopy: false, + note: null, + files: [{ path: 'file.txt', sha256: 'a'.repeat(64), sizeBytes: 12 }], + }]); + await svc.writeManifest(stackName, manifest); + const manifestPath = path.join(tmpDir, 'git-managed', '1', stackName, 'manifest.v1.json'); + const raw = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); + delete raw.buildContexts[0].files[0].sizeBytes; + fs.writeFileSync(manifestPath, JSON.stringify(raw), 'utf8'); + + const read = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (read === null || 'corrupt' in read) throw new Error('expected a legacy manifest'); + expect(read.buildContexts[0].files).toEqual([]); + }); + + it('rejects empty file paths and malformed nested counters', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'deep-validation'; + const manifest = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + await svc.writeManifest(stackName, manifest); + const manifestPath = path.join(tmpDir, 'git-managed', '1', stackName, 'manifest.v1.json'); + + const emptyPath = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); + emptyPath.inputs[0].materializedPath = ''; + fs.writeFileSync(manifestPath, JSON.stringify(emptyPath), 'utf8'); + const emptyRead = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + expect(emptyRead && 'corrupt' in emptyRead).toBe(true); + + await svc.writeManifest(stackName, manifest); + const malformed = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); + malformed.bounds.maxFiles = -1; + fs.writeFileSync(manifestPath, JSON.stringify(malformed), 'utf8'); + const malformedRead = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + expect(malformedRead && 'corrupt' in malformedRead).toBe(true); + }); + + it('returns null after the managed area is deleted', async () => { + const svc = GitProjectManifestService.getInstance(); + const m = buildManifest('deleted-area', [managedEntry({ materializedPath: 'compose.yaml' })]); + await svc.writeManifest('deleted-area', m); + await svc.deleteManagedArea('deleted-area'); + expect(await svc.readManifest('deleted-area', REPO.repo_url, REPO.branch)).toBeNull(); + }); +}); + +describe('promoteGeneration', () => { + it('promotes a candidate transactionally and persists the manifest', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-ok'; + DatabaseService.getInstance().upsertGitSource({ + stack_name: stackName, + repo_url: REPO.repo_url, + branch: REPO.branch, + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + writeStackFile(stackName, 'compose.yaml', 'services:\n web:\n image: nginx:old\n'); + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx:new\n configs: [app]\nconfigs:\n app:\n file: config/app.conf\n', + 'config/app.conf': 'new config\n', + }); + const inventory = await import('../services/ComposeInputDiscoveryService').then((m) => + m.ComposeInputDiscoveryService.getInstance().discoverFromClone({ + cloneDir: clone, + composePaths: ['compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }), + ); + const inputs = inventory.inputs.filter((i) => i.ownership === 'managed'); + const manifest = buildManifest(stackName, inputs); + const candidateRel = await svc.buildCandidate( + stackName, + 'abc123', + clone, + inputs.map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), + inventory.contextCopyPlans, + BOUNDS, + ); + await svc.promoteGeneration(stackName, { sha: 'abc123', candidateRelPath: candidateRel, manifest, priorManifest: null, adoptExistingMaterializedPaths: 'all' }); + + expect(readStackFile(stackName, 'compose.yaml')).toContain('nginx:new'); + expect(readStackFile(stackName, 'config/app.conf')).toContain('new config'); + const read = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (read === null || 'corrupt' in read) throw new Error('expected a manifest'); + expect(read.resolvedRevision.commitSha).toBe('abc123'); + expect(read.generation.appliedDir).toContain('applied-abc123'); + // Marker is gone after a clean promotion. + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + // DB cache agrees with the file. + const row = DatabaseService.getInstance().getGitSource(stackName); + expect(row?.manifest_state).toBe(read.state); + }); + + it('refuses to promote a candidate without the completion marker', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-incomplete'; + writeStackFile(stackName, 'compose.yaml', 'old\n'); + const candidateRel = `generations/candidate-deadbeef`; + const candidateAbs = path.join(tmpDir, 'git-managed', '1', stackName, candidateRel); + fs.mkdirSync(candidateAbs, { recursive: true }); + fs.writeFileSync(path.join(candidateAbs, 'compose.yaml'), 'new\n'); // no completion marker + const manifest = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml', contentSha256: 'x' })]); + await expect( + svc.promoteGeneration(stackName, { sha: 'deadbeef', candidateRelPath: candidateRel, manifest, priorManifest: null }), + ).rejects.toThrow(/Candidate is incomplete/); + expect(readStackFile(stackName, 'compose.yaml')).toBe('old\n'); + }); + + it('cleanup honors deletion authority and tombstones removed paths', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-authority'; + writeStackFile(stackName, 'compose.yaml', 'v1\n'); + writeStackFile(stackName, 'stale.yaml', 'stale\n'); + writeStackFile(stackName, 'user-owned.yaml', 'user\n'); + const priorInputs = [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'stale.yaml' }), + managedEntry({ materializedPath: 'user-owned.yaml', deletionAuthority: 'user' }), + ]; + const prior = buildManifest(stackName, priorInputs); + const priorRel = `generations/applied-prior`; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + for (const p of ['compose.yaml', 'stale.yaml', 'user-owned.yaml']) { + fs.copyFileSync(path.join(stackDir(stackName), p), path.join(priorAbs, p)); + } + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const next = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })], prior); + const clone = makeClone({ 'compose.yaml': 'v2\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'sha2', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }], + [], + BOUNDS, + ); + await svc.promoteGeneration(stackName, { sha: 'sha2', candidateRelPath: candidateRel, manifest: next, priorManifest: prior }); + + expect(fs.existsSync(path.join(stackDir(stackName), 'stale.yaml'))).toBe(false); // sencho authority -> removed + expect(readStackFile(stackName, 'user-owned.yaml')).toBe('user\n'); // user authority -> untouched + const read = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (read === null || 'corrupt' in read) throw new Error('expected a manifest'); + const tombstone = read.inputs.find((i) => i.materializedPath === 'stale.yaml'); + expect(tombstone?.state).toBe('tombstoned'); + }); + + it('keeps the prior snapshot when reapplying the same commit', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-same-sha'; + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-abc123-1'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + const next = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })], prior); + const clone = makeClone({ 'compose.yaml': 'NEW\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'abc123', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }], + [], + BOUNDS, + ); + + await svc.promoteGeneration(stackName, { sha: 'abc123', candidateRelPath: candidateRel, manifest: next, priorManifest: prior }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('NEW\n'); + expect(fs.readFileSync(path.join(priorAbs, 'compose.yaml'), 'utf8')).toBe('PRIOR\n'); + const current = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (current === null || 'corrupt' in current) throw new Error('expected a manifest'); + expect(current.generation.appliedDir).toBe('generations/applied-abc123-2'); + expect(current.generation.previousDir).toBe(priorRel); + expect(fs.existsSync(path.join(priorAbs, 'compose.yaml'))).toBe(true); + }); + + it('refuses case-only managed path changes before mutating the stack', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-case-only'; + writeStackFile(stackName, 'Config.yml', 'PRIOR\n'); + const prior = buildManifest(stackName, [managedEntry({ + materializedPath: 'Config.yml', + role: 'config', + dependencyKind: 'config', + })]); + const incoming = buildManifest(stackName, [managedEntry({ + materializedPath: 'config.yml', + role: 'config', + dependencyKind: 'config', + })], prior); + const clone = makeClone({ 'config.yml': 'NEW\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'case-only', + clone, + [{ srcRel: 'config.yml', destRel: 'config.yml' }], + [], + BOUNDS, + ); + + await expect(svc.promoteGeneration(stackName, { + sha: 'case-only', + candidateRelPath: candidateRel, + manifest: incoming, + priorManifest: prior, + })).rejects.toThrow(/Case-only managed path changes/); + expect(readStackFile(stackName, 'Config.yml')).toBe('PRIOR\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('refuses to overwrite an unowned local file at an introduced path', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-introduced-collision'; + writeStackFile(stackName, 'compose.yaml', 'v1\n'); + // A local file Sencho never owned. + writeStackFile(stackName, 'local-secret.txt', 'user data\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'v1\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const incoming = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'local-secret.txt', dependencyKind: 'config', role: 'config' }), + ], prior); + const clone = makeClone({ 'compose.yaml': 'v2\n', 'local-secret.txt': 'repo version\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'sha-collide', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: 'local-secret.txt', destRel: 'local-secret.txt' }], + [], + BOUNDS, + ); + + await expect(svc.promoteGeneration(stackName, { + sha: 'sha-collide', + candidateRelPath: candidateRel, + manifest: incoming, + priorManifest: prior, + })).rejects.toThrow(/does not manage/); + + // Nothing changed on disk and no promotion marker was written. + expect(readStackFile(stackName, 'local-secret.txt')).toBe('user data\n'); + expect(readStackFile(stackName, 'compose.yaml')).toBe('v1\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('refuses to overwrite an unowned file introduced inside a root-context file set', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-rootctx-collision'; + writeStackFile(stackName, 'compose.yaml', 'v1\n'); + writeStackFile(stackName, 'src/main.go', 'user code\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'v1\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n build: .\n', + 'src/main.go': 'repo code\n', + }); + const inventory = await import('../services/ComposeInputDiscoveryService').then((m) => + m.ComposeInputDiscoveryService.getInstance().discoverFromClone({ + cloneDir: clone, + composePaths: ['compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }), + ); + const incoming = buildManifest(stackName, inventory.inputs.filter((i) => i.ownership === 'managed'), prior, inventory.buildContexts); + const candidateRel = await svc.buildCandidate( + stackName, + 'sha-rootctx', + clone, + inventory.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null) + .map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), + inventory.contextCopyPlans, + BOUNDS, + ); + + await expect(svc.promoteGeneration(stackName, { + sha: 'sha-rootctx', + candidateRelPath: candidateRel, + manifest: incoming, + priorManifest: prior, + })).rejects.toThrow(/does not manage/); + expect(readStackFile(stackName, 'src/main.go')).toBe('user code\n'); + }); + + it('refuses an unowned collision on a pre-manifest stack even with no prior manifest (audit round 9 B-1)', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-premanifest-collision'; + // An existing pre-manifest stack: the legacy compose file plus a local + // file Sencho never owned. + writeStackFile(stackName, 'compose.yaml', 'legacy v1\n'); + writeStackFile(stackName, 'configs/app.json', 'local user data\n'); + + const incoming = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'configs/app.json', dependencyKind: 'config', role: 'config' }), + ]); + const clone = makeClone({ 'compose.yaml': 'v2\n', 'configs/app.json': 'repo version\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'sha-pre', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: 'configs/app.json', destRel: 'configs/app.json' }], + [], + BOUNDS, + ); + + // The allowlist covers only the legacy-owned compose file; the local + // config file must be refused and preserved byte-for-byte. + await expect(svc.promoteGeneration(stackName, { + sha: 'sha-pre', + candidateRelPath: candidateRel, + manifest: incoming, + priorManifest: null, + adoptExistingMaterializedPaths: ['compose.yaml'], + })).rejects.toThrow(/does not manage/); + expect(readStackFile(stackName, 'configs/app.json')).toBe('local user data\n'); + expect(readStackFile(stackName, 'compose.yaml')).toBe('legacy v1\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('adopts exactly the allowlisted legacy paths on a pre-manifest stack (audit round 9 B-1)', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-premanifest-adopt'; + writeStackFile(stackName, 'compose.yaml', 'legacy v1\n'); + writeStackFile(stackName, '.env', 'SYNC=1\n'); + + const incoming = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + { + sourcePath: null, + materializedPath: '.env', + role: 'env', + dependencyKind: 'sync-env', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: 'a'.repeat(64), + sizeBytes: 4, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }, + ]); + const clone = makeClone({ 'compose.yaml': 'v2\n', '.env': 'SYNC=2\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'sha-adopt', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: '.env', destRel: '.env' }], + [], + BOUNDS, + ); + + await svc.promoteGeneration(stackName, { + sha: 'sha-adopt', + candidateRelPath: candidateRel, + manifest: incoming, + priorManifest: null, + adoptExistingMaterializedPaths: ['compose.yaml', '.env'], + }); + expect(readStackFile(stackName, 'compose.yaml')).toBe('v2\n'); + expect(readStackFile(stackName, '.env')).toBe('SYNC=2\n'); + }); + + it('allows the synced stack-root .env to adopt an existing file when sync_env is enabled', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-sync-env-adoption'; + writeStackFile(stackName, 'compose.yaml', 'v1\n'); + writeStackFile(stackName, '.env', 'EXISTING=1\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'v1\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const syncEnvEntry: ComposeInputEntry = { + sourcePath: null, + materializedPath: '.env', + role: 'env', + dependencyKind: 'sync-env', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: 'a'.repeat(64), + sizeBytes: 4, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }; + const incoming = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + syncEnvEntry, + ], prior); + const clone = makeClone({ 'compose.yaml': 'v2\n', '.env': 'NEW=1\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'sha-syncenv', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: '.env', destRel: '.env' }], + [], + BOUNDS, + ); + + await svc.promoteGeneration(stackName, { + sha: 'sha-syncenv', + candidateRelPath: candidateRel, + manifest: incoming, + priorManifest: prior, + }); + expect(readStackFile(stackName, '.env')).toBe('NEW=1\n'); + }); +}); + +describe('sweepManagedArea (crash recovery)', () => { + it('restores the previous applied generation when the marker matches the stack dir', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-restore'; + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + writeStackFile(stackName, 'app.env', 'A=1\n'); + const prior = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'app.env', dependencyKind: 'env_file', role: 'env', sensitivity: 'high' }), + ]); + const priorRel = `generations/applied-prior`; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + fs.writeFileSync(path.join(priorAbs, 'app.env'), 'A=1\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + // Crash mid-promotion: candidate written, compose.yaml already swapped. + const candidateRel = `generations/candidate-crash`; + const candidateAbs = path.join(tmpDir, 'git-managed', '1', stackName, candidateRel); + fs.mkdirSync(candidateAbs, { recursive: true }); + fs.writeFileSync(path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER), 'crash'); + fs.writeFileSync(path.join(candidateAbs, 'compose.yaml'), 'NEW\n'); + writeStackFile(stackName, 'compose.yaml', 'NEW\n'); + writePromotionMarker(stackName, { + sha: 'crash', + manifestVersion: prior.manifestVersion + 1, + candidateRelPath: candidateRel, + appliedRelPath: 'generations/applied-crash-2', + affected: ['app.env', 'compose.yaml'], + }); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('declines to restore over a hand-repaired stack dir and flags migration_required', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-decline'; + DatabaseService.getInstance().upsertGitSource({ + stack_name: stackName, + repo_url: REPO.repo_url, + branch: REPO.branch, + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = `generations/applied-prior`; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const candidateRel = `generations/candidate-crash2`; + const candidateAbs = path.join(tmpDir, 'git-managed', '1', stackName, candidateRel); + fs.mkdirSync(candidateAbs, { recursive: true }); + fs.writeFileSync(path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER), 'crash2'); + fs.writeFileSync(path.join(candidateAbs, 'compose.yaml'), 'NEW\n'); + writeStackFile(stackName, 'compose.yaml', 'OPERATOR FIXED ME\n'); // hand-repaired + writePromotionMarker(stackName, { + sha: 'crash2', + manifestVersion: prior.manifestVersion + 1, + candidateRelPath: candidateRel, + appliedRelPath: 'generations/applied-crash2-2', + affected: ['compose.yaml'], + }); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('OPERATOR FIXED ME\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + const row = DatabaseService.getInstance().getGitSource(stackName); + expect(row?.manifest_state).toBe('migration_required'); + }); + + it('restores through the candidate-to-applied rename window', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-rename-window'; + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const candidateRel = 'generations/candidate-rename'; + const appliedRel = 'generations/applied-rename-2'; + const appliedAbs = path.join(tmpDir, 'git-managed', '1', stackName, appliedRel); + fs.mkdirSync(appliedAbs, { recursive: true }); + fs.writeFileSync(path.join(appliedAbs, CANDIDATE_COMPLETE_MARKER), 'rename'); + fs.writeFileSync(path.join(appliedAbs, 'compose.yaml'), 'NEW\n'); + writeStackFile(stackName, 'compose.yaml', 'NEW\n'); + writePromotionMarker(stackName, { + sha: 'rename', + manifestVersion: prior.manifestVersion + 1, + candidateRelPath: candidateRel, + appliedRelPath: appliedRel, + affected: ['compose.yaml'], + }); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('finalizes a committed manifest instead of rolling it back', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-post-manifest'; + seedGitSource(stackName); + writeStackFile(stackName, 'compose.yaml', 'NEW\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + prior.generation.appliedDir = 'generations/applied-prior'; + const incoming = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })], prior); + incoming.resolvedRevision.commitSha = 'committed'; + const appliedRel = `generations/applied-committed-${incoming.manifestVersion}`; + const appliedAbs = path.join(tmpDir, 'git-managed', '1', stackName, appliedRel); + fs.mkdirSync(appliedAbs, { recursive: true }); + fs.writeFileSync(path.join(appliedAbs, CANDIDATE_COMPLETE_MARKER), 'committed'); + fs.writeFileSync(path.join(appliedAbs, 'compose.yaml'), 'NEW\n'); + incoming.generation = { + candidateDir: 'generations/candidate-committed', + appliedDir: appliedRel, + previousDir: prior.generation.appliedDir, + }; + await svc.writeManifest(stackName, incoming); + writePromotionMarker(stackName, { + phase: 'committing', + sha: 'committed', + manifestVersion: incoming.manifestVersion, + candidateRelPath: incoming.generation.candidateDir, + appliedRelPath: appliedRel, + affected: ['compose.yaml'], + }); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('NEW\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + const read = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (read === null || 'corrupt' in read) throw new Error('expected committed manifest'); + expect(read.manifestVersion).toBe(incoming.manifestVersion); + const row = DatabaseService.getInstance().getGitSource(stackName); + expect(row?.manifest_version).toBe(incoming.manifestVersion); + expect(row?.manifest_state).toBe(incoming.state); + expect(row?.manifest_generation).toBe(appliedRel); + }); + + it('rolls back a committing marker while the prior manifest is still authoritative', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-pre-manifest'; + seedGitSource(stackName); + writeStackFile(stackName, 'compose.yaml', 'NEW\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const appliedRel = `generations/applied-committing-${prior.manifestVersion + 1}`; + const appliedAbs = path.join(tmpDir, 'git-managed', '1', stackName, appliedRel); + fs.mkdirSync(appliedAbs, { recursive: true }); + fs.writeFileSync(path.join(appliedAbs, CANDIDATE_COMPLETE_MARKER), 'committing'); + fs.writeFileSync(path.join(appliedAbs, 'compose.yaml'), 'NEW\n'); + writePromotionMarker(stackName, { + phase: 'committing', + sha: 'committing', + manifestVersion: prior.manifestVersion + 1, + candidateRelPath: 'generations/candidate-committing', + appliedRelPath: appliedRel, + affected: ['compose.yaml'], + }); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n'); + const row = DatabaseService.getInstance().getGitSource(stackName); + expect(row?.manifest_version).toBe(prior.manifestVersion); + expect(row?.manifest_state).toBe(prior.state); + expect(row?.manifest_generation).toBe(priorRel); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('detects a hand edit in the former marker batch tail', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-batch-tail'; + seedGitSource(stackName); + writeStackFile(stackName, 'compose.yaml', 'NEW\n'); + writeStackFile(stackName, 'app.env', 'OPERATOR\n'); + const prior = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'app.env', dependencyKind: 'env_file', role: 'env' }), + ]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + fs.writeFileSync(path.join(priorAbs, 'app.env'), 'A=1\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + const candidateRel = 'generations/candidate-tail'; + const candidateAbs = path.join(tmpDir, 'git-managed', '1', stackName, candidateRel); + fs.mkdirSync(candidateAbs, { recursive: true }); + fs.writeFileSync(path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER), 'tail'); + fs.writeFileSync(path.join(candidateAbs, 'compose.yaml'), 'NEW\n'); + fs.writeFileSync(path.join(candidateAbs, 'app.env'), 'A=2\n'); + writePromotionMarker(stackName, { + sha: 'tail', + manifestVersion: prior.manifestVersion + 1, + candidateRelPath: candidateRel, + appliedRelPath: 'generations/applied-tail-2', + affected: ['app.env', 'compose.yaml'], + }); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe('NEW\n'); + expect(readStackFile(stackName, 'app.env')).toBe('OPERATOR\n'); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + expect(DatabaseService.getInstance().getGitSource(stackName)?.manifest_state).toBe('migration_required'); + }); + + it('drops the managed area when the stack no longer exists', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-orphan'; + await svc.writeManifest(stackName, buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })])); + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: false }); + expect(await svc.readManifest(stackName, REPO.repo_url, REPO.branch)).toBeNull(); + }); +}); + +describe('buildMigratedManifest', () => { + it('builds a conservative single-file migrated manifest', async () => { + const stackName = 'migrate-single'; + writeStackFile(stackName, 'compose.yaml', 'services: {}\n'); + const svc = GitProjectManifestService.getInstance(); + const manifest = await svc.buildMigratedManifest(stackName, { + ...REPO, + sync_env: false, + applied_deploy_spec: null, + }); + expect(manifest.state).toBe('migrated'); + expect(manifest.inputs.some((i) => i.materializedPath === 'compose.yaml' && i.deletionAuthority === 'sencho')).toBe(true); + await svc.writeManifest(stackName, manifest); + const read = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + expect(read).not.toBeNull(); + expect(read && 'corrupt' in read).toBe(false); + }); + + it('grants sencho authority only for spec files and grants none to contextDir contents', async () => { + const stackName = 'migrate-multi'; + writeStackFile(stackName, 'compose.yaml', 'base\n'); + writeStackFile(stackName, 'deploy/prod.yaml', 'prod\n'); + writeStackFile(stackName, 'deploy/settings.env', 'X=1\n'); + writeStackFile(stackName, '.env', 'A=1\n'); + const svc = GitProjectManifestService.getInstance(); + const manifest = await svc.buildMigratedManifest(stackName, { + ...REPO, + sync_env: true, + applied_deploy_spec: { files: ['compose.yaml', 'deploy/prod.yaml'], contextDir: 'deploy' }, + }); + const byPath = (p: string) => manifest.inputs.find((i) => i.materializedPath === p); + expect(byPath('compose.yaml')?.deletionAuthority).toBe('sencho'); + expect(byPath('deploy/prod.yaml')?.deletionAuthority).toBe('sencho'); + expect(byPath('.env')?.deletionAuthority).toBe('sencho'); + expect(byPath('.env')?.dependencyKind).toBe('sync-env'); + // The contextDir subtree is a single note entry with no deletion authority. + const dirNote = manifest.inputs.find((i) => i.materializedPath === 'deploy' && i.role === 'build-context'); + expect(dirNote?.deletionAuthority).toBe('none'); + expect(manifest.generation.appliedDir).toContain('applied-migration'); + // Snapshot dir exists for crash recovery before the first fresh pull. + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, manifest.generation.appliedDir, 'compose.yaml'))).toBe(true); + }); +}); + +describe('exportForDetach', () => { + it('returns the rendered yaml when valid', async () => { + const svc = GitProjectManifestService.getInstance(); + const out = await svc.exportForDetach('detach-ok', async () => 'services:\n web:\n image: nginx\n'); + expect(out).toContain('image: nginx'); + }); + + it('throws on empty render output', async () => { + const svc = GitProjectManifestService.getInstance(); + await expect(svc.exportForDetach('detach-empty', async () => '')).rejects.toThrow(/empty/); + }); + + it('throws on invalid yaml render output', async () => { + const svc = GitProjectManifestService.getInstance(); + await expect(svc.exportForDetach('detach-bad', async () => 'services: [unclosed\n')).rejects.toThrow(/parse/); + }); +}); + +describe('detach crash recovery', () => { + it('restores the durable snapshot when the managed area was staged before a crash', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'detach-crash'; + const original = Buffer.from('services:\n web:\n image: nginx:old\n'); + writeStackFile(stackName, 'compose.yaml', original.toString('utf8')); + await svc.writeManifest(stackName, buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })])); + await svc.prepareDetachRecovery(stackName, REPO.repo_url, REPO.branch, [{ path: 'compose.yaml', existed: true, content: original }]); + writeStackFile(stackName, 'compose.yaml', 'services:\n web:\n image: nginx:new\n'); + expect(await svc.stageManagedAreaForDetach(stackName)).toBe(true); + + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + + expect(readStackFile(stackName, 'compose.yaml')).toBe(original.toString('utf8')); + const restored = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + expect(restored).not.toBeNull(); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', `.detach-${stackName}`))).toBe(false); + }); + + it('round-trips snapshots larger than the former fixed entry limit', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'detach-many-files'; + stackDir(stackName); + const snapshots = Array.from({ length: 17 }, (_, index) => ({ + path: `override-${index}.yaml`, + existed: false as const, + content: null, + })); + await svc.prepareDetachRecovery(stackName, REPO.repo_url, REPO.branch, snapshots); + + expect(await svc.recoverInterruptedDetach(stackName, REPO.repo_url, REPO.branch)).toBe(true); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName))).toBe(false); + }); + + it('rejects duplicate snapshot paths before writing a recovery marker', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'detach-duplicate'; + await expect(svc.prepareDetachRecovery(stackName, REPO.repo_url, REPO.branch, [ + { path: 'compose.yaml', existed: false, content: null }, + { path: 'compose.yaml', existed: false, content: null }, + ])).rejects.toThrow(/duplicate paths/); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName))).toBe(false); + }); + + it('does not treat an unreadable staged detach as absent', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'detach-stage-io'; + const staged = path.join(tmpDir, 'git-managed', '1', `.detach-${stackName}`); + const originalAccess = fs.promises.access.bind(fs.promises); + const accessSpy = vi.spyOn(fs.promises, 'access').mockImplementation(async (...args: Parameters) => { + if (path.resolve(String(args[0])) === path.resolve(staged)) { + throw Object.assign(new Error('permission denied'), { code: 'EACCES' }); + } + return originalAccess(...args); + }); + try { + await expect(svc.recoverInterruptedDetach(stackName, REPO.repo_url, REPO.branch)).rejects.toThrow(/permission denied/); + } finally { + accessSpy.mockRestore(); + } + }); + + it('reports when a staged detach has no recovery snapshot to restore', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'detach-missing-snapshot'; + await svc.writeManifest(stackName, buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })])); + expect(await svc.stageManagedAreaForDetach(stackName)).toBe(true); + + expect(await svc.rollbackStagedDetach(stackName, REPO.repo_url, REPO.branch)).toBe(false); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName))).toBe(true); + }); +}); + +describe('FileSystemService interaction', () => { + it('promotion invalidates the stack file roots', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-invalidate'; + writeStackFile(stackName, 'compose.yaml', 'v1\n'); + const clone = makeClone({ 'compose.yaml': 'v2\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'invalidate', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }], + [], + BOUNDS, + ); + const manifest = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + await svc.promoteGeneration(stackName, { sha: 'invalidate', candidateRelPath: candidateRel, manifest, priorManifest: null, adoptExistingMaterializedPaths: 'all' }); + expect(readStackFile(stackName, 'compose.yaml')).toBe('v2\n'); + expect(await FileSystemService.getInstance().getStackContent(stackName)).toBe('v2\n'); + }); +}); + +describe('promoteGeneration mid-write failure recovery', () => { + it('restores the previous generation and manifest when a write fails mid-promotion', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-midfail'; + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + writeStackFile(stackName, 'app.env', 'A=1\n'); + const prior = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'app.env', dependencyKind: 'env_file', role: 'env', sensitivity: 'high' }), + ]); + const priorRel = `generations/applied-prior`; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + fs.writeFileSync(path.join(priorAbs, 'app.env'), 'A=1\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + // Force the failure at PROMOTION time (a rejected stack write after + // the marker was written), not during candidate staging. + const clone = makeClone({ 'compose.yaml': 'NEW\n', 'app.env': 'B=2\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'midfail', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: 'app.env', destRel: 'app.env' }], + [], + BOUNDS, + ); + const next = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml', contentSha256: 'x' }), + managedEntry({ materializedPath: 'app.env', dependencyKind: 'env_file', role: 'env', sensitivity: 'high' }), + ], prior); + // Fail exactly the promotion's first write; the restore path's writes + // must succeed for the recovery assertions below. + const saveSpy = vi.spyOn(FileSystemService.prototype, 'saveStackContent').mockRejectedValueOnce(new Error('simulated disk failure')); + try { + await expect( + svc.promoteGeneration(stackName, { sha: 'midfail', candidateRelPath: candidateRel, manifest: next, priorManifest: prior }), + ).rejects.toThrow(/simulated disk failure/); + } finally { + saveSpy.mockRestore(); + } + + // The prior generation and the prior manifest FILE are both restored, + // so a subsequent apply reads hashes that match the disk. + expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n'); + const restored = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (restored === null || 'corrupt' in restored) throw new Error('expected a manifest'); + expect(restored.manifestVersion).toBe(prior.manifestVersion); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('restores files, manifest, and cache when the manifest commit write fails', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-commit-fail'; + seedGitSource(stackName); + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = 'generations/applied-prior'; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + DatabaseService.getInstance().setGitSourceManifestState(stackName, prior.manifestVersion, prior.state, priorRel); + + const clone = makeClone({ 'compose.yaml': 'NEW\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'commit-fail', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }], + [], + BOUNDS, + ); + const next = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })], prior); + const manifestWriteSpy = vi.spyOn(svc, 'writeManifest').mockRejectedValueOnce(new Error('simulated manifest write failure')); + try { + await expect(svc.promoteGeneration(stackName, { + sha: 'commit-fail', + candidateRelPath: candidateRel, + manifest: next, + priorManifest: prior, + })).rejects.toThrow(/simulated manifest write failure/); + } finally { + manifestWriteSpy.mockRestore(); + } + + expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n'); + const restored = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (restored === null || 'corrupt' in restored) throw new Error('expected the prior manifest'); + expect(restored.manifestVersion).toBe(prior.manifestVersion); + const row = DatabaseService.getInstance().getGitSource(stackName); + expect(row?.manifest_version).toBe(prior.manifestVersion); + expect(row?.manifest_state).toBe(prior.state); + expect(row?.manifest_generation).toBe(priorRel); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); + + it('cleans protected files after a failed first promotion', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-first-fail'; + seedGitSource(stackName); + stackDir(stackName); + const clone = makeClone({ 'compose.yaml': 'NEW\n', 'new.txt': 'NEW FILE\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'first-fail', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: 'new.txt', destRel: 'new.txt' }], + [], + BOUNDS, + ); + const manifest = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml' }), + managedEntry({ materializedPath: 'new.txt', role: 'other', dependencyKind: 'config' }), + ]); + const writeSpy = vi.spyOn(FileSystemService.prototype, 'writeStackFile').mockRejectedValueOnce(new Error('simulated second write failure')); + try { + await expect( + svc.promoteGeneration(stackName, { sha: 'first-fail', candidateRelPath: candidateRel, manifest, priorManifest: null, adoptExistingMaterializedPaths: 'all' }), + ).rejects.toThrow(/simulated second write failure/); + } finally { + writeSpy.mockRestore(); + } + + expect(fs.existsSync(path.join(stackDir(stackName), 'compose.yaml'))).toBe(false); + expect(fs.existsSync(path.join(stackDir(stackName), 'new.txt'))).toBe(false); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + expect(DatabaseService.getInstance().getGitSource(stackName)?.manifest_state).toBe('migration_required'); + }); + + it('treats a corrupt promotion marker as recovery-required, not a clean slate', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-corrupt-marker'; + writeStackFile(stackName, 'compose.yaml', 'ANY\n'); + DatabaseService.getInstance().upsertGitSource({ + stack_name: stackName, + repo_url: REPO.repo_url, + branch: REPO.branch, + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + fs.mkdirSync(path.join(tmpDir, 'git-managed', '1', stackName), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER), '{"v":3 torn', 'utf8'); + await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true }); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + expect(DatabaseService.getInstance().getGitSource(stackName)?.manifest_state).toBe('migration_required'); + }); + + it('rejects a null candidate path without attempting snapshot recovery', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-null-candidate'; + writeStackFile(stackName, 'compose.yaml', 'ANY\n'); + fs.mkdirSync(path.join(tmpDir, 'git-managed', '1', stackName), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER), JSON.stringify({ + schemaVersion: 2, + phase: 'applying', + sha: 'abc123', + manifestVersion: 1, + candidateRelPath: null, + appliedRelPath: 'generations/applied-abc123-1', + affected: ['compose.yaml'], + introduced: ['compose.yaml'], + }), 'utf8'); + const stateSpy = vi.spyOn(DatabaseService.getInstance(), 'setGitSourceManifestState'); + try { + await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true })).resolves.toBeUndefined(); + expect(stateSpy).toHaveBeenCalledWith(stackName, null, 'migration_required', null); + } finally { + stateSpy.mockRestore(); + } + }); + + it('retains a corrupt promotion marker when persisting recovery state fails', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-state-failure'; + writeStackFile(stackName, 'compose.yaml', 'ANY\n'); + const markerPath = path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER); + fs.mkdirSync(path.dirname(markerPath), { recursive: true }); + fs.writeFileSync(markerPath, '{"broken":', 'utf8'); + const stateSpy = vi.spyOn(DatabaseService.getInstance(), 'setGitSourceManifestState').mockImplementationOnce(() => { + throw new Error('database unavailable'); + }); + try { + await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true })).rejects.toThrow(/database unavailable/); + expect(fs.existsSync(markerPath)).toBe(true); + } finally { + stateSpy.mockRestore(); + } + }); + + it('retains the promotion marker when a recovery snapshot cannot be inspected', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'sweep-snapshot-io'; + writeStackFile(stackName, 'compose.yaml', 'ANY\n'); + const markerPath = path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER); + fs.mkdirSync(path.dirname(markerPath), { recursive: true }); + writePromotionMarker(stackName, { + sha: 'snapshot-io', + manifestVersion: 1, + candidateRelPath: 'generations/candidate-snapshot-io', + appliedRelPath: 'generations/applied-snapshot-io-1', + affected: ['compose.yaml'], + }); + const originalAccess = fs.promises.access.bind(fs.promises); + const accessSpy = vi.spyOn(fs.promises, 'access').mockImplementation(async (...args: Parameters) => { + if (String(args[0]).endsWith(CANDIDATE_COMPLETE_MARKER)) { + throw Object.assign(new Error('permission denied'), { code: 'EACCES' }); + } + return originalAccess(...args); + }); + try { + await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true })).rejects.toThrow(/permission denied/); + expect(fs.existsSync(markerPath)).toBe(true); + } finally { + accessSpy.mockRestore(); + } + }); + + it('does not treat an unreadable introduced path as absent during restore', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'restore-path-error'; + writeStackFile(stackName, 'new.txt', 'NEW\n'); + const pathKindSpy = vi.spyOn(FileSystemService.prototype, 'pathKind').mockRejectedValueOnce(new Error('permission denied')); + const deleteSpy = vi.spyOn(FileSystemService.prototype, 'deleteStackPath'); + try { + await expect(svc.restorePreviousGeneration(stackName, { + priorManifest: null, + incoming: { introducedPaths: ['new.txt'] }, + })).resolves.toBe(false); + expect(deleteSpy).not.toHaveBeenCalled(); + expect(readStackFile(stackName, 'new.txt')).toBe('NEW\n'); + } finally { + pathKindSpy.mockRestore(); + deleteSpy.mockRestore(); + } + }); +}); + +describe('byte-exact materialization (audit C-1)', () => { + it('promotes binary files byte-identically and keeps the divergence guard silent', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'promote-binary'; + writeStackFile(stackName, 'compose.yaml', 'services: {}\n'); + // A real PNG header + arbitrary binary payload. + const pngBytes = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0xff, 0xd8, 0xc0, 0x80, 0x00, 0x01, 0x02, 0xfe, 0xfd]); + const clone = makeClone({ + 'compose.yaml': 'services:\n web:\n image: nginx\n configs: [bin]\nconfigs:\n bin:\n file: blob.bin\n', + 'blob.bin': pngBytes.toString('latin1'), + }); + // makeClone writes via fs.writeFileSync(utf8 default); reconstruct the exact bytes on disk. + fs.writeFileSync(path.join(clone, 'blob.bin'), pngBytes); + + const inventory = await import('../services/ComposeInputDiscoveryService').then((m) => + m.ComposeInputDiscoveryService.getInstance().discoverFromClone({ + cloneDir: clone, + composePaths: ['compose.yaml'], + contextDir: null, + bounds: BOUNDS, + }), + ); + const inputs = inventory.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null); + const blobEntry = inputs.find((i) => i.materializedPath === 'blob.bin'); + expect(blobEntry?.contentSha256).toBeTruthy(); + + const manifest = buildManifest(stackName, inputs); + const candidateRel = await svc.buildCandidate( + stackName, + 'bin-sha', + clone, + inputs.map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), + inventory.contextCopyPlans, + BOUNDS, + ); + await svc.promoteGeneration(stackName, { sha: 'bin-sha', candidateRelPath: candidateRel, manifest, priorManifest: null, adoptExistingMaterializedPaths: 'all' }); + + // The stack-dir file is byte-identical to the clone source. + const onDisk = fs.readFileSync(path.join(stackDir(stackName), 'blob.bin')); + expect(onDisk.equals(pngBytes)).toBe(true); + // The divergence guard hashes the same bytes the manifest recorded. + expect(await svc.hashStackFile(stackName, 'blob.bin')).toBe(blobEntry?.contentSha256); + }); +}); + +describe('partial manifest state', () => { + it('builds a partial-state manifest when refusals exist and the summary surfaces them', () => { + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'partial-state', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'partial-state', + invocation: ['-f', 'compose.yaml', '-p', 'partial-state'], + inputs: [managedEntry({ materializedPath: 'compose.yaml' })], + refusals: [{ sourcePath: 'x.yaml', kind: 'url-include', reason: 'x', actionable: false }], + buildContexts: [], + bounds: BOUNDS, + priorManifest: null, + state: 'partial', + }); + expect(manifest.state).toBe('partial'); + expect(svc.summaryFrom(manifest).state).toBe('partial'); + // Tolerated (non-actionable) refusals reach the manifest; actionable + // ones abort the pull before a manifest is built. + expect(svc.summaryFrom(manifest).refused).toHaveLength(0); + }); +}); + +describe('exact-generation restore (audit round 2 C-1)', () => { + it('removes files the failed promotion introduced', async () => { + const svc = GitProjectManifestService.getInstance(); + const stackName = 'restore-exact'; + writeStackFile(stackName, 'compose.yaml', 'PRIOR\n'); + const prior = buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]); + const priorRel = `generations/applied-prior`; + const priorAbs = path.join(tmpDir, 'git-managed', '1', stackName, priorRel); + fs.mkdirSync(priorAbs, { recursive: true }); + fs.writeFileSync(path.join(priorAbs, 'compose.yaml'), 'PRIOR\n'); + prior.generation.appliedDir = priorRel; + await svc.writeManifest(stackName, prior); + + // The incoming revision introduces new.txt and updates compose.yaml. + const clone = makeClone({ 'compose.yaml': 'NEW\n', 'new.txt': 'fresh\n' }); + const candidateRel = await svc.buildCandidate( + stackName, + 'exact', + clone, + [{ srcRel: 'compose.yaml', destRel: 'compose.yaml' }, { srcRel: 'new.txt', destRel: 'new.txt' }], + [], + BOUNDS, + ); + const next = buildManifest(stackName, [ + managedEntry({ materializedPath: 'compose.yaml', contentSha256: 'x' }), + managedEntry({ materializedPath: 'new.txt', contentSha256: 'y' }), + ], prior); + const saveSpy = vi.spyOn(FileSystemService.prototype, 'saveStackContent').mockRejectedValueOnce(new Error('simulated disk failure')); + try { + await expect( + svc.promoteGeneration(stackName, { sha: 'exact', candidateRelPath: candidateRel, manifest: next, priorManifest: prior }), + ).rejects.toThrow(/simulated disk failure/); + } finally { + saveSpy.mockRestore(); + } + + // The prior generation is exact: compose.yaml restored, new.txt removed. + expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n'); + expect(fs.existsSync(path.join(stackDir(stackName), 'new.txt'))).toBe(false); + const restored = await svc.readManifest(stackName, REPO.repo_url, REPO.branch); + if (restored === null || 'corrupt' in restored) throw new Error('expected a manifest'); + expect(restored.manifestVersion).toBe(prior.manifestVersion); + expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false); + }); +}); + +describe('build-context file-level ownership (audit round 2 C-2)', () => { + it('removes context files deleted upstream and detects local edits', async () => { + const svc = GitProjectManifestService.getInstance(); + const { ComposeInputDiscoveryService } = await import('../services/ComposeInputDiscoveryService'); + const discovery = ComposeInputDiscoveryService.getInstance(); + const stackName = 'context-reconcile'; + writeStackFile(stackName, 'compose.yaml', 'services: {}\n'); + + // Revision 1: context web with keep.txt + drop.txt. + const clone1 = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n', + 'web/keep.txt': 'keep\n', + 'web/drop.txt': 'drop\n', + }); + const inv1 = await discovery.discoverFromClone({ cloneDir: clone1, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + const managed1 = inv1.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null); + const manifest1 = buildManifest(stackName, managed1, null, inv1.buildContexts); + const fileList1 = managed1.filter((i) => i.dependencyKind !== 'build-context'); + const cand1 = await svc.buildCandidate(stackName, 'rev1', clone1, fileList1.map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), inv1.contextCopyPlans, BOUNDS); + await svc.promoteGeneration(stackName, { sha: 'rev1', candidateRelPath: cand1, manifest: manifest1, priorManifest: null, adoptExistingMaterializedPaths: 'all' }); + expect(fs.existsSync(path.join(stackDir(stackName), 'web', 'drop.txt'))).toBe(true); + + // Revision 2: drop.txt removed upstream. + const clone2 = makeClone({ + 'compose.yaml': 'services:\n web:\n build:\n context: web\n', + 'web/keep.txt': 'keep\n', + }); + const inv2 = await discovery.discoverFromClone({ cloneDir: clone2, composePaths: ['compose.yaml'], contextDir: null, bounds: BOUNDS }); + const managed2 = inv2.inputs.filter((i) => i.ownership === 'managed' && i.materializedPath !== null); + const manifest2 = buildManifest(stackName, managed2, manifest1, inv2.buildContexts); + const fileList2 = managed2.filter((i) => i.dependencyKind !== 'build-context'); + const cand2 = await svc.buildCandidate(stackName, 'rev2', clone2, fileList2.map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), inv2.contextCopyPlans, BOUNDS); + await svc.promoteGeneration(stackName, { sha: 'rev2', candidateRelPath: cand2, manifest: manifest2, priorManifest: manifest1 }); + expect(fs.existsSync(path.join(stackDir(stackName), 'web', 'drop.txt'))).toBe(false); + expect(fs.existsSync(path.join(stackDir(stackName), 'web', 'keep.txt'))).toBe(true); + + // A local edit inside the context is divergence on the next apply path. + const diverged = await svc.verifyContextOnDisk(stackName, manifest2.buildContexts[0]); + expect(diverged).toEqual([]); + fs.writeFileSync(path.join(stackDir(stackName), 'web', 'keep.txt'), 'locally edited\n'); + const divergedAfter = await svc.verifyContextOnDisk(stackName, manifest2.buildContexts[0]); + expect(divergedAfter.some((p) => p.includes('keep.txt'))).toBe(true); + }); +}); diff --git a/backend/src/__tests__/git-source-routes.test.ts b/backend/src/__tests__/git-source-routes.test.ts index afa565d3..90088b2b 100644 --- a/backend/src/__tests__/git-source-routes.test.ts +++ b/backend/src/__tests__/git-source-routes.test.ts @@ -18,6 +18,7 @@ import fs from 'fs'; import path from 'path'; import { setupTestDb, cleanupTestDb, TEST_USERNAME, TEST_JWT_SECRET } from './helpers/setupTestDb'; import { DatabaseService } from '../services/DatabaseService'; +import { ComposeService } from '../services/ComposeService'; import { GitSourceService } from '../services/GitSourceService'; // ── Hoisted mocks (must come before importing the app) ───────────────── @@ -299,6 +300,49 @@ describe('GET /api/stacks/:stackName/git-source', () => { expect(res.body.repo_url).toBe('https://github.com/example/repo.git'); expect(res.body.linked).toBeUndefined(); }); + + it('redacts high-sensitivity refusal paths from the summary projection (audit round 9 S-1)', async () => { + const composeDir = process.env.COMPOSE_DIR!; + fs.mkdirSync(path.join(composeDir, 'linked-redacted'), { recursive: true }); + fs.writeFileSync(path.join(composeDir, 'linked-redacted', 'compose.yaml'), 'services:\n x:\n image: nginx\n'); + seedGitSource('linked-redacted'); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'linked-redacted', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc123', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'linked-redacted', + invocation: ['-f', 'compose.yaml', '-p', 'linked-redacted'], + inputs: [], + refusals: [ + { sourcePath: 'secrets/db.env', kind: 'missing-file', reason: 'File not found in repository: secrets/db.env', actionable: true, sensitivity: 'high' }, + { sourcePath: 'compose.yaml', kind: 'missing-file', reason: 'File not found in repository: compose.yaml', actionable: true, sensitivity: 'medium' }, + ], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'partial', + }); + await svc.writeManifest('linked-redacted', manifest); + + const res = await request(app) + .get('/api/stacks/linked-redacted/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + const serialized = JSON.stringify(res.body); + expect(serialized).not.toContain('secrets/db.env'); + const high = res.body.manifest.refused.find( + (r: { kind: string; sourcePath: string | null; reason: string }) => r.kind === 'missing-file' && r.sourcePath === null, + ) as { reason: string } | undefined; + expect(high).toBeTruthy(); + expect(high?.reason).toContain('[redacted]'); + // Non-sensitive refusals keep their actionable path. + expect(res.body.manifest.refused.some((r: { sourcePath: string }) => r.sourcePath === 'compose.yaml')).toBe(true); + }); }); describe('PUT /api/stacks/:stackName/git-source: multi-file selection', () => { @@ -503,35 +547,414 @@ describe('POST /api/stacks/:stackName/git-source/webhook-pull status codes', () }); }); -describe('DELETE /api/stacks/:stackName/git-source — multi-file unlink guard', () => { - it('blocks unlinking a multi-file source with 409 and keeps the row', async () => { +describe('DELETE /api/stacks/:stackName/git-source, detach/export contract', () => { + function mockRender(yaml: string | null): ReturnType { + // ComposeService.getInstance() returns a fresh instance per call, so + // the spy must live on the prototype to reach the route's instance. + return vi + .spyOn(ComposeService.prototype, 'renderComposeYaml') + .mockImplementation(() => (yaml === null ? Promise.reject(new Error('docker unavailable')) : Promise.resolve(yaml))); + } + + it('exports a multi-file source: renders, writes compose.yaml, removes the row', async () => { seedGitSource('mf-unlink'); DatabaseService.getInstance().setGitSourceAppliedSpec('mf-unlink', { files: ['compose.yaml', 'infra/prod.yml'], contextDir: null }); - const res = await request(app) - .delete('/api/stacks/mf-unlink/git-source') - .set('Authorization', `Bearer ${adminToken()}`); - expect(res.status).toBe(409); - expect(res.body.error).toMatch(/multiple compose files/i); - expect(DatabaseService.getInstance().getGitSource('mf-unlink')).toBeTruthy(); + const stackDir = path.join(process.env.COMPOSE_DIR!, 'mf-unlink'); + fs.mkdirSync(stackDir, { recursive: true }); + fs.writeFileSync(path.join(stackDir, 'compose.yaml'), 'services:\n web:\n image: nginx\n'); + const render = mockRender('services:\n web:\n image: nginx\n environment:\n A: b\n'); + try { + const res = await request(app) + .delete('/api/stacks/mf-unlink/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + expect(DatabaseService.getInstance().getGitSource('mf-unlink')).toBeUndefined(); + const exported = fs.readFileSync(path.join(stackDir, 'compose.yaml'), 'utf8'); + expect(exported).toContain('A: b'); + expect(render).toHaveBeenCalled(); + } finally { + render.mockRestore(); + } }); - it('blocks unlinking a context-dir source with 409', async () => { + it('returns 409 and keeps the row when the export render fails', async () => { seedGitSource('ctx-unlink'); DatabaseService.getInstance().setGitSourceAppliedSpec('ctx-unlink', { files: ['compose.yaml'], contextDir: 'app' }); - const res = await request(app) - .delete('/api/stacks/ctx-unlink/git-source') - .set('Authorization', `Bearer ${adminToken()}`); - expect(res.status).toBe(409); - expect(DatabaseService.getInstance().getGitSource('ctx-unlink')).toBeTruthy(); + const render = mockRender(null); + try { + const res = await request(app) + .delete('/api/stacks/ctx-unlink/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(409); + expect(DatabaseService.getInstance().getGitSource('ctx-unlink')).toBeTruthy(); + } finally { + render.mockRestore(); + } + }); + + it('removes auto-discovered override files so the flattened model is final', async () => { + seedGitSource('ov-unlink'); + const stackDir = path.join(process.env.COMPOSE_DIR!, 'ov-unlink'); + fs.mkdirSync(stackDir, { recursive: true }); + fs.writeFileSync(path.join(stackDir, 'compose.yaml'), 'services:\n web:\n image: nginx\n'); + fs.writeFileSync(path.join(stackDir, 'compose.override.yaml'), 'services:\n web:\n environment:\n A: b\n'); + // The managed override file is recorded in the manifest so detach can + // find it; write a manifest entry for it directly. + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'ov-unlink', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'ov-unlink', + invocation: ['-f', 'compose.yaml', '-p', 'ov-unlink'], + inputs: [ + { + sourcePath: 'compose.yaml', materializedPath: 'compose.yaml', role: 'compose-primary', dependencyKind: 'explicit', + ownership: 'managed', provenance: 'fetch', sensitivity: 'medium', contentSha256: null, sizeBytes: 10, + state: 'present', deletionAuthority: 'sencho', note: null, + }, + { + sourcePath: 'compose.override.yaml', materializedPath: 'compose.override.yaml', role: 'compose-override', dependencyKind: 'implicit-override', + ownership: 'managed', provenance: 'fetch', sensitivity: 'medium', contentSha256: null, sizeBytes: 10, + state: 'present', deletionAuthority: 'sencho', note: null, + }, + ], + refusals: [], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'active', + }); + await svc.writeManifest('ov-unlink', manifest); + const render = mockRender('services:\n web:\n image: nginx\n environment:\n A: b\n'); + try { + const res = await request(app) + .delete('/api/stacks/ov-unlink/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + // The flattened model is final: the override file is gone, so plain + // docker compose cannot re-merge it. + expect(fs.existsSync(path.join(stackDir, 'compose.override.yaml'))).toBe(false); + expect(DatabaseService.getInstance().getGitSource('ov-unlink')).toBeUndefined(); + expect(render).toHaveBeenCalled(); + } finally { + render.mockRestore(); + } + }); + + it('keeps an explicitly selected file named compose.override.yaml during detach (audit round 8 B-7)', async () => { + seedGitSource('explicit-override-name'); + const stackDir = path.join(process.env.COMPOSE_DIR!, 'explicit-override-name'); + fs.mkdirSync(stackDir, { recursive: true }); + fs.writeFileSync(path.join(stackDir, 'compose.yaml'), 'services:\n web:\n image: nginx\n'); + fs.writeFileSync(path.join(stackDir, 'compose.override.yaml'), 'services:\n web:\n environment:\n A: b\n'); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'explicit-override-name', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml', 'compose.override.yaml'], + projectName: 'explicit-override-name', + invocation: ['-f', 'compose.yaml', '-f', 'compose.override.yaml', '-p', 'explicit-override-name'], + inputs: [ + { + sourcePath: 'compose.yaml', materializedPath: 'compose.yaml', role: 'compose-primary', dependencyKind: 'explicit', + ownership: 'managed', provenance: 'fetch', sensitivity: 'medium', contentSha256: null, sizeBytes: 10, + state: 'present', deletionAuthority: 'sencho', note: null, + }, + { + // Same basename, but an EXPLICIT -f input, not an + // auto-discovered override. + sourcePath: 'compose.override.yaml', materializedPath: 'compose.override.yaml', role: 'compose-additional', dependencyKind: 'explicit', + ownership: 'managed', provenance: 'fetch', sensitivity: 'medium', contentSha256: null, sizeBytes: 10, + state: 'present', deletionAuthority: 'sencho', note: null, + }, + ], + refusals: [], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'active', + }); + await svc.writeManifest('explicit-override-name', manifest); + const render = mockRender('services:\n web:\n image: nginx\n environment:\n A: b\n'); + try { + const res = await request(app) + .delete('/api/stacks/explicit-override-name/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + // The explicit file is part of the rendered model; detach keeps it. + expect(fs.existsSync(path.join(stackDir, 'compose.override.yaml'))).toBe(true); + expect(DatabaseService.getInstance().getGitSource('explicit-override-name')).toBeUndefined(); + } finally { + render.mockRestore(); + } }); it('allows unlinking a single-file source', async () => { seedGitSource('sf-unlink'); + const stackDir = path.join(process.env.COMPOSE_DIR!, 'sf-unlink'); + fs.mkdirSync(stackDir, { recursive: true }); + fs.writeFileSync(path.join(stackDir, 'compose.yaml'), 'services:\n web:\n image: nginx\n'); + const render = mockRender('services:\n web:\n image: nginx\n'); + try { + const res = await request(app) + .delete('/api/stacks/sf-unlink/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + expect(DatabaseService.getInstance().getGitSource('sf-unlink')).toBeUndefined(); + } finally { + render.mockRestore(); + } + }); + + it('reaps staged managed data after detach cleanup is deferred', async () => { + seedGitSource('deferred-cleanup'); + const stackDir = path.join(process.env.COMPOSE_DIR!, 'deferred-cleanup'); + fs.mkdirSync(stackDir, { recursive: true }); + fs.writeFileSync(path.join(stackDir, 'compose.yaml'), 'services:\n web:\n image: nginx\n'); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifestSvc = GitProjectManifestService.getInstance(); + const finalizeSpy = vi.spyOn(manifestSvc, 'finalizeStagedDetach').mockResolvedValueOnce(false); + const render = mockRender('services:\n web:\n image: nginx\n'); + const staged = path.join(process.env.DATA_DIR!, 'git-managed', '1', '.detach-deferred-cleanup'); + try { + const res = await request(app) + .delete('/api/stacks/deferred-cleanup/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + expect(DatabaseService.getInstance().getGitSource('deferred-cleanup')).toBeUndefined(); + expect(fs.existsSync(staged)).toBe(true); + + await GitSourceService.getInstance().sweepOrphans(); + expect(fs.existsSync(staged)).toBe(false); + } finally { + finalizeSpy.mockRestore(); + render.mockRestore(); + } + }); + + it('restores files and managed data when the database commit fails', async () => { + seedGitSource('db-fail-unlink'); + const stackDir = path.join(process.env.COMPOSE_DIR!, 'db-fail-unlink'); + fs.mkdirSync(stackDir, { recursive: true }); + const original = `${'#'.repeat((2 * 1024 * 1024) + 1)}\nservices:\n web:\n image: nginx:old\n`; + const originalOverride = Buffer.from('services:\n web:\n environment:\n LABEL: café\n', 'utf8'); + fs.writeFileSync(path.join(stackDir, 'compose.yaml'), original); + fs.writeFileSync(path.join(stackDir, 'compose.override.yaml'), originalOverride); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'db-fail-unlink', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'db-fail-unlink', + invocation: ['-f', 'compose.yaml', '-p', 'db-fail-unlink'], + inputs: [ + { + sourcePath: 'compose.yaml', materializedPath: 'compose.yaml', role: 'compose-primary', dependencyKind: 'explicit', + ownership: 'managed', provenance: 'fetch', sensitivity: 'medium', contentSha256: null, sizeBytes: original.length, + state: 'present', deletionAuthority: 'sencho', note: null, + }, + { + sourcePath: 'compose.override.yaml', materializedPath: 'compose.override.yaml', role: 'compose-override', dependencyKind: 'implicit-override', + ownership: 'managed', provenance: 'fetch', sensitivity: 'medium', contentSha256: null, sizeBytes: originalOverride.length, + state: 'present', deletionAuthority: 'sencho', note: null, + }, + ], + refusals: [], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'active', + }); + await svc.writeManifest('db-fail-unlink', manifest); + const render = mockRender('services:\n web:\n image: nginx:new\n'); + const deleteSpy = vi.spyOn(DatabaseService.getInstance(), 'deleteGitSource').mockImplementationOnce(() => { + throw new Error('database unavailable'); + }); + try { + const res = await request(app) + .delete('/api/stacks/db-fail-unlink/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(400); + expect(DatabaseService.getInstance().getGitSource('db-fail-unlink')).toBeTruthy(); + expect(fs.readFileSync(path.join(stackDir, 'compose.yaml'), 'utf8')).toBe(original); + expect(fs.readFileSync(path.join(stackDir, 'compose.override.yaml')).equals(originalOverride)).toBe(true); + const restored = await svc.readManifest('db-fail-unlink', 'https://github.com/example/repo.git', 'main'); + expect(restored).not.toBeNull(); + expect(fs.existsSync(path.join(process.env.DATA_DIR!, 'git-managed', '1', '.detach-db-fail-unlink'))).toBe(false); + } finally { + deleteSpy.mockRestore(); + render.mockRestore(); + } + }); +}); + +describe('GET /api/stacks/:stackName/git-source/manifest', () => { + it('returns the manifest for a stack that has one', async () => { + seedGitSource('manifest-get'); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'manifest-get', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc123', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'manifest-get', + invocation: ['-f', 'compose.yaml', '-p', 'manifest-get'], + inputs: [], + refusals: [], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'active', + }); + await svc.writeManifest('manifest-get', manifest); const res = await request(app) - .delete('/api/stacks/sf-unlink/git-source') + .get('/api/stacks/manifest-get/git-source/manifest') .set('Authorization', `Bearer ${adminToken()}`); expect(res.status).toBe(200); - expect(DatabaseService.getInstance().getGitSource('sf-unlink')).toBeUndefined(); + expect(res.body.manifest.manifestVersion).toBe(1); + expect(res.body.manifest.resolvedCommitSha).toBe('abc123'); + }); + + it('redacts sensitive input paths and omits internal metadata (audit round 8 B-6)', async () => { + seedGitSource('manifest-redact'); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const svc = GitProjectManifestService.getInstance(); + const manifest = svc.buildManifest({ + stackName: 'manifest-redact', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc123', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'manifest-redact', + invocation: ['-f', 'compose.yaml', '-p', 'manifest-redact'], + inputs: [ + { + sourcePath: 'compose.yaml', + materializedPath: 'compose.yaml', + role: 'compose-primary', + dependencyKind: 'explicit', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'medium', + contentSha256: 'a'.repeat(64), + sizeBytes: 120, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }, + { + sourcePath: 'secrets/db.env', + materializedPath: 'secrets/db.env', + role: 'env', + dependencyKind: 'env_file', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: 'b'.repeat(64), + sizeBytes: 40, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }, + { + sourcePath: 'configs/app.conf', + materializedPath: 'configs/app.conf', + role: 'config', + dependencyKind: 'config', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: 'c'.repeat(64), + sizeBytes: 200, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }, + { + sourcePath: 'keys/jwt.pem', + materializedPath: 'keys/jwt.pem', + role: 'secret', + dependencyKind: 'secret', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: 'd'.repeat(64), + sizeBytes: 50, + state: 'present', + deletionAuthority: 'sencho', + note: 'File-backed secret materialized from keys/jwt.pem', + }, + ], + refusals: [], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'active', + }); + await svc.writeManifest('manifest-redact', manifest); + + const res = await request(app) + .get('/api/stacks/manifest-redact/git-source/manifest') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + + // The medium-sensitivity compose input keeps its path... + const compose = res.body.manifest.inputs.find((i: { dependencyKind: string }) => i.dependencyKind === 'explicit'); + expect(compose.path).toBe('compose.yaml'); + // ...and high-sensitivity env/config inputs have their paths redacted. + const env = res.body.manifest.inputs.find((i: { dependencyKind: string }) => i.dependencyKind === 'env_file'); + expect(env.path).toBeNull(); + const cfg = res.body.manifest.inputs.find((i: { dependencyKind: string }) => i.dependencyKind === 'config'); + expect(cfg.path).toBeNull(); + + // Internal metadata never crosses the API. + const serialized = JSON.stringify(res.body); + expect(serialized).not.toContain('contentSha256'); + expect(serialized).not.toContain('sizeBytes'); + expect(serialized).not.toContain('sourcePath'); + expect(serialized).not.toContain('materializedPath'); + expect(serialized).not.toContain('deletionAuthority'); + expect(serialized).not.toContain('provenance'); + expect(serialized).not.toContain('secrets/db.env'); + expect(serialized).not.toContain('configs/app.conf'); + // A path-bearing note on a high-sensitivity entry must not leak either. + expect(serialized).not.toContain('keys/jwt.pem'); + // The redacted projection still counts the entries. + expect(res.body.manifest.inputs).toHaveLength(4); + }); + + it('returns 404 when no manifest exists', async () => { + seedGitSource('manifest-missing'); + const res = await request(app) + .get('/api/stacks/manifest-missing/git-source/manifest') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(404); + }); + + it('denies without the stack:read permission', async () => { + seedGitSource('manifest-denied'); + const res = await request(app) + .get('/api/stacks/manifest-denied/git-source/manifest') + .set('Authorization', `Bearer ${jwt.sign({ username: 'viewer', role: 'viewer' }, TEST_JWT_SECRET, { expiresIn: '1m' })}`); + // viewer lacks stack:read for this stack + expect([401, 403]).toContain(res.status); }); }); @@ -550,6 +973,82 @@ describe('GET /api/git-sources', () => { }); }); +describe('stack_git_sources manifest cache columns', () => { + const MANIFEST_STATES = [ + 'none', + 'migrated', + 'active', + 'partial', + 'unsupported', + 'migration_required', + 'absent', + ] as const; + + it('round-trips every GitSourceManifestState enum member', () => { + const db = DatabaseService.getInstance(); + for (const state of MANIFEST_STATES) { + seedGitSource(`manifest-state-${state}`); + db.setGitSourceManifestState(`manifest-state-${state}`, 7, state, 'generations/applied-abc'); + const row = db.getGitSource(`manifest-state-${state}`)!; + expect(row.manifest_version).toBe(7); + expect(row.manifest_state).toBe(state); + expect(row.manifest_generation).toBe('generations/applied-abc'); + db.deleteGitSource(`manifest-state-${state}`); + } + }); + + it('reads nulls for rows written before the columns existed', () => { + const row = DatabaseService.getInstance().getGitSource('missing-manifest-row'); + expect(row).toBeUndefined(); + }); + + it('upsert does not clobber the manifest cache columns', () => { + const db = DatabaseService.getInstance(); + seedGitSource('manifest-preserved'); + db.setGitSourceManifestState('manifest-preserved', 3, 'active', 'generations/applied-x'); + db.upsertGitSource({ + stack_name: 'manifest-preserved', + repo_url: 'https://github.com/example/repo.git', + branch: 'main', + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + const row = db.getGitSource('manifest-preserved')!; + expect(row.manifest_version).toBe(3); + expect(row.manifest_state).toBe('active'); + expect(row.manifest_generation).toBe('generations/applied-x'); + }); + + it('GET keeps flat manifest_state aligned with the healed summary', async () => { + const composeDir = process.env.COMPOSE_DIR!; + fs.mkdirSync(path.join(composeDir, 'stale-manifest-get'), { recursive: true }); + fs.writeFileSync(path.join(composeDir, 'stale-manifest-get', 'compose.yaml'), 'services:\n x:\n image: nginx\n'); + seedGitSource('stale-manifest-get'); + DatabaseService.getInstance().setGitSourceManifestState('stale-manifest-get', 3, 'active', 'generations/applied-abc-3'); + + const res = await request(app) + .get('/api/stacks/stale-manifest-get/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + expect(res.body.manifest?.state).toBe('migration_required'); + expect(res.body.manifest_state).toBe('migration_required'); + }); +}); + describe('git-source routes: statuses-cache invalidation', () => { // The cached /stacks/statuses payload carries the source label, so link // and unlink must drop the cache; read-only routes must not. @@ -584,13 +1083,22 @@ describe('git-source routes: statuses-cache invalidation', () => { }); it('invalidates node caches when unlinking a Git source', async () => { + seedStackDir('inv-unlink'); seedGitSource('inv-unlink'); - const res = await request(app) - .delete('/api/stacks/inv-unlink/git-source') - .set('Authorization', `Bearer ${adminToken()}`); - expect(res.status).toBe(200); - expect(mockInvalidateNodeCaches).toHaveBeenCalledTimes(1); - expect(mockInvalidateNodeCaches).toHaveBeenCalledWith(expect.any(Number)); + // Stub detach so the assertion stays at the route layer (export/render + // belongs to the service tests); unlink must still drop the cache. + const detachSpy = vi.spyOn(GitSourceService.getInstance(), 'detach') + .mockResolvedValue(undefined); + try { + const res = await request(app) + .delete('/api/stacks/inv-unlink/git-source') + .set('Authorization', `Bearer ${adminToken()}`); + expect(res.status).toBe(200); + expect(mockInvalidateNodeCaches).toHaveBeenCalledTimes(1); + expect(mockInvalidateNodeCaches).toHaveBeenCalledWith(expect.any(Number)); + } finally { + detachSpy.mockRestore(); + } }); it('does not invalidate on GET of the Git source', async () => { diff --git a/backend/src/__tests__/git-source-service.test.ts b/backend/src/__tests__/git-source-service.test.ts index c7a3775b..b8f0c777 100644 --- a/backend/src/__tests__/git-source-service.test.ts +++ b/backend/src/__tests__/git-source-service.test.ts @@ -13,6 +13,8 @@ * - Per-stack mutex serialization ordering */ import crypto from 'crypto'; +import fs from 'fs'; +import path from 'path'; import { describe, it, expect, vi, beforeAll, afterAll, beforeEach } from 'vitest'; import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb'; @@ -217,6 +219,50 @@ describe('GitSourceService.validateCompose (YAML pre-check)', () => { expect(r.ok).toBe(false); expect(r.error).toMatch(/YAML parse error/i); }); + + it('scrubs absolute validation paths from docker compose stderr', async () => { + const instance = svc(); + const leak = + 'open /app/data/git-managed/1/qa-refusal/generations/candidate-abc/case/config.yml: no such file or directory'; + const runSpy = vi.spyOn( + instance as unknown as { runDockerCompose: (a: string[], c: string, t: number) => Promise<{ code: number; stdout: string; stderr: string }> }, + 'runDockerCompose', + ).mockResolvedValue({ code: 1, stdout: '', stderr: leak }); + try { + const r = await instance.validateCompose(asFiles('services:\n web:\n image: nginx\n'), null, null); + expect(r.ok).toBe(false); + expect(r.error).toBeDefined(); + expect(r.error).not.toContain('/app/data'); + expect(r.error).not.toContain('git-managed/1'); + expect(r.error).not.toContain('candidate-abc'); + expect(r.error).toContain('[managed-path]'); + } finally { + runSpy.mockRestore(); + } + }); + + it('does not corrupt unrelated paths when scrubbing a DATA_DIR prefix', async () => { + const instance = svc(); + // Simulate a temp validation dir whose basename is a prefix of another + // word in the message (data vs database). The scrubber must not turn + // "database" into "base". + const runSpy = vi.spyOn( + instance as unknown as { runDockerCompose: (a: string[], c: string, t: number) => Promise<{ code: number; stdout: string; stderr: string }> }, + 'runDockerCompose', + ).mockImplementation(async (_args, cwd) => ({ + code: 1, + stdout: '', + stderr: `open ${cwd}/compose.yaml: failed; see /var/lib/database/notes`, + })); + try { + const r = await instance.validateCompose(asFiles('services:\n web:\n image: nginx\n'), null, null); + expect(r.ok).toBe(false); + expect(r.error).toContain('/var/lib/database/notes'); + expect(r.error).not.toMatch(/\/var\/lib\/base\/notes/); + } finally { + runSpy.mockRestore(); + } + }); }); describe('GitSourceService.upsert (encryption + reachability)', () => { @@ -353,6 +399,138 @@ describe('GitSourceService.upsert (encryption + reachability)', () => { expect(DatabaseService.getInstance().getGitSource('unreachable')).toBeUndefined(); }); + + describe('repository identity changes on managed stacks (audit round 8 B-5)', () => { + async function seedManifest(stackName: string, repoUrl = 'https://github.com/example/repo.git', branch = 'main') { + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifest = GitProjectManifestService.getInstance().buildManifest({ + stackName, + repoUrl, + branch, + commitSha: 'abc123', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: stackName, + invocation: ['-f', 'compose.yaml', '-p', stackName], + inputs: [{ + sourcePath: 'compose.yaml', + materializedPath: 'compose.yaml', + role: 'compose-primary', + dependencyKind: 'explicit', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'medium', + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }], + refusals: [], + buildContexts: [], + bounds: { + maxFiles: 10_000, + maxBytes: 512 * 1024 * 1024, + maxContextBytes: 256 * 1024 * 1024, + maxPathDepth: 64, + maxFileBytes: 10 * 1024 * 1024, + }, + priorManifest: null, + state: 'active', + }); + await GitProjectManifestService.getInstance().writeManifest(stackName, manifest); + } + + const baseInput = { + stackName: 'id-change', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + composePaths: ['compose.yaml'], + contextDir: null, + syncEnv: false, + envPath: null, + authType: 'none' as const, + autoApplyOnWebhook: false, + autoDeployOnApply: false, + }; + + async function seedSource(stackName: string) { + mockSuccessfulClone(); + const svc = GitSourceService.getInstance(); + await svc.upsert({ ...baseInput, stackName }); + } + + it('rejects a repository change when a managed-project manifest exists', async () => { + await seedSource('id-change-repo'); + await seedManifest('id-change-repo'); + const svc = GitSourceService.getInstance(); + mockGitClone.mockClear(); + + await expect(svc.upsert({ + ...baseInput, + stackName: 'id-change-repo', + repoUrl: 'https://github.com/example/other.git', + })).rejects.toMatchObject({ + code: 'GIT_ERROR', + message: expect.stringMatching(/Detach the Git source first/), + }); + + // Nothing persisted, no dry-run fetch attempted. + expect(DatabaseService.getInstance().getGitSource('id-change-repo')?.repo_url).toBe('https://github.com/example/repo.git'); + expect(mockGitClone).not.toHaveBeenCalled(); + }); + + it('rejects a branch change when a managed-project manifest exists', async () => { + await seedSource('id-change-branch'); + await seedManifest('id-change-branch'); + const svc = GitSourceService.getInstance(); + + await expect(svc.upsert({ + ...baseInput, + stackName: 'id-change-branch', + branch: 'develop', + })).rejects.toMatchObject({ code: 'GIT_ERROR', message: expect.stringMatching(/Detach the Git source first/) }); + expect(DatabaseService.getInstance().getGitSource('id-change-branch')?.branch).toBe('main'); + }); + + it('allows a repository change when no manifest exists (legacy stack)', async () => { + await seedSource('id-change-legacy'); + const svc = GitSourceService.getInstance(); + + await svc.upsert({ ...baseInput, stackName: 'id-change-legacy', repoUrl: 'https://github.com/example/other.git' }); + expect(DatabaseService.getInstance().getGitSource('id-change-legacy')?.repo_url).toBe('https://github.com/example/other.git'); + }); + + it('allows non-identity config changes on a managed stack', async () => { + await seedSource('id-change-paths'); + await seedManifest('id-change-paths'); + const svc = GitSourceService.getInstance(); + // The dry-run reachability fetch must find every configured file. + mockSuccessfulClone({ extraFiles: { 'override.yaml': 'services: {}\n' } }); + + await svc.upsert({ ...baseInput, stackName: 'id-change-paths', composePaths: ['compose.yaml', 'override.yaml'] }); + const row = DatabaseService.getInstance().getGitSource('id-change-paths'); + expect(row?.compose_paths).toEqual(['compose.yaml', 'override.yaml']); + }); + + it('apply refuses a stale-identity manifest with a detach-first instruction', async () => { + const sha = 'abc1234567890abc1234567890abc1234567890a'; + await seedSource('id-change-apply'); + // Manifest stamped for a different repository than the source row. + await seedManifest('id-change-apply', 'https://github.com/example/other.git', 'main'); + const svc = GitSourceService.getInstance(); + + mockSuccessfulClone({ sha }); + await svc.pull('id-change-apply'); + const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: true }); + try { + await expect(svc.apply('id-change-apply', sha)) + .rejects.toMatchObject({ code: 'GIT_ERROR', message: expect.stringMatching(/Detach the Git source/) }); + } finally { + validateSpy.mockRestore(); + } + }); + }); }); describe('GitSourceService error mapping', () => { @@ -705,14 +883,16 @@ describe('GitSourceService.handleWebhookPull debounce', () => { autoApplyOnWebhook: false, autoDeployOnApply: false, }); - // upsert runs a dry-run fetch but not validateCompose, so the stub only - // affects the webhook pull below. - const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: false, error: 'bad compose' }); + // Complete-project pulls validate the staged candidate via the docker + // runner; stub it to fail so the webhook pull reports the error. + const runSpy = vi + .spyOn(svc as unknown as { runDockerCompose: (a: string[], c: string, t: number) => Promise<{ code: number; stdout: string; stderr: string }> }, 'runDockerCompose') + .mockResolvedValue({ code: 1, stdout: '', stderr: 'bad compose' }); const result = await svc.handleWebhookPull('webhook-validate-fail'); expect(result.status).toBe('error'); expect(result.message).toMatch(/validation failed/i); - validateSpy.mockRestore(); + runSpy.mockRestore(); }); }); @@ -931,6 +1111,18 @@ describe('GitSourceService.createStackFromGit', () => { expect(result.source.last_applied_commit_sha).toBe(sha); expect(result.source.pending_commit_sha).toBeNull(); + // The manifest cache is persisted after the row insert (audit S-2): + // the immediate response and the DB row report the real state, not + // the default 'absent'. + expect(result.source.manifest_state).toBe('active'); + const row = DatabaseService.getInstance().getGitSource('create-happy'); + expect(row?.manifest_state).toBe('active'); + expect(row?.manifest_version).toBe(1); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifest = await GitProjectManifestService.getInstance().readManifest('create-happy', 'https://github.com/example/repo.git', 'main'); + if (manifest === null || 'corrupt' in manifest) throw new Error('expected a manifest'); + expect(manifest.resolvedRevision.commitSha).toBe(sha); + const { FileSystemService } = await import('../services/FileSystemService'); const onDisk = await FileSystemService.getInstance().getStackContent('create-happy'); expect(onDisk).toContain('image: nginx'); @@ -1222,6 +1414,53 @@ describe('GitSourceService.apply', () => { } }); + it('refuses the first complete-project apply when an unowned local file collides (audit round 9 B-1)', async () => { + const sha = '9999aaaa9999aaaa9999aaaa9999aaaa9999aaaa'; + mockSuccessfulClone({ + compose: 'services:\n web:\n image: nginx\n configs: [app]\nconfigs:\n app:\n file: configs/app.json\n', + extraFiles: { 'configs/app.json': '{"repo": true}\n' }, + sha, + }); + const svc = GitSourceService.getInstance(); + const stackName = 'pre-manifest-collision'; + await svc.upsert({ + stackName, + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + composePaths: ['compose.yaml'], + contextDir: null, + syncEnv: false, + envPath: null, + authType: 'none', + autoApplyOnWebhook: false, + autoDeployOnApply: false, + }); + // Legacy state: only compose.yaml was ever applied (no manifest). + DatabaseService.getInstance().setGitSourceAppliedSpec(stackName, { files: ['compose.yaml'], contextDir: null }); + const { FileSystemService } = await import('../services/FileSystemService'); + const fsSvc = FileSystemService.getInstance(); + await fsSvc.createStack(stackName); + await fsSvc.saveStackContent(stackName, 'services:\n web:\n image: nginx:old\n'); + // A local file Sencho never owned, colliding with the incoming revision. + await fsSvc.writeStackFile(stackName, 'configs/app.json', 'local user data\n'); + + await svc.pull(stackName); + const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: true }); + try { + await expect(svc.apply(stackName, sha)).rejects.toMatchObject({ + code: 'GIT_ERROR', + message: expect.stringMatching(/does not manage/), + }); + // The local file is preserved byte-for-byte. + const onDisk = await fsSvc.readStackFile(stackName, 'configs/app.json'); + expect(onDisk.content).toBe('local user data\n'); + expect(DatabaseService.getInstance().getGitSource(stackName)?.pending_commit_sha).toBe(sha); + } finally { + validateSpy.mockRestore(); + } + await cleanupStackDir(stackName); + }); + it('returns deployError and skips compose deploy when policy blocks apply deploy', async () => { const sha = 'dddd444dddd444dddd444dddd444dddd444dddd4'; const svc = await seedPending('apply-policy-block', 'services:\n x:\n image: nginx:bad\n', sha); @@ -1563,3 +1802,433 @@ describe('GitSourceService multi-file create + apply flow', () => { await cleanupStackDir('pending-config'); }); }); + +describe('GitSourceService pending blob decode branches', () => { + function svc(): unknown { return GitSourceService.getInstance(); } + type DecodeApi = { + crypto: { encrypt(s: string): string; decrypt(s: string): string }; + encodePendingCompose(files: { path: string; content: string }[], ctx: string | null, cand: string | null, inv: unknown): string; + decodePendingCompose(s: string): { files: { path: string; content: string }[]; contextDir: string | null; candidateRelPath: string | null; inventory: unknown }; + }; + + it('round-trips the v3 blob with candidate path and inventory', () => { + const s = svc() as unknown as DecodeApi; + const encoded = s.encodePendingCompose([{ path: 'compose.yaml', content: 'x' }], null, 'generations/candidate-abc', { inputs: [], refusals: [], buildContexts: [] }); + const decoded = s.decodePendingCompose(encoded); + expect(decoded.candidateRelPath).toBe('generations/candidate-abc'); + expect(decoded.files[0].content).toBe('x'); + expect(decoded.inventory).toEqual({ inputs: [], refusals: [], buildContexts: [] }); + }); + + it('decodes a v2 blob without a candidate', () => { + const s = svc() as unknown as DecodeApi; + const encoded = s.crypto.encrypt(JSON.stringify({ v: 2, files: [{ path: 'compose.yaml', content: 'y' }], contextDir: null })); + const decoded = s.decodePendingCompose(encoded); + expect(decoded.candidateRelPath).toBeNull(); + expect(decoded.files[0].content).toBe('y'); + }); + + it('falls back to legacy plaintext for unknown shapes', () => { + const s = svc() as unknown as DecodeApi; + const decoded = s.decodePendingCompose(s.crypto.encrypt('legacy content')); + expect(decoded.files).toEqual([{ path: 'compose.yaml', content: 'legacy content' }]); + expect(decoded.candidateRelPath).toBeNull(); + }); + + it('rejects a corrupt v3 blob as corrupt state instead of falling back to legacy', () => { + const s = svc() as unknown as DecodeApi; + const encoded = s.crypto.encrypt('{"v":3 not json'); + expect(() => s.decodePendingCompose(encoded)).toThrow(/corrupt/); + }); +}); + +describe('GitSourceService managed-area lifecycle', () => { + it('removes the managed area when createStackFromGit fails after staging', async () => { + const sha = 'f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1'; + mockSuccessfulClone({ sha }); + const svc = GitSourceService.getInstance(); + const runSpy = vi + .spyOn(svc as unknown as { runDockerCompose: (a: string[], c: string, t: number) => Promise<{ code: number; stdout: string; stderr: string }> }, 'runDockerCompose') + .mockResolvedValue({ code: 0, stdout: '', stderr: '' }); + const { FileSystemService } = await import('../services/FileSystemService'); + const createSpy = vi + .spyOn(FileSystemService.prototype, 'createStack') + .mockRejectedValue(new Error('simulated create failure')); + try { + await expect( + svc.createStackFromGit({ + stackName: 'rollback-area', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + composePaths: ['compose.yaml'], + contextDir: null, + syncEnv: false, + envPath: null, + authType: 'none', + token: null, + autoApplyOnWebhook: false, + autoDeployOnApply: false, + }), + ).rejects.toThrow(/simulated create failure/); + } finally { + runSpy.mockRestore(); + createSpy.mockRestore(); + } + // The staged candidate lived in the managed area; the rollback must reap it. + const stagedCandidate = path.join(process.env.DATA_DIR!, 'git-managed', '1', 'rollback-area', 'generations', 'candidate-f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1f1'); + expect(fs.existsSync(stagedCandidate)).toBe(false); + expect(DatabaseService.getInstance().getGitSource('rollback-area')).toBeUndefined(); + }); + + it('sweeps managed areas whose stack no longer exists', async () => { + mockSuccessfulClone(); + DatabaseService.getInstance().upsertGitSource({ + stack_name: 'ghost-stack', + repo_url: 'https://github.com/example/repo.git', + branch: 'main', + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifestSvc = GitProjectManifestService.getInstance(); + const manifest = manifestSvc.buildManifest({ + stackName: 'ghost-stack', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: 'ghost-stack', + invocation: ['-f', 'compose.yaml', '-p', 'ghost-stack'], + inputs: [], + refusals: [], + buildContexts: [], + bounds: { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }, + priorManifest: null, + state: 'active', + }); + await manifestSvc.writeManifest('ghost-stack', manifest); + await manifestSvc.prepareDetachRecovery( + 'ghost-stack', + 'https://github.com/example/repo.git', + 'main', + [{ path: 'compose.yaml', existed: false, content: null }], + ); + expect(await manifestSvc.stageManagedAreaForDetach('ghost-stack')).toBe(true); + await GitSourceService.getInstance().sweepOrphans(); + expect(await manifestSvc.readManifest('ghost-stack', 'https://github.com/example/repo.git', 'main')).toBeNull(); + expect(fs.existsSync(path.join(process.env.DATA_DIR!, 'git-managed', '1', '.detach-ghost-stack'))).toBe(false); + }); + + const SWEEP_BOUNDS = { maxFiles: 10_000, maxBytes: 512 * 1024 * 1024, maxContextBytes: 256 * 1024 * 1024, maxPathDepth: 64, maxFileBytes: 10 * 1024 * 1024 }; + + function insertGitSourceRow(stackName: string): void { + DatabaseService.getInstance().upsertGitSource({ + stack_name: stackName, + repo_url: 'https://github.com/example/repo.git', + branch: 'main', + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + } + + /** Live managed-stack fixture: on-disk stack, row, and written manifest. */ + async function seedManagedStack(stackName: string): Promise { + const { FileSystemService } = await import('../services/FileSystemService'); + await FileSystemService.getInstance().createStack(stackName); + await FileSystemService.getInstance().saveStackContent(stackName, 'services: {}\n'); + insertGitSourceRow(stackName); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifestSvc = GitProjectManifestService.getInstance(); + await manifestSvc.writeManifest(stackName, manifestSvc.buildManifest({ + stackName, + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: stackName, + invocation: ['-f', 'compose.yaml', '-p', stackName], + inputs: [], + refusals: [], + buildContexts: [], + bounds: SWEEP_BOUNDS, + priorManifest: null, + state: 'active', + })); + } + + it('does not delete managed areas when the stack listing fails', async () => { + const { FileSystemService } = await import('../services/FileSystemService'); + const svc = GitSourceService.getInstance(); + const stackName = 'live-sweep-listing-fail'; + await seedManagedStack(stackName); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifestSvc = GitProjectManifestService.getInstance(); + // A retained recovery generation, as the sweep would otherwise reap. + const genDir = path.join(process.env.DATA_DIR!, 'git-managed', '1', stackName, 'generations', 'applied-abc-1'); + fs.mkdirSync(genDir, { recursive: true }); + + const strictSpy = vi.spyOn(FileSystemService.prototype, 'getStacksStrict').mockRejectedValue(new Error('EIO: readdir failed')); + // Also mock the SOFT listing: the pre-fix sweep called getStacks(), + // which swallows the failure into an empty list and deletes the area. + // Post-fix the sweep uses the strict variant and is unaffected, so the + // test goes red on the pre-fix call path and green here. + const softSpy = vi.spyOn(FileSystemService.prototype, 'getStacks').mockRejectedValue(new Error('EIO: readdir failed')); + try { + await svc.sweepOrphans(); + } finally { + strictSpy.mockRestore(); + softSpy.mockRestore(); + } + // The manifest and every retained generation survive the failed listing. + expect(await manifestSvc.readManifest(stackName, 'https://github.com/example/repo.git', 'main')).not.toBeNull(); + expect(fs.existsSync(genDir)).toBe(true); + await cleanupStackDir(stackName); + }); + + it('skips a live stack\'s managed area when the listing omits it', async () => { + const { FileSystemService } = await import('../services/FileSystemService'); + const svc = GitSourceService.getInstance(); + const stackName = 'live-sweep-empty-listing'; + await seedManagedStack(stackName); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifestSvc = GitProjectManifestService.getInstance(); + + const strictSpy = vi.spyOn(FileSystemService.prototype, 'getStacksStrict').mockResolvedValue([]); + const softSpy = vi.spyOn(FileSystemService.prototype, 'getStacks').mockResolvedValue([]); + try { + await svc.sweepOrphans(); + } finally { + strictSpy.mockRestore(); + softSpy.mockRestore(); + } + expect(await manifestSvc.readManifest(stackName, 'https://github.com/example/repo.git', 'main')).not.toBeNull(); + await cleanupStackDir(stackName); + }); + + it('reaps a vanished stack\'s managed area while live stacks survive in the same sweep', async () => { + const svc = GitSourceService.getInstance(); + const { GitProjectManifestService } = await import('../services/GitProjectManifestService'); + const manifestSvc = GitProjectManifestService.getInstance(); + const liveA = 'live-sweep-a'; + const liveB = 'live-sweep-b'; + const ghost = 'vanished-sweep'; + for (const name of [liveA, liveB]) { + await seedManagedStack(name); + } + // A row whose stack directory is genuinely gone: row + manifest only. + insertGitSourceRow(ghost); + await manifestSvc.writeManifest(ghost, manifestSvc.buildManifest({ + stackName: ghost, + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + commitSha: 'abc', + projectRoot: null, + composeFiles: ['compose.yaml'], + projectName: ghost, + invocation: ['-f', 'compose.yaml', '-p', ghost], + inputs: [], + refusals: [], + buildContexts: [], + bounds: SWEEP_BOUNDS, + priorManifest: null, + state: 'active', + })); + + await svc.sweepOrphans(); + + // Both live areas and manifests survive; the vanished stack's area is reaped. + expect(await manifestSvc.readManifest(liveA, 'https://github.com/example/repo.git', 'main')).not.toBeNull(); + expect(await manifestSvc.readManifest(liveB, 'https://github.com/example/repo.git', 'main')).not.toBeNull(); + expect(await manifestSvc.readManifest(ghost, 'https://github.com/example/repo.git', 'main')).toBeNull(); + for (const name of [liveA, liveB]) { + await cleanupStackDir(name); + } + }); + + it('reports migration_required when the manifest file is gone but the cache claims an applied state', async () => { + const svc = GitSourceService.getInstance(); + DatabaseService.getInstance().upsertGitSource({ + stack_name: 'stale-cache-summary', + repo_url: 'https://github.com/example/repo.git', + branch: 'main', + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + DatabaseService.getInstance().setGitSourceManifestState('stale-cache-summary', 3, 'active', 'generations/applied-abc-3'); + const summary = await svc.getManifestSummary('stale-cache-summary'); + expect(summary?.state).toBe('migration_required'); + expect(summary?.manifestVersion).toBe(0); + expect(summary?.managedCount).toBe(0); + // Heal-on-read keeps the flat cache aligned with the summary. + expect(DatabaseService.getInstance().getGitSource('stale-cache-summary')?.manifest_state).toBe('migration_required'); + expect(DatabaseService.getInstance().getGitSource('stale-cache-summary')?.manifest_version).toBeNull(); + + // A row that never had a manifest still reports absent. + DatabaseService.getInstance().upsertGitSource({ + stack_name: 'never-manifested', + repo_url: 'https://github.com/example/repo.git', + branch: 'main', + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: null, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + expect((await svc.getManifestSummary('never-manifested'))?.state).toBe('absent'); + }); +}); + +describe('GitSourceService legacy pending apply (migration path)', () => { + it('applies a v2 pending blob via the historical path and builds a migrated manifest', async () => { + const sha = '9999aaa9999aaa9999aaa9999aaa9999aaa9999a'; + const svc = GitSourceService.getInstance(); + const db = DatabaseService.getInstance(); + const { FileSystemService } = await import('../services/FileSystemService'); + const fsSvc = FileSystemService.getInstance(); + await fsSvc.createStack('legacy-apply'); + db.upsertGitSource({ + stack_name: 'legacy-apply', + repo_url: 'https://github.com/example/repo.git', + branch: 'main', + compose_path: 'compose.yaml', + compose_paths: ['compose.yaml'], + context_dir: null, + sync_env: false, + env_path: null, + auth_type: 'none', + encrypted_token: null, + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + last_applied_commit_sha: null, + last_applied_content_hash: null, + pending_commit_sha: sha, + pending_compose_content: null, + pending_env_content: null, + pending_fetched_at: null, + last_debounce_at: null, + }); + // Seed the v2 blob directly, as a pre-upgrade row would carry it. + const svcPriv = svc as unknown as { crypto: { encrypt(s: string): string } }; + db.setGitSourcePending('legacy-apply', sha, svcPriv.crypto.encrypt(JSON.stringify({ v: 2, files: [{ path: 'compose.yaml', content: 'services:\n web:\n image: nginx\n' }], contextDir: null })), null); + const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: true }); + + try { + const applied = await svc.apply('legacy-apply', sha, { deploy: false }); + expect(applied.applied).toBe(true); + expect(await fsSvc.getStackContent('legacy-apply')).toContain('image: nginx'); + const row = db.getGitSource('legacy-apply'); + expect(row?.manifest_state).toBe('migrated'); + } finally { + validateSpy.mockRestore(); + await cleanupStackDir('legacy-apply'); + } + }); +}); + +describe('GitSourceService sync-env stacks with a repo .env (audit C-2)', () => { + it('applies twice without a divergence refusal when the repo carries a root .env', async () => { + const sha = 'abcd1111abcd1111abcd1111abcd1111abcd1111'; + const svc = GitSourceService.getInstance(); + const db = DatabaseService.getInstance(); + const runSpy = vi + .spyOn(svc as unknown as { runDockerCompose: (a: string[], c: string, t: number) => Promise<{ code: number; stdout: string; stderr: string }> }, 'runDockerCompose') + .mockResolvedValue({ code: 0, stdout: '', stderr: '' }); + const { FileSystemService } = await import('../services/FileSystemService'); + const fsSvc = FileSystemService.getInstance(); + await fsSvc.createStack('sync-env-double'); + // Repo carries a root .env; sync_env is on and the sync env path is the same file. + mockSuccessfulClone({ + compose: 'services:\n web:\n image: nginx\n', + env: 'SYNCED=1\n', + envPath: '.env', + extraFiles: { '.env': 'REPO=1\n' }, + sha, + }); + try { + await svc.upsert({ + stackName: 'sync-env-double', + repoUrl: 'https://github.com/example/repo.git', + branch: 'main', + composePaths: ['compose.yaml'], + contextDir: null, + syncEnv: true, + envPath: '.env', + authType: 'none', + autoApplyOnWebhook: false, + autoDeployOnApply: false, + }); + const pull1 = await svc.pull('sync-env-double'); + const apply1 = await svc.apply('sync-env-double', pull1.commitSha, { deploy: false }); + expect(apply1.applied).toBe(true); + // The manifest has exactly one .env entry. + const manifest = await svc.getManifest('sync-env-double'); + const envEntries = manifest?.inputs.filter((i) => i.materializedPath === '.env') ?? []; + expect(envEntries).toHaveLength(1); + expect(envEntries[0].dependencyKind).toBe('sync-env'); + + // Second cycle must not raise the divergence refusal. + const pull2 = await svc.pull('sync-env-double'); + const apply2 = await svc.apply('sync-env-double', pull2.commitSha, { deploy: false }); + expect(apply2.applied).toBe(true); + void db; + } finally { + runSpy.mockRestore(); + await cleanupStackDir('sync-env-double'); + } + }); +}); diff --git a/backend/src/__tests__/update-guard-service.test.ts b/backend/src/__tests__/update-guard-service.test.ts index 2458c00b..cdeaf946 100644 --- a/backend/src/__tests__/update-guard-service.test.ts +++ b/backend/src/__tests__/update-guard-service.test.ts @@ -17,6 +17,7 @@ const { mockGetGlobalSettings, mockFsSize, mockBuildEffectiveServiceModel, + mockGetGitSource, } = vi.hoisted(() => ({ mockListContainers: vi.fn(), mockGetContainer: vi.fn(), @@ -29,6 +30,7 @@ const { mockGetGlobalSettings: vi.fn(), mockFsSize: vi.fn(), mockBuildEffectiveServiceModel: vi.fn(), + mockGetGitSource: vi.fn(), })); vi.mock('../services/DockerController', () => ({ @@ -69,6 +71,9 @@ vi.mock('../services/DatabaseService', () => ({ getOpenDriftFindings: mockGetOpenDriftFindings, getGlobalSettings: mockGetGlobalSettings, getStackActivity: vi.fn().mockReturnValue([]), + // Rollback-readiness partial-revert disclosure reads the git source row; + // no git-managed stacks in these fixtures by default. + getGitSource: mockGetGitSource, }), }, })); @@ -324,3 +329,21 @@ describe('UpdateGuardService.computeRollbackReadiness moving-tag wiring', () => expect(report.overall).toBe('ready'); }); }); + +describe('UpdateGuardService.computeRollbackReadiness git-managed disclosure', () => { + beforeEach(() => { + mockGetGitSource.mockReturnValue(undefined); + }); + + it('adds the partial-revert note when the stack is Git-managed and active', async () => { + mockGetGitSource.mockReturnValue({ manifest_state: 'active' }); + const report = await UpdateGuardService.getInstance().computeRollbackReadiness(1, 'git-stack'); + expect(report.note).toContain('Git-managed'); + expect(report.note).toContain('compose files and .env'); + }); + + it('omits the note when the stack has no Git source or no manifest', async () => { + const report = await UpdateGuardService.getInstance().computeRollbackReadiness(1, 'plain-stack'); + expect(report.note).toBeUndefined(); + }); +}); diff --git a/backend/src/bootstrap/startup.ts b/backend/src/bootstrap/startup.ts index 5b6bda80..37fd2c97 100644 --- a/backend/src/bootstrap/startup.ts +++ b/backend/src/bootstrap/startup.ts @@ -28,7 +28,7 @@ import { applyPilotModeCapabilityFilter } from '../services/CapabilityRegistry'; import { PilotTunnelManager } from '../services/PilotTunnelManager'; import { PilotMetrics } from '../services/PilotMetrics'; import { invalidateRemoteMetaCache } from '../helpers/cacheInvalidation'; -import { sweepStaleTempDirs as sweepStaleGitTempDirs } from '../services/GitSourceService'; +import { sweepStaleTempDirs as sweepStaleGitTempDirs, sweepGitManifestOrphans } from '../services/GitSourceService'; import { PORT } from '../helpers/constants'; import { LOW_MEMORY_FLOOR_BYTES } from '../utils/spawnErrors'; @@ -194,6 +194,9 @@ export async function startServer(server: Server): Promise { sweepStaleGitTempDirs().catch((err) => { console.warn('[GitSource] Temp dir sweep failed:', (err as Error).message); }); + sweepGitManifestOrphans().catch((err) => { + console.warn('[GitManifest] Managed-area sweep failed:', (err as Error).message); + }); sweepStaleTrivyTempDirs().catch((err) => { console.warn('[Trivy] Temp dir sweep failed:', (err as Error).message); }); diff --git a/backend/src/helpers/composeInputParse.ts b/backend/src/helpers/composeInputParse.ts new file mode 100644 index 00000000..5558aa60 --- /dev/null +++ b/backend/src/helpers/composeInputParse.ts @@ -0,0 +1,639 @@ +/** + * Pure Compose input declaration parser for the Git managed-project + * materializer. Walks compose YAML (explicit files plus recursive include / + * extends.file graphs) and emits every repository-local input that can affect + * validation or deployment, WITHOUT touching the filesystem: file contents are + * injected via the `read` callback so this module stays side-effect free and + * testable. + * + * Every declaration is resolved in TWO coordinate systems: + * - source: the repository path of the file (what the clone contains), and + * - materialized: the stack-relative path the file occupies at runtime. + * They diverge when the runtime layout relocates a file: the primary compose + * file always lands at the stack root, so its entire include/extends graph + * (and every project-relative path declared in it) shifts by the primary's + * repository directory prefix. Merged (-f) files keep their own repository + * paths, but their include/extends graph resolves against the base file's + * directory at source and shifts to the stack root at runtime when no context + * dir is configured; include/extends-reached files keep their own project + * directory unless a relocation applies. + * + * Resolution rules (compose spec / compose-go loader): + * - `include:`, `extends.file`, and include map-form `env_file` paths resolve + * against the current level's EFFECTIVE PROJECT base (the compose-go local + * resource loader's WorkingDir is the project directory): the context dir, + * or the base file's directory for merged (-f) files at the top level; the + * including include-entry's project directory for nested includes. + * - An included project's interpolation env defaults to `.env` in its project + * directory; absence is tolerated. + * - For a long-form include path LIST, the FIRST resolved path is the + * included project's main file and defines its directory; later paths are + * overrides of the same project. include map-form `project_directory` + * overrides the included project's directory. + * - service `env_file`, top-level `configs`/`secrets` `file:`, `label_file` + * and `build.context` resolve against the effective project directory: the + * context dir, or the base file's directory for merged (-f) files; files + * reached via include/extends keep their own project directory. An omitted + * build context defaults to that project directory. + * - Absolute (POSIX, Windows drive/UNC, drive-relative, root-relative) and + * home-relative (`~`) paths are HOST paths: emitted with baseDir 'host' so + * the classifier records them as unmanaged (data inputs) or refuses them + * (include/extends), never adopting a same-named repository file. + * + * Never throws: parse errors are collected into `parseErrors` and surface as + * refusals at classification time. + */ +import path from 'path'; +import YAML from 'yaml'; +import type { + DeclaredInput, + DynamicInput, + InputDependencyKind, + InputRole, + ParsedDeclaredInputs, +} from '../types/gitProjectManifest'; + +// Refuse to parse anything beyond this bound so a malformed (or adversarial) +// compose file cannot exhaust heap while walking the project. Mirrors the cap +// in composeDependencyParse.ts / composePreview.ts. +const MAX_COMPOSE_PARSE_BYTES = 1_048_576; // 1 MiB + +// Include/extends recursion bound; deeper graphs are refused as unsupported. +const MAX_INCLUDE_DEPTH = 16; + +export interface ParseOptions { + /** Repo-relative project root (today's context_dir); null = repo root. */ + projectRoot: string | null; + /** Fetches a repo-relative file's content for include/extends recursion; null when unreadable. */ + read: (repoPath: string) => string | null; +} + +interface ComposeRefs { + inputs: DeclaredInput[]; + dynamic: DynamicInput[]; + parseErrors: string[]; +} + +/** + * Walk context for one compose file: its repository path, its runtime + * (materialized) path ('' = stack root), and the project bases used to + * resolve project-relative declarations in both coordinate systems. + */ +interface FileContext { + repoPath: string; + runtimePath: string; + projectBase: string | null; + runtimeProjectBase: string | null; +} + +/** True when the path contains a Compose `$VAR` / `${VAR}` interpolation form. */ +export function isDynamicPath(p: string): boolean { + return /\$\{[^}]*\}|\$[A-Za-z_][A-Za-z0-9_]*/.test(p); +} + +/** Absolute (POSIX, Windows drive/UNC, drive-relative, root-relative) or home-relative host path. */ +export function isHostAbsolutePath(p: string): boolean { + return p.startsWith('/') || p.startsWith('~') || /^[A-Za-z]:/.test(p) || p.startsWith('\\'); +} + +export function isUrl(p: string): boolean { + return /^[a-zA-Z][a-zA-Z0-9+.-]*:\/\//.test(p); +} + +function dirOf(p: string): string | null { + return p && p.includes('/') ? p.slice(0, p.lastIndexOf('/')) : null; +} + +/** Normalize a resolved path; null when it escapes its base (.. / absolute). */ +function normalizeWithinBase(candidate: string): string | null { + // Backslashes are normalized BEFORE collapse so `sub\..\x` (Windows-style + // separators) collapses and escapes are detected, never emitted raw. + const normalized = path.posix.normalize(candidate.replace(/\\/g, '/')).replace(/^\.\//, ''); + if (normalized === '..' || normalized.startsWith('../') || path.posix.isAbsolute(normalized)) return null; + return normalized; +} + +/** Resolve a path against a base; null when the result escapes the base. */ +function resolveWithinBase(base: string | null, target: string): string | null { + return normalizeWithinBase(base ? `${base}/${target}` : target); +} + +function emitInput( + refs: ComposeRefs, + rawPath: string | null, + sourcePath: string | null, + materializedPath: string | null, + kind: InputDependencyKind, + role: InputRole, + fromFile: string, + baseDir: DeclaredInput['baseDir'], + service: string | null = null, + required = true, +): void { + if (rawPath !== null && isDynamicPath(rawPath)) { + refs.dynamic.push({ sourcePath: rawPath, kind, note: 'Path contains a variable; resolved by Compose at deploy time, not enumerated.' }); + return; + } + refs.inputs.push({ sourcePath, materializedPath, baseDir, kind, role, fromFile, service, required }); +} + +/** + * Emit a project-relative declaration resolved in both coordinate systems. + * Host/absolute paths and paths that escape either base are emitted as host + * inputs (the classifier records them unmanaged or refuses include/extends). + */ +function emitProjectRelative( + refs: ComposeRefs, + raw: string, + kind: InputDependencyKind, + role: InputRole, + fromFile: string, + ctx: FileContext, + baseDir: DeclaredInput['baseDir'], + service: string | null = null, + required = true, +): void { + if (isHostAbsolutePath(raw)) { + emitInput(refs, raw, raw, null, kind, role, fromFile, 'host', service, required); + return; + } + const source = resolveWithinBase(ctx.projectBase, raw); + const materialized = resolveWithinBase(ctx.runtimeProjectBase, raw); + if (source === null || materialized === null) { + emitInput(refs, raw, raw, null, kind, role, fromFile, 'host', service, required); + return; + } + emitInput(refs, raw, source, materialized, kind, role, fromFile, baseDir, service, required); +} + +function asString(value: unknown): string | undefined { + if (typeof value === 'string') return value; + if (typeof value === 'number') return String(value); + return undefined; +} + +/** Normalize a string-or-list-of-strings value into its string items. */ +function asStringList(value: unknown): string[] { + if (typeof value === 'string') return [value]; + if (typeof value === 'number') return [String(value)]; + if (Array.isArray(value)) { + return value.map(asString).filter((p): p is string => p !== undefined); + } + return []; +} + +/** Normalize an env_file entry (string, list item, or map {path, required}) to its path and optionality. */ +function envFilePath(value: unknown): { path: string; required: boolean } | undefined { + if (typeof value === 'string') return { path: value, required: true }; + if (value && typeof value === 'object' && !Array.isArray(value)) { + const p = (value as Record).path; + const path = asString(p); + if (path === undefined) return undefined; + const required = (value as Record).required; + return { path, required: required !== false }; + } + return undefined; +} + + +/** Normalize a configs/secrets entry to a file path, or null for external/env forms. */ +function resourceFilePath(value: unknown): string | null { + if (value && typeof value === 'object' && !Array.isArray(value)) { + const record = value as Record; + if (record.external === true) return null; // docker supplies it; only an explicit true applies + if ('env' in record || 'environment' in record) return null; // env-injected + const f = record.file; + if (f !== undefined) return asString(f) ?? null; + return null; // plain {} or name-only reference + } + return null; +} + +function shortFormBindSource(volume: string): string | null { + // A drive-letter prefix (C:\ or C:) keeps its colon as part of the + // source; the source/target separator is the NEXT colon. UNC sources + // (\\server\share) have no colon and split at the separator as usual. + const separator = /^[A-Za-z]:/.test(volume) ? volume.indexOf(':', 2) : volume.indexOf(':'); + if (separator === -1) return null; + const src = volume.slice(0, separator); + // Named volumes have no path separators or leading dot/slash/tilde. + if (/^[A-Za-z0-9_.-]+$/.test(src) && !src.startsWith('.') && !src.startsWith('~')) return null; + return src; +} + +/** Collect bind-mount host sources from a service volumes list. */ +function collectBindMounts(volumes: unknown, fromFile: string, refs: ComposeRefs, ctx: FileContext): void { + const emitBindSource = (src: string): void => { + if (isHostAbsolutePath(src)) { + emitInput(refs, src, null, null, 'bind-mount', 'bind-mount', fromFile, 'host'); + } else { + emitProjectRelative(refs, src, 'bind-mount', 'bind-mount', fromFile, ctx, 'project-root'); + } + }; + if (!Array.isArray(volumes)) return; + for (const entry of volumes) { + if (typeof entry === 'string') { + const src = shortFormBindSource(entry); + if (src !== null) emitBindSource(src); + continue; + } + if (entry && typeof entry === 'object' && !Array.isArray(entry)) { + const record = entry as Record; + if (record.type === 'bind' && typeof record.source === 'string') { + emitBindSource(record.source); + } + } + } +} + +/** Collect build declarations (context, dockerfile, secrets, additional contexts). */ +function collectBuild(build: unknown, fromFile: string, refs: ComposeRefs, ctx: FileContext, service: string | null): void { + if (typeof build === 'string') { + emitProjectRelative(refs, build, 'build-context', 'build-context', fromFile, ctx, 'compose-file-dir', service); + return; + } + if (!build || typeof build !== 'object' || Array.isArray(build)) return; + const record = build as Record; + if (typeof record.context === 'string') { + emitProjectRelative(refs, record.context, 'build-context', 'build-context', fromFile, ctx, 'compose-file-dir', service); + } else { + // An omitted context defaults to the declaring file's project + // directory (compose spec); resolved by emitProjectRelative. + emitProjectRelative(refs, '.', 'build-context', 'build-context', fromFile, ctx, 'compose-file-dir', service); + } + if (typeof record.dockerfile === 'string') { + if (isHostAbsolutePath(record.dockerfile)) { + emitInput(refs, record.dockerfile, record.dockerfile, null, 'dockerfile', 'dockerfile', fromFile, 'host', service); + } else { + // Dockerfile is relative to the context; the classifier rebases it. + emitInput(refs, record.dockerfile, record.dockerfile, null, 'dockerfile', 'dockerfile', fromFile, 'compose-file-dir', service); + } + } + if (record.secrets && Array.isArray(record.secrets)) { + // Long syntax carries only source/target/uid/gid/mode; `source` names + // a TOP-LEVEL SECRET (compose errors if it is not defined in the + // top-level secrets section), never a file path. The referenced + // secret's file, when file-backed, is emitted by the top-level + // secrets walk; the reference itself is recorded as unmanaged, the + // same as the string form. + record.secrets.forEach(() => { + emitInput(refs, null, null, null, 'build-secret', 'build-secret', fromFile, 'host', service); + }); + } + // additional_contexts: mapping (name -> value) or list of NAME=VALUE + // strings (compose build spec). Path values are project-relative inputs; + // type:// and service: values are supplied by the image builder at build + // time and recorded unmanaged. + let additional: Array<[string, unknown]> = []; + if (record.additional_contexts && typeof record.additional_contexts === 'object' && !Array.isArray(record.additional_contexts)) { + additional = Object.entries(record.additional_contexts as Record); + } else if (Array.isArray(record.additional_contexts)) { + for (const item of record.additional_contexts) { + const text = asString(item); + if (text !== undefined) { + const eq = text.indexOf('='); + if (eq > 0) additional.push([text.slice(0, eq), text.slice(eq + 1)]); + } + } + } + for (const [, value] of additional) { + const p = asString(value); + if (p === undefined) continue; + if (isUrl(p) || p.startsWith('service:')) { + emitInput(refs, p, p, null, 'build-additional-context', 'build-additional-context', fromFile, 'host', service); + } else { + emitProjectRelative(refs, p, 'build-additional-context', 'build-additional-context', fromFile, ctx, 'compose-file-dir', service); + } + } +} + +/** Walk one service definition for file-backed inputs. */ +function walkService(serviceName: string, service: unknown, fromFile: string, refs: ComposeRefs, ctx: FileContext): void { + if (!service || typeof service !== 'object' || Array.isArray(service)) return; + const record = service as Record; + + // extends.file (map form) is emitted and recursed by parseFileInner, which + // needs the resolved target for the recursion; the string form extends a + // sibling service in the same file. + + const envFile = record.env_file; + if (envFile !== undefined) { + const entries = Array.isArray(envFile) ? envFile : [envFile]; + for (const entry of entries) { + const p = envFilePath(entry); + if (p !== undefined) { + emitProjectRelative(refs, p.path, 'env_file', 'env', fromFile, ctx, 'compose-file-dir', null, p.required); + } else { + refs.parseErrors.push(`env_file entry in ${fromFile} has no resolvable path`); + } + } + } + + for (const labelFile of asStringList(record.label_file)) { + emitProjectRelative(refs, labelFile, 'label_file', 'label-file', fromFile, ctx, 'compose-file-dir'); + } + + if (record.build !== undefined) collectBuild(record.build, fromFile, refs, ctx, serviceName); + collectBindMounts(record.volumes, fromFile, refs, ctx); + + // Per-service configs/secrets references are keys into the top-level + // maps, which the top-level walk emits; nothing to record here. +} + +/** + * Emit a file-relative declaration (include/extends/include-env), returning + * the resolved source/materialized pair; the caller recurses via + * readAndRecurse when the target is a repository file. + */ +function emitFileRelative( + refs: ComposeRefs, + raw: string, + kind: InputDependencyKind, + role: InputRole, + fromFile: string, + ctx: FileContext, + baseDir: DeclaredInput['baseDir'] = 'repo-root', +): { source: string | null; materialized: string | null } { + if (isHostAbsolutePath(raw)) { + emitInput(refs, raw, raw, null, kind, role, fromFile, 'host'); + return { source: null, materialized: null }; + } + if (isUrl(raw)) { + emitInput(refs, raw, raw.trim(), null, kind, role, fromFile, baseDir); + return { source: null, materialized: null }; + } + // Include, include-env, and extends paths resolve against the current + // level's EFFECTIVE PROJECT base (compose-go: the local resource + // loader's WorkingDir is the project directory), not the declaring + // file's own directory. + const source = resolveWithinBase(ctx.projectBase, raw); + const materialized = resolveWithinBase(ctx.runtimeProjectBase, raw); + if (source === null || materialized === null) { + emitInput(refs, raw, raw, null, kind, role, fromFile, 'host'); + return { source: null, materialized: null }; + } + emitInput(refs, raw, source, materialized, kind, role, fromFile, baseDir); + return { source, materialized }; +} + +/** + * Parse one compose file into declarations, recursing into include/extends. + * Cycle detection uses the RECURSION STACK (a file re-entered while still + * being walked is a real cycle); a file reached again after completion is a + * shared base (diamond / repeated include) and dedupes silently. + */ +function parseFile( + ctx: FileContext, + content: string, + opts: ParseOptions, + refs: ComposeRefs, + visited: Set, + stack: Set, + depth: number, +): void { + const normalized = normalizeRepoPath(ctx.repoPath); + if (stack.has(normalized)) { + refs.parseErrors.push(`Include/extends cycle detected at ${normalized}`); + return; + } + if (visited.has(normalized)) { + return; // shared base file, not a cycle + } + stack.add(normalized); + try { + parseFileInner(ctx, content, opts, refs, visited, stack, depth); + } finally { + stack.delete(normalized); + visited.add(normalized); + } +} + +function normalizeRepoPath(p: string): string { + return path.posix.normalize(p.replace(/\\/g, '/')).replace(/^\/+/, ''); +} + +function parseFileInner( + ctx: FileContext, + content: string, + opts: ParseOptions, + refs: ComposeRefs, + visited: Set, + stack: Set, + depth: number, +): void { + const normalized = normalizeRepoPath(ctx.repoPath); + if (depth > MAX_INCLUDE_DEPTH) { + refs.parseErrors.push(`Include/extends graph exceeds depth ${MAX_INCLUDE_DEPTH} at ${normalized}`); + return; + } + if (content.length > MAX_COMPOSE_PARSE_BYTES) { + refs.parseErrors.push(`Compose file ${normalized} exceeds the ${MAX_COMPOSE_PARSE_BYTES}-byte parse cap`); + return; + } + + let doc: unknown; + try { + doc = YAML.parse(content); + } catch (e) { + refs.parseErrors.push(`Cannot parse compose file ${normalized}: ${e instanceof Error ? e.message : String(e)}`); + return; + } + if (!doc || typeof doc !== 'object' || Array.isArray(doc)) return; + + const root = doc as Record; + + // Top-level include: list of paths or maps {path, env_file, project_directory}. + const include = root.include; + if (include !== undefined) { + const items = Array.isArray(include) ? include : [include]; + for (const item of items) { + if (typeof item === 'string') { + processIncludeEntry(refs, [item], undefined, normalized, ctx, opts, visited, stack, depth); + } else if (item && typeof item === 'object' && !Array.isArray(item)) { + const map = item as Record; + // path accepts a string or a list of strings (merged in order). + const paths = asStringList(map.path); + if (paths.length === 0) { + refs.parseErrors.push(`Include entry in ${normalized} has no resolvable path`); + continue; + } + processIncludeEntry(refs, paths, map, normalized, ctx, opts, visited, stack, depth); + } + } + } + + // Per-file service walk (keeps declaring-file provenance for relative refs). + if (root.services && typeof root.services === 'object' && !Array.isArray(root.services)) { + for (const [serviceName, service] of Object.entries(root.services as Record)) { + walkService(serviceName, service, normalized, refs, ctx); + // extends.file recursion after recording the declaration. + if (service && typeof service === 'object' && !Array.isArray(service)) { + const ext = (service as Record).extends; + if (ext && typeof ext === 'object' && !Array.isArray(ext)) { + const fileTarget = asString((ext as Record).file); + if (fileTarget !== undefined) { + const resolved = emitFileRelative(refs, fileTarget, 'extends', 'compose-additional', normalized, ctx); + readAndRecurse(resolved, ctx, opts, refs, visited, stack, depth, 'extends.file target'); + } + } + } + } + } + + // Top-level configs / secrets file forms. + for (const [key, kind, role] of [ + ['configs', 'config', 'config'], + ['secrets', 'secret', 'secret'], + ] as const) { + const resources = root[key]; + if (!resources || typeof resources !== 'object' || Array.isArray(resources)) continue; + for (const def of Object.values(resources as Record)) { + const file = resourceFilePath(def); + if (file !== null) { + emitProjectRelative(refs, file, kind, role, normalized, ctx, 'compose-file-dir'); + } else if (def !== null && def !== undefined) { + // external / env / name-only forms are docker-supplied or + // unresolvable; record an unmanaged placeholder. + emitInput(refs, null, null, null, kind, role, normalized, 'host'); + } + } + } +} + +/** + * Process one include entry (string path, or map with a path list): resolve + * each path against the current level's project base, derive the included + * project's bases (project_directory, or the FIRST resolved path's directory + * per the compose-go "main file" rule), emit the included project's default + * interpolation .env, and recurse. + */ +function processIncludeEntry( + refs: ComposeRefs, + paths: string[], + map: Record | undefined, + fromFile: string, + ctx: FileContext, + opts: ParseOptions, + visited: Set, + stack: Set, + depth: number, +): void { + const resolvedPaths = paths.map((p) => emitFileRelative(refs, p, 'include', 'compose-additional', fromFile, ctx)); + const first = resolvedPaths[0]; + // interpolation: false disables interpolation for the included project, + // so its default .env is never probed. + const interpolate = map?.interpolation !== false; + const projectDir = asString(map?.project_directory); + let childProjectBase: string | null; + let childRuntimeProjectBase: string | null; + if (projectDir !== undefined) { + childProjectBase = resolveWithinBase(ctx.projectBase, projectDir); + childRuntimeProjectBase = resolveWithinBase(ctx.runtimeProjectBase, projectDir); + if (childProjectBase === null || childRuntimeProjectBase === null) { + refs.parseErrors.push(`Include project_directory ${projectDir} in ${fromFile} escapes its base`); + return; + } + } else if (first !== undefined && first.source !== null && first.materialized !== null) { + // Without project_directory, the FIRST resolved path is the included + // project's main file and defines its directory; later paths are + // overrides of the same project. + childProjectBase = dirOf(first.source); + childRuntimeProjectBase = dirOf(first.materialized); + } else { + return; // host/URL/escape: emitted, the classifier decides + } + + // The included project's interpolation env defaults to .env in its + // project directory (compose include spec); absence is tolerated. + // interpolation: false disables it (handled above). When the included + // project's bases equal the parent's (a same-directory include), the + // entry would duplicate the parent's interpolation env, so it is skipped. + if (interpolate && !(childProjectBase === ctx.projectBase && childRuntimeProjectBase === ctx.runtimeProjectBase)) { + emitInput(refs, '.env', resolveWithinBase(childProjectBase, '.env')!, resolveWithinBase(childRuntimeProjectBase, '.env')!, 'interpolation-env', 'env', fromFile, 'repo-root'); + } + + const childOverride = { projectBase: childProjectBase, runtimeProjectBase: childRuntimeProjectBase }; + for (const resolved of resolvedPaths) { + readAndRecurse(resolved, ctx, opts, refs, visited, stack, depth, 'Included compose file', childOverride); + } + + // Explicit env_file accepts a string or a list of strings; resolves + // against the current level's project base (compose-go include env). + for (const env of asStringList(map?.env_file)) { + emitFileRelative(refs, env, 'include-env', 'env', fromFile, ctx); + } +} + +/** Read a resolved include/extends target and recurse when it is a repository file. */ +function readAndRecurse( + resolved: { source: string | null; materialized: string | null }, + ctx: FileContext, + opts: ParseOptions, + refs: ComposeRefs, + visited: Set, + stack: Set, + depth: number, + errorLabel: string, + childOverride?: { projectBase: string | null; runtimeProjectBase: string | null }, +): void { + // Host/URL/escape paths emit with both sides null; the classifier decides. + if (resolved.source === null || resolved.materialized === null) return; + const nested = opts.read(resolved.source); + if (nested === null) { + refs.parseErrors.push(`${errorLabel} ${resolved.source} is unreadable`); + return; + } + parseFile( + { + repoPath: resolved.source, + runtimePath: resolved.materialized, + projectBase: childOverride?.projectBase ?? dirOf(resolved.source), + runtimeProjectBase: childOverride?.runtimeProjectBase ?? dirOf(resolved.materialized), + }, + nested, + opts, + refs, + visited, + stack, + depth + 1, + ); +} + +export function parseDeclaredInputs( + orderedContents: Array<{ path: string; content: string }>, + opts: ParseOptions, +): ParsedDeclaredInputs { + const refs: ComposeRefs = { inputs: [], dynamic: [], parseErrors: [] }; + const visited = new Set(); + const stack = new Set(); + // Merged (-f) files share the TOP project's bases: the context dir, or + // the base (first) file's directory. The primary file lands at the stack + // root at runtime, so its runtime path is '' while additional files keep + // their repository paths. + const orderedProjectBase = opts.projectRoot ?? dirOf(orderedContents[0]?.path ?? ''); + const orderedRuntimeProjectBase = opts.projectRoot ?? null; + for (const [index, file] of orderedContents.entries()) { + parseFile( + { + repoPath: file.path, + runtimePath: index === 0 ? '' : file.path, + projectBase: orderedProjectBase, + runtimeProjectBase: orderedRuntimeProjectBase, + }, + file.content, + opts, + refs, + visited, + stack, + 0, + ); + } + + // Interpolation env at the project root (compose loads it for variable + // substitution). Always declared; classification decides managed vs absent. + const interpRoot = opts.projectRoot ?? ''; + const interpPath = interpRoot ? `${interpRoot}/.env` : '.env'; + emitInput(refs, interpPath, interpPath, interpPath, 'interpolation-env', 'env', '', 'project-root'); + + return { inputs: refs.inputs, dynamic: refs.dynamic, parseErrors: refs.parseErrors }; +} diff --git a/backend/src/routes/gitSources.ts b/backend/src/routes/gitSources.ts index da77c038..2bc45660 100644 --- a/backend/src/routes/gitSources.ts +++ b/backend/src/routes/gitSources.ts @@ -1,5 +1,6 @@ import { Router, type Request, type Response } from 'express'; import { GitSourceService } from '../services/GitSourceService'; +import { GitProjectManifestService } from '../services/GitProjectManifestService'; import { FileSystemService } from '../services/FileSystemService'; import { DatabaseService } from '../services/DatabaseService'; import { CryptoService } from '../services/CryptoService'; @@ -105,9 +106,19 @@ stackGitSourceRouter.get('/:stackName/git-source', async (req: Request, res: Res } if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return; try { - const source = GitSourceService.getInstance().get(stackName); + const gitSources = GitSourceService.getInstance(); + const source = gitSources.get(stackName); if (source) { - res.json(source); + // The managed-project manifest summary rides the source branch; the + // unlinked {linked:false} shape below is unchanged. Heal-on-read may + // rewrite the DB cache, so re-read the flat row for same-response parity. + const manifest = await gitSources.getManifestSummary(stackName); + const refreshed = gitSources.get(stackName) ?? source; + res.json({ + ...refreshed, + manifest_state: manifest?.state ?? refreshed.manifest_state, + manifest, + }); return; } // No source row. A non-existent stack is a genuine 404, but an existing @@ -248,19 +259,11 @@ stackGitSourceRouter.delete('/:stackName/git-source', async (req: Request, res: } if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return; try { - // The deploy spec lives on the Git-source row, so unlinking a multi-file (or - // project-directory) source would silently drop the spec and revert future - // deploys to root compose.yaml auto-discovery, ignoring the override files - // still on disk. Refuse rather than change deploy semantics out from under the - // user; deleting the stack removes it cleanly. - const spec = DatabaseService.getInstance().getGitSource(stackName)?.applied_deploy_spec; - if (spec && (spec.files.length > 1 || spec.contextDir)) { - res.status(409).json({ - error: 'This stack deploys multiple compose files configured by its Git source. Unlinking would change it to deploy only compose.yaml. Delete the stack to remove it, or keep the Git source.', - }); - return; - } - GitSourceService.getInstance().delete(stackName); + // Detach with the export contract: the effective compose model is rendered + // into a single compose.yaml and the materialized files are kept, so a + // multi-file / project-directory stack stays deployable after unlinking. + // A render failure aborts with 409 and the row is left intact. + await GitSourceService.getInstance().detach(stackName); // The cached /stacks/statuses payload carries the source label; drop it // before responding so a client refetch on this response recomputes. The // full invalidateNodeCaches helper is deliberate here (matching every @@ -268,8 +271,34 @@ stackGitSourceRouter.delete('/:stackName/git-source', async (req: Request, res: // dropping the project-name map and file-root allowlists alongside is // harmless, unlike the high-frequency container-event path. invalidateNodeCaches(req.nodeId); - console.log(`[GitSource] Removed git source for ${stackName}`); + console.log(`[GitSource] Detached git source for ${stackName}`); res.json({ success: true }); + } catch (error) { + const code = (error as { code?: string } | null)?.code; + if (code === 'RENDER_FAILED') { + res.status(409).json({ error: (error as Error).message }); + return; + } + sendGitSourceError(res, error); + } +}); + +stackGitSourceRouter.get('/:stackName/git-source/manifest', async (req: Request, res: Response): Promise => { + const stackName = req.params.stackName as string; + if (!isValidStackName(stackName)) { + res.status(400).json({ error: 'Invalid stack name' }); + return; + } + if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return; + try { + const manifest = await GitSourceService.getInstance().getManifest(stackName); + if (!manifest) { + res.status(404).json({ error: 'No managed-project manifest for this stack' }); + return; + } + // The public projection: no content hashes, size metadata, provenance, or + // deletion authority, and high-sensitivity input paths are redacted. + res.json({ manifest: GitProjectManifestService.getInstance().toPublicManifest(manifest) }); } catch (error) { sendGitSourceError(res, error); } diff --git a/backend/src/services/ComposeInputDiscoveryService.ts b/backend/src/services/ComposeInputDiscoveryService.ts new file mode 100644 index 00000000..3882a0c6 --- /dev/null +++ b/backend/src/services/ComposeInputDiscoveryService.ts @@ -0,0 +1,1040 @@ +/** + * Two-phase Compose input discovery for the Git managed-project materializer. + * + * Phase 1 (pure) lives in helpers/composeInputParse.ts; this service runs + * phase 2 against the cloned tree: resolving every declared input against the + * authorized repo/project boundary, classifying it managed / unmanaged / + * refused, planning dockerignore-aware build contexts, and enforcing the + * materialization bounds. The copy loop (walkAndCopy) is shared with the + * manifest service's candidate builder so discovery and promotion cannot drift. + * + * Refusal policy: actionable refusals (out-of-bounds, url-include, submodule, + * LFS, unbounded context, missing files, unsafe links) are returned with + * actionable: true so the caller aborts the pull; tolerated classes (host + * binds, external resources) become unmanaged entries with a documented- + * limitation note. Dynamic \${VAR} paths are recorded as explicit unmanaged + * entries (they resolve at deploy time from the environment) and are never + * claimed as covered. Never claim coverage for anything else. + */ +import path from 'path'; +import fs from 'fs'; +import { createHash } from 'crypto'; +import { isHostAbsolutePath, isUrl, parseDeclaredInputs } from '../helpers/composeInputParse'; +import { isPathWithinBase } from '../utils/validation'; +import { isLfsPointer } from './GitSourceService'; +import { loadDockerIgnore, type DockerIgnoreMatcher } from '../utils/dockerIgnoreMatch'; +import { PRIMARY_COMPOSE_FILENAME } from '../utils/gitComposeFiles'; +import type { + BuildContextPlan, + ComposeInputEntry, + DeclaredInput, + DynamicInput, + InputDependencyKind, + InputRole, + InputSensitivity, + InventoryResult, + ManifestBounds, + RefusalInfo, +} from '../types/gitProjectManifest'; + +// Compose override filenames docker compose can auto-discover, in priority +// order; mirrors FileSystemService.COMPOSE_OVERRIDE_FILENAMES. Only consulted +// when the invocation passes a single explicit -f (explicit multi-file lists +// suppress auto-discovery). +const COMPOSE_OVERRIDE_FILENAMES = [ + 'compose.override.yaml', + 'compose.override.yml', + 'docker-compose.override.yaml', + 'docker-compose.override.yml', +]; + +const LFS_POINTER_PREFIX_LEN = 48; + +// Include/extends recursion reads stay under this bound (the clone download +// cap bounds the pack, not the decompressed tree). +const MAX_REPO_READ_BYTES = 10 * 1024 * 1024; + +export interface DiscoverFromCloneParams { + /** Root of the cloned tree (immutable resolved revision). */ + cloneDir: string; + /** Ordered explicit repo-relative compose paths. */ + composePaths: string[]; + /** Repo-relative project root (context_dir); null = repo root. */ + contextDir: string | null; + /** True when a synced stack-root .env is deployed (owns that path). */ + syncEnv?: boolean; + bounds: ManifestBounds; +} + +export interface CopyEntry { + srcRel: string; // clone-relative source + destRel: string; // candidate-relative destination +} + +export interface ContextCopyPlan { + context: BuildContextPlan; + /** Clone-relative path of the context root. */ + srcRel: string; + /** Candidate-relative destination (same layout as deploy). */ + destRel: string; + matcher: DockerIgnoreMatcher | null; + /** Clone-relative path of the dockerignore file applied (for diagnostics). */ + dockerignoreRel: string | null; +} + +export interface CopyResult { + copiedFiles: number; + copiedBytes: number; +} + +function posixRel(p: string): string { + return p.replace(/\\/g, '/').replace(/^\.\//, ''); +} + +/** Parent directory of a POSIX relative path; empty string at repo root. */ +function posixDir(p: string): string { + const rel = posixRel(p); + const idx = rel.lastIndexOf('/'); + return idx === -1 ? '' : rel.slice(0, idx); +} + +function caseKey(s: string): string { + return s.toLowerCase(); +} + +/** Map a dynamic declaration's dependency kind to its manifest role. */ +function roleForDynamicKind(kind: DynamicInput['kind']): InputRole { + switch (kind) { + case 'env_file': + case 'include-env': + case 'interpolation-env': + return 'env'; + case 'config': + return 'config'; + case 'secret': + return 'secret'; + case 'build-context': + return 'build-context'; + case 'build-additional-context': + return 'build-additional-context'; + case 'dockerfile': + return 'dockerfile'; + case 'build-secret': + return 'build-secret'; + case 'label_file': + return 'label-file'; + case 'bind-mount': + return 'bind-mount'; + case 'include': + case 'extends': + return 'compose-additional'; + default: + return 'other'; + } +} + +function hasGitMetaSegment(relPath: string): boolean { + return posixRel(relPath) + .split('/') + .filter(Boolean) + .some((seg) => seg.toLowerCase() === '.git'); +} + +/** The submodule whose directory contains the path, or undefined. */ +function submoduleOwner(relPath: string, submodules: string[]): string | undefined { + return submodules.find((s) => relPath === s || relPath.startsWith(`${s}/`)); +} + +/** Kinds whose declared paths must be treated as sensitive. */ +function isSensitiveKind(kind: InputDependencyKind): boolean { + return kind === 'config' || kind === 'secret' || kind === 'env_file' || kind === 'include-env' + || kind === 'interpolation-env' || kind === 'label_file' || kind === 'build-secret'; +} + +function sensitivityFor(kind: InputDependencyKind, fallback: InputSensitivity): InputSensitivity { + return isSensitiveKind(kind) ? 'high' : fallback; +} + +/** User-facing note for an unmanaged host entry. */ +function hostEntryNote(hostAbsolute: boolean, kind: InputDependencyKind, baseDir: DeclaredInput['baseDir']): string { + if (hostAbsolute) return 'Host path; provided by the node, not materialized from the repository'; + if (baseDir === 'host') { + return kind === 'bind-mount' + ? 'Host bind mount; provided by the node, not materialized from the repository' + : 'External resource supplied by Docker or the node; not materialized from the repository'; + } + return 'Declared without a resolvable file path'; +} + +async function readSubmodulePaths(cloneDir: string): Promise { + const gitmodules = path.join(cloneDir, '.gitmodules'); + try { + const raw = await fs.promises.readFile(gitmodules, 'utf8'); + const paths: string[] = []; + for (const line of raw.split(/\r?\n/)) { + const m = /^\s*path\s*=\s*(.+)$/.exec(line); + if (m) paths.push(posixRel(m[1].trim())); + } + return paths; + } catch { + return []; + } +} + +export class ComposeInputDiscoveryService { + private constructor() { + // Stateless service; instantiate via getInstance(). + } + + private static instance: ComposeInputDiscoveryService | null = null; + + static getInstance(): ComposeInputDiscoveryService { + if (!this.instance) this.instance = new ComposeInputDiscoveryService(); + return this.instance; + } + + private refusal(sourcePath: string | null, kind: string, reason: string, actionable: boolean, sensitivity: InputSensitivity = 'medium'): RefusalInfo { + return { sourcePath, kind, reason, actionable, sensitivity }; + } + + /** Unmanaged entry for an optional env_file compose skips at deploy time. */ + private optionalEnvUnmanaged(input: DeclaredInput, sourcePath: string, note: string): ComposeInputEntry { + return { + sourcePath, + materializedPath: null, + role: input.role, + dependencyKind: input.kind, + ownership: 'unmanaged', + provenance: 'fetch', + sensitivity: sensitivityFor(input.kind, 'medium'), + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'none', + note, + }; + } + + /** + * Classify one resolved clone-relative path. Returns the refusal on + * failure; callers record actionable refusals. + */ + private async classifyPath( + cloneDir: string, + relPath: string, + bounds: ManifestBounds, + ): Promise<{ ok: true; sizeBytes: number; contentSha256: string } | { ok: false; refusal: RefusalInfo }> { + const abs = path.resolve(cloneDir, relPath); + if (!isPathWithinBase(abs, path.resolve(cloneDir))) { + return { ok: false, refusal: this.refusal(relPath, 'out-of-bounds', `${relPath} resolves outside the repository`, true) }; + } + if (hasGitMetaSegment(relPath)) { + return { ok: false, refusal: this.refusal(relPath, 'git-meta', `${relPath} targets the .git metadata directory`, true) }; + } + const depth = relPath.split('/').filter(Boolean).length; + if (depth > bounds.maxPathDepth) { + return { ok: false, refusal: this.refusal(relPath, 'path-too-deep', `${relPath} exceeds the path depth limit of ${bounds.maxPathDepth}`, true) }; + } + + let stat: fs.Stats; + try { + stat = await fs.promises.lstat(abs); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') { + return { ok: false, refusal: this.refusal(relPath, 'missing-file', `File not found in repository: ${relPath}`, true) }; + } + // The OS error message embeds the ABSOLUTE path (with the secret + // file name intact) and would leak through public refusals, so it + // is never included. + return { ok: false, refusal: this.refusal(relPath, 'unreadable', `Cannot stat ${relPath}`, true) }; + } + if (stat.isSymbolicLink()) { + return { ok: false, refusal: this.refusal(relPath, 'unsafe-symlink', `${relPath} is a symbolic link`, true) }; + } + if (stat.isDirectory()) { + return { ok: false, refusal: this.refusal(relPath, 'not-a-file', `${relPath} is a directory; a file is required`, true) }; + } + if (stat.isCharacterDevice() || stat.isBlockDevice() || stat.isSocket() || stat.isFIFO()) { + return { ok: false, refusal: this.refusal(relPath, 'special-file', `${relPath} is a device, socket or pipe`, true) }; + } + if (stat.size > bounds.maxFileBytes) { + return { ok: false, refusal: this.refusal(relPath, 'file-too-large', `${relPath} is too large (${stat.size} bytes; maximum ${bounds.maxFileBytes})`, true) }; + } + let content: Buffer; + try { + content = await fs.promises.readFile(abs); + } catch { + // The OS error message embeds the ABSOLUTE path; never included. + return { ok: false, refusal: this.refusal(relPath, 'unreadable', `Cannot read ${relPath}`, true) }; + } + if (isLfsPointer(content.toString('utf8', 0, LFS_POINTER_PREFIX_LEN + 64))) { + return { ok: false, refusal: this.refusal(relPath, 'lfs-pointer', `${relPath} is a Git LFS pointer; LFS content is not fetched`, true) }; + } + return { ok: true, sizeBytes: stat.size, contentSha256: createHash('sha256').update(content).digest('hex') }; + } + + /** + * Plan build contexts with dockerignore semantics and byte accounting. + * Context inputs arrive parser-resolved: sourcePath = the repository + * context root, materializedPath = the stack-relative context root. + */ + private async planBuildContexts( + params: DiscoverFromCloneParams, + submodules: string[], + contextInputs: DeclaredInput[], + dockerfileInputs: DeclaredInput[], + refusals: RefusalInfo[], + ): Promise<{ plans: ContextCopyPlan[]; entries: ComposeInputEntry[] }> { + const { cloneDir, bounds, syncEnv } = params; + const plans: ContextCopyPlan[] = []; + const entries: ComposeInputEntry[] = []; + + for (const input of contextInputs) { + const sourceResolved = input.sourcePath; + const materializedResolved = input.materializedPath; + if (sourceResolved === null || materializedResolved === null) { + // Non-host context inputs always carry both coordinates; a + // missing side means the parser could not reproduce the path + // in the stack layout and must never be guessed. + refusals.push(this.refusal(input.sourcePath, 'out-of-bounds', `Build context ${input.sourcePath ?? '(unnamed)'} cannot be reproduced in the stack layout`, true)); + continue; + } + // A repo-root context (`build: .`) is canonicalized to the empty + // relative path: materializedPath '' is valid per the manifest + // path rules, promotes the candidate root, and never collides + // with '.'-rejection in manifest validation. + const sourceRoot = sourceResolved === '.' || sourceResolved === '' ? '' : sourceResolved; + const materializedRoot = materializedResolved === '.' || materializedResolved === '' ? '' : materializedResolved; + + const abs = path.resolve(cloneDir, sourceRoot); + let stat: fs.Stats; + try { + stat = await fs.promises.lstat(abs); + } catch { + refusals.push(this.refusal(sourceRoot, 'missing-file', `Build context ${sourceRoot} does not exist in the repository`, true)); + continue; + } + if (stat.isSymbolicLink()) { + refusals.push(this.refusal(sourceRoot, 'unsafe-symlink', `Build context ${sourceRoot} is a symbolic link`, true)); + continue; + } + if (!stat.isDirectory()) { + refusals.push(this.refusal(sourceRoot, 'not-a-directory', `Build context ${sourceRoot} is not a directory`, true)); + continue; + } + + // An explicit dockerfile resolves RELATIVE TO THE BUILD CONTEXT + // (docker compose build spec); the parser emits it with + // compose-file-dir provenance, so it is rebased here against the + // context root and validated for containment. + const dockerfileDecl = dockerfileInputs.find( + (i) => + i.kind === 'dockerfile' && + i.fromFile === input.fromFile && + i.service === input.service, + ); + let dockerfileRel: string | null = null; + let dockerfileOutsideContext = false; + // Additional contexts are not the primary build context and never + // inherit the service's dockerfile. + const isPrimaryContext = input.kind === 'build-context'; + if (isPrimaryContext && dockerfileDecl && typeof dockerfileDecl.sourcePath === 'string') { + const rebased = path.posix.normalize(path.posix.join(sourceRoot, dockerfileDecl.sourcePath)); + if (rebased === '..' || rebased.startsWith('../') || path.posix.isAbsolute(rebased)) { + refusals.push(this.refusal(dockerfileDecl.sourcePath, 'out-of-bounds', `Dockerfile ${dockerfileDecl.sourcePath} resolves outside the repository`, true)); + continue; + } + dockerfileRel = rebased; + // Compose resolves the dockerfile relative to the context. A + // `../` form that stays inside the repository is allowed: the + // dockerfile lands outside the context subtree, so it is + // materialized as its own managed input below. Root contexts + // (`''` or `'.'`) contain every repo-relative path. + const inContext = sourceRoot === '' + ? !dockerfileRel.startsWith('../') + : dockerfileRel === sourceRoot || dockerfileRel.startsWith(`${sourceRoot}/`); + dockerfileOutsideContext = !inContext; + } + // Docker's ignore selection: the context-root .dockerignore applies + // by default; a Dockerfile-specific ignore file named + // `.dockerignore` next to the dockerfile takes + // precedence when present. + let matcher = await loadDockerIgnore(abs); + let dockerignoreRel = matcher !== null ? path.relative(cloneDir, path.join(abs, '.dockerignore')).replace(/\\/g, '/') : null; + if (dockerfileRel !== null && !dockerfileOutsideContext) { + const dfBase = dockerfileRel.split('/').pop() ?? ''; + const dfRelDir = dockerfileRel.includes('/') ? dockerfileRel.slice(0, dockerfileRel.lastIndexOf('/')) : ''; + const specificAbs = path.join(cloneDir, dfRelDir); + const specificFile = path.join(specificAbs, `${dfBase}.dockerignore`); + const specificExists = await fs.promises.access(specificFile).then(() => true).catch(() => false); + if (specificExists) { + const specificMatcher = await loadDockerIgnore(specificAbs, `${dfBase}.dockerignore`); + if (specificMatcher !== null) { + matcher = specificMatcher; + dockerignoreRel = path.relative(cloneDir, specificFile).replace(/\\/g, '/'); + } + } + } + // Submodule containment (mirrors the ordinary-input check below): + // a context rooted inside a submodule, or a submodule directory + // inside the context, omits content the author's build would + // include (submodule contents are never fetched). Refuse rather + // than materialize a context that silently lacks it. A submodule + // excluded by the context's dockerignore is excluded by the + // author's build too, so it is not a refusal. + const owner = submoduleOwner(sourceRoot, submodules); + if (owner !== undefined) { + refusals.push(this.refusal(sourceRoot, 'submodule', `Build context ${sourceRoot} is inside Git submodule ${owner}; submodule contents are not fetched`, true)); + continue; + } + const submoduleInContext = submodules.find((s) => { + if (s === sourceRoot) return false; + const rel = sourceRoot ? (s.startsWith(`${sourceRoot}/`) ? s.slice(sourceRoot.length + 1) : null) : s; + if (rel === null) return false; + // The walk prunes a directory subtree when any ancestor (or + // the directory itself) matches; mirror that for the submodule + // path so an ignored submodule is not a refusal. + let prefix = ''; + for (const seg of rel.split('/')) { + prefix = prefix ? `${prefix}/${seg}` : seg; + if (matcher?.matches(prefix, true) ?? false) return false; + } + return true; + }); + if (submoduleInContext !== undefined) { + refusals.push(this.refusal(sourceRoot, 'submodule', `Build context ${sourceRoot} contains Git submodule ${submoduleInContext}; submodule contents are not fetched`, true)); + continue; + } + + if (dockerfileOutsideContext) { + // Materialize the out-of-context dockerfile as its own managed + // input (it is outside the context subtree the walk copies), + // classified with the same lstat/symlink/containment/size/LFS + // guards as every other repository input. Its materialized + // path follows the same runtime base as the context itself + // (compose resolves the dockerfile relative to the context at + // its materialized location). + const dfMaterialized = path.posix.normalize(path.posix.join(materializedRoot, dockerfileDecl!.sourcePath!)); + if (dfMaterialized === '..' || dfMaterialized.startsWith('../') || path.posix.isAbsolute(dfMaterialized)) { + refusals.push(this.refusal(dockerfileRel, 'out-of-bounds', `Dockerfile ${dockerfileRel} cannot be reproduced in the stack layout`, true)); + continue; + } + const dfClassified = await this.classifyPath(cloneDir, dockerfileRel!, bounds); + if (!dfClassified.ok) { + refusals.push(dfClassified.refusal); + continue; + } + entries.push({ + sourcePath: dockerfileRel, + materializedPath: dfMaterialized, + role: 'dockerfile', + dependencyKind: 'dockerfile', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'medium', + contentSha256: dfClassified.contentSha256, + sizeBytes: dfClassified.sizeBytes, + state: 'present', + deletionAuthority: 'sencho', + note: 'Dockerfile outside the build context; materialized at its stack-relative path', + }); + } + + // Walk the context subtree: filtered bytes, ignored count, LFS and + // special-file detection, and the per-file hash inventory that + // gives the context file-granular ownership. + let contextBytes = 0; + let ignoredCount = 0; + let lfsInContext = false; + let specialInContext: string | null = null; + const contextFiles: Array<{ path: string; sha256: string; sizeBytes: number }> = []; + const walk = async (dir: string, rel: string): Promise => { + let entriesList: fs.Dirent[]; + try { + entriesList = await fs.promises.readdir(dir, { withFileTypes: true }); + } catch (e) { + specialInContext = `Cannot read ${rel}: ${(e as Error).message}`; + return; + } + for (const entry of entriesList) { + // Never counted or copied: `.git` is excluded from the + // materialized context by the copier, and counting it here + // would trip the size caps on repo-root contexts. + if (entry.name.toLowerCase() === '.git') continue; + const childRel = rel ? `${rel}/${entry.name}` : entry.name; + if (materializedRoot === '' && childRel === '.env' && syncEnv) { + // The synced stack-root .env owns this path; the repo + // copy must not be hash-guarded against it. + continue; + } + if (matcher?.matches(childRel, entry.isDirectory())) { + ignoredCount += 1; + continue; + } + if (entry.isSymbolicLink()) { + specialInContext = `${childRel} is a symbolic link inside the build context`; + return; + } + if (entry.isDirectory()) { + await walk(path.join(dir, entry.name), childRel); + if (specialInContext) return; + continue; + } + if (entry.isCharacterDevice() || entry.isBlockDevice() || entry.isSocket() || entry.isFIFO()) { + specialInContext = `${childRel} is a device, socket or pipe inside the build context`; + return; + } + const st = await fs.promises.stat(path.join(dir, entry.name)); + if (st.size > bounds.maxFileBytes) { + specialInContext = `${childRel} is too large for the build context (${st.size} bytes)`; + return; + } + contextBytes += st.size; + const content = await fs.promises.readFile(path.join(dir, entry.name)); + contextFiles.push({ path: childRel, sha256: createHash('sha256').update(content).digest('hex'), sizeBytes: st.size }); + if (!lfsInContext) { + const handle = await fs.promises.open(path.join(dir, entry.name), 'r'); + try { + const buf = Buffer.alloc(LFS_POINTER_PREFIX_LEN + 64); + const { bytesRead } = await handle.read(buf, 0, buf.length, 0); + if (isLfsPointer(buf.toString('utf8', 0, bytesRead))) lfsInContext = true; + } finally { + await handle.close(); + } + } + } + }; + await walk(abs, ''); + + const isRepoRoot = sourceRoot === ''; + let note: string | null = null; + if (specialInContext !== null) { + refusals.push(this.refusal(sourceRoot, 'unsafe-context', specialInContext, true)); + continue; + } + if (lfsInContext) { + refusals.push(this.refusal(sourceRoot, 'lfs-in-context', `Build context ${sourceRoot} contains Git LFS pointers`, true)); + continue; + } + if (isRepoRoot) { + note = `Context is the repository root; bounded by GITSOURCE_MAX_BUILD_CONTEXT_BYTES (${bounds.maxContextBytes} bytes)`; + } + + const plan: BuildContextPlan = { + repoPath: materializedRoot, + dockerfile: dockerfileRel, + contextBytes, + ignoredCount, + dockerignoreApplied: matcher !== null, + excludedFromCopy: false, + note, + files: contextFiles, + }; + plans.push({ + context: plan, + srcRel: sourceRoot, + destRel: materializedRoot, + matcher, + dockerignoreRel, + }); + // Contexts are tracked in buildContexts with per-file inventories; + // a materializedPath entry in the input list is only emitted for + // non-root contexts (a directory on disk). The root context (empty + // path) must never become a stack-relative path for promotion or + // stale cleanup to write/delete. + if (materializedRoot !== '') { + entries.push({ + sourcePath: sourceRoot, + materializedPath: materializedRoot, + role: 'build-context', + dependencyKind: 'build-context', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'low', + contentSha256: null, + sizeBytes: contextBytes, + state: 'present', + deletionAuthority: 'sencho', + note, + }); + } + } + // Merge context plans sharing the same root: multiple services in + // one compose file referencing the same context with different + // Dockerfiles produce one plan with the union of all file inventories + // so no service loses required files. + const mergedPlans: ContextCopyPlan[] = []; + for (const plan of plans) { + const existing = mergedPlans.find((mp) => caseKey(mp.context.repoPath) === caseKey(plan.context.repoPath)); + if (existing) { + for (const f of plan.context.files) { + if (!existing.context.files.some((ef) => caseKey(ef.path) === caseKey(f.path))) { + existing.context.files.push(f); + existing.context.contextBytes += f.sizeBytes; + } + } + } else { + mergedPlans.push(plan); + } + } + return { plans: mergedPlans, entries }; + } + + /** + * Run full discovery against the clone. Returns the classified inventory + * plus the copy plans (dockerignore matchers) the candidate builder needs; + * actionable refusals are included and the caller decides abort vs + * tolerate. + */ + async discoverFromClone( + params: DiscoverFromCloneParams, + ): Promise { + const { cloneDir, composePaths, contextDir, bounds } = params; + const projectRoot = contextDir ?? null; + const refusals: RefusalInfo[] = []; + const dynamic: DynamicInput[] = []; + const submodules = await readSubmodulePaths(cloneDir); + + // Read the ordered explicit compose files (non-throwing). + const ordered: Array<{ path: string; content: string }> = []; + for (const p of composePaths) { + const result = await this.classifyPath(cloneDir, p, bounds); + if (!result.ok) { + refusals.push(result.refusal); + continue; + } + try { + ordered.push({ path: p, content: await fs.promises.readFile(path.join(cloneDir, p), 'utf8') }); + } catch (e) { + refusals.push(this.refusal(p, 'unreadable', `Cannot read ${p}: ${(e as Error).message}`, true)); + } + } + if (ordered.length === 0) { + return { inputs: [], refusals, buildContexts: [], contextCopyPlans: [], dynamic, counts: { managed: 0, unmanaged: 0, refused: refusals.length } }; + } + + // Implicit override: only when the runtime invocation stays plain + // `docker compose` auto-discovery. A single explicit -f with no project + // directory keeps auto-discovery (deriveAppliedSpec returns null), but + // a configured project directory forces explicit -f arguments, which + // suppress auto-discovery (compose merge docs: explicit -f disables it). + // + // Search next to the primary compose file (Compose's effective project + // directory), not at projectRoot / the repository root when those + // differ. A monorepo subproject must not absorb a sibling project's + // root-level override. + let implicitOverridePath: string | null = null; + if (composePaths.length === 1 && !contextDir) { + const primaryDir = posixDir(composePaths[0]); + for (const candidate of COMPOSE_OVERRIDE_FILENAMES) { + const rel = primaryDir ? `${primaryDir}/${candidate}` : candidate; + const result = await this.classifyPath(cloneDir, rel, bounds); + if (result.ok) { + implicitOverridePath = rel; + ordered.push({ path: rel, content: await fs.promises.readFile(path.join(cloneDir, rel), 'utf8') }); + break; + } + } + } + + const parsed = parseDeclaredInputs(ordered, { + projectRoot, + read: (repoPath) => { + // Containment + size bound at the read boundary: include/extends + // targets can carry `..` segments, and an attacker-controlled + // repo must never make the parser read outside the clone or pull + // an unbounded file into memory. + if (hasGitMetaSegment(repoPath)) return null; + const abs = path.resolve(cloneDir, repoPath); + if (!isPathWithinBase(abs, path.resolve(cloneDir))) return null; + try { + const st = fs.statSync(abs); + if (!st.isFile() || st.size > MAX_REPO_READ_BYTES) return null; + return fs.readFileSync(abs, 'utf8'); + } catch { + return null; + } + }, + }); + for (const err of parsed.parseErrors) refusals.push(this.refusal(null, 'parse-error', err, true)); + dynamic.push(...parsed.dynamic); + + // Track running aggregate counts against the bounds (explicit compose + // files and the implicit override count toward the caps too). + let managedCount = 0; + let managedBytes = 0; + const inputs: ComposeInputEntry[] = []; + + // Explicit compose files (ordered). The content is already in memory, + // so the content hash is computed here: the apply-time divergence guard + // needs a sha for every managed present file, compose.yaml included, or + // a hand-edited compose file would be silently overwritten. + composePaths.forEach((p, index) => { + const local = index === 0 ? PRIMARY_COMPOSE_FILENAME : posixRel(p); + const role: InputRole = index === 0 ? 'compose-primary' : 'compose-additional'; + const content = ordered.find((o) => o.path === p)?.content ?? null; + inputs.push({ + sourcePath: p, + materializedPath: local, + role, + dependencyKind: 'explicit', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'medium', + contentSha256: content !== null ? createHash('sha256').update(content).digest('hex') : null, + sizeBytes: content !== null ? Buffer.byteLength(content, 'utf8') : 0, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }); + managedCount += 1; + managedBytes += content !== null ? Buffer.byteLength(content, 'utf8') : 0; + }); + if (implicitOverridePath) { + const content = ordered.find((o) => o.path === implicitOverridePath)?.content ?? ''; + // Primary compose relocates to stack-root compose.yaml; the override + // must land beside it (basename only) so plain auto-discovery finds it. + inputs.push({ + sourcePath: implicitOverridePath, + materializedPath: path.posix.basename(posixRel(implicitOverridePath)), + role: 'compose-override', + dependencyKind: 'implicit-override', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'medium', + contentSha256: createHash('sha256').update(content).digest('hex'), + sizeBytes: Buffer.byteLength(content, 'utf8'), + state: 'present', + deletionAuthority: 'sencho', + note: 'Implicit compose override auto-discovered for single-file stacks', + }); + managedCount += 1; + managedBytes += Buffer.byteLength(content, 'utf8'); + } + + // Classify the parser's declarations. The parser resolved every path + // in both coordinate systems (source = repo, materialized = runtime); + // this loop only classifies against the clone and records entries. + for (const input of parsed.inputs) { + const kind = input.kind; + // Host forms (absolute, home-relative, and out-of-repo paths, + // external resources, name-only references): unmanaged, never + // copied, never deleted. Include/extends cannot be enumerated + // from a host path, so they are refused rather than silently + // dropped from the model. + if (input.sourcePath === null || input.baseDir === 'host') { + if (kind === 'include' || kind === 'extends') { + refusals.push(this.refusal(input.sourcePath, 'out-of-bounds', `${input.sourcePath} is outside the repository and cannot be fetched`, true, sensitivityFor(kind, 'medium'))); + continue; + } + const hostAbsolute = input.sourcePath !== null && isHostAbsolutePath(input.sourcePath); + inputs.push({ + sourcePath: null, + materializedPath: null, + role: input.role, + dependencyKind: kind, + ownership: 'unmanaged', + provenance: 'fetch', + sensitivity: sensitivityFor(kind, 'low'), + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'none', + note: hostEntryNote(hostAbsolute, kind, input.baseDir), + }); + continue; + } + if (isUrl(input.sourcePath)) { + // URL includes can carry embedded credentials; treat them as + // high sensitivity so the redaction covers them. + refusals.push(this.refusal(input.sourcePath, 'url-include', `${input.sourcePath} is a URL; remote includes are not fetched`, true, 'high')); + continue; + } + + // Build contexts and their dockerfiles are planned separately + // below (the dockerfile is rebased against its context there). + if (kind === 'build-context' || kind === 'build-additional-context' || kind === 'dockerfile') continue; + + const sourcePath = input.sourcePath; + const materializedPath = input.materializedPath; + if (materializedPath === null) { + // Non-host inputs always carry both coordinates; a missing + // materialized side must never be guessed from the source. + refusals.push(this.refusal(sourcePath, 'out-of-bounds', `${sourcePath} cannot be reproduced in the stack layout`, true)); + continue; + } + + // Submodule containment check. An OPTIONAL env_file inside a + // submodule is absent at deploy time and compose skips it, so it + // is recorded as unmanaged like any other missing optional file. + const inSubmodule = submoduleOwner(sourcePath, submodules) ?? null; + if (inSubmodule !== null) { + if (input.required === false) { + inputs.push(this.optionalEnvUnmanaged(input, sourcePath, 'Optional env file inside a Git submodule; compose skips it at deploy time')); + continue; + } + refusals.push(this.refusal(sourcePath, 'submodule', `${sourcePath} is inside Git submodule ${inSubmodule}; submodule contents are not fetched`, true, sensitivityFor(kind, 'medium'))); + continue; + } + + // When the synced stack-root .env owns the same path (no project + // dir), the interpolation env must NOT be hash-guarded: the apply + // stages the sync content over it, so a managed entry would record + // the repo hash and the next apply would refuse .env as locally + // modified forever. Record it unmanaged in that case. + if (kind === 'interpolation-env' && params.syncEnv === true && sourcePath === '.env') { + inputs.push({ + sourcePath, + materializedPath: null, + role: 'env', + dependencyKind: 'interpolation-env', + ownership: 'unmanaged', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'none', + note: 'Owned by the synced stack-root .env; not hash-guarded by the repository copy', + }); + continue; + } + + // A missing interpolation .env is not a refusal: compose tolerates an + // absent project env (variables resolve from the environment), and the + // synced stack-root .env covers the common case. Record it unmanaged. + if (kind === 'interpolation-env' && !fs.existsSync(path.join(cloneDir, sourcePath))) { + inputs.push({ + sourcePath, + materializedPath: null, + role: 'env', + dependencyKind: 'interpolation-env', + ownership: 'unmanaged', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'none', + note: 'No project .env in the repository; interpolation falls back to the environment at deploy time', + }); + continue; + } + + const classified = await this.classifyPath(cloneDir, sourcePath, bounds); + if (!classified.ok) { + if (classified.refusal.kind === 'missing-file' && input.required === false) { + // env_file map form with required: false: compose skips a + // missing optional file instead of failing. Record it as + // an unmanaged entry (never a refusal). + inputs.push(this.optionalEnvUnmanaged(input, sourcePath, 'Optional env file not present in the repository; compose skips it')); + continue; + } + refusals.push({ ...classified.refusal, sensitivity: sensitivityFor(kind, 'medium') }); + continue; + } + if (managedCount + 1 > bounds.maxFiles) { + refusals.push(this.refusal(sourcePath, 'too-many-files', `Materialization would exceed ${bounds.maxFiles} files`, true, sensitivityFor(kind, 'medium'))); + continue; + } + if (managedBytes + classified.sizeBytes > bounds.maxBytes) { + refusals.push(this.refusal(sourcePath, 'too-many-bytes', `Materialization would exceed ${bounds.maxBytes} bytes (${managedBytes} so far)`, true, sensitivityFor(kind, 'medium'))); + continue; + } + managedCount += 1; + managedBytes += classified.sizeBytes; + const sensitivity: InputSensitivity = isSensitiveKind(kind) ? 'high' : 'medium'; + inputs.push({ + sourcePath, + materializedPath, + role: input.role, + dependencyKind: kind, + ownership: 'managed', + provenance: 'fetch', + sensitivity, + contentSha256: classified.contentSha256, + sizeBytes: classified.sizeBytes, + state: 'present', + deletionAuthority: 'sencho', + note: null, + }); + } + + // Dynamic ${VAR} paths resolve at deploy time from the environment and + // can never be enumerated against the clone. Persist each as an + // explicit unmanaged entry so the manifest inventory never silently + // drops a declared input (create and pull both consume this inventory). + // Dynamic include/extends cannot be materialized at all (the candidate + // can never contain the variable-resolved file), so they are refused. + for (const dyn of dynamic) { + if (dyn.kind === 'include' || dyn.kind === 'extends') { + refusals.push(this.refusal(dyn.sourcePath, 'dynamic-include', `${dyn.sourcePath} is a dynamic ${dyn.kind} path; the included file cannot be materialized`, true)); + continue; + } + inputs.push({ + sourcePath: dyn.sourcePath, + materializedPath: null, + role: roleForDynamicKind(dyn.kind), + dependencyKind: dyn.kind, + ownership: 'unmanaged', + provenance: 'fetch', + sensitivity: sensitivityFor(dyn.kind, 'medium'), + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'none', + note: dyn.note, + }); + } + + // Build contexts. + const contextInputs = parsed.inputs.filter((i) => (i.kind === 'build-context' || i.kind === 'build-additional-context') && i.baseDir !== 'host'); + const dockerfileInputs = parsed.inputs.filter((i) => i.kind === 'dockerfile' && i.baseDir !== 'host'); + const { plans, entries: contextEntries } = await this.planBuildContexts(params, submodules, contextInputs, dockerfileInputs, refusals); + inputs.push(...contextEntries); + + // Sync env entry (stack-root .env) is recorded by the caller (it knows + // sync_env + env_path); interpolation-env classification is covered above. + + // Root contexts share the stack root with managed inputs (compose.yaml, + // .env, configs). Remove context files that already have a managed-input + // owner so the manifest collision check and candidate copy never dupe. + const managedPaths = new Set(inputs.filter((i) => i.materializedPath !== null).map((i) => caseKey(i.materializedPath!))); + const reconciledBuildContexts = plans.map((p) => { + const files = p.context.files.filter((f) => { + const key = caseKey(p.context.repoPath ? `${p.context.repoPath}/${f.path}` : f.path); + return !managedPaths.has(key); + }); + return { + ...p, + context: { + ...p.context, + files, + contextBytes: files.reduce((total, file) => total + file.sizeBytes, 0), + }, + }; + }); + for (const plan of reconciledBuildContexts) { + if (plan.context.contextBytes > bounds.maxContextBytes) { + refusals.push( + this.refusal( + plan.context.repoPath || '.', + 'context-unbounded', + `Merged build context ${plan.context.repoPath || '.'} is ${plan.context.contextBytes} bytes after ownership reconciliation; the maximum is ${bounds.maxContextBytes}`, + true, + ), + ); + } + } + + // Deduplicate managed inputs by stack-relative path: two services + // referencing the same file (shared env_file, config, or Dockerfile) + // produce one entry so the candidate writer never rejects a duplicate. + // Case-only collisions (Config.yml vs config.yml) are refused: silently + // dropping one leaves compose config looking for a file that was never + // staged, and leaks internal candidate paths into the validation error. + const dedupedInputs: ComposeInputEntry[] = []; + const seenByCase = new Map(); + for (const entry of inputs) { + if (entry.materializedPath !== null) { + const key = caseKey(entry.materializedPath); + const prior = seenByCase.get(key); + if (prior) { + if (prior.materializedPath !== entry.materializedPath) { + refusals.push( + this.refusal( + entry.sourcePath ?? entry.materializedPath, + 'case-collision', + `Case-only path collision between ${prior.materializedPath} and ${entry.materializedPath}; both cannot be materialized on a case-insensitive filesystem`, + true, + ), + ); + } + continue; + } + seenByCase.set(key, entry); + } + dedupedInputs.push(entry); + } + + return { + inputs: dedupedInputs, + refusals, + buildContexts: reconciledBuildContexts.map((p) => p.context), + contextCopyPlans: reconciledBuildContexts, + dynamic, + counts: { + managed: dedupedInputs.filter((i) => i.ownership === 'managed').length, + unmanaged: dedupedInputs.filter((i) => i.ownership === 'unmanaged').length, + refused: refusals.length, + }, + }; + } + + /** + * Copy the materialized project into the candidate dir: managed files at + * their stack-relative paths plus dockerignore-filtered build contexts. + * Enforces aggregate bounds mid-copy and throws with running counts on + * violation (callers convert to a refusal). Never copies `.git`. + */ + async walkAndCopy( + cloneDir: string, + destDir: string, + files: CopyEntry[], + contexts: ContextCopyPlan[], + bounds: ManifestBounds, + ): Promise { + let copiedFiles = 0; + let copiedBytes = 0; + const seen = new Set(); + + const writeOne = async (srcAbs: string, destRel: string): Promise => { + const stat = await fs.promises.stat(srcAbs); + if (!stat.isFile()) throw new Error(`Not a regular file: ${destRel}`); + if (stat.size > bounds.maxFileBytes) { + throw new Error(`File too large to materialize: ${destRel} (${stat.size} bytes)`); + } + if (copiedFiles + 1 > bounds.maxFiles) { + throw new Error(`Materialization exceeds ${bounds.maxFiles} files`); + } + if (copiedBytes + stat.size > bounds.maxBytes) { + throw new Error(`Materialization exceeds ${bounds.maxBytes} bytes (${copiedBytes} so far)`); + } + const destAbs = path.resolve(destDir, destRel); + if (!isPathWithinBase(destAbs, path.resolve(destDir))) { + throw new Error(`Destination escapes the candidate dir: ${destRel}`); + } + const key = caseKey(destRel); + if (seen.has(key)) throw new Error(`Duplicate materialized path (case-insensitive): ${destRel}`); + seen.add(key); + await fs.promises.mkdir(path.dirname(destAbs), { recursive: true }); + await fs.promises.copyFile(srcAbs, destAbs); + copiedFiles += 1; + copiedBytes += stat.size; + }; + + for (const file of files) { + await writeOne(path.join(cloneDir, file.srcRel), file.destRel); + } + + for (const plan of contexts) { + const srcRoot = path.resolve(cloneDir, plan.srcRel); + // Context files are copied from the INVENTORY (plan.context.files), + // not from a re-walk of the directory with the first plan's matcher. + // This means merged plans (multiple services sharing a context with + // different Dockerfiles) copy the exact union their manifests record. + for (const f of plan.context.files) { + const src = path.resolve(srcRoot, f.path); + const destRel = plan.destRel && plan.destRel !== '.' ? `${plan.destRel}/${f.path}` : f.path; + // Sencho metadata must never reach the live stack. + if (f.path === '.candidate-complete') continue; + // A repo-root context overlaps the managed file set: paths + // already copied as managed inputs must not be copied again. + if (seen.has(caseKey(destRel))) continue; + await writeOne(src, destRel); + } + } + + return { copiedFiles, copiedBytes }; + } +} diff --git a/backend/src/services/ComposeService.ts b/backend/src/services/ComposeService.ts index dfa7213d..68d48b81 100644 --- a/backend/src/services/ComposeService.ts +++ b/backend/src/services/ComposeService.ts @@ -1236,6 +1236,96 @@ export class ComposeService { }); } + /** + * Render the effective compose model as YAML (the default `docker compose + * config` output) with the exact authored invocation and NO mesh override. + * Used by the Git source detach/export contract: the rendered model becomes + * the stack's single compose.yaml. Throws when the render fails or times + * out, so the detach transaction aborts before anything changes. + */ + public async renderComposeYaml(stackName: string): Promise { + if (!isValidStackName(stackName)) { + throw new Error('Invalid stack path'); + } + const baseResolved = path.resolve(this.baseDir); + const stackDir = path.resolve(baseResolved, stackName); + if (!stackDir.startsWith(baseResolved + path.sep)) { + throw new Error('Invalid stack path'); + } + let filePrefix: string[]; + try { + filePrefix = authoredComposeFileArgs(stackName, this.nodeId); + } catch (err) { + throw err instanceof Error ? err : new Error(String(err)); + } + const envFileArgs = await authoredComposeEnvFileArgs(stackName, this.nodeId); + const child = spawn('docker', ['compose', ...filePrefix, ...envFileArgs, 'config'], { + cwd: stackDir, + env: { + ...process.env, + PATH: process.env.PATH || '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', + }, + }); + return new Promise((resolve, reject) => { + const MAX_OUTPUT = 5 * 1024 * 1024; // 5 MiB cap on each stream + const TIMEOUT_MS = 30_000; + // Accumulate Buffer chunks and decode ONCE at the end: chunk-wise + // toString() can split a multi-byte UTF-8 sequence across a chunk + // boundary and mangle non-ASCII values. + const outChunks: Buffer[] = []; + let outBytes = 0; + let stderr = ''; + let capped = false; + let settled = false; + const timer = setTimeout(() => { + settled = true; + clearTimeout(timer); + try { + child.kill('SIGKILL'); + } catch { + // best effort + } + reject(new Error(`docker compose config timed out after ${TIMEOUT_MS / 1000}s`)); + }, TIMEOUT_MS); + const finish = (error: Error | null) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (error) reject(error); + else resolve(Buffer.concat(outChunks).toString('utf8')); + }; + child.stdout.on('data', (data: Buffer) => { + if (capped) return; + outBytes += data.length; + if (outBytes > MAX_OUTPUT) { + // A truncated model frequently still parses as YAML; overwriting a + // working compose.yaml with it would be silent corruption. The cap is + // an error, not a truncation. + capped = true; + settled = true; + clearTimeout(timer); + try { + child.kill('SIGKILL'); + } catch { + // best effort + } + reject(new Error(`docker compose config output exceeded ${MAX_OUTPUT} bytes`)); + return; + } + outChunks.push(data); + }); + child.stderr.on('data', (data: Buffer) => { + if (stderr.length < MAX_OUTPUT) stderr += data.toString(); + }); + child.on('close', (code) => { + if (capped) return; + if (code === 0) finish(null); + else finish(new Error(stderr.trim() || `docker compose config exited with code ${code}`)); + }); + child.on('error', (err) => finish(err)); + }); + } + /** * Render the fully-resolved effective Compose model via `docker compose * config --format json`. This is the AUTHORED model: it does NOT splice in diff --git a/backend/src/services/DatabaseService.ts b/backend/src/services/DatabaseService.ts index b987ca03..1fac8fd8 100644 --- a/backend/src/services/DatabaseService.ts +++ b/backend/src/services/DatabaseService.ts @@ -16,6 +16,7 @@ import { } from '../helpers/notificationSchedule'; import { readSnapshotFileRow, type SnapshotFileReadResult, type SnapshotFileRow } from '../helpers/snapshotFileDecrypt'; import { sanitizeForLog } from '../utils/safeLog'; +import type { GitSourceManifestState } from '../types/gitProjectManifest'; export type { SnapshotFileReadResult } from '../helpers/snapshotFileDecrypt'; @@ -446,6 +447,9 @@ export interface StackGitSource { pending_fetched_at: number | null; last_debounce_at: number | null; applied_deploy_spec: GitSourceAppliedSpec | null; // deploy-time materialized file set; null = single-file auto-discovery + manifest_version: number | null; // cache of the managed-project manifest's manifestVersion (file is the source of truth) + manifest_state: GitSourceManifestState | null; // DB-only enum, wider than the file state; see types/gitProjectManifest.ts + manifest_generation: string | null; // stack-relative path of the applied generation dir created_at: number; updated_at: number; } @@ -1140,6 +1144,7 @@ export class DatabaseService { this.migrateFleetSyncStickyError(); this.migrateStackDossierHashes(); this.migrateGitSourceMultiFile(); + this.migrateGitSourceManifest(); this.migrateNodeUpdateSkips(); this.migrateStackAlertServiceScope(); @@ -2509,6 +2514,14 @@ export class DatabaseService { this.tryAddColumn('stack_dossiers', 'last_drift_check_at', 'INTEGER'); } + private migrateGitSourceManifest(): void { + // Cache columns for the managed-project manifest (the manifest FILE in + // /git-managed/// is the source of truth). + this.tryAddColumn('stack_git_sources', 'manifest_version', 'INTEGER'); + this.tryAddColumn('stack_git_sources', 'manifest_state', 'TEXT'); + this.tryAddColumn('stack_git_sources', 'manifest_generation', 'TEXT'); + } + private migrateGitSourceMultiFile(): void { this.tryAddColumn('stack_git_sources', 'compose_paths', 'TEXT'); this.tryAddColumn('stack_git_sources', 'context_dir', 'TEXT'); @@ -6171,6 +6184,9 @@ export class DatabaseService { compose_paths: this.normalizeComposePaths(row.compose_paths, composePath), context_dir: (row.context_dir as string | null) ?? null, applied_deploy_spec: this.parseAppliedDeploySpec(row.applied_deploy_spec), + manifest_version: row.manifest_version !== undefined && row.manifest_version !== null ? Number(row.manifest_version) : null, + manifest_state: (row.manifest_state as GitSourceManifestState | null) ?? null, + manifest_generation: (row.manifest_generation as string | null) ?? null, sync_env: Number(row.sync_env) === 1, env_path: (row.env_path as string | null) ?? null, auth_type: row.auth_type as GitSourceAuthType, @@ -6199,7 +6215,7 @@ export class DatabaseService { return rows.map(r => this.parseGitSource(r)!); } - public upsertGitSource(source: Omit): number { + public upsertGitSource(source: Omit): number { const now = Date.now(); const existing = this.getGitSource(source.stack_name); const composePathsJson = JSON.stringify(source.compose_paths ?? [source.compose_path]); @@ -6253,6 +6269,18 @@ export class DatabaseService { this.db.prepare('DELETE FROM stack_git_sources WHERE stack_name = ?').run(stackName); } + /** + * Persist the managed-project manifest cache columns. The manifest FILE in + * the managed area is the source of truth; these columns are cheap + * projections for GET/dashboard reads and carry the two states the file + * cannot express ('migration_required', 'absent'). + */ + public setGitSourceManifestState(stackName: string, version: number | null, state: GitSourceManifestState | null, generation: string | null): void { + this.db.prepare( + `UPDATE stack_git_sources SET manifest_version = ?, manifest_state = ?, manifest_generation = ?, updated_at = ? WHERE stack_name = ?` + ).run(version, state, generation, Date.now(), stackName); + } + public setGitSourcePending(stackName: string, commitSha: string, composeContent: string, envContent: string | null): void { this.db.prepare( `UPDATE stack_git_sources SET diff --git a/backend/src/services/DeployedStackDeletionService.ts b/backend/src/services/DeployedStackDeletionService.ts index 17dace08..9da17242 100644 --- a/backend/src/services/DeployedStackDeletionService.ts +++ b/backend/src/services/DeployedStackDeletionService.ts @@ -16,6 +16,7 @@ import { import { ComposeService } from './ComposeService'; import DockerController from './DockerController'; import { FileSystemService } from './FileSystemService'; +import { GitProjectManifestService } from './GitProjectManifestService'; import { NodeRegistry } from './NodeRegistry'; import { MeshService } from './MeshService'; import { NotificationService } from './NotificationService'; @@ -363,6 +364,9 @@ export class DeployedStackDeletionService { db.clearStackScanAttempts(nodeId, stackName); db.deleteRoleAssignmentsByStack(nodeId, stackName); db.deleteGitSource(stackName); + // R6: the managed-project area must not outlive the stack; failures are + // logged inside, never fatal to the deletion. + await GitProjectManifestService.getInstance().deleteManagedArea(stackName); db.deleteStackDossier(nodeId, stackName); db.deleteStackDriftFindings(nodeId, stackName); db.deleteStackExposureIntents(nodeId, stackName); diff --git a/backend/src/services/FileSystemService.ts b/backend/src/services/FileSystemService.ts index d803469d..c16cb3c9 100644 --- a/backend/src/services/FileSystemService.ts +++ b/backend/src/services/FileSystemService.ts @@ -300,22 +300,26 @@ export class FileSystemService { return null; } + private async listStacksRaw(): Promise { + const items = await fsPromises.readdir(this.baseDir, { withFileTypes: true }); + const stackNames: string[] = []; + + for (const item of items) { + if (!item.isDirectory()) continue; + if (!item.name || typeof item.name !== 'string') continue; + + const stackDir = path.join(this.baseDir, item.name); + if (await this.hasComposeFile(stackDir)) { + stackNames.push(item.name); + } + } + + return stackNames; + } + async getStacks(): Promise { try { - const items = await fsPromises.readdir(this.baseDir, { withFileTypes: true }); - const stackNames: string[] = []; - - for (const item of items) { - if (!item.isDirectory()) continue; - if (!item.name || typeof item.name !== 'string') continue; - - const stackDir = path.join(this.baseDir, item.name); - if (await this.hasComposeFile(stackDir)) { - stackNames.push(item.name); - } - } - - return stackNames; + return await this.listStacksRaw(); } catch (error: any) { if (error?.code === 'ENOMEM') { const freeMiB = Math.round(os.freemem() / (1024 * 1024)); @@ -327,6 +331,16 @@ export class FileSystemService { } } + /** + * Like getStacks(), but PROPAGATES listing errors instead of returning an + * empty list. Callers that must distinguish "no stacks" from "could not + * list stacks" (the boot orphan sweep) use this variant: a swallowed read + * failure must never look like every stack disappeared. + */ + async getStacksStrict(): Promise { + return this.listStacksRaw(); + } + async getStackContent(stackName: string): Promise { try { const filePath = await this.getComposeFilePath(stackName); @@ -363,11 +377,14 @@ export class FileSystemService { } } - async saveStackContent(stackName: string, content: string): Promise { + async saveStackContent(stackName: string, content: string | Buffer): Promise { const stackDir = this.resolveStackDir(stackName); const filePath = path.join(stackDir, 'compose.yaml'); await this.assertRealWithinBase(filePath); try { + // Buffer input is written byte-exact (the encoding option is ignored for + // Buffers); string input keeps the utf-8 write. Byte-exactness matters to + // the Git materializer, whose content hashes are computed over raw bytes. await fsPromises.writeFile(filePath, content, 'utf-8'); } catch (error) { console.error('Error writing file:', error); @@ -1806,7 +1823,7 @@ export class FileSystemService { async writeStackFile( stackName: string, relPath: string, - content: string, + content: string | Buffer, opts?: { exclusive?: boolean }, ): Promise { const safePath = await this.resolveSafeStackPath(stackName, relPath); diff --git a/backend/src/services/GitProjectManifestService.ts b/backend/src/services/GitProjectManifestService.ts new file mode 100644 index 00000000..5a1dbe43 --- /dev/null +++ b/backend/src/services/GitProjectManifestService.ts @@ -0,0 +1,1567 @@ +/** + * Managed-project manifest service: the single canonical inventory for + * Git-managed stacks, stored OUTSIDE the stack directory at + * /git-managed/// so it is unreachable from the + * file explorer and never enters a Docker build context. + * + * The manifest file is the source of truth and is treated as UNTRUSTED on + * every read: shape, enum membership, and the identity stamp (nodeId, + * stackName, repo url, branch) are validated before any field is honored. A + * mismatch is corruption, never partial trust. + * + * Promotion is transactional: a promotion.json marker records the full file + * journal and commit phase. The boot sweep either finalizes a committed + * manifest or restores the previous applied generation under the per-stack + * lock. If live files match neither snapshot, it declines and flags recovery. + */ +import path from 'path'; +import fs from 'fs'; +import { createHash } from 'crypto'; +import YAML from 'yaml'; +import { NodeRegistry } from './NodeRegistry'; +import { FileSystemService } from './FileSystemService'; +import { StackFileRootsService } from './StackFileRootsService'; +import { DatabaseService } from './DatabaseService'; +import { ComposeInputDiscoveryService, type ContextCopyPlan, type CopyEntry } from './ComposeInputDiscoveryService'; +import { authoredComposeFileArgs, authoredComposeEnvFileArgs } from '../utils/authoredComposeArgs'; +import { sanitizeForLog } from '../utils/safeLog'; +import { isPathWithinBase, isValidStackName } from '../utils/validation'; +import type { + BuildContextPlan, + ComposeInputEntry, + DeletionAuthority, + GitProjectManifest, + InputDependencyKind, + InputOwnership, + InputRole, + InputSensitivity, + InputState, + ManifestBounds, + ManifestIdentity, + ManifestProvenance, + ManifestState, + ManifestSummary, + PublicManifest, + RefusalInfo, +} from '../types/gitProjectManifest'; + +export const MANAGED_ROOT_NAME = 'git-managed'; +export const MANIFEST_FILENAME = 'manifest.v1.json'; +export const PROMOTION_MARKER = 'promotion.json'; +export const CANDIDATE_COMPLETE_MARKER = '.candidate-complete'; +export const GENERATIONS_DIR = 'generations'; +const DETACH_RECOVERY_MARKER = 'detach-recovery.v1.json'; + +// Retention: current applied generation + one previous (prune keeps the +// manifest's previousDir explicitly, not by count). +const ORPHAN_CANDIDATE_AGE_MS = 24 * 60 * 60 * 1000; + +type PromotionPhase = 'applying' | 'committing'; + +interface PromotionMarker { + schemaVersion: 2; + phase: PromotionPhase; + sha: string; + manifestVersion: number; + candidateRelPath: string; + appliedRelPath: string; + /** + * Complete stack-relative file operation journal, persisted before the + * first live write. Recovery validates every path against either the prior + * or incoming snapshot, including writes after the last marker flush. + */ + affected: string[]; + /** Incoming files absent from the prior generation. */ + introduced: string[]; +} + +const PROMOTION_PHASES: readonly PromotionPhase[] = ['applying', 'committing']; + +type DetachRecoveryFile = + | { path: string; existed: true; contentBase64: string } + | { path: string; existed: false; contentBase64: null }; + +type DetachRecoveryInput = + | { path: string; existed: true; content: Buffer } + | { path: string; existed: false; content: null }; + +interface DetachRecoveryMarker { + schemaVersion: 1; + identity: { stackName: string; repoUrl: string; branch: string }; + managedAreaExisted: boolean; + files: DetachRecoveryFile[]; +} + +type RecoveryIncoming = + | { inputs: ComposeInputEntry[]; buildContexts: BuildContextPlan[] } + | { introducedPaths: string[] }; + +const MANIFEST_STATES: readonly ManifestState[] = ['none', 'migrated', 'active', 'partial', 'unsupported']; +const DEPENDENCY_KINDS: readonly InputDependencyKind[] = [ + 'explicit', 'implicit-override', 'include', 'include-env', 'extends', 'env_file', + 'interpolation-env', 'config', 'secret', 'label_file', 'build-context', 'dockerfile', + 'build-secret', 'build-additional-context', 'sync-env', 'bind-mount', +]; +const INPUT_ROLES: readonly InputRole[] = [ + 'compose-primary', 'compose-additional', 'compose-override', 'env', 'config', 'secret', + 'label-file', 'build-context', 'dockerfile', 'build-secret', 'build-additional-context', + 'bind-mount', 'other', +]; +const OWNERSHIPS: readonly InputOwnership[] = ['managed', 'unmanaged']; +const PROVENANCES: readonly ManifestProvenance[] = ['fetch', 'migration', 'adopted']; +const SENSITIVITIES: readonly InputSensitivity[] = ['high', 'medium', 'low']; +const INPUT_STATES: readonly InputState[] = ['present', 'tombstoned']; +const DELETION_AUTHORITIES: readonly DeletionAuthority[] = ['sencho', 'user', 'none']; + +function isOneOf(value: unknown, allowed: readonly T[]): value is T { + return typeof value === 'string' && (allowed as readonly string[]).includes(value); +} + +function sha256Of(buf: Buffer): string { + return createHash('sha256').update(buf).digest('hex'); +} + +/** A manifest path field must be relative and free of `..` / absolute escapes. Empty string is allowed (unset generation dirs). */ +function isSafeRelPath(value: unknown): boolean { + if (value === null) return true; + if (typeof value !== 'string') return false; + if (value === '') return true; + const normalized = value.replace(/\\/g, '/'); + if (path.posix.isAbsolute(normalized)) return false; + if (/^[A-Za-z]:/.test(normalized)) return false; + const segments = normalized.split('/'); + return !segments.some((seg) => seg === '..' || seg === '.'); +} + +/** + * A file-level path in a marker or manifest must be non-empty because an + * empty path resolves to the stack root and would delete/overwrite it. + */ +function isNonEmptyRelPath(value: unknown): value is string { + if (typeof value !== 'string' || value.length === 0) return false; + return isSafeRelPath(value); +} + +function isNonNegativeInteger(value: unknown): value is number { + return typeof value === 'number' && Number.isSafeInteger(value) && value >= 0; +} + +function isPositiveInteger(value: unknown): value is number { + return typeof value === 'number' && Number.isSafeInteger(value) && value > 0; +} + +function isSha256(value: unknown): value is string { + return typeof value === 'string' && /^[0-9a-f]{64}$/.test(value); +} + +export class GitProjectManifestService { + private constructor() { + // Singleton; node scoping follows the executing node's default node id, + // mirroring GitSourceService (proxied requests run on the owning node). + } + + private static instance: GitProjectManifestService | null = null; + + static getInstance(): GitProjectManifestService { + if (!this.instance) this.instance = new GitProjectManifestService(); + return this.instance; + } + + private dataRoot(): string { + return process.env.DATA_DIR || path.join(process.cwd(), 'data'); + } + + private managedRoot(stackName: string): string { + if (!isValidStackName(stackName)) throw new Error('Invalid stack name for managed project data'); + const nodeId = NodeRegistry.getInstance().getDefaultNodeId(); + return path.join(this.dataRoot(), MANAGED_ROOT_NAME, String(nodeId), stackName); + } + + private generationsDir(stackName: string): string { + return path.join(this.managedRoot(stackName), GENERATIONS_DIR); + } + + private async pathExists(absPath: string): Promise { + try { + await fs.promises.access(absPath); + return true; + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return false; + throw e; + } + } + + // ─── Bounds ─────────────────────────────────────────────────────────────── + + /** Materialization bounds; env-overridable following GITSOURCE_MAX_CLONE_BYTES. */ + boundsConfig(): ManifestBounds { + const num = (key: string, fallback: number): number => { + const raw = process.env[key]; + if (!raw) return fallback; + const parsed = Number.parseInt(raw, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; + }; + return { + maxFiles: num('GITSOURCE_MAX_MATERIALIZED_FILES', 10_000), + maxBytes: num('GITSOURCE_MAX_MATERIALIZED_BYTES', 512 * 1024 * 1024), + maxContextBytes: num('GITSOURCE_MAX_BUILD_CONTEXT_BYTES', 256 * 1024 * 1024), + maxPathDepth: num('GITSOURCE_MAX_PATH_DEPTH', 64), + maxFileBytes: num('GITSOURCE_MAX_FILE_BYTES', 10 * 1024 * 1024), + }; + } + + // ─── Manifest read/write (untrusted on read) ───────────────────────────── + + private expectedIdentity(stackName: string, repoUrl: string, branch: string): ManifestIdentity { + return { + nodeId: String(NodeRegistry.getInstance().getDefaultNodeId()), + stackName, + repoUrl, + branch, + }; + } + + /** + * Validate an untrusted manifest. Returns a reason when the manifest must + * be treated as corrupt; the identity stamp must match the expected stack + * so a same-named successor can never adopt an orphan. + */ + private validateManifest( + raw: unknown, + stackName: string, + expected: ManifestIdentity, + ): string | null { + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return 'not an object'; + const m = raw as Record; + if (m.schemaVersion !== 1) return `unsupported schemaVersion ${String(m.schemaVersion)}`; + if (!isPositiveInteger(m.manifestVersion)) return 'invalid manifestVersion'; + if (!isOneOf(m.state, MANIFEST_STATES)) return `invalid state ${String(m.state)}`; + if (!isNonNegativeInteger(m.generatedAt)) return 'invalid generatedAt'; + + const identity = m.identity as Record | undefined; + if (!identity || typeof identity !== 'object') return 'missing identity'; + if (identity.nodeId !== expected.nodeId) return `identity nodeId mismatch (${String(identity.nodeId)} vs ${expected.nodeId})`; + if (identity.stackName !== stackName) return `identity stackName mismatch (${String(identity.stackName)} vs ${stackName})`; + if (identity.repoUrl !== expected.repoUrl || identity.branch !== expected.branch) { + return `identity repository mismatch (${String(identity.repoUrl)}#${String(identity.branch)})`; + } + + const repo = m.repo as Record | undefined; + if (!repo || repo.url !== expected.repoUrl || repo.branch !== expected.branch) return 'repo mismatch'; + const revision = m.resolvedRevision as Record | undefined; + if (!revision + || typeof revision.commitSha !== 'string' + || (revision.commitSha.length === 0 && m.state !== 'migrated') + || !isNonNegativeInteger(revision.fetchedAt)) { + return 'invalid resolvedRevision'; + } + const project = m.project as Record | undefined; + if (!project || typeof project !== 'object') return 'invalid project'; + if (!Array.isArray(project.composeFiles) || !project.composeFiles.every((f) => isNonEmptyRelPath(f))) return 'invalid project.composeFiles'; + if (!Array.isArray(project.invocation) || !project.invocation.every((a) => typeof a === 'string')) return 'invalid project.invocation'; + if (typeof project.projectName !== 'string' || project.projectName.length === 0) return 'invalid project.projectName'; + + if (!Array.isArray(m.inputs)) return 'invalid inputs'; + for (const entry of m.inputs as unknown[]) { + if (!entry || typeof entry !== 'object' || Array.isArray(entry)) return 'invalid input entry'; + const e = entry as Record; + if (e.sourcePath !== null && typeof e.sourcePath !== 'string') return 'invalid input sourcePath'; + if (e.materializedPath !== null && !isNonEmptyRelPath(e.materializedPath)) return `invalid input materializedPath ${String(e.materializedPath)}`; + if (!isOneOf(e.dependencyKind, DEPENDENCY_KINDS)) return `invalid input kind ${String(e.dependencyKind)}`; + if (!isOneOf(e.role, INPUT_ROLES)) return `invalid input role ${String(e.role)}`; + if (!isOneOf(e.ownership, OWNERSHIPS)) return `invalid input ownership ${String(e.ownership)}`; + if (!isOneOf(e.provenance, PROVENANCES)) return `invalid input provenance ${String(e.provenance)}`; + if (!isOneOf(e.sensitivity, SENSITIVITIES)) return `invalid input sensitivity ${String(e.sensitivity)}`; + if (!isOneOf(e.state, INPUT_STATES)) return `invalid input state ${String(e.state)}`; + if (!isOneOf(e.deletionAuthority, DELETION_AUTHORITIES)) return `invalid deletionAuthority ${String(e.deletionAuthority)}`; + if (e.contentSha256 !== null && !isSha256(e.contentSha256)) return 'invalid input contentSha256'; + if (e.sizeBytes !== null && !isNonNegativeInteger(e.sizeBytes)) return 'invalid input sizeBytes'; + if (e.note !== null && typeof e.note !== 'string') return 'invalid input note'; + } + if (!Array.isArray(m.refusals) || !Array.isArray(m.buildContexts)) return 'invalid refusals/buildContexts'; + for (const refusal of m.refusals as unknown[]) { + if (!refusal || typeof refusal !== 'object' || Array.isArray(refusal)) return 'invalid refusal entry'; + const r = refusal as Record; + if (r.sourcePath !== null && typeof r.sourcePath !== 'string') return 'invalid refusal sourcePath'; + if (typeof r.kind !== 'string' || r.kind.length === 0 || typeof r.reason !== 'string' || typeof r.actionable !== 'boolean') { + return 'invalid refusal fields'; + } + if (r.sensitivity !== undefined && !isOneOf(r.sensitivity, SENSITIVITIES)) return 'invalid refusal sensitivity'; + } + for (const ctx of m.buildContexts as unknown[]) { + if (!ctx || typeof ctx !== 'object' || Array.isArray(ctx)) return 'invalid build context entry'; + const c = ctx as Record; + if (!isSafeRelPath(c.repoPath)) return `invalid build context root ${String(c.repoPath)}`; + if (c.dockerfile !== null && !isNonEmptyRelPath(c.dockerfile)) return `invalid build context dockerfile ${String(c.dockerfile)}`; + if (!isNonNegativeInteger(c.contextBytes) || !isNonNegativeInteger(c.ignoredCount)) return 'invalid build context counters'; + if (typeof c.dockerignoreApplied !== 'boolean' || typeof c.excludedFromCopy !== 'boolean') return 'invalid build context flags'; + if (c.note !== null && typeof c.note !== 'string') return 'invalid build context note'; + const files = c.files; + if (files !== undefined) { + if (!Array.isArray(files)) return 'invalid build context files'; + const seenPaths = new Set(); + let inventoryBytes = 0; + let inventoryHasSizes = true; + for (const f of files as unknown[]) { + if (!f || typeof f !== 'object' || Array.isArray(f)) return 'invalid build context file entry'; + const fe = f as Record; + if (!isNonEmptyRelPath(fe.path)) return `invalid build context file path ${String(fe.path)}`; + if (!isSha256(fe.sha256)) return 'invalid build context file hash'; + if (fe.sizeBytes !== undefined && !isNonNegativeInteger(fe.sizeBytes)) return 'invalid build context file size'; + if (typeof fe.sizeBytes === 'number') inventoryBytes += fe.sizeBytes; + else inventoryHasSizes = false; + const key = String(fe.path).toLowerCase(); + if (seenPaths.has(key)) return `duplicate build context file ${String(fe.path)}`; + seenPaths.add(key); + } + if (inventoryHasSizes && inventoryBytes !== c.contextBytes) return 'build context byte count does not match its file inventory'; + } + } + const generation = m.generation as Record | undefined; + if (!generation || typeof generation !== 'object') return 'invalid generation'; + if (!isSafeRelPath(generation.candidateDir) || !isSafeRelPath(generation.appliedDir) || !isSafeRelPath(generation.previousDir)) { + return 'invalid generation paths'; + } + if (!isSafeRelPath((m.project as Record | undefined)?.root)) return 'invalid project.root'; + if (!m.counts || typeof m.counts !== 'object') return 'invalid counts'; + const counts = m.counts as Record; + if (!isNonNegativeInteger(counts.managed) || !isNonNegativeInteger(counts.unmanaged) || !isNonNegativeInteger(counts.refused)) { + return 'invalid counts'; + } + const inputEntries = m.inputs as Record[]; + const expectedManaged = inputEntries.filter((entry) => entry.ownership === 'managed' && entry.state === 'present').length; + const expectedUnmanaged = inputEntries.filter((entry) => entry.ownership === 'unmanaged').length; + if (counts.managed !== expectedManaged || counts.unmanaged !== expectedUnmanaged || counts.refused !== (m.refusals as unknown[]).length) { + return 'manifest counts do not match its inventory'; + } + if (!m.bounds || typeof m.bounds !== 'object') return 'invalid bounds'; + const bounds = m.bounds as Record; + if (!isPositiveInteger(bounds.maxFiles) || !isPositiveInteger(bounds.maxBytes) || !isPositiveInteger(bounds.maxContextBytes) + || !isPositiveInteger(bounds.maxPathDepth) || !isPositiveInteger(bounds.maxFileBytes)) { + return 'invalid bounds'; + } + // Cross-check: no managed input path collides with any context file path. + const inputSeen = new Set(); + for (const e of m.inputs as Record[]) { + if (e.materializedPath === null || e.materializedPath === undefined) continue; + const key = String(e.materializedPath).toLowerCase(); + if (inputSeen.has(key)) return `duplicate input path ${String(e.materializedPath)}`; + inputSeen.add(key); + } + for (const c of m.buildContexts as Record[]) { + if (!c || typeof c !== 'object') continue; + const files = (c as Record).files; + if (!Array.isArray(files)) continue; + for (const f of files as Record[]) { + const ctxPath = String((c as Record).repoPath ?? ''); + const filePath = String(f.path ?? ''); + const key = (ctxPath ? `${ctxPath}/${filePath}` : filePath).toLowerCase(); + if (inputSeen.has(key)) return `context file path ${key} collides with an input path`; + inputSeen.add(key); + } + } + return null; + } + + /** + * Read + validate the manifest. Returns the manifest, `{ corrupt: reason }` + * when a file exists but cannot be trusted, or null when absent. + */ + async readManifest( + stackName: string, + repoUrl: string, + branch: string, + ): Promise { + const manifestPath = path.join(this.managedRoot(stackName), MANIFEST_FILENAME); + let raw: string; + try { + raw = await fs.promises.readFile(manifestPath, 'utf8'); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return null; + return { corrupt: `Cannot read manifest: ${(e as Error).message}` }; + } + let parsed: unknown; + try { + parsed = JSON.parse(raw); + } catch (e) { + return { corrupt: `Manifest is not valid JSON: ${(e as Error).message}` }; + } + const reason = this.validateManifest(parsed, stackName, this.expectedIdentity(stackName, repoUrl, branch)); + if (reason !== null) return { corrupt: `Manifest failed validation: ${reason}` }; + // Normalize pre-correction manifests: context entries written before + // the per-file inventory existed have no `files` array; degrade them + // to directory granularity instead of rejecting or throwing later. + const manifest = parsed as GitProjectManifest; + manifest.buildContexts = manifest.buildContexts.map((ctx) => ({ + ...ctx, + files: Array.isArray(ctx.files) && ctx.files.every((file) => isNonNegativeInteger(file.sizeBytes)) + ? ctx.files + : [], + })); + return manifest; + } + + /** Atomic manifest write (tmp + rename). */ + async writeManifest(stackName: string, manifest: GitProjectManifest): Promise { + const dir = this.managedRoot(stackName); + await fs.promises.mkdir(dir, { recursive: true }); + const target = path.join(dir, MANIFEST_FILENAME); + const tmp = path.join(dir, `${MANIFEST_FILENAME}.tmp`); + await fs.promises.writeFile(tmp, JSON.stringify(manifest, null, 2), 'utf8'); + await fs.promises.rename(tmp, target); + } + + /** + * Public projection for the manifest read endpoint: hashes, size metadata, + * provenance, and deletion authority are internal-only, and for + * high-sensitivity inputs the display path AND the note are redacted + * (null) so secret file names never cross the API. + */ + toPublicManifest(manifest: GitProjectManifest): PublicManifest { + return { + manifestVersion: manifest.manifestVersion, + state: manifest.state, + resolvedCommitSha: manifest.resolvedRevision.commitSha, + projectRoot: manifest.project.root, + composeFiles: manifest.project.composeFiles, + projectName: manifest.project.projectName, + inputs: manifest.inputs.map((entry) => ({ + path: entry.sensitivity === 'high' ? null : (entry.materializedPath ?? entry.sourcePath), + role: entry.role, + dependencyKind: entry.dependencyKind, + ownership: entry.ownership, + sensitivity: entry.sensitivity, + state: entry.state, + note: entry.sensitivity === 'high' ? null : entry.note, + })), + counts: manifest.counts, + }; + } + + /** + * Public projection of refusals for every API surface (summary, pull, and + * abort messages): high-sensitivity refusals lose their source path and + * the path text is scrubbed (case-insensitively) from the reason so + * secret file names never cross the API. + */ + toPublicRefusals(refusals: RefusalInfo[]): RefusalInfo[] { + return refusals.map((r) => { + if (r.sensitivity !== 'high' || r.sourcePath === null) return r; + const pattern = r.sourcePath.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + const scrubbedReason = r.reason.replace(new RegExp(pattern, 'gi'), '[redacted]'); + return { ...r, sourcePath: null, reason: scrubbedReason }; + }); + } + + summaryFrom(manifest: GitProjectManifest): ManifestSummary { + const actionable = this.toPublicRefusals(manifest.refusals.filter((r) => r.actionable)); + return { + state: manifest.state, + manifestVersion: manifest.manifestVersion, + resolvedCommitSha: manifest.resolvedRevision.commitSha, + managedCount: manifest.counts.managed, + unmanagedCount: manifest.counts.unmanaged, + refusedCount: manifest.counts.refused, + refused: actionable, + hasBuildContexts: manifest.buildContexts.length > 0, + generatedAt: manifest.generatedAt, + }; + } + + /** Build a fresh manifest for a fetched revision (provenance: fetch). */ + buildManifest(opts: { + stackName: string; + repoUrl: string; + branch: string; + commitSha: string; + projectRoot: string | null; + composeFiles: string[]; + projectName: string; + invocation: string[]; + inputs: ComposeInputEntry[]; + refusals: RefusalInfo[]; + buildContexts: BuildContextPlan[]; + bounds: ManifestBounds; + priorManifest: GitProjectManifest | null; + state: ManifestState; + }): GitProjectManifest { + const now = Date.now(); + return { + schemaVersion: 1, + manifestVersion: (opts.priorManifest?.manifestVersion ?? 0) + 1, + state: opts.state, + generatedAt: now, + identity: this.expectedIdentity(opts.stackName, opts.repoUrl, opts.branch), + repo: { url: opts.repoUrl, branch: opts.branch }, + resolvedRevision: { commitSha: opts.commitSha, fetchedAt: now }, + project: { + root: opts.projectRoot, + composeFiles: opts.composeFiles, + effectiveProjectDir: opts.projectRoot, + projectName: opts.projectName, + invocation: opts.invocation, + }, + inputs: opts.inputs, + refusals: opts.refusals, + buildContexts: opts.buildContexts, + generation: { + candidateDir: '', + appliedDir: '', + previousDir: null, + }, + counts: { + managed: opts.inputs.filter((i) => i.ownership === 'managed').length, + unmanaged: opts.inputs.filter((i) => i.ownership === 'unmanaged').length, + refused: opts.refusals.length, + }, + bounds: opts.bounds, + }; + } + + // ─── Candidate build ────────────────────────────────────────────────────── + + /** + * Build the staged candidate for a sha: copy managed files + filtered + * build contexts into the managed area, then write the completion marker. + * Returns the generations-relative candidate path. + */ + async buildCandidate( + stackName: string, + sha: string, + cloneDir: string, + files: CopyEntry[], + contexts: ContextCopyPlan[], + bounds: ManifestBounds, + ): Promise { + const candidateRel = `${GENERATIONS_DIR}/candidate-${sha}`; + const candidateAbs = path.join(this.managedRoot(stackName), candidateRel); + await fs.promises.rm(candidateAbs, { recursive: true, force: true }); + await fs.promises.mkdir(candidateAbs, { recursive: true }); + await ComposeInputDiscoveryService.getInstance().walkAndCopy(cloneDir, candidateAbs, files, contexts, bounds); + // Completion marker: candidate reuse is gated on this, never on + // directory existence (a partial build must not be promoted). + await fs.promises.writeFile(path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER), sha, 'utf8'); + return candidateRel; + } + + // ─── Promotion ─────────────────────────────────────────────────────────── + + private async markerPath(stackName: string): Promise { + return path.join(this.managedRoot(stackName), PROMOTION_MARKER); + } + + /** + * Read the promotion marker. Distinguishes ABSENT (no crash happened) from + * CORRUPT (a crash mid-marker-write): a corrupt marker must never be + * treated as a clean slate, or recovery would skip a half-written stack. + */ + private async readMarker(stackName: string): Promise { + let raw: string; + try { + raw = await fs.promises.readFile(await this.markerPath(stackName), 'utf8'); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return null; + return { corrupt: `Cannot read promotion marker: ${(e as Error).message}` }; + } + try { + const parsed: unknown = JSON.parse(raw); + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return { corrupt: 'Promotion marker is not an object' }; + const marker = parsed as Record; + if (marker.schemaVersion !== 2) return { corrupt: 'Promotion marker has an unsupported schema version' }; + if (!isOneOf(marker.phase, PROMOTION_PHASES)) return { corrupt: 'Promotion marker has an invalid phase' }; + if (typeof marker.sha !== 'string' || marker.sha.length === 0) return { corrupt: 'Promotion marker has an invalid sha' }; + if (!isPositiveInteger(marker.manifestVersion)) return { corrupt: 'Promotion marker has an invalid manifest version' }; + if (!isNonEmptyRelPath(marker.candidateRelPath)) return { corrupt: 'Promotion marker has an invalid candidate path' }; + if (!isNonEmptyRelPath(marker.appliedRelPath)) return { corrupt: 'Promotion marker has an invalid applied path' }; + if (!Array.isArray(marker.affected) || !marker.affected.every((w) => isNonEmptyRelPath(w))) { + return { corrupt: 'Promotion marker has an invalid affected set' }; + } + if (!Array.isArray(marker.introduced) || !marker.introduced.every((w) => isNonEmptyRelPath(w))) { + return { corrupt: 'Promotion marker has an invalid introduced set' }; + } + const affected = marker.affected.filter((value): value is string => typeof value === 'string'); + const introduced = marker.introduced.filter((value): value is string => typeof value === 'string'); + const markerPathLimit = this.boundsConfig().maxFiles * 2; + if (affected.length > markerPathLimit || introduced.length > markerPathLimit) { + return { corrupt: 'Promotion marker exceeds the path journal bound' }; + } + if (new Set(affected.map((w) => w.toLowerCase())).size !== affected.length) { + return { corrupt: 'Promotion marker has duplicate affected paths' }; + } + if (new Set(introduced.map((w) => w.toLowerCase())).size !== introduced.length) { + return { corrupt: 'Promotion marker has duplicate introduced paths' }; + } + const affectedKeys = new Set(affected.map((w) => w.toLowerCase())); + if (!introduced.every((w) => affectedKeys.has(w.toLowerCase()))) { + return { corrupt: 'Promotion marker introduced paths are not in the affected set' }; + } + return { + schemaVersion: 2, + phase: marker.phase, + sha: marker.sha, + manifestVersion: marker.manifestVersion, + candidateRelPath: marker.candidateRelPath, + appliedRelPath: marker.appliedRelPath, + affected, + introduced, + }; + } catch (e) { + return { corrupt: `Promotion marker is not valid JSON: ${(e as Error).message}` }; + } + } + + /** Atomic marker write (tmp + rename) so a crash never leaves a half-written marker. */ + private async writeMarker(stackName: string, marker: PromotionMarker): Promise { + const target = await this.markerPath(stackName); + const tmp = `${target}.tmp`; + await fs.promises.writeFile(tmp, JSON.stringify(marker), 'utf8'); + await fs.promises.rename(tmp, target); + } + + /** Hash of the stack-dir file at a materialized path, or null when absent. */ + async hashStackFile(stackName: string, relPath: string): Promise { + const abs = await this.stackFileAbs(stackName, relPath); + try { + return sha256Of(await fs.promises.readFile(abs)); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return null; + throw e; + } + } + + /** + * Verify a build-context subtree on disk against the manifest's file-level + * inventory. Returns the context-relative paths that diverge: files whose + * hash differs, files missing from the stack, and files present in the + * stack that the manifest does not own (locally added). This gives contexts + * the same local-modification protection as plain managed files. + */ + async verifyContextOnDisk(stackName: string, context: BuildContextPlan, managedInputPaths?: Set): Promise { + const abs = await this.stackFileAbs(stackName, context.repoPath); + const diverged: string[] = []; + const owned = new Set(context.files.map((f) => f.path)); + const walk = async (dir: string, rel: string): Promise => { + let entriesList: fs.Dirent[]; + try { + entriesList = await fs.promises.readdir(dir, { withFileTypes: true }); + } catch { + return; // missing context dir reported by the owned-file loop below + } + for (const entry of entriesList) { + const childRel = rel ? `${rel}/${entry.name}` : entry.name; + if (entry.isDirectory()) { + await walk(path.join(dir, entry.name), childRel); + continue; + } + if (entry.isSymbolicLink()) { + diverged.push(`${childRel} (symbolic link)`); + continue; + } + // Files not in the context inventory: if they have a + // managed-input owner (stack-relative path), they are owned + // by another manifest entry. The managed set uses stack- + // relative paths; the walk uses context-relative paths. + if (!owned.has(childRel)) { + const stackRel = context.repoPath ? `${context.repoPath}/${childRel}` : childRel; + if (managedInputPaths && managedInputPaths.has(stackRel)) continue; + diverged.push(`${childRel} (locally added, not in the managed context)`); + continue; + } + const expected = context.files.find((f) => f.path === childRel)?.sha256; + const actual = await this.hashStackFile(stackName, context.repoPath ? `${context.repoPath}/${childRel}` : childRel); + if (expected === undefined || actual !== expected) { + diverged.push(childRel); + } + } + }; + await walk(abs, ''); + for (const ownedFile of context.files) { + if (!owned.has(ownedFile.path)) continue; + const present = await fs.promises + .access(path.join(abs, ownedFile.path)) + .then(() => true) + .catch(() => false); + if (!present) diverged.push(`${ownedFile.path} (missing)`); + } + return diverged; + } + + private async stackFileAbs(stackName: string, relPath: string): Promise { + // Same resolution chain as FileSystemService: node.compose_dir -> + // COMPOSE_DIR -> /app/compose. The stack name was validated upstream + // (isValidStackName); writes still go through the guarded FS service. + const composeDir = NodeRegistry.getInstance().getComposeDir(NodeRegistry.getInstance().getDefaultNodeId()); + if (!isValidStackName(stackName) || !isSafeRelPath(relPath)) throw new Error('Invalid stack file path'); + const stackRoot = path.resolve(composeDir, stackName); + const resolved = path.resolve(stackRoot, relPath); + if (!isPathWithinBase(resolved, stackRoot)) throw new Error('Stack file path escapes the stack root'); + return resolved; + } + + /** Exact file paths owned by one manifest, excluding directory inventory entries. */ + private manifestFilePaths(manifest: Pick): string[] { + const paths = new Map(); + for (const entry of manifest.inputs) { + if (entry.ownership !== 'managed' || entry.state !== 'present' || entry.materializedPath === null) continue; + if (entry.dependencyKind === 'build-context' || entry.dependencyKind === 'build-additional-context') continue; + paths.set(entry.materializedPath.toLowerCase(), entry.materializedPath); + } + for (const context of manifest.buildContexts) { + for (const file of context.files) { + const rel = context.repoPath ? `${context.repoPath}/${file.path}` : file.path; + paths.set(rel.toLowerCase(), rel); + } + } + return [...paths.values()].sort((a, b) => a.localeCompare(b)); + } + + /** Hash one snapshot file, preserving the distinction between missing and unreadable. */ + private async hashSnapshotFile(baseDir: string, relPath: string): Promise { + const resolved = path.resolve(baseDir, relPath); + if (!isPathWithinBase(resolved, baseDir)) throw new Error(`Snapshot path escapes its generation: ${relPath}`); + try { + return sha256Of(await fs.promises.readFile(resolved)); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return null; + throw e; + } + } + + /** + * Copy one candidate path into the stack dir through the guarded FS + * service. Content is written BYTE-EXACT: build + * contexts, configs, and secrets can be binary, and the manifest hashes + * are computed over raw bytes, so a lossy string conversion would corrupt + * files and permanently trip the divergence guard on re-apply. + */ + private async writeStackFileFromCandidate(stackName: string, candidateAbs: string, destRel: string, maxFileBytes: number): Promise { + if (!isNonEmptyRelPath(destRel)) throw new Error('Invalid candidate file path'); + const src = path.resolve(candidateAbs, destRel); + if (!isPathWithinBase(src, candidateAbs)) throw new Error(`Candidate file path escapes its generation: ${destRel}`); + const stat = await fs.promises.stat(src); + if (stat.isDirectory()) { + throw new Error(`Expected a materialized file but found a directory: ${destRel}`); + } + if (stat.size > maxFileBytes) { + throw new Error(`Materialized file too large: ${destRel} (${stat.size} bytes)`); + } + const content = await fs.promises.readFile(src); + const fsSvc = FileSystemService.getInstance(); + if (destRel === 'compose.yaml' || destRel === 'compose.yml' || destRel === 'docker-compose.yaml' || destRel === 'docker-compose.yml') { + await fsSvc.saveStackContent(stackName, content); + } else { + await fsSvc.writeStackFile(stackName, destRel, content); + } + } + + /** + * Promote the validated candidate into the live stack dir. Crash-safe: + * the promotion marker records the intermediate state; any mid-write + * failure restores the previous applied generation and rethrows. + * + * The introduced-path collision guard protects every path the incoming + * generation owns that the prior generation did not. What may overwrite + * an existing file at such a path depends on the caller: + * - a managed stack (priorManifest present): nothing except the synced + * stack-root .env, the file is unowned and the apply refuses; + * - fresh stack creation: 'all' (the directory was just created and + * every file is adoption boilerplate); + * - an existing pre-manifest stack (legacy Git source): the caller's + * allowlist of known legacy-owned paths (compose files + synced .env), + * matched exactly as STACK-RELATIVE materialized paths; + * - omitted: fail closed, nothing may be adopted. + */ + async promoteGeneration(stackName: string, opts: { + sha: string; + candidateRelPath: string; + manifest: GitProjectManifest; + priorManifest: GitProjectManifest | null; + /** Paths the incoming generation may overwrite even though they already exist and were never managed. 'all' = fresh creation. Omitted = nothing may be adopted. */ + adoptExistingMaterializedPaths?: string[] | 'all'; + }): Promise { + const { sha, candidateRelPath, manifest, priorManifest } = opts; + const candidateAbs = path.join(this.managedRoot(stackName), candidateRelPath); + const markerPath = await this.markerPath(stackName); + const bounds = this.boundsConfig(); + const appliedRel = `${GENERATIONS_DIR}/applied-${sha}-${manifest.manifestVersion}`; + const appliedAbs = path.join(this.managedRoot(stackName), appliedRel); + const incomingFiles = this.manifestFilePaths(manifest); + const priorFiles = priorManifest ? this.manifestFilePaths(priorManifest) : []; + const priorKeys = new Set(priorFiles.map((rel) => rel.toLowerCase())); + const priorByCaseFold = new Map(priorFiles.map((rel) => [rel.toLowerCase(), rel])); + const caseOnlyChange = incomingFiles.find((rel) => { + const priorRel = priorByCaseFold.get(rel.toLowerCase()); + return priorRel !== undefined && priorRel !== rel; + }); + if (caseOnlyChange !== undefined) { + throw new Error(`Case-only managed path changes are not supported: ${priorByCaseFold.get(caseOnlyChange.toLowerCase())} -> ${caseOnlyChange}`); + } + const introduced = incomingFiles.filter((rel) => !priorKeys.has(rel.toLowerCase())); + const affected = [...new Map([...priorFiles, ...incomingFiles].map((rel) => [rel.toLowerCase(), rel])).values()] + .sort((a, b) => a.localeCompare(b)); + const markerBase: Omit = { + schemaVersion: 2, + sha, + manifestVersion: manifest.manifestVersion, + candidateRelPath, + appliedRelPath: appliedRel, + affected, + introduced, + }; + let liveMutationStarted = false; + + try { + // The candidate must be complete; a partial build is never promoted. + if (!(await this.pathExists(path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER)))) { + throw new Error('Candidate is incomplete; pull again before applying'); + } + + // Introduced-path collision guard: a path the incoming generation + // owns that the prior generation did not, and that ALREADY EXISTS + // in the live stack, is a local file Sencho never owned. + // Overwriting it would destroy user data, so refuse BEFORE the + // first live mutation. Exceptions are explicit and caller-driven: + // the synced stack-root .env (enabling sync_env is the explicit + // adoption of that path, its content is staged by design), 'all' + // for fresh creation (the directory was just created and every + // file is adoption boilerplate), and the caller's legacy-ownership + // allowlist for an existing pre-manifest stack. Omitted (managed + // stack, or an apply with no allowlist) fails closed. Allowlist + // entries are STACK-RELATIVE materialized paths matched exactly: + // a case-variant path is a different file on case-sensitive + // filesystems and must not be adopted. + { + const fsSvc = FileSystemService.getInstance(); + const syncEnvOwnsEnv = manifest.inputs.some((i) => i.dependencyKind === 'sync-env' && i.materializedPath === '.env'); + const adoptAll = opts.adoptExistingMaterializedPaths === 'all'; + const allowlist = new Set(Array.isArray(opts.adoptExistingMaterializedPaths) ? opts.adoptExistingMaterializedPaths : []); + const colliding: string[] = []; + for (const rel of introduced) { + if (rel === '.env' && syncEnvOwnsEnv) continue; + if (adoptAll || allowlist.has(rel)) continue; + const kind = await fsSvc.pathKind(stackName, rel); + if (kind !== null) colliding.push(rel); + } + if (colliding.length > 0) { + throw new Error( + `Refusing to overwrite ${colliding.length > 1 ? 'files' : 'a file'} that Sencho does not manage: ${colliding.join(', ')}. Remove or rename ${colliding.length > 1 ? 'them' : 'it'} in the stack directory, or detach the Git source first.`, + ); + } + } + + await fs.promises.mkdir(this.managedRoot(stackName), { recursive: true }); + // Persist the complete journal before the first live write. This + // covers every write and deletion even if the process exits between + // individual operations. + await this.writeMarker(stackName, { ...markerBase, phase: 'applying' }); + liveMutationStarted = true; + + // 1. Write every owned file exactly once. Context directory input + // entries are inventory only; their files are listed explicitly by + // manifestFilePaths. + const managed = manifest.inputs.filter((i) => i.ownership === 'managed' && i.state === 'present' && i.materializedPath !== null); + for (const rel of incomingFiles) { + await this.writeStackFileFromCandidate(stackName, candidateAbs, rel, bounds.maxFileBytes); + } + + // 2. Stale cleanup: prior-manifest paths Sencho owns (deletionAuthority + // sencho), absent from the new set. Only sencho-authority paths are + // ever unlinked; user/none authority stays untouched. A failed + // unlink FAILS the promotion (the transaction restores the prior + // generation) rather than recording a tombstone for a file that + // still exists and can silently change the deployed model. + const newPaths = new Set(managed.map((i) => i.materializedPath!)); + const removed: ComposeInputEntry[] = []; + const fsSvc = FileSystemService.getInstance(); + // Context files are reconciled FILE-LEVEL: a file removed from the + // repository inside a retained context must disappear from the + // stack context too, or the deployed/build context would keep + // deleted (possibly secret-bearing) content. + const newContextFiles = new Map>(); + for (const ctx of manifest.buildContexts) { + newContextFiles.set(ctx.repoPath, new Set(ctx.files.map((f) => f.path))); + } + if (priorManifest) { + for (const entry of priorManifest.inputs) { + if (entry.ownership !== 'managed' || entry.state !== 'present' || entry.materializedPath === null) continue; + if (entry.deletionAuthority !== 'sencho') continue; // never touch user/none authority + if (newPaths.has(entry.materializedPath)) continue; + // Directories (build contexts) need a recursive unlink; a + // non-recursive attempt would throw and fail the promotion + // even though the directory is legitimately removable. + const isDir = await fsSvc + .pathKind(stackName, entry.materializedPath) + .then((kind) => kind === 'directory') + .catch(() => false); + await fsSvc.deleteStackPath(stackName, entry.materializedPath, isDir, { protectedEnabled: false }); + removed.push({ ...entry, state: 'tombstoned', contentSha256: null, sizeBytes: null }); + } + // Context-file reconciliation for contexts retained in both sets. + for (const priorCtx of priorManifest.buildContexts) { + const newFiles = newContextFiles.get(priorCtx.repoPath); + if (!newFiles) continue; // context removed entirely; handled above + for (const priorFile of priorCtx.files) { + if (newFiles.has(priorFile.path)) continue; + const rel = priorCtx.repoPath ? `${priorCtx.repoPath}/${priorFile.path}` : priorFile.path; + await fsSvc.deleteStackPath(stackName, rel, false, { protectedEnabled: false }); + } + } + } + + // 3. Move candidate to a versioned applied snapshot. The manifest + // version keeps a same-revision re-apply from deleting its own + // previous recovery snapshot. + manifest.generation = { + candidateDir: candidateRelPath, + appliedDir: appliedRel, + previousDir: priorManifest?.generation.appliedDir ?? null, + }; + manifest.inputs = [...manifest.inputs, ...removed]; + manifest.counts = { + managed: manifest.inputs.filter((i) => i.ownership === 'managed' && i.state === 'present').length, + unmanaged: manifest.inputs.filter((i) => i.ownership === 'unmanaged').length, + refused: manifest.refusals.length, + }; + await fs.promises.rm(appliedAbs, { recursive: true, force: true }); + await fs.promises.rename(candidateAbs, appliedAbs); + await this.pruneGenerations(stackName, appliedRel, manifest.generation.previousDir); + + // 4. Mark the commit phase before replacing the manifest. Recovery + // can now distinguish an old-manifest rollback from a committed + // manifest that only needs its DB cache finalized. + await this.writeMarker(stackName, { ...markerBase, phase: 'committing' }); + DatabaseService.getInstance().setGitSourceManifestState(stackName, manifest.manifestVersion, manifest.state, appliedRel); + await this.writeManifest(stackName, manifest); + StackFileRootsService.invalidate(NodeRegistry.getInstance().getDefaultNodeId(), stackName); + + // Marker cleanup is post-commit housekeeping. If it fails, the + // next boot recognizes the committed manifest and finalizes it. + try { + await fs.promises.rm(markerPath, { force: true }); + } catch (e) { + console.warn('[GitManifest] committed promotion marker cleanup failed:', (e as Error).message); + } + } catch (error) { + if (!liveMutationStarted) throw error; + // Mid-write failure: restore the previous applied generation and + // rethrow so the caller reports the failure honestly. + try { + await this.restorePreviousGeneration(stackName, { + priorManifest: opts.priorManifest, + incoming: { inputs: opts.manifest.inputs, buildContexts: opts.manifest.buildContexts }, + }); + } catch (restoreError) { + console.error('[GitManifest] promotion failed and recovery restore also failed:', (restoreError as Error).message); + } + throw error; + } + } + + /** + * Exact introduced-set computation: every path the incoming generation + * owns that the prior generation did not. Includes context files so a + * failed promotion cannot leave a mixed file set inside a retained + * context. Stack-relative paths. + */ + private introducedPaths( + prior: GitProjectManifest | null, + incoming: { inputs: ComposeInputEntry[]; buildContexts: BuildContextPlan[] }, + ): string[] { + const priorPaths = new Set((prior ? this.manifestFilePaths(prior) : []).map((rel) => rel.toLowerCase())); + return this.manifestFilePaths(incoming).filter((rel) => !priorPaths.has(rel.toLowerCase())); + } + + /** + * Restore the previous applied generation's managed files AND the manifest + * FILE into the stack dir, then clear the promotion marker. Used after a + * mid-write crash or failed promotion. Files the incoming generation + * introduced (top-level or inside retained contexts) are removed so the + * prior generation is exact. If any restore step fails, the marker is KEPT + * and the DB state is set to migration_required so the boot sweep retries + * and the UI flags the stack instead of declaring a false recovery. + */ + async restorePreviousGeneration( + stackName: string, + opts: { priorManifest: GitProjectManifest | null; incoming?: RecoveryIncoming | null }, + ): Promise { + const prior = opts.priorManifest; + let failures = 0; + const fsSvc = FileSystemService.getInstance(); + const removeStackPath = async (relPath: string): Promise => { + const kind = await fsSvc.pathKind(stackName, relPath); + if (kind !== null) { + await fsSvc.deleteStackPath(stackName, relPath, kind === 'directory', { protectedEnabled: false }); + } + }; + if (prior) { + const priorDir = path.join(this.managedRoot(stackName), prior.generation.appliedDir); + for (const rel of this.manifestFilePaths(prior)) { + try { + await this.writeStackFileFromCandidate(stackName, priorDir, rel, this.boundsConfig().maxFileBytes); + } catch (e) { + failures += 1; + console.error(`[GitManifest] restore could not write ${rel}:`, (e as Error).message); + } + } + // Exact-generation restore: paths the failed promotion introduced + // (present in the incoming inventory, absent from the prior one) + // must be removed, or the stack would keep a mixed old/new file + // set. Context files are included. + if (failures === 0 && opts.incoming) { + const removeList = 'introducedPaths' in opts.incoming + ? opts.incoming.introducedPaths + : this.introducedPaths(prior, opts.incoming); + for (const rel of removeList) { + try { + await removeStackPath(rel); + } catch (e) { + failures += 1; + console.error(`[GitManifest] restore could not remove ${rel}:`, (e as Error).message); + } + } + } + if (failures === 0) { + // The manifest FILE must agree with the restored disk state, + // or the next apply would read the new manifest against the old + // files and refuse every input as locally modified forever. + await this.writeManifest(stackName, prior); + } + DatabaseService.getInstance().setGitSourceManifestState( + stackName, + failures === 0 ? prior.manifestVersion : null, + failures === 0 ? prior.state : 'migration_required', + failures === 0 ? prior.generation.appliedDir : null, + ); + } else { + // First-ever promotion failed with no prior generation to restore: + // remove everything the failed promotion wrote and flag the row. + if (opts.incoming && failures === 0) { + const removeList = 'introducedPaths' in opts.incoming + ? opts.incoming.introducedPaths + : this.introducedPaths(null, opts.incoming); + for (const rel of removeList) { + try { + await removeStackPath(rel); + } catch (e) { + failures += 1; + console.error(`[GitManifest] restore could not remove ${rel}:`, (e as Error).message); + } + } + } + DatabaseService.getInstance().setGitSourceManifestState(stackName, null, 'migration_required', null); + } + if (failures === 0) { + await fs.promises.rm(await this.markerPath(stackName), { force: true }); + } + StackFileRootsService.invalidate(NodeRegistry.getInstance().getDefaultNodeId(), stackName); + return failures === 0; + } + + private async pruneGenerations(stackName: string, keepAppliedRel: string, previousDir: string | null): Promise { + const dir = this.generationsDir(stackName); + let entries; + try { + entries = await fs.promises.readdir(dir, { withFileTypes: true }); + } catch { + return; + } + const keepBase = path.basename(keepAppliedRel); + // Retention is previousDir-explicit, never lexicographic: sha hex order + // says nothing about recency, and the manifest's previousDir is what a + // crash restore reads from. + const keep = new Set([keepBase, previousDir ? path.basename(previousDir) : null].filter((v): v is string => v !== null)); + for (const entry of entries) { + if (!entry.isDirectory() || !entry.name.startsWith('applied-')) continue; + if (keep.has(entry.name)) continue; + await fs.promises.rm(path.join(dir, entry.name), { recursive: true, force: true }); + } + } + + private async flagRecoveryRequired(stackName: string, reason: string): Promise { + DatabaseService.getInstance().setGitSourceManifestState(stackName, null, 'migration_required', null); + await fs.promises.rm(await this.markerPath(stackName), { force: true }); + console.warn(`[GitManifest] ${reason}; flagged migration_required`); + } + + // ─── Boot sweep ────────────────────────────────────────────────────────── + + /** + * One stack's crash recovery and orphan sweep. Callers run this under the + * per-stack lock. Every journaled path must match either the prior or the + * incoming snapshot before recovery writes anything. A committed manifest + * is finalized; an uncommitted promotion restores the prior generation. + * A third state is treated as an operator edit, so recovery declines and + * flags migration_required. Interrupted detach snapshots are restored first. + */ + async sweepManagedArea( + stackName: string, + opts: { repoUrl: string; branch: string; stackExists: boolean }, + ): Promise { + const { repoUrl, branch, stackExists } = opts; + if (!stackExists) { + await this.deleteManagedArea(stackName); + return; + } + await this.recoverInterruptedDetach(stackName, repoUrl, branch); + const marker = await this.readMarker(stackName); + if (marker) { + if ('corrupt' in marker) { + // A corrupt marker (crash mid-marker-write) is NOT a clean slate: + // the stack dir may be half-written. Flag recovery-required. + await this.flagRecoveryRequired(stackName, `promotion marker for ${sanitizeForLog(stackName)} is corrupt (${marker.corrupt})`); + return; + } + const current = await this.readManifest(stackName, repoUrl, branch); + + // Once the incoming manifest is visible, promotion is committed. + // A remaining marker only means DB cache or marker cleanup did not + // finish, so finalize without touching live stack files. + if (marker.phase === 'committing' + && current !== null + && !('corrupt' in current) + && current.manifestVersion === marker.manifestVersion + && current.resolvedRevision.commitSha === marker.sha + && current.generation.appliedDir === marker.appliedRelPath) { + DatabaseService.getInstance().setGitSourceManifestState( + stackName, + current.manifestVersion, + current.state, + current.generation.appliedDir, + ); + StackFileRootsService.invalidate(NodeRegistry.getInstance().getDefaultNodeId(), stackName); + await fs.promises.rm(await this.markerPath(stackName), { force: true }); + console.warn(`[GitManifest] finalized committed promotion for ${sanitizeForLog(stackName)} after a crash`); + } else { + const candidateAbs = path.join(this.managedRoot(stackName), marker.candidateRelPath); + const appliedAbs = path.join(this.managedRoot(stackName), marker.appliedRelPath); + let incomingAbs: string | null = null; + if (await this.pathExists(path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER))) { + incomingAbs = candidateAbs; + } else if (await this.pathExists(path.join(appliedAbs, CANDIDATE_COMPLETE_MARKER))) { + incomingAbs = appliedAbs; + } + if (incomingAbs === null) { + await this.flagRecoveryRequired(stackName, `promotion snapshots for ${sanitizeForLog(stackName)} are missing`); + return; + } + if (current !== null && 'corrupt' in current) { + await this.flagRecoveryRequired(stackName, `promotion marker found for ${sanitizeForLog(stackName)} but the prior manifest is corrupt`); + return; + } + + const prior = current; + const priorAbs = prior ? path.join(this.managedRoot(stackName), prior.generation.appliedDir) : null; + let mismatch: string | null = null; + for (const rel of marker.affected) { + try { + const actual = await this.hashStackFile(stackName, rel); + const before = priorAbs ? await this.hashSnapshotFile(priorAbs, rel) : null; + const after = await this.hashSnapshotFile(incomingAbs, rel); + if (actual !== before && actual !== after) { + mismatch = `${rel} does not match either recovery snapshot`; + break; + } + } catch (e) { + mismatch = `${rel} could not be verified: ${(e as Error).message}`; + break; + } + } + if (mismatch !== null) { + await this.flagRecoveryRequired( + stackName, + `promotion marker for ${sanitizeForLog(stackName)} does not match the stack dir (${mismatch}); restore declined`, + ); + return; + } + const restored = await this.restorePreviousGeneration(stackName, { + priorManifest: prior, + incoming: marker.introduced.length > 0 ? { introducedPaths: marker.introduced } : null, + }); + if (restored) { + console.warn(`[GitManifest] restored previous applied generation for ${sanitizeForLog(stackName)} after a crash`); + } + } + } + + // Orphan candidates: incomplete or stale. + const dir = this.generationsDir(stackName); + try { + const entries = await fs.promises.readdir(dir, { withFileTypes: true }); + const now = Date.now(); + for (const entry of entries) { + if (!entry.isDirectory() || !entry.name.startsWith('candidate-')) continue; + const abs = path.join(dir, entry.name); + const complete = await fs.promises + .access(path.join(abs, CANDIDATE_COMPLETE_MARKER)) + .then(() => true) + .catch(() => false); + if (!complete) { + await fs.promises.rm(abs, { recursive: true, force: true }); + continue; + } + const st = await fs.promises.stat(abs); + if (now - st.mtimeMs > ORPHAN_CANDIDATE_AGE_MS) { + await fs.promises.rm(abs, { recursive: true, force: true }); + } + } + } catch { + // no generations dir yet + } + } + + private detachStagedRoot(stackName: string): string { + const root = this.managedRoot(stackName); + return path.join(path.dirname(root), `.detach-${stackName}`); + } + + /** Persist exact stack-file snapshots before detach mutates the live files. */ + async prepareDetachRecovery( + stackName: string, + repoUrl: string, + branch: string, + files: DetachRecoveryInput[], + ): Promise { + const root = this.managedRoot(stackName); + const managedAreaExisted = await this.pathExists(root); + const bounds = this.boundsConfig(); + if (files.length > bounds.maxFiles) throw new Error('Detach recovery snapshot exceeds the file bound'); + if (files.some((file) => !isNonEmptyRelPath(file.path))) throw new Error('Invalid detach recovery path'); + if (new Set(files.map((file) => file.path.toLowerCase())).size !== files.length) { + throw new Error('Detach recovery snapshot has duplicate paths'); + } + let snapshotBytes = 0; + for (const file of files) { + if (!file.existed) continue; + if (file.content.length > bounds.maxFileBytes) throw new Error(`Detach recovery file exceeds the size bound: ${file.path}`); + snapshotBytes += file.content.length; + if (snapshotBytes > bounds.maxBytes) throw new Error('Detach recovery snapshot exceeds the byte bound'); + } + const marker: DetachRecoveryMarker = { + schemaVersion: 1, + identity: { stackName, repoUrl, branch }, + managedAreaExisted, + files: files.map((file): DetachRecoveryFile => file.existed + ? { path: file.path, existed: true, contentBase64: file.content.toString('base64') } + : { path: file.path, existed: false, contentBase64: null }), + }; + await fs.promises.mkdir(root, { recursive: true }); + const target = path.join(root, DETACH_RECOVERY_MARKER); + const tmp = `${target}.tmp`; + await fs.promises.writeFile(tmp, JSON.stringify(marker), 'utf8'); + await fs.promises.rename(tmp, target); + } + + /** Restore a detach snapshot and remove its marker after every file succeeds. */ + private async restoreDetachRecoveryFromRoot(stackName: string, repoUrl: string, branch: string, root: string): Promise { + const markerPath = path.join(root, DETACH_RECOVERY_MARKER); + let raw: string; + try { + raw = await fs.promises.readFile(markerPath, 'utf8'); + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return false; + throw e; + } + const rawMarker: unknown = JSON.parse(raw); + if (!rawMarker || typeof rawMarker !== 'object' || Array.isArray(rawMarker)) throw new Error('Detach recovery marker is invalid'); + const marker = rawMarker as Record; + const identity = marker.identity as Record | undefined; + const bounds = this.boundsConfig(); + if (marker.schemaVersion !== 1 + || !identity + || identity.stackName !== stackName + || identity.repoUrl !== repoUrl + || identity.branch !== branch + || typeof marker.managedAreaExisted !== 'boolean' + || !Array.isArray(marker.files) + || marker.files.length > bounds.maxFiles) { + throw new Error('Detach recovery marker is invalid'); + } + let snapshotBytes = 0; + for (const file of marker.files as unknown[]) { + if (!file || typeof file !== 'object' || Array.isArray(file)) throw new Error('Detach recovery file entry is invalid'); + const entry = file as Record; + if (!isNonEmptyRelPath(entry.path) || typeof entry.existed !== 'boolean') throw new Error('Detach recovery file entry is invalid'); + if (entry.existed && typeof entry.contentBase64 !== 'string') throw new Error('Detach recovery content is missing'); + if (!entry.existed && entry.contentBase64 !== null) throw new Error('Detach recovery absent file has content'); + if (typeof entry.contentBase64 === 'string') { + const maxEncodedLength = Math.ceil(bounds.maxFileBytes / 3) * 4; + if (entry.contentBase64.length > maxEncodedLength + || !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(entry.contentBase64)) { + throw new Error('Detach recovery content is invalid'); + } + const contentBytes = Buffer.byteLength(entry.contentBase64, 'base64'); + if (contentBytes > bounds.maxFileBytes) throw new Error('Detach recovery content exceeds the file-size bound'); + snapshotBytes += contentBytes; + if (snapshotBytes > bounds.maxBytes) throw new Error('Detach recovery content exceeds the byte bound'); + } + } + const parsed = marker as unknown as DetachRecoveryMarker; + if (new Set(parsed.files.map((file) => file.path.toLowerCase())).size !== parsed.files.length) { + throw new Error('Detach recovery marker has duplicate paths'); + } + + const fsSvc = FileSystemService.getInstance(); + for (const file of parsed.files) { + if (file.existed) { + const content = Buffer.from(file.contentBase64, 'base64'); + if (file.path === 'compose.yaml') { + await fsSvc.saveStackContent(stackName, content); + } else { + await fsSvc.writeStackFile(stackName, file.path, content); + } + } else { + try { + await fsSvc.deleteStackPath(stackName, file.path, false, { protectedEnabled: false }); + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== 'ENOENT') throw e; + } + } + } + await fs.promises.rm(markerPath, { force: true }); + if (!parsed.managedAreaExisted) { + await fs.promises.rm(root, { recursive: true, force: true }); + } + return true; + } + + /** Recover an interrupted detach while its Git source row still exists. */ + async recoverInterruptedDetach(stackName: string, repoUrl: string, branch: string): Promise { + const root = this.managedRoot(stackName); + const staged = this.detachStagedRoot(stackName); + const stagedExists = await this.pathExists(staged); + if (stagedExists) { + const rootExists = await this.pathExists(root); + if (rootExists) throw new Error('Detach recovery has both live and staged managed areas'); + await fs.promises.rename(staged, root); + try { + return await this.restoreDetachRecoveryFromRoot(stackName, repoUrl, branch, root); + } catch (e) { + try { + const restoredRootExists = await this.pathExists(root); + if (restoredRootExists) await fs.promises.rename(root, staged); + } catch (cleanupError) { + const cleanupMessage = cleanupError instanceof Error ? cleanupError.message : String(cleanupError); + console.error('[GitManifest] detach recovery restaging failed:', sanitizeForLog(cleanupMessage)); + } + throw e; + } + } + return this.restoreDetachRecoveryFromRoot(stackName, repoUrl, branch, root); + } + + /** Move the managed area aside until the Git source row deletion commits. */ + async stageManagedAreaForDetach(stackName: string): Promise { + const root = this.managedRoot(stackName); + const staged = this.detachStagedRoot(stackName); + await fs.promises.rm(staged, { recursive: true, force: true }); + try { + await fs.promises.rename(root, staged); + return true; + } catch (e) { + if ((e as NodeJS.ErrnoException).code === 'ENOENT') return false; + throw e; + } + } + + /** Put a staged managed area back, then restore its durable file snapshot. */ + async rollbackStagedDetach(stackName: string, repoUrl: string, branch: string): Promise { + const root = this.managedRoot(stackName); + const staged = this.detachStagedRoot(stackName); + await fs.promises.rename(staged, root); + return this.restoreDetachRecoveryFromRoot(stackName, repoUrl, branch, root); + } + + /** Delete a staged managed area after the database row is gone. */ + async finalizeStagedDetach(stackName: string): Promise { + try { + await fs.promises.rm(this.detachStagedRoot(stackName), { recursive: true, force: true }); + return true; + } catch (e) { + console.warn('[GitManifest] staged detach cleanup failed:', sanitizeForLog(stackName), (e as Error).message); + return false; + } + } + + /** + * Delete the whole managed area. Failures are logged and reported as + * false so callers can decide whether the operation should proceed + * (stack deletion tolerates a lingering area; detach must not drop the + * row while secret-bearing generations survive). + */ + async deleteManagedArea(stackName: string): Promise { + const root = this.managedRoot(stackName); + try { + await fs.promises.rm(root, { recursive: true, force: true }); + return true; + } catch (e) { + console.warn('[GitManifest] could not delete managed area:', sanitizeForLog(stackName), (e as Error).message); + return false; + } + } + + // ─── Migration ─────────────────────────────────────────────────────────── + + /** + * Build a conservative manifest from historical state (applied_deploy_spec + * + disk). Deletion authority is granted ONLY for the exact paths + * historical code wrote (spec files + synced .env); contextDir subtrees + * were never enumerated, so their files get authority 'none'. Never infer + * deletion authority from incomplete historical metadata. + */ + async buildMigratedManifest( + stackName: string, + source: { repo_url: string; branch: string; sync_env: boolean; applied_deploy_spec: { files: string[]; contextDir: string | null } | null }, + priorVersion = 0, + ): Promise { + const nodeId = NodeRegistry.getInstance().getDefaultNodeId(); + const bounds = this.boundsConfig(); + const now = Date.now(); + const inputs: ComposeInputEntry[] = []; + const spec = source.applied_deploy_spec; + + const addEntry = async ( + materializedPath: string, + role: InputRole, + kind: InputDependencyKind, + authority: DeletionAuthority, + note: string | null, + ): Promise => { + let hash: string | null = null; + let size: number | null = null; + try { + const abs = await this.stackFileAbs(stackName, materializedPath); + const buf = await fs.promises.readFile(abs); + hash = sha256Of(buf); + size = buf.length; + } catch { + // file absent on disk; entry records the historical intent + } + inputs.push({ + sourcePath: materializedPath, + materializedPath, + role, + dependencyKind: kind, + ownership: 'managed', + provenance: 'migration', + sensitivity: kind === 'sync-env' ? 'high' : 'medium', + contentSha256: hash, + sizeBytes: size, + state: 'present', + deletionAuthority: authority, + note, + }); + }; + + if (spec && spec.files.length > 0) { + for (const [index, file] of spec.files.entries()) { + // Must be awaited: addEntry reads the disk before pushing, and + // the manifest + counts below are built from the final array. + await addEntry(file, index === 0 ? 'compose-primary' : 'compose-additional', 'explicit', 'sencho', null); + } + if (spec.contextDir) { + inputs.push({ + sourcePath: spec.contextDir, + materializedPath: spec.contextDir, + role: 'build-context', + dependencyKind: 'build-context', + ownership: 'managed', + provenance: 'migration', + sensitivity: 'low', + contentSha256: null, + sizeBytes: null, + state: 'present', + deletionAuthority: 'none', + note: 'Project directory subtree was not enumerated historically; no deletion authority inferred for files inside it', + }); + } + } else { + await addEntry('compose.yaml', 'compose-primary', 'explicit', 'sencho', null); + } + if (source.sync_env) { + await addEntry('.env', 'env', 'sync-env', 'sencho', null); + } + + const invocation: string[] = []; + try { + invocation.push(...(await authoredComposeFileArgs(stackName, nodeId))); + invocation.push(...(await authoredComposeEnvFileArgs(stackName, nodeId))); + } catch { + // invocation is best-effort at migration time; a fresh pull rebuilds it + } + + const manifest: GitProjectManifest = { + schemaVersion: 1, + manifestVersion: priorVersion + 1, + state: 'migrated', + generatedAt: now, + identity: this.expectedIdentity(stackName, source.repo_url, source.branch), + repo: { url: source.repo_url, branch: source.branch }, + resolvedRevision: { commitSha: '', fetchedAt: now }, + project: { + root: spec?.contextDir ?? null, + composeFiles: spec?.files ?? ['compose.yaml'], + effectiveProjectDir: spec?.contextDir ?? null, + projectName: stackName, + invocation, + }, + inputs, + refusals: [], + buildContexts: [], + generation: { candidateDir: '', appliedDir: '', previousDir: null }, + counts: { + managed: inputs.filter((i) => i.ownership === 'managed').length, + unmanaged: 0, + refused: 0, + }, + bounds, + }; + // Backfill the previous applied generation so restore works after a + // crash even before the first fresh pull. + const appliedRel = `${GENERATIONS_DIR}/applied-migration`; + const appliedAbs = path.join(this.managedRoot(stackName), appliedRel); + await fs.promises.mkdir(appliedAbs, { recursive: true }); + for (const entry of inputs) { + if (entry.materializedPath === null) continue; + try { + const abs = await this.stackFileAbs(stackName, entry.materializedPath); + const dest = path.join(appliedAbs, entry.materializedPath); + await fs.promises.mkdir(path.dirname(dest), { recursive: true }); + await fs.promises.copyFile(abs, dest); + } catch { + // best-effort snapshot of the migrated state + } + } + manifest.generation.appliedDir = appliedRel; + return manifest; + } + + // ─── Detach/export ─────────────────────────────────────────────────────── + + /** + * Render the effective compose model with the exact authored invocation + * (no mesh). Throws when the render fails or the output is not usable, so + * the detach transaction aborts before anything changes. + */ + async exportForDetach(stackName: string, render: () => Promise): Promise { + let rendered: string; + try { + rendered = await render(); + } catch (e) { + // A render failure aborts the detach transaction; tag it so the + // route can answer 409 (row kept) rather than a generic 500. + throw Object.assign(new Error(`Detach render failed: ${e instanceof Error ? e.message : String(e)}`), { code: 'RENDER_FAILED' }); + } + if (!rendered || !rendered.trim()) { + throw Object.assign(new Error('Detach render produced empty output; nothing exported'), { code: 'RENDER_FAILED' }); + } + try { + const parsed = YAML.parse(rendered); + if (!parsed || typeof parsed !== 'object') { + throw new Error('Detach render produced invalid YAML'); + } + } catch (e) { + throw Object.assign(new Error(`Detach render failed to parse: ${e instanceof Error ? e.message : String(e)}`), { code: 'RENDER_FAILED' }); + } + return rendered; + } +} diff --git a/backend/src/services/GitSourceService.ts b/backend/src/services/GitSourceService.ts index fdaaa56c..de35b626 100644 --- a/backend/src/services/GitSourceService.ts +++ b/backend/src/services/GitSourceService.ts @@ -17,6 +17,10 @@ import { isDebugEnabled } from '../utils/debug'; import { sanitizeForLog } from '../utils/safeLog'; import { isPathWithinBase, isValidRelativeStackPath } from '../utils/validation'; import { gitSourceLocalComposeFiles, PRIMARY_COMPOSE_FILENAME } from '../utils/gitComposeFiles'; +import { ComposeInputDiscoveryService, type ContextCopyPlan } from './ComposeInputDiscoveryService'; +import { GitProjectManifestService } from './GitProjectManifestService'; +import { authoredComposeFileArgs, authoredComposeEnvFileArgs } from '../utils/authoredComposeArgs'; +import type { ComposeInputEntry, GitProjectManifest, GitSourceManifestState, InventoryResult, ManifestSummary, RefusalInfo } from '../types/gitProjectManifest'; import type { GitHttpRequest, GitHttpResponse, HttpClient } from 'isomorphic-git/http/node'; // isomorphic-git is the heaviest dependency in the backend (~5 MB) and only @@ -173,6 +177,18 @@ export class GitSourceError extends Error { } } +/** + * Merge the synthesized sync-env entry into the discovery inventory with a + * one-entry-per-path invariant: the synced stack-root .env owns the path, so + * any discovery entry for it is dropped. The discovery guard already marks the + * repo's interpolation .env unmanaged when syncEnv is on; this dedupe enforces + * the invariant regardless of which branch produced the entries. + */ +function mergeSyncEnvEntry(inputs: ComposeInputEntry[], syncEntry: ComposeInputEntry | null): ComposeInputEntry[] { + if (!syncEntry) return inputs; + return [...inputs.filter((i) => i.materializedPath !== syncEntry.materializedPath), syncEntry]; +} + /** A single compose file fetched from a repo, keyed by its repo-relative path. */ export interface ComposeFile { path: string; @@ -186,6 +202,12 @@ export interface FetchParams { envPath?: string | null; token?: string | null; timeoutMs?: number; + /** + * Runs inside the clone lifecycle (before the temp dir is removed) so the + * caller can discover and stage the complete project from the checkout. + * Used by the pull/create paths for complete-project materialization. + */ + onClone?: (cloneDir: string, commitSha: string, envContent: string | null) => Promise; } export interface FetchResult { @@ -198,6 +220,16 @@ export interface FetchResult { * UI should surface these so the user is not surprised later. */ warnings: string[]; + /** Set when an onClone hook ran (complete-project materialization). */ + materialization?: MaterializationResult | null; +} + +/** Result of discovery + candidate staging inside the clone lifecycle. */ +export interface MaterializationResult { + inventory: InventoryResult; + contextCopyPlans: ContextCopyPlan[]; + candidateRelPath: string; + validation: { ok: boolean; error?: string }; } export interface UpsertInput { @@ -243,6 +275,14 @@ export interface PullResult { currentEnv: string | null; validation: { ok: boolean; error?: string }; hasLocalChanges: boolean; + /** Tolerated (non-actionable) refusals from complete-project discovery. */ + refusals: RefusalInfo[]; + /** Projection of the current managed-project manifest, when one exists. */ + manifestSummary: ManifestSummary | null; + /** True when a validated candidate is staged and ready to apply. */ + candidateReady: boolean; + /** Clone-time warnings (submodules present, for example). */ + warnings: string[]; } export interface PublicGitSource { @@ -264,6 +304,8 @@ export interface PublicGitSource { pending_fetched_at: number | null; created_at: number; updated_at: number; + /** Managed-project manifest cache state (DB-only enum, see gitProjectManifest.ts). */ + manifest_state: GitSourceManifestState | null; } // ─── Constants ─────────────────────────────────────────────────────────────── @@ -315,6 +357,41 @@ function scrubCredentials(message: string): string { .replace(/(password[:=]\s*)[^\s,;]+/gi, '$1***'); } +/** + * Strip absolute DATA_DIR / candidate paths from compose validation stderr + * before it reaches the API, so operators never see host layout details. + * Only remove directory prefixes (root + separator), never a bare substring + * that could corrupt a longer path (for example `data` inside `database`). + */ +function scrubInternalPaths(message: string, ...roots: Array): string { + let out = message; + for (const root of roots) { + if (!root) continue; + for (const variant of new Set([root, root.replace(/\\/g, '/'), root.replace(/\//g, '\\')])) { + if (!variant) continue; + const prefixes = + variant.endsWith('/') || variant.endsWith('\\') + ? [variant] + : [`${variant}/`, `${variant}\\`]; + for (const prefix of prefixes) { + out = out.split(prefix).join(''); + } + } + } + // Catch any remaining absolute .../git-managed/... path (temp dirs, other nodes). + out = out.replace(/(?:[A-Za-z]:)?(?:\/|\\)[^\s"']*?(?:\/|\\)git-managed(?:\/|\\)[^\s"']*/gi, '[managed-path]'); + return out.replace(/\/{2,}/g, '/').replace(/\\{2,}/g, '\\').trim(); +} + +function publicComposeValidationError( + stderr: string, + exitCode: number, + ...roots: Array +): string { + const text = stderr.trim() || `docker compose exited with code ${exitCode}`; + return scrubCredentials(scrubInternalPaths(text, ...roots)); +} + /** * Extract just the hostname for log lines so we never echo a full * repo URL that could contain an inline credential. Falls back to @@ -391,7 +468,7 @@ function transportError(code: string, host: string): GitSourceError | null { */ const LFS_POINTER_PREFIX = 'version https://git-lfs.github.com/spec/v'; -function isLfsPointer(content: string): boolean { +export function isLfsPointer(content: string): boolean { // Pointer files are a few lines of ASCII, always starting with the // version header on the first line. Check just the leading bytes so // a very large plain file does not trigger a full scan. @@ -494,6 +571,11 @@ async function removeTempDir(dir: string): Promise { } } +/** Module-level boot hook for the managed-area sweep (see sweepOrphans). */ +export async function sweepGitManifestOrphans(): Promise { + await GitSourceService.getInstance().sweepOrphans(); +} + /** * Sweep any leftover sencho-git-* temp dirs older than 1 hour. Runs once at * service boot to clean up after a crashed process. @@ -562,6 +644,7 @@ export class GitSourceService { pending_fetched_at: src.pending_fetched_at, created_at: src.created_at, updated_at: src.updated_at, + manifest_state: src.manifest_state ?? 'absent', }; } @@ -598,6 +681,24 @@ export class GitSourceService { throw new GitSourceError('GIT_ERROR', 'Auto-deploy requires auto-apply-on-webhook to be enabled.'); } + // Repository identity changes on a managed stack deadlock: the manifest + // file is stamped with the old repo/branch, and every subsequent apply + // refuses it as untrusted forever (a pull stages a new pending blob but + // never replaces the manifest file). Require a detach (the export + // contract flattens the effective model into compose.yaml) before + // re-pointing the source. + const identityChanged = !!existing && (existing.repo_url !== input.repoUrl || existing.branch !== input.branch); + if (identityChanged) { + const manifestSvc = GitProjectManifestService.getInstance(); + const manifest = await manifestSvc.readManifest(input.stackName, existing.repo_url, existing.branch); + if (manifest !== null) { + throw new GitSourceError( + 'GIT_ERROR', + 'Changing the repository or branch of a stack with a managed project is not supported. Detach the Git source first (the effective compose model is exported to compose.yaml), then re-link the source to the new repository or branch.', + ); + } + } + // Dry-run reachability check before persisting. Fetches every configured // file so a bad path in the ordered list is caught at save time. const token = encryptedToken ? this.crypto.decrypt(encryptedToken) : null; @@ -651,8 +752,212 @@ export class GitSourceService { return this.get(input.stackName)!; } - public delete(stackName: string): void { - DatabaseService.getInstance().deleteGitSource(stackName); + /** + * Detach a Git source under the export contract: render the effective + * compose model into a single compose.yaml, keep the materialized files, + * remove the managed area, then drop the row. Ordering guarantees every + * failure leaves the stack and the row intact; the render is deterministic + * per disk state, so a late failure leaves detach safely re-runnable. + */ + public async detach(stackName: string): Promise { + return this.withStackLock(stackName, async () => { + const src = DatabaseService.getInstance().getGitSource(stackName); + if (!src) throw new GitSourceError('GIT_ERROR', 'No Git source configured for this stack.'); + const manifestSvc = GitProjectManifestService.getInstance(); + await manifestSvc.recoverInterruptedDetach(stackName, src.repo_url, src.branch); + + // Phase 1: render the effective model. A render failure aborts + // before anything on disk changes. + const rendered = await manifestSvc.exportForDetach(stackName, () => + ComposeService.getInstance().renderComposeYaml(stackName), + ); + + // Phase 2: snapshot compose.yaml and every managed override so a + // mid-detach failure can restore the exact pre-detach state. + const manifest = await manifestSvc.readManifest(stackName, src.repo_url, src.branch); + // A corrupt manifest means the stack's ownership status is unknown: + // detach cannot know which files are auto-discovered overrides and + // must not proceed. A missing manifest means the stack was never + // materialized, so there are no managed overrides to clean up. + if (manifest !== null && 'corrupt' in manifest) { + throw new GitSourceError('GIT_ERROR', 'Managed-project manifest cannot be trusted; detach aborted. Pull to rebuild before detaching.'); + } + const snapshotFileLimit = manifest?.bounds.maxFileBytes ?? manifestSvc.boundsConfig().maxFileBytes; + let priorCompose: Buffer | null = null; + try { + const content = await FileSystemService.getInstance().readStackFile(stackName, 'compose.yaml', snapshotFileLimit, { forceText: true }); + if (content.oversized || content.content === undefined) { + throw new GitSourceError('GIT_ERROR', 'compose.yaml cannot be snapshotted within the managed file-size limit; detach aborted.'); + } + priorCompose = Buffer.from(content.content, 'utf8'); + } catch (e) { + if (e instanceof GitSourceError) throw e; + if ((e as NodeJS.ErrnoException).code !== 'ENOENT') { + const cause = e instanceof Error ? e.message : String(e); + console.error(`[GitSource] compose snapshot read failed for ${sanitizeForLog(stackName)}:`, sanitizeForLog(cause)); + throw new GitSourceError('GIT_ERROR', 'Cannot read compose.yaml for snapshot; detach aborted.'); + } + } + const overrideSnapshots: Array<{ path: string; content: Buffer }> = []; + // Only AUTO-DISCOVERED implicit overrides are removed: an explicit + // compose file, config, secret, or include that happens to share the + // basename is part of the rendered model and must survive detach. + const managedOverridePaths = manifest?.inputs + .flatMap((entry) => entry.ownership === 'managed' && entry.dependencyKind === 'implicit-override' && entry.materializedPath !== null ? [entry.materializedPath] : []) ?? []; + for (const overridePath of managedOverridePaths) { + try { + const content = await FileSystemService.getInstance().readStackFile(stackName, overridePath, snapshotFileLimit, { forceText: true }); + if (content.oversized || content.content === undefined) { + throw new GitSourceError('GIT_ERROR', `Override ${overridePath} cannot be snapshotted within the managed file-size limit; detach aborted.`); + } + overrideSnapshots.push({ path: overridePath, content: Buffer.from(content.content, 'utf8') }); + } catch (e) { + if (e instanceof GitSourceError) throw e; + if ((e as NodeJS.ErrnoException).code !== 'ENOENT') { + const cause = e instanceof Error ? e.message : String(e); + console.error(`[GitSource] override snapshot read failed for ${sanitizeForLog(stackName)}:`, sanitizeForLog(cause)); + throw new GitSourceError('GIT_ERROR', `Cannot read override ${overridePath} for snapshot; detach aborted.`); + } + // ENOENT: file not present on disk; nothing to snapshot. + } + } + + // The snapshot lives in the managed area and is restored by the + // boot sweep if the process exits before the database commit point. + const detachSnapshots = [ + priorCompose !== null + ? { path: 'compose.yaml', existed: true as const, content: priorCompose } + : { path: 'compose.yaml', existed: false as const, content: null }, + ...overrideSnapshots.map((snapshot) => ({ path: snapshot.path, existed: true as const, content: snapshot.content })), + ]; + await manifestSvc.prepareDetachRecovery(stackName, src.repo_url, src.branch, detachSnapshots); + + // Phase 3: write the flattened compose.yaml, then delete + // overrides. If anything fails, restore compose.yaml and every + // deleted override so the stack is byte-identical to pre-detach. + let areaStaged = false; + const rollback = async (): Promise<'complete' | 'missing' | 'failed'> => { + try { + let restored: boolean; + if (areaStaged) { + restored = await manifestSvc.rollbackStagedDetach(stackName, src.repo_url, src.branch); + areaStaged = false; + } else { + restored = await manifestSvc.recoverInterruptedDetach(stackName, src.repo_url, src.branch); + } + if (!restored) { + console.error(`[GitSource] detach rollback for ${sanitizeForLog(stackName)} found no recovery snapshot`); + return 'missing'; + } + } catch (e) { + const cause = e instanceof Error ? e.message : String(e); + console.error(`[GitSource] detach rollback failed for ${sanitizeForLog(stackName)}:`, sanitizeForLog(cause)); + return 'failed'; + } + return 'complete'; + }; + const rollbackAndThrow = async (message: string, cause?: unknown): Promise => { + if (cause !== undefined) { + const causeMessage = cause instanceof Error ? cause.message : String(cause); + console.error(`[GitSource] detach failed for ${sanitizeForLog(stackName)}:`, sanitizeForLog(causeMessage)); + } + const rollbackResult = await rollback(); + let outcome = 'detach rolled back.'; + if (rollbackResult === 'failed') { + outcome = 'detach rollback is incomplete; restart Sencho to retry recovery.'; + } else if (rollbackResult === 'missing') { + outcome = 'detach rollback could not find its recovery snapshot; inspect the stack files before retrying.'; + } + throw new GitSourceError('GIT_ERROR', `${message}; ${outcome} Retry to complete it.`); + }; + try { + await FileSystemService.getInstance().saveStackContent(stackName, rendered); + } catch (e) { + await rollbackAndThrow('Could not write the detached compose model', e); + } + + for (const snapshot of overrideSnapshots) { + try { + await FileSystemService.getInstance().deleteStackPath(stackName, snapshot.path, false); + } catch (e) { + await rollbackAndThrow(`Could not remove auto-discovered override ${snapshot.path}`, e); + } + } + + // Phase 4: stage the managed area, then delete the row as the + // commit point. A database failure puts the area back and restores + // the durable file snapshot. A crash before the commit is handled + // by the boot sweep; a crash after it leaves an orphan stage that + // the normal orphan sweep removes. + try { + areaStaged = await manifestSvc.stageManagedAreaForDetach(stackName); + } catch (e) { + await rollbackAndThrow('Could not stage the managed project data', e); + } + if (!areaStaged) { + await rollbackAndThrow('Managed project data disappeared during detach'); + } + try { + DatabaseService.getInstance().deleteGitSource(stackName); + } catch (e) { + await rollbackAndThrow('Could not commit the Git source removal', e); + } + if (!(await manifestSvc.finalizeStagedDetach(stackName))) { + console.warn(`[GitManifest] detach for ${sanitizeForLog(stackName)} committed with managed cleanup pending`); + } + }); + } + + /** The managed-project manifest for a stack, when it exists and is trusted. */ + public async getManifest(stackName: string): Promise { + const src = DatabaseService.getInstance().getGitSource(stackName); + if (!src) return null; + const manifest = await GitProjectManifestService.getInstance().readManifest(stackName, src.repo_url, src.branch); + return manifest !== null && !('corrupt' in manifest) ? manifest : null; + } + + /** + * Summary projection of the managed-project manifest. When the manifest + * FILE is absent or cannot be trusted, the summary is synthesized from the + * DB cache so the UI can render the actual state ('absent' / + * 'migration_required') instead of treating corruption as "nothing". + */ + public async getManifestSummary(stackName: string): Promise { + const src = DatabaseService.getInstance().getGitSource(stackName); + if (!src) return null; + const manifestSvc = GitProjectManifestService.getInstance(); + const manifest = await manifestSvc.readManifest(stackName, src.repo_url, src.branch); + if (manifest !== null && !('corrupt' in manifest)) { + return manifestSvc.summaryFrom(manifest); + } + // No manifest file (or an untrusted one). When the DB cache claims an + // applied state but the file is gone, report migration_required + // instead of manufacturing a healthy state: the manifest may have + // been lost and the stack's ownership is unknown. + let state: GitSourceManifestState; + if (manifest !== null && 'corrupt' in manifest) { + state = 'migration_required'; + } else { + const cached = src.manifest_state ?? 'absent'; + state = cached === 'absent' || cached === 'none' ? 'absent' : 'migration_required'; + } + // Heal the flat cache so the same GET payload cannot report + // manifest_state:"active" beside manifest.state:"migration_required". + // Synthesized absent/migration_required never carries a trusted version. + if ((src.manifest_state ?? 'absent') !== state) { + DatabaseService.getInstance().setGitSourceManifestState(stackName, null, state, null); + } + return { + state, + manifestVersion: 0, + resolvedCommitSha: null, + managedCount: 0, + unmanagedCount: 0, + refusedCount: 0, + refused: [], + hasBuildContexts: false, + generatedAt: null, + }; } // ─── Fetch ─────────────────────────────────────────────────────────────── @@ -798,12 +1103,17 @@ export class GitSourceService { } } + let materialization: MaterializationResult | null = null; + if (params.onClone) { + materialization = (await params.onClone(dir, commitSha, envContent)) as MaterializationResult | null; + } + if (diag) { console.log( - `[GitSource:diag] fetch ok host=${sanitizeForLog(repoHost(repoUrl))} branch=${sanitizeForLog(branch)} sha=${commitSha.slice(0, 7)} files=${composeFiles.length} env=${envContent !== null ? 'present' : 'absent'} warnings=${warnings.length} elapsedMs=${Date.now() - startedAt}` + `[GitSource:diag] fetch ok host=${sanitizeForLog(repoHost(repoUrl))} branch=${sanitizeForLog(branch)} sha=${commitSha.slice(0, 7)} files=${composeFiles.length} env=${envContent !== null ? 'present' : 'absent'} warnings=${warnings.length} materialized=${materialization !== null} elapsedMs=${Date.now() - startedAt}` ); } - return { composeFiles, envContent, commitSha, warnings }; + return { composeFiles, envContent, commitSha, warnings, materialization }; }); } catch (err) { if (diag) { @@ -995,12 +1305,132 @@ export class GitSourceService { args.push('config', '--quiet'); const result = await this.runDockerCompose(args, dir, 10_000); if (result.code === 0) return { ok: true }; - return { ok: false, error: result.stderr.trim() || `docker compose exited with code ${result.code}` }; + return { ok: false, error: publicComposeValidationError(result.stderr, result.code, dir) }; } finally { await removeTempDir(dir); } } + /** + * Complete-project materialization inside the clone lifecycle: discover + + * classify every declared input, abort on actionable refusals, build the + * staged candidate (managed files + filtered build contexts), and validate + * the exact candidate with the exact deploy invocation (including -p). + * Runs only when the complete-project contract applies. + */ + private async buildMaterialization( + stackName: string, + cloneDir: string, + commitSha: string, + src: { compose_paths: string[]; context_dir: string | null; sync_env: boolean }, + envContent: string | null, + ): Promise { + const manifestSvc = GitProjectManifestService.getInstance(); + const bounds = manifestSvc.boundsConfig(); + + const inventory = await ComposeInputDiscoveryService.getInstance().discoverFromClone({ + cloneDir, + composePaths: src.compose_paths, + contextDir: src.context_dir, + // The synced stack-root .env owns that path when sync_env is on; the + // discovery guard makes the repo's interpolation .env unmanaged so it + // is never hash-guarded against the staged sync content. + syncEnv: src.sync_env, + bounds, + }); + + const actionable = inventory.refusals.filter((r) => r.actionable); + if (actionable.length > 0) { + // The abort message is a public surface: high-sensitivity refusal + // paths are redacted before they reach the API. + const publicRefusals = manifestSvc.toPublicRefusals(actionable); + const detail = publicRefusals.slice(0, 5).map((r) => r.reason).join('; '); + throw new GitSourceError('GIT_ERROR', `Cannot materialize the complete project: ${detail}${publicRefusals.length > 5 ? ` (and ${publicRefusals.length - 5} more)` : ''}`); + } + + // Build-context entries are directories; their content is copied via + // the context copy plans (dockerignore-filtered), never as files. + const managed = inventory.inputs.filter( + (i) => + i.ownership === 'managed' && + i.materializedPath !== null && + i.state === 'present' && + i.dependencyKind !== 'build-context' && + i.dependencyKind !== 'build-additional-context', + ); + const candidateRel = await manifestSvc.buildCandidate( + stackName, + commitSha, + cloneDir, + managed.map((i) => ({ srcRel: i.sourcePath!, destRel: i.materializedPath! })), + inventory.contextCopyPlans, + bounds, + ); + + // Stage the synced env into the candidate so validation exercises the + // exact deploy layout (stack-root .env). + if (src.sync_env && envContent !== null) { + const candidateAbs = path.join(process.env.DATA_DIR || path.join(process.cwd(), 'data'), 'git-managed', String(NodeRegistry.getInstance().getDefaultNodeId()), stackName, candidateRel); + await fsPromises.mkdir(candidateAbs, { recursive: true }); + await fsPromises.writeFile(path.join(candidateAbs, '.env'), envContent, 'utf8'); + } + + const validation = await this.validateCandidate(stackName, candidateRel, src.compose_paths, src.context_dir); + return { inventory, contextCopyPlans: inventory.contextCopyPlans, candidateRelPath: candidateRel, validation }; + } + + /** + * Validate the staged candidate with the exact deploy invocation: the same + * relative -f order, -p project name, --project-directory, and --env-file + * the deploy uses, run inside the candidate dir. Candidate validation gets a + * larger budget than the pull preview (30s) and names the timeout. + */ + private async validateCandidate(stackName: string, candidateRelPath: string, composePaths: string[], contextDir: string | null): Promise<{ ok: boolean; error?: string }> { + const nodeId = NodeRegistry.getInstance().getDefaultNodeId(); + const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data'); + const candidateAbs = path.join(dataDir, 'git-managed', String(nodeId), stackName, candidateRelPath); + const localFiles = gitSourceLocalComposeFiles(composePaths); + + const args = ['compose']; + for (const local of localFiles) { + // The candidate mirrors the stack layout 1:1 (primary -> compose.yaml, + // additional files at their repo-relative path), so the -f path is + // used AS-IS; basename-collapsing would break nested additional files. + const safeRel = local.replace(/\\/g, '/'); + const abs = path.resolve(candidateAbs, safeRel); + if (!isPathWithinBase(abs, candidateAbs)) { + return { ok: false, error: `Compose path escapes the candidate dir: ${local}` }; + } + args.push('-f', safeRel); + } + args.push('-p', stackName); + if (contextDir) { + const baseResolved = path.resolve(candidateAbs); + const ctxAbs = path.resolve(baseResolved, contextDir); + if (!ctxAbs.startsWith(baseResolved + path.sep)) { + return { ok: false, error: 'Context directory escapes the candidate dir.' }; + } + args.push('--project-directory', ctxAbs); + } + // Mirror the deploy-time env resolution: only pass --env-file when the + // candidate actually carries the env (sync-env stacks stage it there). + const candidateEnv = path.join(candidateAbs, '.env'); + try { + await fsPromises.access(candidateEnv); + args.push('--env-file', candidateEnv); + } catch { + // no staged env; compose falls back to environment interpolation + } + args.push('config', '--quiet'); + const result = await this.runDockerCompose(args, candidateAbs, 30_000); + if (result.code === 0) return { ok: true }; + const timeoutHint = result.stderr.includes('Validation timed out') ? ' (docker compose config timed out after 30s)' : ''; + return { + ok: false, + error: `${publicComposeValidationError(result.stderr, result.code, candidateAbs, dataDir)}${timeoutHint}`, + }; + } + private runDockerCompose(args: string[], cwd: string, timeoutMs: number): Promise<{ code: number; stdout: string; stderr: string }> { return new Promise((resolve) => { const child = spawn('docker', args, { cwd }); @@ -1062,32 +1492,59 @@ export class GitSourceService { return { files: gitSourceLocalComposeFiles(composePaths), contextDir: contextDir ?? null }; } - /** Encrypt the ordered compose file set as the v2 pending blob (carries contextDir). */ - private encodePendingCompose(files: ComposeFile[], contextDir: string | null): string { - return this.crypto.encrypt(JSON.stringify({ v: 2, files, contextDir })); + /** Encrypt the ordered compose file set as the v3 pending blob (carries contextDir + staged candidate + inventory). */ + private encodePendingCompose(files: ComposeFile[], contextDir: string | null, candidateRelPath: string | null, inventory: InventoryResult | null): string { + return this.crypto.encrypt(JSON.stringify({ v: 3, files, contextDir, candidateRelPath, inventory })); } /** * Decrypt a stored pending compose blob into its ordered file set + contextDir. - * Detects the v2 marker; anything else is a legacy single-file plaintext string. + * Detects the v3 marker first, then v2, then legacy single-file plaintext. + * The `{"v":2` / `{"v":3` prefixes are mutually exclusive, so ordering is for + * readability only; a parse failure under a version marker is corrupt state + * (logged, never silently treated as legacy). */ - private decodePendingCompose(stored: string): { files: ComposeFile[]; contextDir: string | null } { + private decodePendingCompose(stored: string): { files: ComposeFile[]; contextDir: string | null; candidateRelPath: string | null; inventory: InventoryResult | null } { const raw = this.crypto.decrypt(stored); + if (raw.startsWith('{"v":3')) { + try { + const parsed = JSON.parse(raw) as { v: number; files?: ComposeFile[]; contextDir?: string | null; candidateRelPath?: string | null; inventory?: InventoryResult | null }; + // Shape gate: the inventory drives the manifest build, so a + // structurally wrong inventory must be corrupt, never silently + // filtered into a half-populated manifest. + const inventoryValid = + parsed.inventory === null || + (parsed.inventory !== undefined && + Array.isArray(parsed.inventory.inputs) && + Array.isArray(parsed.inventory.refusals) && + Array.isArray(parsed.inventory.buildContexts)); + if (Array.isArray(parsed.files) && parsed.files.length > 0 && inventoryValid) { + return { + files: parsed.files, + contextDir: parsed.contextDir ?? null, + candidateRelPath: typeof parsed.candidateRelPath === 'string' ? parsed.candidateRelPath : null, + inventory: parsed.inventory ?? null, + }; + } + } catch (e) { + console.error('[GitSource] pending compose blob carried the v3 marker but failed to parse:', (e as Error).message); + } + // A v3-marker blob that fails to parse is corrupt state, not a + // legacy row: applying it as plaintext compose would deploy garbage + // or a misleading validation error. + throw new GitSourceError('GIT_ERROR', 'Pending update is corrupt; pull again to rebuild it.'); + } if (raw.startsWith('{"v":2')) { try { const parsed = JSON.parse(raw) as { v: number; files?: ComposeFile[]; contextDir?: string | null }; if (Array.isArray(parsed.files) && parsed.files.length > 0) { - return { files: parsed.files, contextDir: parsed.contextDir ?? null }; + return { files: parsed.files, contextDir: parsed.contextDir ?? null, candidateRelPath: null, inventory: null }; } } catch (e) { - // The v2 marker proves this was written as multi-file, so a parse - // failure signals a corrupt pending blob, not a legacy row. Log it - // so the misleading downstream validation error is traceable; the - // re-validate in apply still blocks deploying the garbled content. console.error('[GitSource] pending compose blob carried the v2 marker but failed to parse; treating as legacy:', (e as Error).message); } } - return { files: [{ path: PRIMARY_COMPOSE_FILENAME, content: raw }], contextDir: null }; + return { files: [{ path: PRIMARY_COMPOSE_FILENAME, content: raw }], contextDir: null, candidateRelPath: null, inventory: null }; } private async readDiskContent(stackName: string, syncEnv: boolean, relFiles: string[]): Promise<{ files: ComposeFile[]; env: string | null }> { @@ -1114,7 +1571,8 @@ export class GitSourceService { if (syncEnv) { try { env = await fsSvc.getEnvContent(stackName); - } catch { + } catch (e) { + console.warn(`[GitSource] could not read .env for ${sanitizeForLog(stackName)} diff:`, (e as Error).message); env = null; } } @@ -1205,15 +1663,24 @@ export class GitSourceService { } const token = src.encrypted_token ? this.crypto.decrypt(src.encrypted_token) : null; + const manifestSvc = GitProjectManifestService.getInstance(); + // Object holder: property access is not narrowed by control-flow + // analysis, so the closure assignment below stays visible. + const materialization: { value: MaterializationResult | null } = { value: null }; const fetched = await this.fetchFromGit({ repoUrl: src.repo_url, branch: src.branch, composePaths: src.compose_paths, envPath: src.sync_env ? src.env_path : null, token, + onClone: async (cloneDir, commitSha, envContent) => { + materialization.value = await this.buildMaterialization(stackName, cloneDir, commitSha, src, envContent); + }, }); - const validation = await this.validateCompose(fetched.composeFiles, fetched.envContent, src.context_dir); + const validation = materialization.value + ? materialization.value.validation + : await this.validateCompose(fetched.composeFiles, fetched.envContent, src.context_dir); const appliedFiles = src.applied_deploy_spec?.files ?? [PRIMARY_COMPOSE_FILENAME]; const disk = await this.readDiskContent(stackName, src.sync_env, appliedFiles); const currentHash = this.hashContent(disk.files, disk.env); @@ -1222,17 +1689,26 @@ export class GitSourceService { // Store pending so a subsequent apply doesn't re-fetch. Compose files // routinely contain secrets inlined as env interpolations or passwords, - // so the v2 blob (ordered files + contextDir) is encrypted at rest. + // so the v3 blob (ordered files + contextDir + candidate + inventory) + // is encrypted at rest. db.setGitSourcePending( stackName, fetched.commitSha, - this.encodePendingCompose(fetched.composeFiles, src.context_dir), + this.encodePendingCompose(fetched.composeFiles, src.context_dir, materialization.value?.candidateRelPath ?? null, materialization.value?.inventory ?? null), fetched.envContent !== null ? this.crypto.encrypt(fetched.envContent) : null, ); - console.log(`[GitSource] Pending update ready for ${stackName} at ${fetched.commitSha.slice(0, 7)} (validation=${validation.ok ? 'ok' : 'fail'}, localEdits=${hasLocalChanges})`); + // Prior manifest summary for the pull response (managed/unmanaged/refused + // counts + pinned revision); a corrupt manifest is surfaced as such. + let manifestSummary: ManifestSummary | null = null; + const prior = await manifestSvc.readManifest(stackName, src.repo_url, src.branch); + if (prior !== null && !('corrupt' in prior)) { + manifestSummary = manifestSvc.summaryFrom(prior); + } + + console.log(`[GitSource] Pending update ready for ${stackName} at ${fetched.commitSha.slice(0, 7)} (validation=${validation.ok ? 'ok' : 'fail'}, localEdits=${hasLocalChanges}, candidate=${materialization.value?.candidateRelPath ?? "none"})`); if (diag) { - console.log(`[GitSource:diag] pull done stack=${stackName} sha=${fetched.commitSha.slice(0, 7)} validation=${validation.ok} localEdits=${hasLocalChanges}`); + console.log(`[GitSource:diag] pull done stack=${stackName} sha=${fetched.commitSha.slice(0, 7)} validation=${validation.ok} localEdits=${hasLocalChanges} candidate=${materialization.value !== null}`); } return { @@ -1243,6 +1719,11 @@ export class GitSourceService { currentEnv: disk.env, validation, hasLocalChanges, + // High-sensitivity refusals are redacted on every public surface. + refusals: manifestSvc.toPublicRefusals(materialization.value?.inventory.refusals ?? []), + manifestSummary, + candidateReady: materialization.value !== null && materialization.value.validation.ok, + warnings: fetched.warnings, }; } @@ -1287,24 +1768,189 @@ export class GitSourceService { // Materialize from the pending blob (its files + contextDir), never the // live config: a config edit between pull and apply must not change what - // gets written. The v2 blob is decoded here; legacy plaintext is treated - // as a single compose.yaml. + // gets written. The v3 blob is decoded here (legacy v2/plaintext blobs + // fall back to the historical file-set path below). const pending = this.decodePendingCompose(src.pending_compose_content); const envContent = src.pending_env_content !== null ? this.crypto.decrypt(src.pending_env_content) : null; + const manifestSvc = GitProjectManifestService.getInstance(); - // Re-validate before writing. - const validation = await this.validateCompose(pending.files, envContent, pending.contextDir); - if (!validation.ok) { - if (diag) console.log(`[GitSource:diag] apply validation fail stack=${stackName}`); - throw new GitSourceError('GIT_ERROR', `Compose validation failed: ${validation.error}`); + let appliedSpec: GitSourceAppliedSpec | null; + if (pending.candidateRelPath !== null && pending.inventory !== null) { + // ── Complete-project path (v3 pending) ─────────────────────────── + const prior = await manifestSvc.readManifest(stackName, src.repo_url, src.branch); + if (prior !== null && 'corrupt' in prior) { + // Any identity-stamp corruption (missing identity, node/stack/ + // repo/branch mismatch) is unrecoverable by pulling (a pull + // never replaces the manifest file); the actionable escape is + // detach + re-link. + const identityMismatch = prior.corrupt.includes('identity'); + throw new GitSourceError( + 'GIT_ERROR', + identityMismatch + ? `The managed-project manifest for ${stackName} is stamped for a different repository or branch. Detach the Git source, then re-link it to the current repository and branch.` + : `The managed-project manifest for ${stackName} cannot be trusted (${prior.corrupt}). Detach the Git source and re-link it to rebuild the managed project.`, + ); + } + + // The staged candidate must still exist and be complete; a deleted + // candidate (or a node restart that swept it) invalidates the pull. + const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data'); + const candidateAbs = path.join(dataDir, 'git-managed', String(NodeRegistry.getInstance().getDefaultNodeId()), stackName, pending.candidateRelPath); + try { + await fsPromises.access(candidateAbs); + } catch { + throw new GitSourceError('GIT_ERROR', 'Pending update was invalidated; pull again.'); + } + + // Re-validate the exact candidate before touching the live project. + const candValidation = await this.validateCandidate(stackName, pending.candidateRelPath, src.compose_paths, src.context_dir); + if (!candValidation.ok) { + if (diag) console.log(`[GitSource:diag] apply candidate validation fail stack=${stackName}`); + throw new GitSourceError('GIT_ERROR', `Candidate validation failed: ${candValidation.error}`); + } + + // Local-modification refusal (convergence prelude): every managed, + // present input must still match the manifest hash, or the incoming + // commit would overwrite user edits. Abort before any write. + if (prior) { + const diverged: string[] = []; + const unreadable: string[] = []; + for (const entry of prior.inputs) { + if (entry.ownership !== 'managed' || entry.state !== 'present' || entry.materializedPath === null) continue; + if (entry.contentSha256 === null) continue; + const diskHash = await manifestSvc.hashStackFile(stackName, entry.materializedPath); + if (diskHash === null) unreadable.push(entry.materializedPath); + else if (diskHash !== entry.contentSha256) diverged.push(entry.materializedPath); + } + // Build contexts are file-granular: local edits, added files, + // and missing files inside a retained context are divergence. + const managedInputPaths = new Set( + prior.inputs + .filter((i) => i.ownership === 'managed' && i.materializedPath !== null) + .map((i) => i.materializedPath!), + ); + for (const context of prior.buildContexts) { + if (context.files.length === 0) continue; + const contextDiverged = await manifestSvc.verifyContextOnDisk(stackName, context, managedInputPaths); + for (const rel of contextDiverged) { + diverged.push(`${context.repoPath}/${rel}`); + } + } + if (diverged.length > 0) { + throw new GitSourceError( + 'GIT_ERROR', + `Local modifications detected on ${diverged.join(', ')}. Detach the Git source or restore these files; the incoming commit will not overwrite local changes.`, + ); + } + if (unreadable.length > 0) { + throw new GitSourceError( + 'GIT_ERROR', + `Could not read ${unreadable.join(', ')} to verify it is unchanged; restore it or detach the Git source before applying.`, + ); + } + } + + // Assemble the new manifest from the pull-time inventory. + const syncEnvEntry: ComposeInputEntry | null = + src.sync_env && envContent !== null + ? { + sourcePath: null, + materializedPath: '.env', + role: 'env', + dependencyKind: 'sync-env', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: crypto.createHash('sha256').update(envContent).digest('hex'), + sizeBytes: Buffer.byteLength(envContent, 'utf8'), + state: 'present', + deletionAuthority: 'sencho', + note: null, + } + : null; + const invocation: string[] = []; + try { + invocation.push(...(await authoredComposeFileArgs(stackName, NodeRegistry.getInstance().getDefaultNodeId()))); + invocation.push(...(await authoredComposeEnvFileArgs(stackName, NodeRegistry.getInstance().getDefaultNodeId()))); + } catch (e) { + console.warn(`[GitSource] invocation build failed for ${stackName}:`, (e as Error).message); + } + const manifest = manifestSvc.buildManifest({ + stackName, + repoUrl: src.repo_url, + branch: src.branch, + commitSha, + projectRoot: src.context_dir, + composeFiles: src.compose_paths, + projectName: stackName, + invocation, + inputs: mergeSyncEnvEntry(pending.inventory.inputs, syncEnvEntry), + refusals: pending.inventory.refusals, + buildContexts: pending.inventory.buildContexts, + bounds: manifestSvc.boundsConfig(), + priorManifest: prior ?? null, + state: pending.inventory.refusals.length > 0 ? 'partial' : 'active', + }); + // An existing pre-manifest stack (legacy Git source) adopts ONLY + // the paths the legacy format owned: the applied compose files and + // the synced .env. Every other existing file at an introduced path + // is a local file the incoming generation must not overwrite. + const legacyOwnedPaths = prior + ? undefined + : [ + ...(src.applied_deploy_spec?.files ?? [PRIMARY_COMPOSE_FILENAME]), + ...(src.sync_env ? ['.env'] : []), + ]; + try { + await manifestSvc.promoteGeneration(stackName, { + sha: commitSha, + candidateRelPath: pending.candidateRelPath, + manifest, + priorManifest: prior ?? null, + adoptExistingMaterializedPaths: legacyOwnedPaths, + }); + } catch (e) { + // Pre-mutation refusals (collision guard, case-only changes) + // and promotion failures surface as clean GitSourceErrors. + // The original error is logged with its stack for diagnosis, + // and the message is scrubbed of credentials and of the + // incoming manifest's high-sensitivity paths. + if (e instanceof GitSourceError) throw e; + const raw = e instanceof Error ? e.message : String(e); + console.error(`[GitSource] promotion failed for ${sanitizeForLog(stackName)}:`, e instanceof Error ? e.stack ?? e.message : raw); + const sensitivePaths = manifest.inputs + .filter((i) => i.sensitivity === 'high' && i.materializedPath !== null) + .map((i) => i.materializedPath!); + let redacted = raw; + for (const rel of sensitivePaths) { + redacted = redacted.split(rel).join('[redacted]'); + } + throw new GitSourceError('GIT_ERROR', scrubCredentials(redacted)); + } + appliedSpec = this.deriveAppliedSpec(src.compose_paths, src.context_dir); + } else { + // ── Legacy path (v2/plaintext pending from before the upgrade) ── + const validation = await this.validateCompose(pending.files, envContent, pending.contextDir); + if (!validation.ok) { + if (diag) console.log(`[GitSource:diag] apply validation fail stack=${stackName}`); + throw new GitSourceError('GIT_ERROR', `Compose validation failed: ${validation.error}`); + } + appliedSpec = await this.materialize( + stackName, pending.files, pending.contextDir, src.sync_env, envContent, src.applied_deploy_spec, + ); + // Migration: build the conservative manifest from spec + disk. + const migrated = await manifestSvc.buildMigratedManifest(stackName, { + repo_url: src.repo_url, + branch: src.branch, + sync_env: src.sync_env, + applied_deploy_spec: appliedSpec, + }); + await manifestSvc.writeManifest(stackName, migrated); + db.setGitSourceManifestState(stackName, migrated.manifestVersion, migrated.state, migrated.generation.appliedDir); } - const appliedSpec = await this.materialize( - stackName, pending.files, pending.contextDir, src.sync_env, envContent, src.applied_deploy_spec, - ); - const hash = this.hashContent(pending.files, envContent); db.markGitSourceApplied(stackName, commitSha, hash); db.setGitSourceAppliedSpec(stackName, appliedSpec); @@ -1380,32 +2026,109 @@ export class GitSourceService { } // 1. Fetch from git BEFORE touching disk or DB. If the fetch - // fails there is nothing to clean up. + // fails there is nothing to clean up. The onClone hook stages + // the complete-project candidate inside the clone lifecycle. + const manifestSvc = GitProjectManifestService.getInstance(); + const materialization: { value: MaterializationResult | null } = { value: null }; const fetched = await this.fetchFromGit({ repoUrl: input.repoUrl, branch: input.branch, composePaths: input.composePaths, envPath: input.syncEnv ? input.envPath : null, token: input.token, + onClone: async (cloneDir, commitSha, envContent) => { + materialization.value = await this.buildMaterialization(input.stackName, cloneDir, commitSha, { + compose_paths: input.composePaths, + context_dir: input.contextDir, + sync_env: input.syncEnv, + }, envContent); + }, }); // 2. Validate against the same `docker compose config` check the // apply path uses. Reject before creating anything on disk. - const validation = await this.validateCompose(fetched.composeFiles, fetched.envContent, input.contextDir); + const validation = materialization.value + ? materialization.value.validation + : await this.validateCompose(fetched.composeFiles, fetched.envContent, input.contextDir); if (!validation.ok) { throw new GitSourceError('GIT_ERROR', `Compose validation failed: ${validation.error}`); } - // 3. Create directory + boilerplate, then materialize the fetched - // files. createStack() throws if the directory already exists, so a + // 3. Create directory + boilerplate, then promote the staged + // candidate (or fall back to the historical file-set write). + // createStack() throws if the directory already exists, so a // name collision is caught here. let stackCreated = false; + let rowInserted = false; + // Promotion persists the manifest cache columns BEFORE the row + // exists (zero rows updated); the cache is written again after the + // insert below so list and immediate projections report the real + // state instead of 'absent'. + let completeProjectManifest: GitProjectManifest | null = null; try { await fsSvc.createStack(input.stackName); stackCreated = true; - const appliedSpec = await this.materialize( - input.stackName, fetched.composeFiles, input.contextDir, input.syncEnv, fetched.envContent, null, - ); + let appliedSpec: GitSourceAppliedSpec | null; + if (materialization.value) { + const inputs = materialization.value.inventory.inputs.filter( + (i) => i.ownership === 'managed' && i.state === 'present' && i.materializedPath !== null, + ); + const syncEnvEntry: ComposeInputEntry | null = + input.syncEnv && fetched.envContent !== null + ? { + sourcePath: null, + materializedPath: '.env', + role: 'env', + dependencyKind: 'sync-env', + ownership: 'managed', + provenance: 'fetch', + sensitivity: 'high', + contentSha256: crypto.createHash('sha256').update(fetched.envContent).digest('hex'), + sizeBytes: Buffer.byteLength(fetched.envContent, 'utf8'), + state: 'present', + deletionAuthority: 'sencho', + note: null, + } + : null; + const invocation: string[] = []; + try { + invocation.push(...(await authoredComposeFileArgs(input.stackName, NodeRegistry.getInstance().getDefaultNodeId()))); + invocation.push(...(await authoredComposeEnvFileArgs(input.stackName, NodeRegistry.getInstance().getDefaultNodeId()))); + } catch { + // best-effort; a fresh pull rebuilds the invocation + } + const manifest = manifestSvc.buildManifest({ + stackName: input.stackName, + repoUrl: input.repoUrl, + branch: input.branch, + commitSha: fetched.commitSha, + projectRoot: input.contextDir, + composeFiles: input.composePaths, + projectName: input.stackName, + invocation, + inputs: mergeSyncEnvEntry(inputs, syncEnvEntry), + refusals: materialization.value.inventory.refusals, + buildContexts: materialization.value.inventory.buildContexts, + bounds: manifestSvc.boundsConfig(), + priorManifest: null, + state: materialization.value.inventory.refusals.length > 0 ? 'partial' : 'active', + }); + await manifestSvc.promoteGeneration(input.stackName, { + sha: fetched.commitSha, + candidateRelPath: materialization.value.candidateRelPath, + manifest, + priorManifest: null, + // The stack directory was just created by this flow; + // every existing file is adoption boilerplate. + adoptExistingMaterializedPaths: 'all', + }); + completeProjectManifest = manifest; + appliedSpec = this.deriveAppliedSpec(input.composePaths, input.contextDir); + } else { + appliedSpec = await this.materialize( + input.stackName, fetched.composeFiles, input.contextDir, input.syncEnv, fetched.envContent, null, + ); + } const envWritten = input.syncEnv && fetched.envContent !== null; // 4. Insert the git-source row, then mark it applied so future @@ -1438,7 +2161,18 @@ export class GitSourceService { }); db.markGitSourceApplied(input.stackName, fetched.commitSha, hash); db.setGitSourceAppliedSpec(input.stackName, appliedSpec); + if (completeProjectManifest) { + // The promotion's cache write predated the row; persist the + // cache now so list and response projections are truthful. + db.setGitSourceManifestState( + input.stackName, + completeProjectManifest.manifestVersion, + completeProjectManifest.state, + completeProjectManifest.generation.appliedDir, + ); + } + rowInserted = true; const source = this.get(input.stackName); if (!source) { throw new GitSourceError('GIT_ERROR', 'Failed to read back created git source.'); @@ -1460,12 +2194,136 @@ export class GitSourceService { console.error(`[GitSource] Rollback: failed to remove partial stack dir ${input.stackName}:`, cleanupErr); } } - db.deleteGitSource(input.stackName); + // R6: the managed area must not outlive a create THIS invocation + // staged. A pre-existing stack (TOCTOU race or a create failure + // for a non-existence reason) must never lose its previous + // applied generations to someone else's rollback: when the stack + // dir was NOT created by us, remove only the candidate we staged. + if (stackCreated) { + await GitProjectManifestService.getInstance().deleteManagedArea(input.stackName); + } else if (materialization.value) { + const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data'); + const stagedCandidate = path.join( + dataDir, + 'git-managed', + String(NodeRegistry.getInstance().getDefaultNodeId()), + input.stackName, + materialization.value.candidateRelPath, + ); + await fsPromises.rm(stagedCandidate, { recursive: true, force: true }); + } + if (rowInserted) { + db.deleteGitSource(input.stackName); + } throw e; } }); } + /** + * Boot sweep for every managed-project area, under the per-stack lock: + * crash-recovery restore, orphan candidates, and areas whose stack no + * longer exists (the row is gone, or the directory is gone). A stack + * whose directory exists but has no discoverable compose file is left in + * place (the managed area lingers until the row is removed), a deliberate + * fail-safe trade: never delete on uncertainty. + * + * The stack listing is read STRICTLY: a listing failure (EIO, EACCES, + * ENOMEM on the compose base dir) must never look like every stack + * disappeared, or the sweep would delete the manifest and every retained + * recovery generation of live Git-managed stacks. A failed listing aborts + * the whole sweep (orphan cleanup is deferred to the next boot), and each + * candidate is verified to be genuinely gone before its area is deleted + * (this also covers the per-stack read errors the listing's compose-file + * probe swallows). + */ + /** + * Whether the stack directory still exists. Read errors other than + * ENOENT are logged and treated as "exists": the sweep must never delete + * a managed area it could not verify was gone. + */ + private async stackDirExists(stackRoot: string, stackName: string): Promise { + try { + await fsPromises.lstat(stackRoot); + return true; + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== 'ENOENT') { + console.warn(`[GitSource] cannot verify stack ${stackName} is gone; skipping managed cleanup:`, (e as Error).message); + return true; + } + return false; + } + } + + public async sweepOrphans(): Promise { + const fsSvc = FileSystemService.getInstance(); + const manifestSvc = GitProjectManifestService.getInstance(); + const rows = DatabaseService.getInstance().getGitSources(); + const composeDir = NodeRegistry.getInstance().getComposeDir(NodeRegistry.getInstance().getDefaultNodeId()); + const stackRootBase = path.resolve(composeDir); + let stacks: Set; + try { + stacks = new Set(await fsSvc.getStacksStrict()); + } catch (e) { + console.error('[GitSource] stack listing failed; aborting the orphan sweep to protect managed areas:', e instanceof Error ? e.stack ?? e.message : String(e)); + return; + } + for (const row of rows) { + // One failing stack must never abort recovery for the rest. + try { + if (!stacks.has(row.stack_name)) { + const stackRoot = path.resolve(composeDir, row.stack_name); + if (!isPathWithinBase(stackRoot, stackRootBase)) { + console.warn(`[GitSource] stack ${row.stack_name} escapes the compose directory; skipping managed cleanup`); + continue; + } + // The probe and the delete run under the per-stack lock so + // a concurrent create for the same row cannot stage a + // candidate into the area this branch is about to reap. + await this.withStackLock(row.stack_name, async () => { + if (await this.stackDirExists(stackRoot, row.stack_name)) { + console.warn(`[GitSource] stack ${row.stack_name} is missing from the listing but its directory exists; skipping managed cleanup`); + return; + } + console.log(`[GitSource] removing managed area for vanished stack ${row.stack_name}: not listed and the stack directory is gone`); + const liveRemoved = await manifestSvc.deleteManagedArea(row.stack_name); + const stagedRemoved = await manifestSvc.finalizeStagedDetach(row.stack_name); + if (!liveRemoved || !stagedRemoved) throw new Error('Could not remove orphaned managed project data'); + }); + continue; + } + await this.withStackLock(row.stack_name, () => + manifestSvc.sweepManagedArea(row.stack_name, { repoUrl: row.repo_url, branch: row.branch, stackExists: true }), + ); + } catch (e) { + console.error(`[GitManifest] sweep failed for ${row.stack_name}:`, (e as Error).message); + } + } + // Areas whose stack row is gone entirely (source deleted, stack kept) + // must not linger either. + const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data'); + const managedRoot = path.join(dataDir, 'git-managed', String(NodeRegistry.getInstance().getDefaultNodeId())); + const known = new Set(rows.map((r) => r.stack_name)); + let entries; + try { + entries = await fsPromises.readdir(managedRoot, { withFileTypes: true }); + } catch (e) { + if ((e as NodeJS.ErrnoException).code !== 'ENOENT') { + console.error('[GitManifest] could not read the managed project root:', (e as Error).message); + } + return; + } + for (const entry of entries) { + if (!entry.isDirectory() || known.has(entry.name)) continue; + if (entry.name.startsWith('.detach-') && known.has(entry.name.slice('.detach-'.length))) continue; + try { + await fsPromises.rm(path.join(managedRoot, entry.name), { recursive: true, force: true }); + } catch (e) { + console.error(`[GitManifest] could not remove orphaned area ${sanitizeForLog(entry.name)}:`, (e as Error).message); + } + } + } + // ─── Webhook-triggered pull ────────────────────────────────────────────── /** diff --git a/backend/src/services/UpdateGuardService.ts b/backend/src/services/UpdateGuardService.ts index 14d6db53..40f0d002 100644 --- a/backend/src/services/UpdateGuardService.ts +++ b/backend/src/services/UpdateGuardService.ts @@ -257,7 +257,17 @@ export class UpdateGuardService { containers, }, now); - return { stack: stackName, computedAt: now, overall: aggregateRollbackOverall(items), items }; + // Partial-revert disclosure for Git-managed stacks: rollback restores only + // compose files and .env; the rest of the materialized project is not + // reverted by the backup slot. State the scope rather than imply a + // complete revert. + let note: string | undefined; + const gitSource = db.getGitSource(stackName); + if (gitSource && (gitSource.manifest_state === 'active' || gitSource.manifest_state === 'partial' || gitSource.manifest_state === 'migrated')) { + note = 'This stack is Git-managed. Rollback restores compose files and .env; other materialized inputs are not reverted. Re-apply the previous revision from Git to restore them.'; + } + + return { stack: stackName, computedAt: now, overall: aggregateRollbackOverall(items), items, note }; } /** Host disk use percent for the main filesystem, or null when unavailable. */ diff --git a/backend/src/services/updateGuard/types.ts b/backend/src/services/updateGuard/types.ts index 7b756f82..7ad60b43 100644 --- a/backend/src/services/updateGuard/types.ts +++ b/backend/src/services/updateGuard/types.ts @@ -52,6 +52,12 @@ export interface RollbackReadinessReport { computedAt: number; overall: RollbackOverall; items: RollbackReadinessItem[]; + /** + * Scope disclosure for stacks whose project is partly Git-managed: rollback + * restores compose files and .env only; other materialized inputs are not + * reverted. Set when the stack has a managed-project manifest. + */ + note?: string; } /** Normalized per-container probe used by readiness and the health gate. */ diff --git a/backend/src/types/gitProjectManifest.ts b/backend/src/types/gitProjectManifest.ts new file mode 100644 index 00000000..99cfa7a8 --- /dev/null +++ b/backend/src/types/gitProjectManifest.ts @@ -0,0 +1,260 @@ +/** + * Canonical types for the Git managed-project materialization contract (the + * single repository file inventory consumed by every GitOps feature). + * + * The manifest FILE is the source of truth; `stack_git_sources` cache columns + * are cheap projections for GET/dashboard reads. The manifest is treated as + * untrusted input on every read: shape, enum membership, and the identity + * stamp are validated before any field is honored. + */ + +/** File-side manifest state: what the manifest file itself can express. */ +export type ManifestState = 'none' | 'migrated' | 'active' | 'partial' | 'unsupported'; + +/** + * DB-column manifest state, strictly wider than the file-side state. + * `migration_required` means a manifest was expected but could not be trusted + * (corrupt shape, identity mismatch, declined crash-recovery restore); + * `absent` means no manifest file exists yet. Neither can be expressed by a + * manifest file, so they live only in the DB column and the GET projection. + */ +export type GitSourceManifestState = ManifestState | 'migration_required' | 'absent'; + +export type InputOwnership = 'managed' | 'unmanaged'; +export type ManifestProvenance = 'fetch' | 'migration' | 'adopted'; +export type InputSensitivity = 'high' | 'medium' | 'low'; +export type InputState = 'present' | 'tombstoned'; +export type DeletionAuthority = 'sencho' | 'user' | 'none'; + +export type InputRole = + | 'compose-primary' + | 'compose-additional' + | 'compose-override' + | 'env' + | 'config' + | 'secret' + | 'label-file' + | 'build-context' + | 'dockerfile' + | 'build-secret' + | 'build-additional-context' + | 'bind-mount' + | 'other'; + +export type InputDependencyKind = + | 'explicit' + | 'implicit-override' + | 'include' + | 'include-env' + | 'extends' + | 'env_file' + | 'interpolation-env' + | 'config' + | 'secret' + | 'label_file' + | 'build-context' + | 'dockerfile' + | 'build-secret' + | 'build-additional-context' + | 'sync-env' + | 'bind-mount'; + +/** One materialized or referenced input. Exactly one authoritative entry per input. */ +export interface ComposeInputEntry { + /** Repo-relative source path; null for host/unmanaged references. */ + sourcePath: string | null; + /** Stack-relative path after materialization; null when not copied. */ + materializedPath: string | null; + role: InputRole; + dependencyKind: InputDependencyKind; + ownership: InputOwnership; + provenance: ManifestProvenance; + sensitivity: InputSensitivity; + /** Null for unmanaged/refused entries. */ + contentSha256: string | null; + sizeBytes: number | null; + state: InputState; + /** Who may delete this path during stale cleanup (manifest-scoped authority). */ + deletionAuthority: DeletionAuthority; + /** Refusal reason / documented limitation. */ + note: string | null; +} + +export interface RefusalInfo { + sourcePath: string | null; + kind: string; + reason: string; + actionable: boolean; + /** + * Sensitivity of the declared input the refusal is about; public + * projections redact high-sensitivity refusals. Absent on refusals from + * older manifests (treated as not sensitive). + */ + sensitivity?: InputSensitivity; +} + +export interface BuildContextPlan { + /** Context root in the materialized (stack-relative) layout. */ + repoPath: string; + /** Repo-relative dockerfile path within the context, if declared. */ + dockerfile: string | null; + /** Materialized context size after dockerignore filtering. */ + contextBytes: number; + ignoredCount: number; + dockerignoreApplied: boolean; + /** True when the context is not copied (refused). */ + excludedFromCopy: boolean; + note: string | null; + /** + * File-level inventory of the materialized context (context-relative paths + * with content hashes). Gives the context file-granular ownership: local + * edits and files removed upstream are detected per file, so a removed file + * can be cleared on promotion and a locally edited one refuses apply. + */ + files: Array<{ path: string; sha256: string; sizeBytes: number }>; +} + +export interface ManifestBounds { + maxFiles: number; + maxBytes: number; + maxContextBytes: number; + maxPathDepth: number; + maxFileBytes: number; +} + +/** Stamp that binds a manifest to exactly one stack on one node. */ +export interface ManifestIdentity { + nodeId: string; + stackName: string; + repoUrl: string; + branch: string; +} + +export interface GitProjectManifest { + schemaVersion: 1; + /** Incremented on every successful write. */ + manifestVersion: number; + state: ManifestState; + generatedAt: number; + identity: ManifestIdentity; + repo: { url: string; branch: string }; + resolvedRevision: { commitSha: string; fetchedAt: number }; + project: { + /** Repo-relative project root (today's context_dir); null = repo root. */ + root: string | null; + /** Ordered explicit repo-relative compose file set. */ + composeFiles: string[]; + /** Stack-relative, passed as --project-directory. */ + effectiveProjectDir: string | null; + /** Pinned via -p. */ + projectName: string; + /** Ordered compose invocation args (relative paths). */ + invocation: string[]; + }; + inputs: ComposeInputEntry[]; + refusals: RefusalInfo[]; + buildContexts: BuildContextPlan[]; + generation: { candidateDir: string; appliedDir: string; previousDir: string | null }; + counts: { managed: number; unmanaged: number; refused: number }; + bounds: ManifestBounds; +} + +/** + * Public projection of one manifest input, served by the manifest read + * endpoint. Hashes, size metadata, and provenance are internal-only; the + * display path is redacted (null) for high-sensitivity inputs so secret file + * names never cross the API. + */ +export interface PublicManifestInput { + /** Display label; null when redacted (high-sensitivity input) or when the input has no path. */ + path: string | null; + role: InputRole; + dependencyKind: InputDependencyKind; + ownership: InputOwnership; + sensitivity: InputSensitivity; + state: InputState; + note: string | null; +} + +/** Public projection of the internal manifest (no hashes, no sensitive paths). */ +export interface PublicManifest { + manifestVersion: number; + state: ManifestState; + resolvedCommitSha: string | null; + projectRoot: string | null; + composeFiles: string[]; + projectName: string; + inputs: PublicManifestInput[]; + counts: { managed: number; unmanaged: number; refused: number }; +} + +/** Projection served by GET /git-source and the manifest read endpoint. */ +export interface ManifestSummary { + state: GitSourceManifestState; + manifestVersion: number; + resolvedCommitSha: string | null; + managedCount: number; + unmanagedCount: number; + refusedCount: number; + /** Actionable refusals surfaced to the UI. */ + refused: RefusalInfo[]; + hasBuildContexts: boolean; + generatedAt: number | null; +} + +// --- Discovery result types (declared here so they land once) --- + +/** A declared input found by the pure parser (no I/O yet). */ +export interface DeclaredInput { + /** Repo-relative or host path; null for non-path forms. */ + sourcePath: string | null; + /** + * Stack-relative path the input occupies at runtime; null for host paths, + * non-path forms, and the dockerfile declaration (which is resolved + * relative to its build context in planBuildContexts). Diverges from + * sourcePath when the runtime layout relocates a file: the primary compose + * file lands at the stack root, so its include/extends graph AND every + * project-relative declaration in the primary or in merged (-f) files + * shifts by the primary's repository directory prefix. + */ + materializedPath: string | null; + baseDir: 'repo-root' | 'project-root' | 'compose-file-dir' | 'host'; + kind: InputDependencyKind; + role: InputRole; + /** + * When false (env_file map form with required: false), a missing file is + * recorded as an unmanaged entry (never a refusal). + */ + required: boolean; + /** Compose source context for diagnostics. */ + fromFile: string | null; + /** + * Service name that declared this input (build declarations, env_file, + * configs references). Pairs a build context with its own dockerfile, + * build secrets, and additional contexts when a file declares several + * services. Null for top-level declarations. + */ + service: string | null; +} + +/** Path with ${VAR} interpolation that Compose resolves at deploy time. */ +export interface DynamicInput { + sourcePath: string; + kind: InputDependencyKind; + note: string; +} + +export interface ParsedDeclaredInputs { + inputs: DeclaredInput[]; + dynamic: DynamicInput[]; + parseErrors: string[]; +} + +/** Classification of every declared input against the cloned tree. */ +export interface InventoryResult { + inputs: ComposeInputEntry[]; + refusals: RefusalInfo[]; + buildContexts: BuildContextPlan[]; + dynamic: DynamicInput[]; + counts: { managed: number; unmanaged: number; refused: number }; +} diff --git a/backend/src/utils/dockerIgnoreMatch.ts b/backend/src/utils/dockerIgnoreMatch.ts new file mode 100644 index 00000000..b4d42ee4 --- /dev/null +++ b/backend/src/utils/dockerIgnoreMatch.ts @@ -0,0 +1,185 @@ +/** + * Minimal Docker .dockerignore matcher, vendored to avoid a dependency for the + * git managed-project materializer. Implements the semantics docker's + * patternmatcher uses (docs.docker.com "Context" / moby/patternmatcher): + * + * - Patterns are matched against the path relative to the context root. + * - A pattern with no slash matches the basename at any depth (`*.md` matches + * `README.md` and `sub/README.md`); a pattern with a slash (with or without a + * leading `/`) is anchored to the root. + * - `**` crosses directories; `*`, `?` and `[...]` apply within one segment. + * - A trailing `/` restricts the pattern to directories. + * - `!` negates; the LAST matching pattern wins. + * - `#` starts a comment; `\#` is a literal `#`. Empty lines are ignored. + * - Dotfiles are matched by default (unlike gitignore). + * + * Callers must prune directory subtrees: when a directory path matches, every + * file beneath it is ignored as well. + */ + +export interface DockerIgnoreMatcher { + /** True when `relPath` (context-root-relative, posix) is ignored. */ + matches(relPath: string, isDir?: boolean): boolean; +} + +function escapeRegExpSegment(input: string): string { + return input.replace(/[.+^${}()|[\]\\]/g, '\\$&'); +} + +/** + * Convert one dockerignore segment (`**`, `*`, `?`, `[...]`, literal text) into + * a regex source that does not cross `/`. + */ +function segmentToRegex(segment: string): string { + let out = ''; + let i = 0; + while (i < segment.length) { + const ch = segment[i]; + if (ch === '*') { + if (segment[i + 1] === '*') { + // `**` inside a segment is docker's "match across directories"; + // when it is a full segment the caller emits `.*` instead. Here + // it can only appear as a partial-segment oddity; treat as `.*`. + out += '.*'; + i += 2; + continue; + } + out += '[^/]*'; + i += 1; + continue; + } + if (ch === '?') { + out += '[^/]'; + i += 1; + continue; + } + if (ch === '[') { + const close = segment.indexOf(']', i + 1); + if (close === -1) { + out += '\\['; + i += 1; + continue; + } + const inner = segment.slice(i + 1, close); + // Preserve negation and ranges; escape backslashes; keep the class + // as-is otherwise (docker passes character classes through). + out += '[' + inner.replace(/\\/g, '\\\\') + ']'; + i = close + 1; + continue; + } + if (ch === '\\' && i + 1 < segment.length) { + // Escaped char: `\#` yields a literal `#`; any other escape is + // passed through as the literal character. + out += escapeRegExpSegment(segment[i + 1]); + i += 2; + continue; + } + out += escapeRegExpSegment(ch); + i += 1; + } + return out; +} + +interface CompiledPattern { + regex: RegExp; + negate: boolean; + dirOnly: boolean; + basenameOnly: boolean; +} + +function compilePattern(rawLine: string): CompiledPattern | null { + let line = rawLine.trim(); + if (!line) return null; + + // Escaped comment marker: `\#` starts a literal pattern; an unescaped `#` + // at the start is a comment. + if (line.startsWith('#')) return null; + if (line.startsWith('\\#')) { + line = line.slice(1); + } + + let negate = false; + if (line.startsWith('!')) { + negate = true; + line = line.slice(1).trim(); + if (!line) return null; // bare `!` is a no-op in docker + } + + let dirOnly = false; + if (line.endsWith('/')) { + dirOnly = true; + line = line.slice(0, -1); + } + if (!line) return null; + + // A leading `/` anchors to the root; strip it (root-anchoring is the + // default for slash-bearing patterns). + const anchored = line.startsWith('/'); + if (anchored) line = line.slice(1); + + const segments = line.split('/'); + const basenameOnly = !anchored && segments.length === 1; + + let source = ''; + if (basenameOnly) { + // Match the basename at any depth. + source = '(?:^|.*/)' + segmentToRegex(segments[0]) + '$'; + } else { + // A `**` segment compiles to `(?:.*/)?` (zero or more directories, + // consuming its own trailing slash) or `.*` when trailing. Separators + // are inserted between consecutive parts only when the previous part + // was NOT compiled from `**` (those parts already carry their separator). + const parts: string[] = []; + for (let i = 0; i < segments.length; i++) { + if (segments[i] === '**') { + parts.push(i === segments.length - 1 ? '.*' : '(?:.*/)?'); + } else { + parts.push(segmentToRegex(segments[i])); + } + } + source = parts[0]; + for (let i = 1; i < parts.length; i++) { + if (segments[i - 1] !== '**') source += '/'; + source += parts[i]; + } + } + + return { regex: new RegExp('^' + source + '$'), negate, dirOnly, basenameOnly }; +} + +export function compileDockerIgnore(lines: string[]): DockerIgnoreMatcher { + const patterns = lines.map(compilePattern).filter((p): p is CompiledPattern => p !== null); + return { + matches(relPath: string, isDir: boolean): boolean { + const normalized = relPath.replace(/\\/g, '/').replace(/^\/+/, ''); + if (!normalized) return false; + const base = normalized.slice(normalized.lastIndexOf('/') + 1); + let ignored = false; + for (const p of patterns) { + if (p.dirOnly && !isDir) continue; + const target = p.basenameOnly ? base : normalized; + if (p.regex.test(target)) { + ignored = !p.negate; + } + } + return ignored; + }, + }; +} + +/** + * Load a Docker ignore file from a directory. Defaults to `.dockerignore`; + * pass an explicit filename for the Dockerfile-specific + * `.dockerignore` precedence form. Returns null when no file + * exists or it cannot be read (nothing ignored). + */ +export async function loadDockerIgnore(rootDir: string, filename = '.dockerignore'): Promise { + const fs = await import('fs'); + const path = await import('path'); + try { + const raw = await fs.promises.readFile(path.join(rootDir, filename), 'utf8'); + return compileDockerIgnore(raw.split(/\r?\n/)); + } catch { + return null; + } +} diff --git a/docs/features/git-sources.mdx b/docs/features/git-sources.mdx index 693e9f5b..12efa58c 100644 --- a/docs/features/git-sources.mdx +++ b/docs/features/git-sources.mdx @@ -29,7 +29,7 @@ The panel groups four regions: - **Pending update banner.** Appears at the top when a webhook in **Review only** mode has fetched a new commit. Click **Review** to re-fetch the incoming commit and open the diff dialog. - **Form fields.** Repository URL, branch, the ordered compose-file picker, an optional project directory, optional sibling `.env` sync, authentication toggle, and the apply behavior radio group. - **Last applied stat strip.** Shows the short SHA of the last commit Sencho applied to disk, plus the timestamp of the most recent successful save or pull. -- **Footer actions.** **Remove** disconnects the source without touching the stack files; **Pull now** fetches the configured branch's HEAD; **Save** or **Update** persists form changes after a reachability check passes. +- **Footer actions.** **Remove** disconnects the source by exporting the effective compose model into a single `compose.yaml` and removing auto-discovered override files; the remaining materialized files are kept. **Pull now** fetches the configured branch's HEAD; **Save** or **Update** persists form changes after a reachability check passes. ## Create a stack from a Git repository @@ -221,7 +221,7 @@ Pulls, applies, and create-from-git operations on the same stack are serialized - Repositories that use Git submodules do not have their submodule contents cloned during a Git Source fetch. Sencho surfaces a warning on create when `.gitmodules` is present. If the compose file references paths inside a submodule (build contexts, volume mounts, include directives), inline the referenced files into the main repository or flatten the submodule so the paths resolve at deploy time. + Repositories that use Git submodules do not have their submodule contents cloned during a Git Source fetch. Sencho surfaces a warning when `.gitmodules` is present, and refuses inputs and build contexts that reference submodule contents with an actionable message. Inline the referenced files into the main repository or flatten the submodule so the paths resolve at deploy time. @@ -229,7 +229,7 @@ Pulls, applies, and create-from-git operations on the same stack are serialized - A Git Source materializes only the compose and `.env` files you select, not the rest of the repository. Build contexts, bind-mount sources, and cross-file `extends:` targets referenced by relative path are not pulled, so they can be missing at deploy time. Set a **Project directory** to fix the base for relative paths, and for build-based stacks keep the referenced files in the repository alongside the compose files or build the image out of band. + Git Sources materializes the complete project, including recursive `include:` and `extends.file` dependencies, service env files, file-backed configs and secrets, and build contexts. If a referenced file is still missing, the pull refused it and the refusal message names the path and the reason (out-of-bound path, Git LFS pointer, submodule, symlink, or a size cap). Fix the declaration in the repository and pull again. @@ -241,8 +241,12 @@ Pulls, applies, and create-from-git operations on the same stack are serialized - **HTTPS only.** SSH URLs and SSH keys are not supported. Use a Personal Access Token for private repos. - **No Git LFS.** Compose and env files stored via LFS are rejected. Commit plain files instead. -- **No submodules.** Submodule contents are not fetched; paths inside a submodule directory will be missing at deploy time. A warning is shown on create when `.gitmodules` is present. -- **Branch-tracking only.** Sources follow the head of a branch. Specific commit SHAs and tags are not pinnable. +- **No submodules.** Submodule contents are not fetched. Inputs and build contexts that reference submodule contents are refused with an actionable message; a warning is shown when `.gitmodules` is present. +- **Branch-tracking only.** Sources follow the head of a branch. Specific commit SHAs and tags are not pinnable. Each pull resolves and pins the exact commit SHA, so apply always materializes the reviewed revision. - **Clone size cap.** A clone is bounded on how much it downloads (and each compose/env file is capped on read), so very large repositories are rejected. Operators can adjust the download ceiling with `GITSOURCE_MAX_CLONE_BYTES`. -- **Referenced files are not materialized.** Only the selected compose and `.env` files are written to disk. Relative build contexts, bind-mount sources, and `extends:` targets are not pulled, so build-based stacks that depend on other repository files need those files committed alongside the compose files (or the image built out of band). +- **Complete project materialization.** Every repository-local input the project needs is materialized: the ordered compose files, implicit `compose.override.*` files, recursive `include:` and `extends.file` dependencies, service env files, file-backed configs and secrets, label files, and build contexts with `.dockerignore` semantics. The materialized set is recorded in a versioned managed-project manifest, and each pull stages a candidate that is validated with the exact deployment invocation before anything on disk changes. If apply is interrupted, Sencho completes the accepted generation or restores the previous generation. If files were edited during the interruption and no longer match either generation, Sencho preserves them and requires manual recovery instead of overwriting them. +- **Unsupported inputs are refused, not guessed.** Inputs that cannot be safely reproduced fail the pull with an actionable message: URL includes, Git LFS pointers, submodule contents, symbolic links, build contexts that exceed the size bounds, and include or extends declarations that point outside the repository or use dynamic `\${VAR}` paths (their contents cannot be enumerated). Nothing is applied until the declaration is fixed. Absolute host paths, host bind mounts, external resources, and dynamic `\${VAR}` data paths are never claimed as covered: they resolve at deploy time from the environment or the node, and the manifest records them as unmanaged. +- **Materialization bounds.** The materialized project is bounded by file count, total bytes, per-file size, path depth, and build-context size, each adjustable with a `GITSOURCE_*` variable (see configuration). Crossing a bound refuses the pull with the counts so far rather than producing a partial project. +- **Detach and export.** Removing a Git source renders the effective compose model into a single `compose.yaml`, keeps the remaining materialized files, removes auto-discovered override files so the exported model is final, and removes Git tracking. Resolved environment values are baked into the exported file. If removal is interrupted before it completes, Sencho restores the original files automatically. +- **Rollback scope.** Rollback of a Git-managed stack restores compose files and `.env`. Other materialized inputs are not reverted by rollback; re-apply the previous revision from Git to restore them. - **Some read-only views read the primary file.** The dependency graph, drift snapshot, and networking inspector summarize the primary compose file, so a service declared only in an override may not appear in those views. Deploy, update, image-update checks, and mesh attachment use the full merged set. diff --git a/docs/getting-started/configuration.mdx b/docs/getting-started/configuration.mdx index 451ba29f..ebed5c79 100644 --- a/docs/getting-started/configuration.mdx +++ b/docs/getting-started/configuration.mdx @@ -49,6 +49,11 @@ These tune optional subsystems. Most deployments never set them; the defaults ar | `SENCHO_MESH_RECONCILE_INTERVAL_MS` | `60000` | How often the central instance re-checks proxy-mode mesh tunnels to detect a peer that rebooted. Lower it for faster peer-reboot detection at the cost of more frequent checks. See [Sencho Mesh](/features/sencho-mesh). | | `SENCHO_MESH_PROXY_TUNNEL_IDLE_MS` | `0` | Idle timeout before a proxy-mode mesh tunnel tears down and reopens on demand. `0` keeps the tunnel open for the life of the connection. See [Sencho Mesh](/features/sencho-mesh). | | `GITSOURCE_MAX_CLONE_BYTES` | `104857600` | Maximum bytes a single [Git Source](/features/git-sources) clone may download before it is aborted (100 MB). A shallow Compose clone is tiny; raise it only if you track Compose files in a legitimately large repository. | +| `GITSOURCE_MAX_MATERIALIZED_FILES` | `10000` | Maximum number of files the complete-project materializer may write for one Git-managed stack. Crossing it refuses the pull with the count so far. | +| `GITSOURCE_MAX_MATERIALIZED_BYTES` | `536870912` | Maximum total bytes the complete-project materializer may write for one Git-managed stack (512 MB). Crossing it refuses the pull with the count so far. | +| `GITSOURCE_MAX_BUILD_CONTEXT_BYTES` | `268435456` | Maximum size of a single materialized build context after `.dockerignore` filtering (256 MB). A context that exceeds it, including a repository-root context, is refused. | +| `GITSOURCE_MAX_PATH_DEPTH` | `64` | Maximum directory depth for a materialized repository path. Deeper paths are refused. | +| `GITSOURCE_MAX_FILE_BYTES` | `10485760` | Maximum size of a single materialized file (10 MB). Oversized files are refused. | | `SENCHO_PUBLIC_URL` | *(request host)* | Set on the primary instance. Its externally reachable `http(s)://` URL, no trailing slash, baked into pilot enrollment so remote agents dial the public hostname rather than the address the admin used at setup. | | `SENCHO_COMPOSE_COMMAND_TIMEOUT_MS` | `1800000` | Hard timeout for a single Compose command (pull, up, down) during deploy and update, in milliseconds (30 minutes). Sencho kills the command and reports failure if it runs longer than this, regardless of whether it is still producing output. Raise it only for very large images or slow storage. | | `SENCHO_COMPOSE_STALL_TIMEOUT_MS` | `600000` | Idle-output backstop for deploy and update Compose steps (pull and recreate), separate from the hard timeout above. If a step produces no output for this long while still running, Sencho stops it so a hung image pull surfaces a clear failure and the in-app recovery actions instead of spinning. Raise it on slow links or for heavy local image builds. | diff --git a/e2e/fixtures/git-ca.key b/e2e/fixtures/git-ca.key new file mode 100644 index 00000000..9a0b36ae --- /dev/null +++ b/e2e/fixtures/git-ca.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCjHmvwxoL/JYqr ++aKvQ7tBF9ZCtmSKlgyf3+CxakbcVsfLJEFJsnVpDrYb8QPlEqNYuLpjBo1CJT9i +V3Zmzs7ZFOdtixuO8TFz3+ggNfWENVnCRmT4QU/k9V0ZCkrebf4NMQuYLrmt9nLm +0E+UVFgf+a9yKE/r50YSjoNnNVxP78ES3E1tuJbDD1VPJCTdNSbBkdRs+/wQdrg3 +pFd2LvS60Enhp0TTLzfMUErIMD1kXnetA8kxaeBugAwdpCtbeVNOuY4QdyUkVlmK +Rrl35vCIwD33ezYyXTpO1xIB40kQiN3VJTJxgyIIcvoF5vrrN+c4xlL3hUrllbWp +rVd/rgMbAgMBAAECggEAStbX1lH140NhstKnofsU7GIX42bUjUMXyrhIWo30sf48 +z4a0T7BJryhZREuZql2ZRUkH9wwX/muhf6i6QaQMAkxVfAxxWe1ub4gg87peCUkD +BCARDlfE5Lrwel8fB5t2jq3ccHerqFWk1SJpCJiEEDOaG+nD9WuYWkY5or44OtH8 +qBWtH89xQQQaYusGxSwswlZEl78krOU/vZhoQWcAz2NjuSPFhPzX0Uv4316SpVwW +LmQnwQYFjSVxChQUk3ic3WqNp23R0BjgiHYL+FMrp/5MuiBZWRqNM6tEr4zm2s2v +qTKTgfF/+Q654F+gJpsx+qC1uQAu5Nw5/Qu7gv4tdQKBgQDg1o16CpvrSitbAtwF +yyVU0FF0U3r7c0/t/Ll0Wnpb3Z1cUWpFgybDzGr2im9Uq4eqGhZ59a7nbVF1L6U+ +FAhoS2u0lQioNaCBbDRPkVlA7jBofV+V2ekD64rgsIyITB4UWNVBL9odu5RRpLJS +o/HGEQgZeFT6nScb+DIuc7OHPQKBgQC5ugRYK4HXAQ3wdKsJ1udKT8xLFlODaZim +PBdX/RMUB/4I0x1tK2Cr5o9tPC/9XVI9yYhAqzV8qXr93eHa2JattEDYDmT2kjfQ +8iXoJUy2W67QZ0DVEF8272Xz4g2TrzeaEYZV/zjUdufJAiO5Fk2nspLBAaTGqxKu +cHeVgfMZNwKBgQC2sD7GkkY8ucherAUhQ/5yWs5Eqew80mZ45qe+DiJr8LeDIrgq +ATYHVFr5NmTdtH6ITahDsshKTT4p7OvkdBycueOrGImvO9vOLaCXom+WXbPBw1Ve +inBWehYfGfUmbkrml4O38uzUyezrJdqrYYD7Qi2FnIvYEseLZ3FG40ZVtQKBgG2g +m0fgBnf+q1evxfW84DFBAPmhaBI9llkpRy4st8IvJYfX9Zqm7B0LrsVvrXQETbMi +7kYdySiYcXzAJ7yh8+78YvolJPtWO5QeGrn4qltJqtpg2CfrzggDL07Rs+nklxFe +HslMKSNgPFit5qAtxhCim3VJnxWVjSViRXP+jJvpAoGBAKQhoyhGCGfolHSddy7N +efUDu+AJFLhGOZWuNvi4ExWF2hFXJAhH8LMuZ0/AY8E+ZJAKmkIPaRBYAj+Zvcub +XrkH1Z6BwgFKUk/dOaoePItQYkdCFFAFslRj7l062bTUs2dKInCWvjgoa69SPCn7 +U4lpMm2XZomIRSXjAkpJdJXG +-----END PRIVATE KEY----- diff --git a/e2e/fixtures/git-ca.pem b/e2e/fixtures/git-ca.pem new file mode 100644 index 00000000..cf13984c --- /dev/null +++ b/e2e/fixtures/git-ca.pem @@ -0,0 +1,19 @@ +-----BEGIN CERTIFICATE----- +MIIDGzCCAgOgAwIBAgIUAl9Xewt3pNuCTvCaW6XFJjfI/I4wDQYJKoZIhvcNAQEL +BQAwHTEbMBkGA1UEAwwSU2VuY2hvIEUyRSBUZXN0IENBMB4XDTI2MDgwNjE4MzMx +MVoXDTM2MDgwMzE4MzMxMVowHTEbMBkGA1UEAwwSU2VuY2hvIEUyRSBUZXN0IENB +MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAox5r8MaC/yWKq/mir0O7 +QRfWQrZkipYMn9/gsWpG3FbHyyRBSbJ1aQ62G/ED5RKjWLi6YwaNQiU/Yld2Zs7O +2RTnbYsbjvExc9/oIDX1hDVZwkZk+EFP5PVdGQpK3m3+DTELmC65rfZy5tBPlFRY +H/mvcihP6+dGEo6DZzVcT+/BEtxNbbiWww9VTyQk3TUmwZHUbPv8EHa4N6RXdi70 +utBJ4adE0y83zFBKyDA9ZF53rQPJMWngboAMHaQrW3lTTrmOEHclJFZZika5d+bw +iMA993s2Ml06TtcSAeNJEIjd1SUycYMiCHL6Beb66zfnOMZS94VK5ZW1qa1Xf64D +GwIDAQABo1MwUTAdBgNVHQ4EFgQUemat8qGDr9A4UViNzQ3Zog+4IKwwHwYDVR0j +BBgwFoAUemat8qGDr9A4UViNzQ3Zog+4IKwwDwYDVR0TAQH/BAUwAwEB/zANBgkq +hkiG9w0BAQsFAAOCAQEAaiDHpjNen0WNVJ+OzPNEBT32KsylyZikyEONULg3753T +YKThj6cAmjtuZvllD3xW7A6pd31x0pARMdsowAFiQOl+dyPB64QViIm9MB+1Fqb9 +vGGpRTM4e4vaJItwAkm7nkbJgBhSVhm1NhHvXSOrqPf9oqyRS3Cxfta+KNoTJcwa +eHC1+w5i5BVQXy4ZaD7raOPICwcFKHz6bOBly25DHgIzQQQrS57LgM9hMAJGJvuD +4tAZjUIcXaEH1UQq6xtooBPpmBbUumvw1Xa2vV7md5ZN5YJt/8Kx+2t3JM77lFwd +AxuuitUnZwEQlCgp567Z6XAuT4r3Q1dNW5oH88G/rw== +-----END CERTIFICATE----- diff --git a/e2e/fixtures/git-server.key b/e2e/fixtures/git-server.key new file mode 100644 index 00000000..2d58c6ff --- /dev/null +++ b/e2e/fixtures/git-server.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCUoeS5sZx32C3e +RDTMtOa3BATYBVjyN/QEQ8JIp7wuHSlqMlbOgwwXNOYzWgjzmFV3wGADt7b3gt3c +7UrM+NKDgzqMXnLtLiGbXowsUch/VuZNOcfdiqD4x18obpS6T/W9nTwu4NIzYLPe +440PUfNNYw65W+YOvgzJVNtJdjSzQQAP8KAfVT9A4uXYaFUpd5HN7cda5Nb40ycI +mmlN0oraBLzoIcNuR/GczDvTPZqIDcIUoMfwkCtmK95kwGPfJpiBPui0bVMo0JbJ +u2xi2nHYldOJyCi+qMMF+ArDGIoftrEUbGcZvC6bXu8yPP4DEcGP8P7C3gkZ/8ct +MwuYVv35AgMBAAECggEABx2XByLPJFennn/ybSht8S4Sk0ryPqpaFsgtqW/KTQTd +YjWlvT492nCQYr35NrxAvzpo/lSRwFi2CWkczyJMZbpnF6g/5UXzmu4UVNzXde0x +mlndTShhW3ekARoGKcNBNwIriuz9czM6eT24l5arSLWo00vogCUuFIdo9iPgEwLD +cYp6bfZM+3p/jmJpazeIf5OC1CuZvkmKiaXd1jVN4NPbO8d0+LN1tNywO0oecV4u +xJSrWz/bkLgep6p8iwKK+MLOr/9q7dPjrRe40Rq8rvyQEe+u49JQPUXTQXdaedtr +z7z0iD/s1ic2a1gFpmZfaXautI41aH0ZiRmUax63gQKBgQDGM+mnHQt5191DL3f7 +nHhd9jkCiHFVFWGBBFCTRcJuB4chSqZHnFAzbP2xjpgiTPpSBoVSXqeDwyNGbTqu +BSPvqEiK/RxmoKTHhRU+cP8fvJvZ4VTxFf8LIfuVuPtSXHTXhnaxJmtRYp7eVILH +1ub+ozu/LOo80CE+nye+TPIe4QKBgQC/+X0p2PumV1FZ+DWt/Vrv3b5IQS30syh5 +KmaEePodtYF6hJSs96trbK8o41E+/nEOKXyu5WcEhmENJx7ipZRYsqGio0a7EO/W +iA3FHHG3ECqE0OZcNl/fAfdv1aP5cuVZhWlY/6PWe9tk0kStLuYIvcix1pTd5Yyc +h2z3zxM6GQKBgEtbL2m3FEbl+JzzrkV+jxECbVh2MciskV6xGkV0D2EwAYN501CE +sMVsmePpGBRqef23tvbDQCNLjNzY6KeDEs+qhrI5W5P1XdDx20rbQVR6rDKhhl8H +AunjqLibFQqSVmYfHH9r4P+XZFmZfgOmxDpqK2wbEo++ffVGI6EptiBhAoGAdfnJ +rHT3OaNBkEvUGUewoeYgsOC1cELFpaij9dcuxiEsH/HoOF/ADbVt82+3F0JgkfZ2 +9DhwVbyLWfznoxtkjhnA8WTr67wd2DYmWDMBwyGBL7v9RT/5LOBVgnnFWl/8iEZP +lm0L75yQGGaL4+4FWevfsUKQm8kc33juQ2ATjckCgYB1QHG26lDAEGDQN6oKtus1 +Jsh7Q7jyxSARqxbW53EBCI5P5CRhJsqD6VRZ621qWAd03RnvDENwHg+77Wh+DUQJ +FNuaetYvVdWy/OR6vIJexlDkM6Eb++Dxd6zYoAjx+O0yulq4c1ApTdKvfvkFUfOV +BKTCjelpTBvwUXh0K+LQKg== +-----END PRIVATE KEY----- diff --git a/e2e/fixtures/git-server.pem b/e2e/fixtures/git-server.pem new file mode 100644 index 00000000..3a568ecf --- /dev/null +++ b/e2e/fixtures/git-server.pem @@ -0,0 +1,19 @@ +-----BEGIN CERTIFICATE----- +MIIDHTCCAgWgAwIBAgIUdDKj2ycU64rpDK1kYSR++P7ZEIEwDQYJKoZIhvcNAQEL +BQAwHTEbMBkGA1UEAwwSU2VuY2hvIEUyRSBUZXN0IENBMB4XDTI2MDgwNjE4MzMx +MVoXDTM2MDgwMzE4MzMxMVowFDESMBAGA1UEAwwJMTI3LjAuMC4xMIIBIjANBgkq +hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAlKHkubGcd9gt3kQ0zLTmtwQE2AVY8jf0 +BEPCSKe8Lh0pajJWzoMMFzTmM1oI85hVd8BgA7e294Ld3O1KzPjSg4M6jF5y7S4h +m16MLFHIf1bmTTnH3Yqg+MdfKG6Uuk/1vZ08LuDSM2Cz3uOND1HzTWMOuVvmDr4M +yVTbSXY0s0EAD/CgH1U/QOLl2GhVKXeRze3HWuTW+NMnCJppTdKK2gS86CHDbkfx +nMw70z2aiA3CFKDH8JArZiveZMBj3yaYgT7otG1TKNCWybtsYtpx2JXTicgovqjD +BfgKwxiKH7axFGxnGbwum17vMjz+AxHBj/D+wt4JGf/HLTMLmFb9+QIDAQABo14w +XDAaBgNVHREEEzARhwR/AAABgglsb2NhbGhvc3QwHQYDVR0OBBYEFOG8ctpn+N0n +Y/7EgbpfAdy9fGdwMB8GA1UdIwQYMBaAFHpmrfKhg6/QOFFYjc0N2aIPuCCsMA0G +CSqGSIb3DQEBCwUAA4IBAQA4by3KANgv5xMMCm8yuFR89dmYx3sPtm9QH/qN7IZS +aPO83LEB8oQUE8MMCEm3HzkMMAsvA27t8PLhy1Pu2m+i2JJxdXWFsYv3BVEqvtCn +1uLxM+2jmXTFyrplQZRnMhGQQsvsouLvvwjy+xnR+eSlygavSUmAxUJJvl43tFoJ +1L44J/Aqpk7xOjX7yFZenkBlPU3Q8LNMR/XIqhtHjtHNbPVuLmGR2vNASi7Hmf9n +rtwwG8FwFnEWJ4XEYQpuNIbSAX2Ob81AV2km3+4e3J+RwsWj+qNlthYFEH6uIrek +jdVobDhu9+gVozG/itWcLx9kM2SIBQxlcwCx+L8zVnKD +-----END CERTIFICATE----- diff --git a/e2e/git-sources.spec.ts b/e2e/git-sources.spec.ts index 22f0204a..2e733e87 100644 --- a/e2e/git-sources.spec.ts +++ b/e2e/git-sources.spec.ts @@ -8,6 +8,7 @@ */ import { test, expect, Page } from '@playwright/test'; import { loginAs } from './helpers'; +import { gitAvailable, buildFixtureRepo, serveRepos, fullProjectFiles, multiFileFiles, refusalFiles } from './gitServer.helper'; const TEST_STACK = 'e2e-git-source-stack'; @@ -192,10 +193,10 @@ test.describe('Git Sources', () => { // footer button and not the picker's per-file "Remove " buttons. await page.getByRole('dialog').getByRole('button', { name: 'Remove', exact: true }).click(); await expect(page.getByRole('alertdialog')).toBeVisible({ timeout: 5_000 }); - await page.getByRole('alertdialog').getByRole('button', { name: /^Remove$/ }).click(); + await page.getByRole('alertdialog').getByRole('button', { name: /^Detach$/ }).click(); - // After removal, the "Remove" button is gone from the panel footer. - await expect(page.getByRole('dialog').getByRole('button', { name: /^Remove$/ })).not.toBeVisible({ timeout: 5_000 }); + // After detach, the "Detach" button is gone from the panel footer. + await expect(page.getByRole('dialog').getByRole('button', { name: /^Detach$/ })).not.toBeVisible({ timeout: 5_000 }); }); }); @@ -382,3 +383,226 @@ test.describe('Create stack from Git', () => { } }); }); + +test.describe('Git Sources complete-project materialization (local git server)', () => { + test.skip(!gitAvailable(), 'system git binary is not available'); + + let server: { url: string; close: () => void }; + let stackName: string; + + test.beforeAll(async () => { + server = await serveRepos({ + app: buildFixtureRepo(fullProjectFiles()), + multi: buildFixtureRepo(multiFileFiles()), + bad: buildFixtureRepo(refusalFiles()), + }); + }); + + test.afterAll(() => { + server?.close(); + }); + + test.beforeEach(async () => { + stackName = `e2e-mater-${Date.now()}`; + }); + + test.afterEach(async ({ page }) => { + await page.evaluate(async (name) => { + await fetch(`/api/stacks/${name}/git-source`, { method: 'DELETE', credentials: 'include' }).catch(() => {}); + await fetch(`/api/stacks/${name}`, { method: 'DELETE', credentials: 'include' }).catch(() => {}); + }, stackName); + }); + + function saveSource(page: Page, repoUrl: string, composePaths: string[], contextDir: string | null = null) { + return page.evaluate(async ({ name, repoUrl, composePaths, contextDir }) => { + const res = await fetch(`/api/stacks/${name}/git-source`, { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ + repo_url: repoUrl, + branch: 'main', + compose_paths: composePaths, + context_dir: contextDir, + sync_env: false, + auth_type: 'none', + auto_apply_on_webhook: false, + auto_deploy_on_apply: false, + }), + }); + return res.status; + }, { name: stackName, repoUrl, composePaths, contextDir }); + } + + test('materializes the complete project and records it in the manifest', async ({ page }) => { + await loginAs(page); + await page.evaluate(async (name) => { + await fetch('/api/stacks', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ stackName: name }), + }); + }, stackName); + const repoUrl = `${server.url}/app.git`; + expect(await saveSource(page, repoUrl, ['compose.yaml'])).toBe(200); + + const pull = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source/pull`, { method: 'POST', credentials: 'include' }); + return { status: res.status, body: await res.json() }; + }, stackName); + expect(pull.status, JSON.stringify(pull.body)).toBe(200); + expect(pull.body.candidateReady).toBe(true); + + const applied = await page.evaluate(async ({ name, sha }) => { + const res = await fetch(`/api/stacks/${name}/git-source/apply`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ commitSha: sha, deploy: false }), + }); + return { status: res.status, body: await res.json() }; + }, { name: stackName, sha: pull.body.commitSha }); + expect(applied.status).toBe(200); + expect(applied.body.applied).toBe(true); + + // The managed-project manifest records the complete materialized set. + const manifest = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source/manifest`, { credentials: 'include' }); + return res.ok ? await res.json() : null; + }, stackName); + expect(manifest).not.toBeNull(); + const kinds = manifest.manifest.inputs.map((i: { dependencyKind: string }) => i.dependencyKind); + expect(kinds).toContain('config'); + expect(kinds).toContain('env_file'); + expect(kinds).toContain('build-context'); + expect(manifest.manifest.state).toBe('active'); + }); + + test('refuses to overwrite local modifications on apply', async ({ page }) => { + await loginAs(page); + await page.evaluate(async (name) => { + await fetch('/api/stacks', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ stackName: name }), + }); + }, stackName); + const repoUrl = `${server.url}/app.git`; + expect(await saveSource(page, repoUrl, ['compose.yaml'])).toBe(200); + const pull = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source/pull`, { method: 'POST', credentials: 'include' }); + return await res.json(); + }, stackName); + const applied = await page.evaluate(async ({ name, sha }) => { + const res = await fetch(`/api/stacks/${name}/git-source/apply`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ commitSha: sha, deploy: false }), + }); + return res.status; + }, { name: stackName, sha: pull.commitSha }); + expect(applied).toBe(200); + + // Locally modify a managed input through the file editor API. + const writeStatus = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/files/content?path=web.env`, { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ content: 'FOO=locally-edited\n' }), + }); + return res.status; + }, stackName); + expect([200, 204]).toContain(writeStatus); + + const secondPull = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source/pull`, { method: 'POST', credentials: 'include' }); + return await res.json(); + }, stackName); + const refused = await page.evaluate(async ({ name, sha }) => { + const res = await fetch(`/api/stacks/${name}/git-source/apply`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ commitSha: sha, deploy: false }), + }); + return { status: res.status, body: await res.json() }; + }, { name: stackName, sha: secondPull.commitSha }); + expect(refused.status).toBe(400); + expect(JSON.stringify(refused.body)).toContain('Local modifications'); + expect(JSON.stringify(refused.body)).toContain('web.env'); + }); + + test('detaches a multi-file stack with the export contract', async ({ page }) => { + await loginAs(page); + await page.evaluate(async (name) => { + await fetch('/api/stacks', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ stackName: name }), + }); + }, stackName); + const repoUrl = `${server.url}/multi.git`; + expect(await saveSource(page, repoUrl, ['deploy/base.yaml', 'deploy/prod.yaml'], 'deploy')).toBe(200); + const pull = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source/pull`, { method: 'POST', credentials: 'include' }); + return await res.json(); + }, stackName); + const applied = await page.evaluate(async ({ name, sha }) => { + const res = await fetch(`/api/stacks/${name}/git-source/apply`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ commitSha: sha, deploy: false }), + }); + return res.status; + }, { name: stackName, sha: pull.commitSha }); + expect(applied).toBe(200); + + const detached = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source`, { method: 'DELETE', credentials: 'include' }); + return { status: res.status, body: await res.json() }; + }, stackName); + expect(detached.status).toBe(200); + + // The stack still deploys as a plain compose project: the effective model + // was exported into compose.yaml and the source row is gone. + const after = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source`, { credentials: 'include' }); + return { sourceStatus: res.status, body: await res.json() }; + }, stackName); + expect(after.sourceStatus).toBe(200); + expect(after.body).toEqual({ linked: false }); + const exported = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/files/content?path=compose.yaml`, { credentials: 'include' }); + return res.ok ? await res.text() : ''; + }, stackName); + expect(exported).toContain('image: nginx'); + }); + + test('surfaces the refusal callout for an out-of-bound include on pull', async ({ page }) => { + await loginAs(page); + await page.evaluate(async (name) => { + await fetch('/api/stacks', { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ stackName: name }), + }); + }, stackName); + const repoUrl = `${server.url}/bad.git`; + expect(await saveSource(page, repoUrl, ['compose.yaml'])).toBe(200); + + const pull = await page.evaluate(async (name) => { + const res = await fetch(`/api/stacks/${name}/git-source/pull`, { method: 'POST', credentials: 'include' }); + return { status: res.status, body: await res.json() }; + }, stackName); + // The actionable refusal aborts the pull with an actionable message. + expect(pull.status).toBe(400); + expect(JSON.stringify(pull.body)).toMatch(/outside the repository|Cannot materialize/); + }); +}); diff --git a/e2e/gitServer.helper.ts b/e2e/gitServer.helper.ts new file mode 100644 index 00000000..4cced1dd --- /dev/null +++ b/e2e/gitServer.helper.ts @@ -0,0 +1,178 @@ +/** + * Local smart-HTTP Git server for the Git Sources E2E specs. + * + * Builds fixture repositories with the system git binary and serves them over + * HTTPS, so the full clone -> pull -> apply pipeline runs without network + * egress. Implements the two smart-HTTP endpoints isomorphic-git needs + * (GET info/refs advertise + POST upload-pack) directly; git-http-backend's + * stream internals break on modern Node. + * + * Git Sources requires HTTPS URLs, so the server speaks TLS with the committed + * dev-only CA (e2e/fixtures/git-ca.pem). The backend must trust that CA via + * NODE_EXTRA_CA_CERTS (wired in CI and in the local validation lifecycle). + * The key is a throwaway test certificate with no security value. + * + * Soft-skips when the system git binary is unavailable. + */ +import { spawn, spawnSync } from 'child_process'; +import fs from 'fs'; +import https from 'https'; +import os from 'os'; +import path from 'path'; + +export function gitAvailable(): boolean { + const probe = spawnSync('git', ['--version'], { stdio: 'ignore' }); + return probe.status === 0; +} + +/** Build a git repository with the given files on `branch`, returns the repo dir. */ +export function buildFixtureRepo(files: Record, branch = 'main'): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-e2e-repo-')); + for (const [rel, content] of Object.entries(files)) { + const abs = path.join(dir, rel); + fs.mkdirSync(path.dirname(abs), { recursive: true }); + fs.writeFileSync(abs, content); + } + const run = (args: string[]) => { + const r = spawnSync('git', args, { cwd: dir, encoding: 'utf8' }); + if (r.status !== 0) throw new Error(`git ${args[0]} failed: ${r.stderr}`); + }; + run(['init', '-b', branch]); + run(['config', 'user.email', 'e2e@sencho.test']); + run(['config', 'user.name', 'Sencho E2E']); + run(['add', '-A']); + run(['commit', '-m', 'fixture']); + return dir; +} + +/** + * Serve the given repos (keyed by served name) over smart HTTPS. Returns the + * base URL; repos are reachable at `/.git`. + */ +export function serveRepos(repoDirs: Record): Promise<{ url: string; close: () => void }> { + return new Promise((resolve, reject) => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-e2e-git-')); + for (const [name, dir] of Object.entries(repoDirs)) { + const bare = path.join(root, `${name}.git`); + const r = spawnSync('git', ['clone', '--bare', '--quiet', dir, bare], { encoding: 'utf8' }); + if (r.status !== 0) throw new Error(`git clone --bare failed: ${r.stderr}`); + repoDirs[`${name}.git`] = bare; + } + + const fixtures = path.join(process.cwd(), 'e2e', 'fixtures'); + const server = https.createServer( + { + cert: fs.readFileSync(path.join(fixtures, 'git-server.pem')), + key: fs.readFileSync(path.join(fixtures, 'git-server.key')), + }, + (req, res) => { + const url = req.url ?? '/'; + const repoName = url.split('/')[1] ?? ''; + const bare = repoDirs[repoName]; + if (!bare) { + res.statusCode = 404; + res.end('unknown repo'); + return; + } + const pathname = url.slice(url.indexOf(repoName) + repoName.length).split('?')[0]; + if (pathname === '/info/refs' && (req.method === 'GET' || req.method === 'POST')) { + const ps = spawn('git', ['upload-pack', '--stateless-rpc', '--advertise-refs', bare]); + let out = Buffer.alloc(0); + let err = ''; + ps.stdout.on('data', (d: Buffer) => { + out = Buffer.concat([out, d]); + }); + ps.stderr.on('data', (d: Buffer) => { + err += d.toString(); + }); + ps.on('error', (e) => { + console.error('[gitServer.helper] upload-pack spawn failed:', e.message); + res.statusCode = 500; + res.end('git upload-pack failed to start'); + }); + ps.on('close', (code) => { + if (code !== 0) { + console.error('[gitServer.helper] upload-pack exited', code, err); + res.statusCode = 500; + res.end(err || 'git upload-pack failed'); + return; + } + res.setHeader('content-type', 'application/x-git-upload-pack-advertisement'); + res.end(Buffer.concat([Buffer.from('001e# service=git-upload-pack\n0000'), out])); + }); + return; + } + if (pathname === '/git-upload-pack' && req.method === 'POST') { + const ps = spawn('git', ['upload-pack', '--stateless-rpc', bare]); + res.setHeader('content-type', 'application/x-git-upload-pack-result'); + ps.stdout.pipe(res); + ps.on('error', (e) => { + console.error('[gitServer.helper] upload-pack spawn failed:', e.message); + if (!res.headersSent) { + res.statusCode = 500; + res.end('git upload-pack failed to start'); + } + }); + ps.stdin.on('error', () => { + // client aborted mid-stream; the response is already ending + }); + req.pipe(ps.stdin); + ps.stderr.on('data', (d: Buffer) => console.error('[gitServer.helper] upload-pack stderr:', d.toString())); + return; + } + res.statusCode = 404; + res.end('unsupported git endpoint'); + }, + ); + server.on('error', reject); + server.listen(0, '127.0.0.1', () => { + const address = server.address(); + if (address === null || typeof address === 'string') { + reject(new Error('server did not bind')); + return; + } + resolve({ + url: `https://127.0.0.1:${address.port}`, + close: () => server.close(), + }); + }); + }); +} + +/** The full-project fixture: compose + env file + config + build context. */ +export function fullProjectFiles(): Record { + return { + 'compose.yaml': `services: + web: + image: nginx + env_file: web.env + configs: [app-conf] + build: + context: web +configs: + app-conf: + file: config/app.conf +`, + 'web.env': 'FOO=bar\n', + 'config/app.conf': 'server {}\n', + 'web/.dockerignore': 'node_modules\n', + 'web/Dockerfile': 'FROM nginx\n', + 'web/index.html': '

fixture

\n', + }; +} + +/** Multi-file fixture: base + override under a project dir. */ +export function multiFileFiles(): Record { + return { + 'deploy/base.yaml': 'services:\n web:\n image: nginx\n env_file: web.env\n', + 'deploy/prod.yaml': 'services:\n web:\n environment:\n - MODE=prod\n', + 'deploy/web.env': 'FOO=bar\n', + }; +} + +/** Refusal fixture: an include that escapes the repository. */ +export function refusalFiles(): Record { + return { + 'compose.yaml': 'include:\n - ../outside.yaml\nservices: {}\n', + }; +} diff --git a/e2e/mobile-check.spec.ts b/e2e/mobile-check.spec.ts new file mode 100644 index 00000000..8dd13318 --- /dev/null +++ b/e2e/mobile-check.spec.ts @@ -0,0 +1,59 @@ +import { test, expect, type Page } from '@playwright/test'; +import { loginAs, waitForStacksLoaded } from './helpers'; +import { buildFixtureRepo, serveRepos, fullProjectFiles } from './gitServer.helper'; + +const STACK = 'mobile-check'; + +let server: { url: string; close: () => void }; + +test.beforeAll(async () => { + // Deterministic seed: the local TLS fixture git server (no external network). + server = await serveRepos({ full: buildFixtureRepo(fullProjectFiles()) }); +}); + +test.afterAll(() => { + server.close(); +}); + +async function openStack(page: Page): Promise { + await page.reload(); + await waitForStacksLoaded(page); + await page.getByText(STACK, { exact: true }).first().click(); +} + +test('git source panel renders at phone width', async ({ page }) => { + await page.setViewportSize({ width: 390, height: 844 }); + await loginAs(page); + // Pre-clean any stack left by an interrupted run so the seeding assertions + // below fail only on genuine errors, never on stale state. + await page.evaluate(async (name) => { + await fetch(`/api/stacks/${name}`, { method: 'DELETE', credentials: 'include' }).catch(() => {}); + }, STACK); + // Seed the stack and the git-source row. Seeding failures fail the test + // loudly instead of silently degrading the assertions to a no-op. + const seed = await page.evaluate(async ({ name, repoUrl }) => { + const stackRes = await fetch('/api/stacks', { method: 'POST', headers: { 'Content-Type': 'application/json' }, credentials: 'include', body: JSON.stringify({ stackName: name }) }); + const gitRes = await fetch(`/api/stacks/${name}/git-source`, { + method: 'PUT', + headers: { 'Content-Type': 'application/json' }, + credentials: 'include', + body: JSON.stringify({ repo_url: repoUrl, branch: 'main', compose_paths: ['compose.yaml'], context_dir: null, sync_env: false, auth_type: 'none', auto_apply_on_webhook: false, auto_deploy_on_apply: false }), + }); + return { stackOk: stackRes.ok, gitOk: gitRes.ok, gitStatus: gitRes.status, gitBody: await gitRes.text() }; + }, { name: STACK, repoUrl: `${server.url}/full.git` }); + expect(seed.stackOk).toBe(true); + expect(seed.gitOk, `git-source seed failed (HTTP ${seed.gitStatus}): ${seed.gitBody}`).toBe(true); + await openStack(page); + await page.getByRole('tab', { name: 'Compose' }).click(); + await page.getByRole('button', { name: 'Git Source' }).click(); + await expect(page.getByRole('dialog').getByRole('heading', { name: /git source/i })).toBeVisible({ timeout: 10_000 }); + // No horizontal overflow at phone width. + const overflow = await page.evaluate(() => document.documentElement.scrollWidth > document.documentElement.clientWidth); + expect(overflow).toBe(false); + // The manifest section renders inside the scrollable dialog. + await expect(page.getByText('Managed project').first()).toBeVisible({ timeout: 5_000 }); + await page.screenshot({ path: 'e2e/report/mobile-panel.png' }); + await page.evaluate(async (name) => { + await fetch(`/api/stacks/${name}`, { method: 'DELETE', credentials: 'include' }).catch(() => {}); + }, STACK); +}); diff --git a/e2e/routing.spec.ts b/e2e/routing.spec.ts index ee3fe950..a69ce72e 100644 --- a/e2e/routing.spec.ts +++ b/e2e/routing.spec.ts @@ -6,10 +6,16 @@ import { test, expect } from '@playwright/test'; import { loginAs, waitForStacksLoaded } from './helpers'; async function firstStackName(page: import('@playwright/test').Page): Promise { - const row = page.locator('[role="listbox"] [role="option"]').first(); - if (!(await row.isVisible().catch(() => false))) return null; - const text = await row.textContent(); - return text?.trim() ?? null; + const response = await page.request.get('/api/stacks'); + await expect(response).toBeOK(); + const stacks = await response.json() as string[]; + return stacks[0] ?? null; +} + +async function openStack(page: import('@playwright/test').Page, stackName: string): Promise { + const option = page.locator(`[data-stacks-loaded="true"] [cmdk-item][data-value="${stackName}"]`); + await expect(option).toBeVisible(); + await option.click(); } test.describe('URL routing', () => { @@ -53,7 +59,7 @@ test.describe('URL routing', () => { const stackName = await firstStackName(page); test.skip(!stackName, 'No stacks available to open'); - await page.locator('[role="listbox"]').getByText(stackName!, { exact: true }).click(); + await openStack(page, stackName!); const slug = stackName!.replace(/^-+/, '').replace(/\.(ya?ml)$/i, ''); const escaped = slug.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); await expect(page).toHaveURL(new RegExp(`/nodes/local/stacks/${escaped}/?$`)); @@ -95,7 +101,7 @@ test.describe('URL routing', () => { test.skip(!stackName, 'No stacks available to open'); const slug = stackName!.replace(/^-+/, '').replace(/\.(ya?ml)$/i, ''); - await page.locator('[role="listbox"]').getByText(stackName!, { exact: true }).click(); + await openStack(page, stackName!); await expect(page).toHaveURL(new RegExp(`/nodes/local/stacks/`)); await expect(page).not.toHaveURL(/\/compose$/); diff --git a/frontend/src/components/stack/GitManifestSummary.tsx b/frontend/src/components/stack/GitManifestSummary.tsx new file mode 100644 index 00000000..5b1c77f7 --- /dev/null +++ b/frontend/src/components/stack/GitManifestSummary.tsx @@ -0,0 +1,204 @@ +import { useEffect, useState } from 'react'; +import { ChevronDown, ChevronRight, FileBox } from 'lucide-react'; +import { Badge } from '@/components/ui/badge'; +import { apiFetch } from '@/lib/api'; +import { cn } from '@/lib/utils'; + +/** Backend projections of the managed-project manifest (types/gitProjectManifest.ts). */ +export interface ManifestSummary { + state: 'none' | 'migrated' | 'active' | 'partial' | 'unsupported' | 'migration_required' | 'absent'; + manifestVersion: number; + resolvedCommitSha: string | null; + managedCount: number; + unmanagedCount: number; + refusedCount: number; + refused: Array<{ sourcePath: string | null; kind: string; reason: string; actionable: boolean }>; + hasBuildContexts: boolean; + generatedAt: number | null; +} + +/** Public manifest projection served by the manifest endpoint (no hashes, sensitive paths redacted). */ +interface ManifestInput { + /** Display label; null for high-sensitivity inputs whose path is redacted. */ + path: string | null; + role: string; + dependencyKind: string; + ownership: 'managed' | 'unmanaged'; + sensitivity: 'high' | 'medium' | 'low'; + state: 'present' | 'tombstoned'; + note: string | null; +} + +interface GitManifest { + manifestVersion: number; + state: string; + inputs: ManifestInput[]; +} + +const LIST_CAP = 200; + +const STATE_LABEL: Record = { + none: 'Unmanaged', + migrated: 'Migrated', + active: 'Active', + partial: 'Partial', + unsupported: 'Unsupported', + migration_required: 'Migration required', + absent: 'Not materialized', +}; + +function stateVariant(state: string): 'default' | 'secondary' | 'destructive' | 'outline' { + if (state === 'migration_required' || state === 'unsupported') return 'destructive'; + if (state === 'active' || state === 'migrated') return 'default'; + return 'secondary'; +} + +interface GitManifestSummaryProps { + stackName: string; + summary: ManifestSummary | null; +} + +/** + * Managed-project manifest summary for a Git-sourced stack: pinned revision, + * input counts, and the materialized-file inventory. The full manifest is + * fetched lazily, only when the section is expanded, to keep the panel light. + */ +export function GitManifestSummary({ stackName, summary }: GitManifestSummaryProps) { + const [expanded, setExpanded] = useState(false); + const [manifest, setManifest] = useState(null); + const [loading, setLoading] = useState(false); + const [error, setError] = useState(null); + // One fetch attempt per mount: a failed request must not re-trigger the + // effect (loading flipping false would otherwise loop forever). The attempt + // flag flips only in the fetch's finally, so the effect never refires on the + // loading state; retrying is an explicit user action. + const [attempted, setAttempted] = useState(false); + + useEffect(() => { + if (!expanded || manifest !== null || attempted) return; + setLoading(true); + setError(null); + apiFetch(`/stacks/${encodeURIComponent(stackName)}/git-source/manifest`) + .then(async (res) => { + if (res.ok) { + const data = (await res.json()) as { manifest: GitManifest }; + setManifest(data.manifest); + } else { + setError('Could not load the managed-project manifest.'); + } + }) + .catch(() => setError('Could not load the managed-project manifest.')) + .finally(() => { + setLoading(false); + setAttempted(true); + }); + }, [expanded, manifest, attempted, stackName]); + + if (!summary) return null; + + const state = summary.state; + const shownInputs = manifest ? manifest.inputs.slice(0, LIST_CAP) : []; + const visibleCount = manifest?.inputs.length ?? 0; + + return ( +
+ + + {expanded && ( +
+
+ + {summary.managedCount} managed + + + {summary.unmanagedCount} unmanaged + + + {summary.refusedCount} refused + + {summary.hasBuildContexts && build contexts} + + manifest v{summary.manifestVersion} + +
+ + {state === 'migrated' && ( +

+ This project was adopted from the previous Git-source format. Pull once to rebuild the + complete inventory from the repository. +

+ )} + {state === 'migration_required' && ( +

+ The managed-project manifest cannot be trusted. Pull now to rebuild it before applying changes. +

+ )} + + {loading &&

Loading inventory...

} + {error && !loading && ( +
+

{error}

+ +
+ )} + + {manifest && ( + <> +
+ {shownInputs.map((input, i) => ( +
+ + {input.path ?? input.dependencyKind} + + + + {input.dependencyKind} + + + {input.state === 'tombstoned' ? 'removed' : input.ownership} + + +
+ ))} +
+ {visibleCount > LIST_CAP && ( +

Showing {LIST_CAP} of {visibleCount} inputs.

+ )} + + )} +
+ )} +
+ ); +} diff --git a/frontend/src/components/stack/GitSourceDiffDialog.tsx b/frontend/src/components/stack/GitSourceDiffDialog.tsx index 3b4bb819..8b5a9269 100644 --- a/frontend/src/components/stack/GitSourceDiffDialog.tsx +++ b/frontend/src/components/stack/GitSourceDiffDialog.tsx @@ -16,6 +16,10 @@ export interface PullResult { currentEnv: string | null; validation: { ok: boolean; error?: string }; hasLocalChanges: boolean; + /** Tolerated refusals from complete-project discovery; intentionally not surfaced in this dialog (actionable refusals abort the pull, so this is always empty). */ + refusals?: Array<{ sourcePath: string | null; kind: string; reason: string; actionable: boolean }>; + /** Clone-time warnings (submodules present, for example). */ + warnings?: string[]; } interface GitSourceDiffDialogProps { @@ -97,7 +101,6 @@ export function GitSourceDiffDialog({ )} - {envAvailable && ( setDiffTab(v as 'compose' | 'env')}> diff --git a/frontend/src/components/stack/GitSourcePanel.test.tsx b/frontend/src/components/stack/GitSourcePanel.test.tsx index 6669ded8..0433322f 100644 --- a/frontend/src/components/stack/GitSourcePanel.test.tsx +++ b/frontend/src/components/stack/GitSourcePanel.test.tsx @@ -70,6 +70,8 @@ const LINKED_SOURCE = { pending_fetched_at: null, created_at: 0, updated_at: 0, + manifest_state: 'absent' as const, + manifest: null, }; function panel() { @@ -137,3 +139,54 @@ describe('GitSourcePanel deploy-mode apply node binding', () => { expect(dfCtl.params).toEqual(expect.objectContaining({ action: 'deploy', nodeId: 4 })); }); }); + +describe('GitSourcePanel manifest summary', () => { + it('renders the managed-project section when the source carries a manifest', async () => { + const summary = { + state: 'active', + manifestVersion: 2, + resolvedCommitSha: 'abc1234567890abc1234567890abc1234567890a', + managedCount: 3, + unmanagedCount: 1, + refusedCount: 0, + refused: [], + hasBuildContexts: true, + generatedAt: 1, + }; + vi.mocked(apiFetch).mockImplementation(async (url: string) => + url.includes('/git-source/manifest') + ? jsonRes({ + // The manifest endpoint serves the redacted PUBLIC projection + // (path, not sourcePath/materializedPath; no hashes or internals). + manifest: { + manifestVersion: 2, + state: 'active', + inputs: [ + { path: 'compose.yaml', role: 'compose-primary', dependencyKind: 'explicit', ownership: 'managed', sensitivity: 'medium', state: 'present', note: null }, + ], + }, + }) + : jsonRes({ ...LINKED_SOURCE, manifest_state: 'active', manifest: summary }), + ); + render(panel()); + const toggle = await screen.findByText('Managed project'); + expect(screen.getByText('abc1234')).toBeTruthy(); + expect(screen.getByText('Active')).toBeTruthy(); + // Counts render in the expanded section; the inventory is lazy-fetched. + fireEvent.click(toggle); + await waitFor(() => expect(screen.getByText('3')).toBeTruthy()); + expect(screen.getByText('1')).toBeTruthy(); + expect(screen.getByText('unmanaged')).toBeTruthy(); + await waitFor(() => expect(screen.getByText('explicit')).toBeTruthy()); + }); + + it('renders the manifest section with the DB state when the source has no manifest file', async () => { + vi.mocked(apiFetch).mockResolvedValue(jsonRes(LINKED_SOURCE)); + render(panel()); + await waitFor(() => expect(screen.getByText('Last applied commit')).toBeTruthy()); + // The section is driven by the DB manifest_state ('absent') when the file + // has not been materialized yet. + expect(screen.getByText('Managed project')).toBeTruthy(); + expect(screen.getByText('Not materialized')).toBeTruthy(); + }); +}); diff --git a/frontend/src/components/stack/GitSourcePanel.tsx b/frontend/src/components/stack/GitSourcePanel.tsx index 36a5a03e..55ce7e54 100644 --- a/frontend/src/components/stack/GitSourcePanel.tsx +++ b/frontend/src/components/stack/GitSourcePanel.tsx @@ -10,6 +10,7 @@ import { useNodes } from '@/context/NodeContext'; import { toast } from '@/components/ui/toast-store'; import { GitSourceDiffDialog, type PullResult } from './GitSourceDiffDialog'; import { GitSourceFields, type ApplyMode } from './GitSourceFields'; +import { GitManifestSummary, type ManifestSummary } from './GitManifestSummary'; import type { GitBrowseResult } from './GitComposeFilePicker'; export interface GitSource { @@ -31,6 +32,8 @@ export interface GitSource { pending_fetched_at: number | null; created_at: number; updated_at: number; + manifest_state: ManifestSummary['state'] | null; + manifest: ManifestSummary | null; } interface GitSourcePanelProps { @@ -248,6 +251,9 @@ export function GitSourcePanel({ }); if (res.ok) { const data: PullResult = await res.json(); + if (data.warnings && data.warnings.length > 0) { + toast.warning(data.warnings.join(' ')); + } setPull(data); setDiffOpen(true); onSourceChanged?.(); @@ -413,6 +419,28 @@ export function GitSourcePanel({ )} + + {source && ( + + )} )} @@ -479,13 +507,16 @@ export function GitSourcePanel({ onOpenChange={setRemoveConfirmOpen} variant="destructive" kicker={`${stackName.toUpperCase()} · GIT · DISCONNECT`} - title="Remove Git source" - confirmLabel={deleting ? 'Removing...' : 'Remove'} + title="Detach and export" + confirmLabel={deleting ? 'Detaching...' : 'Detach'} confirming={deleting} onConfirm={remove} >

- Disconnects the stack from its Git source. The stack files on disk are left in place and you can reconfigure the source later at any time. + Detaches the stack from its Git source. Sencho renders the effective compose model into a single + compose.yaml, keeps the materialized files, and removes Git tracking. Resolved values are baked into + the exported file: anything interpolated from .env or env_file files, including credentials, becomes + readable in compose.yaml. Reconfiguring the source later is always possible.

diff --git a/frontend/src/components/stack/RollbackReadinessSection.tsx b/frontend/src/components/stack/RollbackReadinessSection.tsx index d893b208..ac389d58 100644 --- a/frontend/src/components/stack/RollbackReadinessSection.tsx +++ b/frontend/src/components/stack/RollbackReadinessSection.tsx @@ -20,6 +20,8 @@ interface RollbackReadinessReport { computedAt: number; overall: RollbackOverall; items: RollbackReadinessItem[]; + /** Partial-revert scope disclosure for Git-managed stacks. */ + note?: string; } const LABEL_CLASS = 'font-mono text-[10px] uppercase tracking-[0.18em] text-stat-subtitle'; @@ -87,6 +89,11 @@ export function RollbackReadinessSection({ stackName }: { stackName: string }) { {overall.label} + {report.note && ( +
+ {report.note} +
+ )}
{report.items.map(item => { const meta = STATE_META[item.state] ?? STATE_META.unknown; diff --git a/frontend/src/components/stack/__tests__/GitManifestSummary.test.tsx b/frontend/src/components/stack/__tests__/GitManifestSummary.test.tsx new file mode 100644 index 00000000..01d5473f --- /dev/null +++ b/frontend/src/components/stack/__tests__/GitManifestSummary.test.tsx @@ -0,0 +1,83 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; +import { render, screen, waitFor, fireEvent } from '@testing-library/react'; +import { GitManifestSummary, type ManifestSummary } from '../GitManifestSummary'; + +vi.mock('@/lib/api', () => ({ apiFetch: vi.fn() })); + +import { apiFetch } from '@/lib/api'; + +const summary: ManifestSummary = { + state: 'active', + manifestVersion: 1, + resolvedCommitSha: 'abc1234', + managedCount: 2, + unmanagedCount: 0, + refusedCount: 0, + refused: [], + hasBuildContexts: false, + generatedAt: Date.now(), +}; + +describe('GitManifestSummary', () => { + beforeEach(() => { + vi.mocked(apiFetch).mockReset(); + }); + + it('renders nothing without a summary', () => { + const { container } = render(); + expect(container).toBeEmptyDOMElement(); + }); + + it('fetches the manifest once per expansion and renders the inventory', async () => { + vi.mocked(apiFetch).mockResolvedValue( + new Response( + JSON.stringify({ + manifest: { + manifestVersion: 1, + state: 'active', + inputs: [ + { path: 'compose.yaml', dependencyKind: 'explicit', ownership: 'managed', sensitivity: 'medium', state: 'present', note: null }, + { path: null, dependencyKind: 'secret', ownership: 'managed', sensitivity: 'high', state: 'present', note: null }, + ], + }, + }), + { status: 200 }, + ), + ); + render(); + fireEvent.click(screen.getByRole('button', { name: /Managed project/ })); + await waitFor(() => expect(screen.getByText('compose.yaml')).toBeInTheDocument()); + expect(apiFetch).toHaveBeenCalledTimes(1); + }); + + it('does not refetch after a failed request (no retry loop)', async () => { + vi.mocked(apiFetch).mockRejectedValue(new Error('down')); + render(); + fireEvent.click(screen.getByRole('button', { name: /Managed project/ })); + await waitFor(() => expect(screen.getByText('Could not load the managed-project manifest.')).toBeInTheDocument()); + expect(apiFetch).toHaveBeenCalledTimes(1); + // The pre-fix effect re-fired on the loading flip; give it a beat to prove + // the failure does not restart the request. + await new Promise((resolve) => setTimeout(resolve, 100)); + expect(apiFetch).toHaveBeenCalledTimes(1); + }); + + it('retries only through the explicit retry action', async () => { + vi.mocked(apiFetch) + .mockRejectedValueOnce(new Error('down')) + .mockResolvedValueOnce( + new Response( + JSON.stringify({ manifest: { manifestVersion: 1, state: 'active', inputs: [] } }), + { status: 200 }, + ), + ); + render(); + fireEvent.click(screen.getByRole('button', { name: /Managed project/ })); + await waitFor(() => expect(screen.getByText('Could not load the managed-project manifest.')).toBeInTheDocument()); + expect(apiFetch).toHaveBeenCalledTimes(1); + + fireEvent.click(screen.getByRole('button', { name: 'Retry' })); + await waitFor(() => expect(apiFetch).toHaveBeenCalledTimes(2)); + await waitFor(() => expect(screen.queryByText('Could not load the managed-project manifest.')).not.toBeInTheDocument()); + }); +});