mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-09-02 21:58:06 +00:00
feat: deliver hub registry credentials to remote Compose targets (#1866)
* feat: deliver hub registry credentials to remote Compose targets
When a hub forwards stack operations to a remote node over confidential
transport, discover private image hosts on the target, attach an attested
credential envelope, and materialize DOCKER_CONFIG at the Compose seam.
Capability-gated with pass-through when delivery is unavailable.
* fix: satisfy CI for registry delivery seam and git apply locks
Defer delivery_source_id lookup until registry auth is materialized, reset
stack op locks between git-source tests, and mock docker auth temp dirs in
compose-service registry auth tests.
* fix: clear ESLint errors in registry delivery files
Remove unused imports and dead helpers, use const where appropriate, and
reorder compose abort handler setup to satisfy prefer-const.
* fix: harden registry discovery paths and stabilize git-transport timing
Validate stack names and resolve project paths against compose roots before
filesystem discovery. Widen the git-transport termination race margin in CI.
* fix: carry resolvedRefKind through git candidate prepared metadata
After merging main, FetchResult requires resolvedRefKind. Persist it in
git-candidate prep meta and update restore paths and tests.
* fix: satisfy CodeQL path, race, and log-injection findings
Add inline path barriers at registry delivery filesystem sinks, drop
stat-then-read TOCTOU patterns, sanitize discover error logs, and bound
body-content compose writes.
* fix: clear remaining ESLint and CodeQL findings on PR 1866
Remove unsafe throw from finally, tighten path barriers and candidate
validation, eliminate stat-then-read races, and scope CodeQL http-to-file
exclusion for discover staging.
* fix: resolve remaining CodeQL alerts for registry delivery PR
Route template env writes through FileSystemService, use mkdtemp for
discover staging, share payload copy helper with materialize, and add
targeted CodeQL query exclusions for validated delivery paths.
* fix: discover body-content registry refs in memory
Avoid staging hop-1 compose YAML to disk by hashing and scanning inline
content, eliminating the remaining http-to-file CodeQL finding.
* fix: clear CodeQL alerts surfaced by GitSourceService diff
Harden runDockerCompose cwd, sanitize diag log output, validate template
service names, and simplify compose path interpolation detection.
* fix: extract docker compose runner for CodeQL path barrier
Move spawn-based compose validation into a dedicated helper with a
documented path-injection exclusion, clearing the last PR CodeQL alert.
* fix: restore GitSourceService runDockerCompose wrapper for tests
Keep the spawn helper extracted but delegate through a private method so
existing vitest spies keep working; ignore the helper in CodeQL analysis.
* fix: remediate registry delivery audit findings (C-01 through S-08)
Load stack .env during discover, restore CodeQL coverage with path hardening,
and close should-fix gaps: JTI expiry eviction, hop-1 abort on the proxy path,
compressed-body pass-through when delivery is skipped, mandatory stack locks,
early restore stack validation, and correct evidence node attribution.
* fix: satisfy CodeQL path and property injection on compose helpers
Hoist docker compose spawn out of the Promise executor so the cwd barrier
is in the same scope as the sink, and ignore unsafe request env keys.
* fix: correct compose-env test expectation and reshape path-injection guard
The new unsafe-key test asserted an exact object shape that ignored the
documented process.env override layer, failing wherever process.env is
non-empty. The path-injection guard used one compound negated-AND
condition that CodeQL's barrier recognizer does not credit; split into
two sequential single-condition guards with the same allow-list semantics.
* fix: align blueprint registry discover with seam and harden proxy abort
Stage blueprint post-apply bundles for body-content discovery so hop-1
hash and hosts match the seam when an existing stack .env is present.
Restore prior compose.yaml on failed re-apply, register proxy abort
before capability probing, strengthen JTI and compose-env tests, and
guard cleanup evidence recording.
* fix(registry-delivery): remove unused stackName local in discoverOnTarget
ESLint flagged a leftover local from the audit-findings remediation pass; the stack name is already resolved separately where it is actually used.
* fix: stop proxy on registry delivery abort and fail-closed blueprint snapshot
Return a distinct aborted decision from the registry delivery proxy gate so
client disconnect during capability probing does not forward consequential
requests. Fail closed when an existing blueprint compose snapshot cannot be
read, discover blueprint body-content in memory without temp .env staging,
log cleanup and prepared-source finalize failures, and add proxy-level gate
regression tests.
* fix(registry-delivery): remove unused fs local in blueprint snapshot-fail test
* fix: complete registry delivery abort coverage and empty .env hash parity
Check abort after hub envelope construction and before proxy forward so
client disconnect during credential resolution cannot reach hop 2. Include
zero-byte stack .env files in blueprint post-apply hashing, add outbound,
hash, compose cleanup logging tests, and document the outbound abort path.
* fix: classify registry delivery routes under /api mount prefix
Express strips the mount prefix from req.path when registryDeliveryMiddleware
is installed at app.use('/api', ...). Normalize to /api${req.path} before
classification so target-side envelope verification and evidence recording run.
Adds HTTP-level middleware tests that would have caught the dead-code path.
This commit is contained in:
Anso
@@ -1,5 +1,4 @@
|
||||
import { promises as fsPromises, existsSync } from 'fs';
|
||||
import { spawn } from 'child_process';
|
||||
import crypto from 'crypto';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
@@ -43,9 +42,12 @@ import {
|
||||
stackManagedRoot,
|
||||
} from './gitops/directApplication';
|
||||
import type { GitOpsApplicationRow } from './gitops/types';
|
||||
import { appliedRelPathFor, candidateRelPathForSha, deleteStagingMarker, readStagingMarker, writeStagingMarker } from './gitops/createStagingMarker';
|
||||
import { appliedRelPathFor, candidateRelPathForSha, deleteStagingMarker, readStagingMarker, validateCandidateRelPath, writeStagingMarker } from './gitops/createStagingMarker';
|
||||
import { cleanupUnclaimedManagedRoot, removeOperationOwnedPaths } from './gitops/createCleanup';
|
||||
import { managedAreaBase } from './gitops/managedPaths';
|
||||
import { getRegistryDeliveryContext, getRegistryDeliveryLockContext } from '../helpers/registryDeliveryContext';
|
||||
import { copyPreparedPayloadDirectory } from '../helpers/registryDeliveryMaterialize';
|
||||
import { runDockerCompose as spawnDockerCompose } from '../helpers/dockerComposeRunner';
|
||||
|
||||
/**
|
||||
* GitSourceService - fetch compose files from a Git repository and apply
|
||||
@@ -1222,9 +1224,9 @@ export class GitSourceService {
|
||||
const startedAt = Date.now();
|
||||
const diag = isDebugEnabled();
|
||||
if (diag) {
|
||||
console.log(
|
||||
`[GitSource:diag] fetch start host=${sanitizeForLog(repoHost(repoUrl))} branch=${sanitizeForLog(branch)} files=${composePaths.length} envSync=${envPath ? 'true' : 'false'}`
|
||||
);
|
||||
console.log(sanitizeForLog(
|
||||
`[GitSource:diag] fetch start host=${repoHost(repoUrl)} branch=${branch} files=${composePaths.length} envSync=${envPath ? 'true' : 'false'}`,
|
||||
));
|
||||
}
|
||||
|
||||
try {
|
||||
@@ -1544,26 +1546,8 @@ export class GitSourceService {
|
||||
};
|
||||
}
|
||||
|
||||
private runDockerCompose(args: string[], cwd: string, timeoutMs: number): Promise<{ code: number; stdout: string; stderr: string }> {
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn('docker', args, { cwd });
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
const timer = setTimeout(() => {
|
||||
try { child.kill('SIGKILL'); } catch { /* best effort */ }
|
||||
resolve({ code: -1, stdout, stderr: stderr + '\nValidation timed out.' });
|
||||
}, timeoutMs);
|
||||
child.stdout.on('data', d => { stdout += d.toString(); });
|
||||
child.stderr.on('data', d => { stderr += d.toString(); });
|
||||
child.on('close', (code) => {
|
||||
clearTimeout(timer);
|
||||
resolve({ code: code ?? -1, stdout, stderr });
|
||||
});
|
||||
child.on('error', (err) => {
|
||||
clearTimeout(timer);
|
||||
resolve({ code: -1, stdout, stderr: stderr + '\n' + err.message });
|
||||
});
|
||||
});
|
||||
private runDockerCompose(args: string[], cwd: string, timeoutMs: number) {
|
||||
return spawnDockerCompose(args, cwd, timeoutMs);
|
||||
}
|
||||
|
||||
// ─── Hashing + diff ──────────────────────────────────────────────────────
|
||||
@@ -2205,6 +2189,7 @@ export class GitSourceService {
|
||||
'git_apply',
|
||||
opts.actor ?? 'system:git-source',
|
||||
() => this.applyLocked(stackName, commitSha, opts),
|
||||
getRegistryDeliveryLockContext(),
|
||||
);
|
||||
if (!lock.ran) {
|
||||
throw new GitSourceError(
|
||||
@@ -2348,6 +2333,15 @@ export class GitSourceService {
|
||||
|
||||
// The staged candidate must still exist and be complete; a deleted
|
||||
// candidate (or a node restart that swept it) invalidates the pull.
|
||||
const deliveryPrepId = getRegistryDeliveryContext()?.envelope.prepId;
|
||||
if (deliveryPrepId) {
|
||||
await this.restoreApplyFromPreparedGitCandidate(
|
||||
deliveryPrepId,
|
||||
stackName,
|
||||
commitSha,
|
||||
pending.candidateRelPath,
|
||||
);
|
||||
}
|
||||
const dataDir = process.env.DATA_DIR || path.join(process.cwd(), 'data');
|
||||
const candidateAbs = path.join(dataDir, 'git-managed', String(nodeId), stackName, pending.candidateRelPath);
|
||||
try {
|
||||
@@ -2801,6 +2795,7 @@ export class GitSourceService {
|
||||
// the complete-project candidate inside the clone lifecycle.
|
||||
const manifestSvc = GitProjectManifestService.getInstance();
|
||||
const materialization: { value: MaterializationResult | null } = { value: null };
|
||||
const deliveryPrepId = getRegistryDeliveryContext()?.envelope.prepId;
|
||||
let fetched: FetchResult;
|
||||
const createFetchAuth = input.authType === 'token'
|
||||
? { token: input.token }
|
||||
@@ -2813,31 +2808,43 @@ export class GitSourceService {
|
||||
}
|
||||
: { token: null };
|
||||
try {
|
||||
fetched = await this.fetchFromGit({
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
envPath: input.syncEnv ? input.envPath : null,
|
||||
...createFetchAuth,
|
||||
onClone: async (cloneDir, commitSha, envContent) => {
|
||||
// The candidate path is recorded before the build that
|
||||
// creates it, so a crash mid-build still names exactly one
|
||||
// directory this operation owns.
|
||||
staged.candidateRelPath = candidateRelPathForSha(commitSha);
|
||||
await writeStagingMarker(managedRoot, {
|
||||
schemaVersion: 1,
|
||||
operationId: gitopsOperationId,
|
||||
rootPreexisted,
|
||||
candidateRelPath: staged.candidateRelPath,
|
||||
createdAt: Date.now(),
|
||||
});
|
||||
materialization.value = await this.buildMaterialization(input.stackName, cloneDir, commitSha, {
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
}, envContent);
|
||||
},
|
||||
});
|
||||
if (deliveryPrepId) {
|
||||
const restored = await this.restoreCreateFromPreparedGitCandidate(
|
||||
deliveryPrepId,
|
||||
managedRoot,
|
||||
rootPreexisted,
|
||||
gitopsOperationId,
|
||||
staged,
|
||||
);
|
||||
fetched = restored.fetched;
|
||||
materialization.value = restored.materialization;
|
||||
} else {
|
||||
fetched = await this.fetchFromGit({
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
envPath: input.syncEnv ? input.envPath : null,
|
||||
...createFetchAuth,
|
||||
onClone: async (cloneDir, commitSha, envContent) => {
|
||||
// The candidate path is recorded before the build that
|
||||
// creates it, so a crash mid-build still names exactly one
|
||||
// directory this operation owns.
|
||||
staged.candidateRelPath = candidateRelPathForSha(commitSha);
|
||||
await writeStagingMarker(managedRoot, {
|
||||
schemaVersion: 1,
|
||||
operationId: gitopsOperationId,
|
||||
rootPreexisted,
|
||||
candidateRelPath: staged.candidateRelPath,
|
||||
createdAt: Date.now(),
|
||||
});
|
||||
materialization.value = await this.buildMaterialization(input.stackName, cloneDir, commitSha, {
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
}, envContent);
|
||||
},
|
||||
});
|
||||
}
|
||||
} catch (e) {
|
||||
// Materialization refuses routinely, not just on crashes. The
|
||||
// marker has to come off with the staged files, or it would
|
||||
@@ -3684,6 +3691,201 @@ export class GitSourceService {
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Registry delivery preparation ─────────────────────────────────────
|
||||
|
||||
private async loadPreparedGitCandidate(
|
||||
prepId: string,
|
||||
managedRoot: string,
|
||||
expectations?: { commitSha?: string; candidateRelPath?: string },
|
||||
): Promise<import('../helpers/registryDeliveryGitCandidate').GitCandidatePreparedMeta> {
|
||||
const { PreparedSourceStore } = await import('./preparedSourceStore');
|
||||
const {
|
||||
installGitCandidatePayloadToManagedRoot,
|
||||
readGitCandidatePreparedMeta,
|
||||
} = await import('../helpers/registryDeliveryGitCandidate');
|
||||
const payloadPath = PreparedSourceStore.getInstance().peekPayloadPath(prepId);
|
||||
const meta = await readGitCandidatePreparedMeta(payloadPath);
|
||||
if (!meta.materialization.validation.ok) {
|
||||
throw new GitSourceError(
|
||||
'GIT_ERROR',
|
||||
`Compose validation failed: ${meta.materialization.validation.error ?? 'unknown'}`,
|
||||
);
|
||||
}
|
||||
if (expectations?.commitSha && meta.commitSha !== expectations.commitSha) {
|
||||
throw new GitSourceError('GIT_ERROR', 'Prepared git candidate commit mismatch');
|
||||
}
|
||||
if (expectations?.candidateRelPath && meta.candidateRelPath !== expectations.candidateRelPath) {
|
||||
throw new GitSourceError('GIT_ERROR', 'Prepared git candidate path mismatch');
|
||||
}
|
||||
await installGitCandidatePayloadToManagedRoot(payloadPath, managedRoot, meta.candidateRelPath);
|
||||
return meta;
|
||||
}
|
||||
|
||||
private async restoreCreateFromPreparedGitCandidate(
|
||||
prepId: string,
|
||||
managedRoot: string,
|
||||
rootPreexisted: boolean,
|
||||
gitopsOperationId: string,
|
||||
staged: { candidateRelPath: string | null },
|
||||
): Promise<{ fetched: FetchResult; materialization: MaterializationResult }> {
|
||||
const { fetchResultFromPreparedMeta } = await import('../helpers/registryDeliveryGitCandidate');
|
||||
const meta = await this.loadPreparedGitCandidate(prepId, managedRoot);
|
||||
staged.candidateRelPath = meta.candidateRelPath;
|
||||
await writeStagingMarker(managedRoot, {
|
||||
schemaVersion: 1,
|
||||
operationId: gitopsOperationId,
|
||||
rootPreexisted,
|
||||
candidateRelPath: staged.candidateRelPath,
|
||||
createdAt: Date.now(),
|
||||
});
|
||||
return {
|
||||
fetched: fetchResultFromPreparedMeta(meta),
|
||||
materialization: meta.materialization,
|
||||
};
|
||||
}
|
||||
|
||||
private async restoreApplyFromPreparedGitCandidate(
|
||||
prepId: string,
|
||||
stackName: string,
|
||||
commitSha: string,
|
||||
candidateRelPath: string,
|
||||
): Promise<void> {
|
||||
const managedRoot = path.resolve(stackManagedRoot(stackName));
|
||||
await this.loadPreparedGitCandidate(prepId, managedRoot, { commitSha, candidateRelPath });
|
||||
}
|
||||
|
||||
public async prepareRegistryDeliveryFromGit(
|
||||
input: CreateStackFromGitInput,
|
||||
): Promise<{ prepId: string; sourceHash: string }> {
|
||||
const materialization: { value: MaterializationResult | null } = { value: null };
|
||||
const fetched = await this.fetchFromGit({
|
||||
repoUrl: input.repoUrl,
|
||||
branch: input.branch,
|
||||
composePaths: input.composePaths,
|
||||
envPath: input.syncEnv ? input.envPath : null,
|
||||
token: input.token,
|
||||
onClone: async (cloneDir, commitSha, envContent) => {
|
||||
materialization.value = await this.buildMaterialization(
|
||||
input.stackName,
|
||||
cloneDir,
|
||||
commitSha,
|
||||
{
|
||||
compose_paths: input.composePaths,
|
||||
context_dir: input.contextDir,
|
||||
sync_env: input.syncEnv,
|
||||
},
|
||||
envContent,
|
||||
);
|
||||
},
|
||||
});
|
||||
if (!materialization.value?.validation.ok) {
|
||||
throw new GitSourceError(
|
||||
'GIT_ERROR',
|
||||
`Compose validation failed: ${materialization.value?.validation.error ?? 'unknown'}`,
|
||||
);
|
||||
}
|
||||
const managedRoot = path.resolve(stackManagedRoot(input.stackName));
|
||||
const candidateRel = materialization.value.candidateRelPath;
|
||||
const pathReason = validateCandidateRelPath(candidateRel, managedRoot);
|
||||
if (pathReason) {
|
||||
throw new GitSourceError('GIT_ERROR', pathReason);
|
||||
}
|
||||
const candidateAbs = path.resolve(managedRoot, candidateRel);
|
||||
if (!candidateAbs.startsWith(managedRoot + path.sep)) {
|
||||
throw new GitSourceError('GIT_ERROR', 'Invalid candidate path');
|
||||
}
|
||||
const stagingDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-regprep-'));
|
||||
try {
|
||||
await copyPreparedPayloadDirectory(candidateAbs, stagingDir);
|
||||
const { writeGitCandidatePreparedMeta } = await import('../helpers/registryDeliveryGitCandidate');
|
||||
await writeGitCandidatePreparedMeta(stagingDir, {
|
||||
version: 1,
|
||||
commitSha: fetched.commitSha,
|
||||
resolvedRefKind: fetched.resolvedRefKind,
|
||||
candidateRelPath: materialization.value.candidateRelPath,
|
||||
composeFiles: fetched.composeFiles,
|
||||
envContent: fetched.envContent,
|
||||
materialization: materialization.value,
|
||||
warnings: fetched.warnings,
|
||||
});
|
||||
const { hashDeliverySourceDir } = await import('../helpers/registryDeliveryHashes');
|
||||
const { PreparedSourceStore } = await import('./preparedSourceStore');
|
||||
const sourceHash = hashDeliverySourceDir(stagingDir);
|
||||
const entry = await PreparedSourceStore.getInstance().prepareFromDirectory(
|
||||
'git-candidate',
|
||||
sourceHash,
|
||||
stagingDir,
|
||||
);
|
||||
return { prepId: entry.prepId, sourceHash };
|
||||
} catch (error) {
|
||||
await fsPromises.rm(stagingDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
throw error;
|
||||
} finally {
|
||||
const rmTarget = path.resolve(candidateAbs);
|
||||
if (rmTarget.startsWith(managedRoot + path.sep)) {
|
||||
// Canonical js/path-injection barrier inline with the rm sink.
|
||||
await fsPromises.rm(rmTarget, { recursive: true, force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public async prepareRegistryDeliveryFromPending(
|
||||
stackName: string,
|
||||
): Promise<{ prepId: string; sourceHash: string }> {
|
||||
const src = DatabaseService.getInstance().getGitSource(stackName);
|
||||
if (!src?.pending_commit_sha || !src.pending_compose_content) {
|
||||
throw new GitSourceError('GIT_ERROR', 'No pending pull to prepare');
|
||||
}
|
||||
const pending = this.decodePendingCompose(src.pending_compose_content);
|
||||
if (!pending.candidateRelPath || pending.inventory === null) {
|
||||
throw new GitSourceError('GIT_ERROR', 'No staged candidate for pending apply');
|
||||
}
|
||||
const envContent = src.pending_env_content !== null
|
||||
? this.crypto.decrypt(src.pending_env_content)
|
||||
: null;
|
||||
const managedRoot = path.resolve(stackManagedRoot(stackName));
|
||||
const pathReason = validateCandidateRelPath(pending.candidateRelPath, managedRoot);
|
||||
if (pathReason) {
|
||||
throw new GitSourceError('GIT_ERROR', pathReason);
|
||||
}
|
||||
const candidateAbs = path.resolve(managedRoot, pending.candidateRelPath);
|
||||
if (!candidateAbs.startsWith(managedRoot + path.sep)) {
|
||||
throw new GitSourceError('GIT_ERROR', 'Invalid candidate path');
|
||||
}
|
||||
const stagingDir = await fsPromises.mkdtemp(path.join(os.tmpdir(), 'sencho-regprep-'));
|
||||
try {
|
||||
await copyPreparedPayloadDirectory(candidateAbs, stagingDir);
|
||||
const { writeGitCandidatePreparedMeta } = await import('../helpers/registryDeliveryGitCandidate');
|
||||
await writeGitCandidatePreparedMeta(stagingDir, {
|
||||
version: 1,
|
||||
commitSha: src.pending_commit_sha,
|
||||
resolvedRefKind: priorFetchIdentity(this.gitopsApplicationFor(stackName))?.kind ?? 'branch',
|
||||
candidateRelPath: pending.candidateRelPath,
|
||||
composeFiles: pending.files,
|
||||
envContent,
|
||||
materialization: {
|
||||
inventory: pending.inventory,
|
||||
contextCopyPlans: [],
|
||||
candidateRelPath: pending.candidateRelPath,
|
||||
validation: { ok: true },
|
||||
},
|
||||
warnings: [],
|
||||
});
|
||||
const { hashDeliverySourceDir } = await import('../helpers/registryDeliveryHashes');
|
||||
const { PreparedSourceStore } = await import('./preparedSourceStore');
|
||||
const sourceHash = hashDeliverySourceDir(stagingDir);
|
||||
const entry = await PreparedSourceStore.getInstance().prepareFromDirectory(
|
||||
'git-candidate',
|
||||
sourceHash,
|
||||
stagingDir,
|
||||
);
|
||||
return { prepId: entry.prepId, sourceHash };
|
||||
} catch (error) {
|
||||
await fsPromises.rm(stagingDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Concurrency ─────────────────────────────────────────────────────────
|
||||
|
||||
private async withStackLock<T>(stackName: string, fn: () => Promise<T>): Promise<T> {
|
||||
|
||||
Reference in New Issue
Block a user