feat(gitops): resolve branch, tag, and SHA refs to immutable commits before fetch (#1864)

* feat(gitops): resolve branch, tag, and SHA refs to immutable commits before fetch

The ref model now resolves a configured branch, tag, or full commit SHA to
an immutable commit before any content is downloaded, and records both the
configured and the resolved identity where revision state persists.

- RefKind (branch | tag | sha) is a resolved property, not caller-asserted.
  A bare string resolves branch-first, then tag; a full 40/64-hex SHA
  self-resolves with no remote round-trip. Branch and tag both fetch via a
  bare --branch name; a SHA uses init + shallow fetch + detached checkout.
- A single ls-remote with narrow heads/tags refspecs pins the configured ref
  to an immutable SHA; rev-parse HEAD must equal the resolved SHA or the
  fetch refuses (tip-changed) instead of materializing unreviewed content.
- Error union grew: REF_NOT_FOUND (ref-neutral, replaces BRANCH_NOT_FOUND),
  UNSUPPORTED_REF (a pinned SHA the host will not serve), and a service-level
  REF_DELETED upgrade that fires when a classified REF_NOT_FOUND occurs for a
  source with prior fetch history (a vanished ref reads as delete/force-push,
  not a fresh typo). Status mapping: REF_NOT_FOUND/REF_DELETED to 404,
  UNSUPPORTED_REF to 400.
- Configured-vs-resolved identity is recorded via a nullable resolved_ref_kind
  column on gitops_generations (added to CREATE TABLE and re-added for legacy
  installs through maybeAddCol). The kind is deliberately NOT in the plan
  fingerprint: two sources naming the same commit differently are the same plan.

Docs updated (git-sources feature page, connect-a-git-source tutorial, and the
native-git-transport internal deep-dive) to the ref-neutral naming.

* fix(gitops): harden ref resolution after pre-merge audit

Request peeled annotated-tag refs from ls-remote, detect force-pushes and
ref-kind changes against prior fetch identity, persist resolved kind on
application rows, and add real-git tag/SHA integration coverage plus
ref-neutral UI and operator docs.

* test(gitops): mock verifyFastForward in direct producer suite

The producer tests stub the transport seam but were missing resolved kind
on resolveRef and a verifyFastForward stub, so second pulls tripped the new
ref-continuity checks as REF_DELETED.

* test(git): remove unused buildBareFixtureRepo helper

Fixes backend lint failure after the integration fixture was refactored
to buildRichFixtureRepo without dropping the old wrapper.

* fix(gitops): correct fast-forward ancestry verification under size bounds

Replace the dual shallow-fetch ancestry probe with a single-tip deepen
strategy, keep verifier Git work inside the transport watchdog, and add
real-Git regression coverage for linear advances and rewritten history.

* fix(gitops): bound fast-forward verification with exponential deepen

Replace per-commit deepen loops with exponential steps, cap remote fetch
rounds, and share one deadline across verifier Git calls. Budget exhaustion
now surfaces as a classified timeout instead of REF_DELETED.

* fix(gitops): classify fast-forward probe failures accurately

Normalize verifier probe timeouts and unexpected exit codes into transport
failures, interpret merge-base status 1 as proven non-ancestry only, and
treat shallow stagnation as timeout instead of REF_DELETED.

* fix(gitops): satisfy tsc on probeFailure never returns

* fix(gitops): address Phase E QA findings on ref verification

Remove the fast-forward scratch repo after verification so pull size
caps are not inflated, classify GitHub not-our-ref as UNSUPPORTED_REF,
persist fetched_resolved_ref_kind on create-from-git, and broaden
REF_DELETED copy for retagged tags.
This commit is contained in:
Anso
2026-08-28 20:15:37 +00:00
committed by GitHub
parent 7cd42699d1
commit 48f010475b
38 changed files with 1397 additions and 137 deletions
+4
View File
@@ -1950,6 +1950,10 @@ export class DatabaseService {
maybeAddCol('stack_update_recovery_generations', 'content_path', 'TEXT');
maybeAddCol('stack_update_recovery_generations', 'operation_kind', 'TEXT');
maybeAddCol('stack_update_cleanup_pending', 'required_blueprint_id', 'INTEGER');
// Resolved ref kind for existing GitOps generations. New installs get it
// from the CREATE TABLE; older DBs need the additive column here.
maybeAddCol('gitops_generations', 'resolved_ref_kind', 'TEXT NULL');
maybeAddCol('gitops_applications', 'fetched_resolved_ref_kind', 'TEXT NULL');
// Distributed API model columns
maybeAddCol('nodes', 'api_url', "TEXT DEFAULT ''");
+93 -9
View File
@@ -29,7 +29,8 @@ import type { GitChangePlan, PublicGitChangePlan, GitChangePlanCounts, PublicGit
import { GIT_CHANGE_PLAN_SCHEMA_VERSION } from '../types/gitChangePlan';
import type { NotificationCategory } from './NotificationService';
import { classifyGitFailure, isTransportFailure } from './git/errors';
import { nativeGitTransport } from './git/nativeGitTransport';
import type { RefKind } from './git/types';
import { nativeGitTransport, verifyFastForward } from './git/nativeGitTransport';
import { GitOpsStore } from './gitops/store';
import { GitOpsTransitions, GitOpsTransitionError } from './gitops/transitions';
import {
@@ -58,7 +59,9 @@ import { managedAreaBase } from './gitops/managedPaths';
export type GitSourceErrorCode =
| 'REPO_NOT_FOUND'
| 'AUTH_FAILED'
| 'BRANCH_NOT_FOUND'
| 'REF_NOT_FOUND'
| 'REF_DELETED'
| 'UNSUPPORTED_REF'
| 'FILE_NOT_FOUND'
| 'NETWORK_TIMEOUT'
| 'GIT_ERROR'
@@ -111,12 +114,29 @@ export interface FetchParams {
* Used by the pull/create paths for complete-project materialization.
*/
onClone?: (cloneDir: string, commitSha: string, envContent: string | null) => Promise<unknown>;
/**
* True when this source has fetched successfully before. Turns a ref that
* now fails to resolve into REF_DELETED (removed or force-pushed) instead
* of a plain REF_NOT_FOUND, which would read as a mis-typed ref.
*/
hasPriorHistory?: boolean;
/**
* The commit and resolved namespace from the last successful fetch. Used to
* detect force-pushes and ref-kind changes when the symbolic ref still exists.
*/
priorIdentity?: { commitSha: string; kind: RefKind };
}
export interface FetchResult {
composeFiles: ComposeFile[];
envContent: string | null;
commitSha: string;
/**
* The namespace the configured ref resolved through. Recorded wherever the
* commit is persisted so "tag v1 -> <sha>" and "branch v1 -> <sha>" stay
* distinguishable in revision state.
*/
resolvedRefKind: RefKind;
/**
* Non-fatal issues detected during the fetch (e.g. the repo uses
* submodules that are not cloned). The stack is still usable but the
@@ -407,6 +427,24 @@ async function readRepoFile(rootDir: string, relPath: string, label: string): Pr
const SUBMODULE_WARNING =
'Repository contains Git submodules. Their contents are not cloned; any paths referenced from them will be missing at deploy time.';
const REF_DELETED_MESSAGE =
'The configured branch, tag, or commit no longer points at the same revision as before. It may have been deleted, force-pushed, or moved to a different commit (for example a retagged release).';
function priorFetchIdentity(app: GitOpsApplicationRow | null | undefined): FetchParams['priorIdentity'] {
if (!app?.fetched_commit_sha) return undefined;
let kind = app.fetched_resolved_ref_kind;
if (!kind) {
const genId = app.candidate_generation_id ?? app.accepted_generation_id;
if (genId) {
kind = GitOpsStore.getInstance().getGeneration(genId)?.resolved_ref_kind ?? null;
}
}
return {
commitSha: app.fetched_commit_sha,
kind: kind ?? 'branch',
};
}
/**
* Reject any relative path that resolves into the `.git` metadata
* directory. The path-traversal check in `fetchFromGit` already bounds
@@ -917,12 +955,20 @@ export class GitSourceService {
* structured transport failures mapped below.
*/
private async withClonedRepo<T>(
params: { repoUrl: string; branch: string; token?: string | null; timeoutMs?: number },
fn: (dir: string, commitSha: string, warnings: string[]) => Promise<T>,
params: {
repoUrl: string;
branch: string;
token?: string | null;
timeoutMs?: number;
hasPriorHistory?: boolean;
priorIdentity?: { commitSha: string; kind: RefKind };
},
fn: (dir: string, commitSha: string, warnings: string[], resolvedRefKind: RefKind) => Promise<T>,
): Promise<T> {
const { repoUrl, branch, token } = params;
const timeoutMs = params.timeoutMs ?? DEFAULT_FETCH_TIMEOUT_MS;
const root = await createTempDir();
const hasPriorHistory = params.hasPriorHistory === true || params.priorIdentity != null;
try {
const resolved = await nativeGitTransport.resolveRef({
@@ -932,9 +978,30 @@ export class GitSourceService {
timeoutMs,
workspaceRoot: root,
});
if (params.priorIdentity) {
const prior = params.priorIdentity;
if (prior.kind !== resolved.kind) {
throw new GitSourceError('REF_DELETED', REF_DELETED_MESSAGE);
}
if (prior.kind !== 'sha' && prior.commitSha !== resolved.commitSha) {
const fastForward = await verifyFastForward({
repoUrl,
ancestorSha: prior.commitSha,
descendantSha: resolved.commitSha,
token,
timeoutMs,
workspaceRoot: root,
maxBytes: maxCloneBytes(),
});
if (!fastForward) {
throw new GitSourceError('REF_DELETED', REF_DELETED_MESSAGE);
}
}
}
const fetched = await nativeGitTransport.fetchAtCommit({
repoUrl,
ref: branch,
refKind: resolved.kind,
token,
timeoutMs,
commitSha: resolved.commitSha,
@@ -951,7 +1018,7 @@ export class GitSourceService {
warnings.push(SUBMODULE_WARNING);
}
return await fn(fetched.dir, fetched.commitSha, warnings);
return await fn(fetched.dir, fetched.commitSha, warnings, resolved.kind);
} catch (e) {
if (isTransportFailure(e)) {
// The classified message operators see is deliberately
@@ -966,6 +1033,11 @@ export class GitSourceService {
console.error(`[GitSource:transport] argv=[${e.argv.map((a) => sanitizeForLog(a)).join(' ')}]`);
}
const classified = classifyGitFailure(e);
// A ref that resolved before but no longer does is a deletion
// or force-push, distinct from a mis-typed ref on first link.
if (classified.code === 'REF_NOT_FOUND' && hasPriorHistory) {
throw new GitSourceError('REF_DELETED', REF_DELETED_MESSAGE);
}
throw new GitSourceError(classified.code, classified.message);
}
throw e;
@@ -993,7 +1065,14 @@ export class GitSourceService {
}
try {
return await this.withClonedRepo({ repoUrl, branch, token, timeoutMs: params.timeoutMs }, async (dir, commitSha, warnings) => {
return await this.withClonedRepo({
repoUrl,
branch,
token,
timeoutMs: params.timeoutMs,
hasPriorHistory: params.hasPriorHistory,
priorIdentity: params.priorIdentity,
}, async (dir, commitSha, warnings, resolvedRefKind) => {
const composeFiles: ComposeFile[] = [];
for (const composePath of composePaths) {
const content = await readRepoFile(dir, composePath, 'Compose path');
@@ -1040,7 +1119,7 @@ export class GitSourceService {
`[GitSource:diag] fetch ok host=${sanitizeForLog(repoHost(repoUrl))} branch=${sanitizeForLog(branch)} sha=${commitSha.slice(0, 7)} files=${composeFiles.length} env=${envContent !== null ? 'present' : 'absent'} warnings=${warnings.length} materialized=${materialization !== null} elapsedMs=${Date.now() - startedAt}`
);
}
return { composeFiles, envContent, commitSha, warnings, materialization };
return { composeFiles, envContent, commitSha, resolvedRefKind, warnings, materialization };
});
} catch (err) {
if (diag) {
@@ -1739,12 +1818,15 @@ export class GitSourceService {
const materialization: { value: MaterializationResult | null } = { value: null };
// Every throw from here on, including this fetch, is closed by the
// caller's handler, so nothing is recorded locally.
const priorIdentity = priorFetchIdentity(gitopsApp);
const fetched: FetchResult = await this.fetchFromGit({
repoUrl: src.repo_url,
branch: src.branch,
composePaths: src.compose_paths,
envPath: src.sync_env ? src.env_path : null,
token,
hasPriorHistory: priorIdentity != null,
priorIdentity,
onClone: async (cloneDir, commitSha, envContent) => {
materialization.value = await this.buildMaterialization(stackName, cloneDir, commitSha, src, envContent);
},
@@ -1792,10 +1874,10 @@ export class GitSourceService {
// generation while the projection reports the older commit.
DatabaseService.getInstance().getDb().transaction(() => {
if (!validation.ok) {
tx.fetchedInvalid(gitopsApp.id, fetched.commitSha, gitopsEnv);
tx.fetchedInvalid(gitopsApp.id, fetched.commitSha, gitopsEnv, fetched.resolvedRefKind);
return;
}
tx.fetched(gitopsApp.id, fetched.commitSha, gitopsEnv);
tx.fetched(gitopsApp.id, fetched.commitSha, gitopsEnv, fetched.resolvedRefKind);
if (!materialization.value) return;
const identity = directSourceIdentity({
repoUrl: src.repo_url,
@@ -1831,6 +1913,7 @@ export class GitSourceService {
commitSha: fetched.commitSha,
identity,
configuredRef: src.branch,
resolvedRefKind: fetched.resolvedRefKind,
candidateRelPath: materialization.value.candidateRelPath,
appliedRelPath: appliedRelPathFor(fetched.commitSha, nextManifestVersion),
manifestVersion: nextManifestVersion,
@@ -2725,6 +2808,7 @@ export class GitSourceService {
commitSha: fetched.commitSha,
identity: gitopsIdentity,
configuredRef: input.branch,
resolvedRefKind: fetched.resolvedRefKind,
candidateRelPath: staged.candidateRelPath,
appliedRelPath: appliedRelPathFor(fetched.commitSha, completeProjectManifest.manifestVersion),
manifestVersion: completeProjectManifest.manifestVersion,
+17 -4
View File
@@ -18,7 +18,8 @@
export type TransportFacingCode =
| 'REPO_NOT_FOUND'
| 'AUTH_FAILED'
| 'BRANCH_NOT_FOUND'
| 'REF_NOT_FOUND'
| 'UNSUPPORTED_REF'
| 'NETWORK_TIMEOUT'
| 'GIT_ERROR';
@@ -29,6 +30,7 @@ export type TransportFailureReason =
| 'git-missing'
| 'git-old'
| 'ref-not-found'
| 'unsupported-ref'
| 'tip-changed'
| 'size'
| 'timeout'
@@ -52,6 +54,7 @@ export type TransportFailure = TransportFailureBase & (
| { reason: 'git-missing'; stderr?: string }
| { reason: 'git-old'; stderr?: string }
| { reason: 'ref-not-found' }
| { reason: 'unsupported-ref' }
| { reason: 'tip-changed' }
| { reason: 'size'; maxBytes: number }
| { reason: 'timeout' }
@@ -103,13 +106,15 @@ export function classifyGitFailure(
case 'invalid-url':
return { code: 'GIT_ERROR', message: 'Unsupported repository URL. Use an https:// URL without embedded credentials.' };
case 'invalid-ref':
return { code: 'GIT_ERROR', message: 'Unsupported branch name. Use the branch name as the remote reports it.' };
return { code: 'GIT_ERROR', message: 'Unsupported ref name. Use a branch name, a tag name, or a full commit SHA as the remote reports it.' };
case 'git-missing':
return { code: 'GIT_ERROR', message: failure.stderr || 'The git command was not found on PATH.' };
case 'git-old':
return { code: 'GIT_ERROR', message: failure.stderr || 'The installed git client is too old.' };
case 'ref-not-found':
return { code: 'BRANCH_NOT_FOUND', message: 'Branch not found in the repository.' };
return { code: 'REF_NOT_FOUND', message: 'The configured branch, tag, or commit was not found in the repository.' };
case 'unsupported-ref':
return { code: 'UNSUPPORTED_REF', message: 'The configured commit is not reachable on this repository host. Use a branch or tag, or a commit the host advertises.' };
case 'tip-changed':
return { code: 'GIT_ERROR', message: 'Repository tip changed during fetch; retry the pull.' };
case 'size':
@@ -145,7 +150,15 @@ export function classifyGitFailure(
};
}
if (/remote branch .+ not found in upstream|branch not found/.test(raw)) {
return { code: 'BRANCH_NOT_FOUND', message: 'Branch not found in the repository.' };
return { code: 'REF_NOT_FOUND', message: 'The configured branch, tag, or commit was not found in the repository.' };
}
// A host that refuses to serve an unadvertised object (SHA fetch without
// allowAnySHA1InWant/allowReachableSHA1InWant) still exits non-zero, but
// the failure is about server capability, not the SHA existing. Hosts word
// the refusal differently (GitHub vs GitLab/Gitea), so match stable phrases
// rather than one vendor's full sentence.
if (/unadvertised object|not our ref/.test(raw)) {
return { code: 'UNSUPPORTED_REF', message: 'The configured commit is not reachable on this repository host. Use a branch or tag, or a commit the host advertises.' };
}
if (/repository[\s\S]*\bnot found\b|not found in upstream/.test(raw)) {
return {
+337 -56
View File
@@ -10,7 +10,7 @@ import {
writeCredentialHelper,
} from './credentialHelper';
import { isTransportFailure, type TransportFailure } from './errors';
import type { FetchRequest, FetchResult, GitTransport, ResolveRequest } from './types';
import type { FetchRequest, FetchResult, GitTransport, ResolveRequest, ResolveResult } from './types';
/**
* Native git transport: every Git operation is an `execFile`-style spawn of
@@ -590,26 +590,31 @@ async function ensureBinaryReady(hasToken: boolean): Promise<void> {
}
}
function parseLsRemoteLine(line: string, fullRef: string): string | null {
const tabIndex = line.indexOf('\t');
if (tabIndex === -1) return null;
if (line.slice(tabIndex + 1).trim() !== fullRef) return null;
const sha = line.slice(0, tabIndex).trim();
return SHA_PATTERN.test(sha) ? sha.toLowerCase() : null;
interface ResolvedRemoteRefs {
branchSha: string | null;
tagSha: string | null;
}
async function lsRemoteHead(
/**
* Ask the remote where a bare ref name lives. One `ls-remote` with explicit
* refspecs for both namespaces keeps the response tiny (a name matches at
* most a couple of lines even on huge repos), so the stdout cap can never
* truncate the answer we need. For an annotated tag the peeled `^{}` entry
* carries the commit, so it wins over the raw tag-object line; a lightweight
* tag's raw line already points at the commit.
*/
async function lsRemoteRefs(
url: URL,
ref: string,
env: NodeJS.ProcessEnv,
baseArgs: string[],
timeoutMs: number,
hasToken: boolean,
): Promise<string> {
): Promise<ResolvedRemoteRefs> {
let res: RunResult;
try {
res = await runGit(
[...baseArgs, 'ls-remote', '--heads', url.href, `refs/heads/${ref}`],
[...baseArgs, 'ls-remote', url.href, `refs/heads/${ref}`, `refs/tags/${ref}`, `refs/tags/${ref}^{}`],
{ env, timeoutMs: Math.min(timeoutMs, LS_REMOTE_MAX_MS) },
);
} catch (e) {
@@ -623,27 +628,289 @@ async function lsRemoteHead(
if (res.exitCode !== 0) {
throw { transportFailure: true as const, reason: 'exit', stderr: res.stderr, exitCode: res.exitCode, argv: baseArgs, host: url.host, hasToken } satisfies TransportFailure;
}
const fullRef = `refs/heads/${ref}`;
const found: ResolvedRemoteRefs = { branchSha: null, tagSha: null };
for (const line of res.stdout.split(/\r?\n/)) {
const sha = parseLsRemoteLine(line, fullRef);
if (sha) return sha;
const tabIndex = line.indexOf('\t');
if (tabIndex === -1) continue;
const sha = line.slice(0, tabIndex).trim();
if (!SHA_PATTERN.test(sha)) continue;
const full = line.slice(tabIndex + 1).trim();
if (full === `refs/heads/${ref}`) {
found.branchSha = sha.toLowerCase();
} else if (full === `refs/tags/${ref}^{}`) {
found.tagSha = sha.toLowerCase();
} else if (full === `refs/tags/${ref}` && found.tagSha === null) {
found.tagSha = sha.toLowerCase();
}
}
return found;
}
/** Remote fetch rounds allowed after the initial shallow tip fetch. */
const MAX_FF_FETCH_ROUNDS = 14;
/** Per-round deepen step cap for exponential backoff. */
const MAX_FF_DEEPEN_STEP = 2048;
/**
* Whether `descendantSha` is a fast-forward from `ancestorSha` on the remote.
* Distinguishes a normal branch advance from a force-push after ls-remote has
* already resolved the ref to a new tip.
*
* Fetches the descendant tip once, then deepens that shallow boundary with
* exponentially increasing steps until the prior commit is reachable, the
* downloaded history is complete, or a safety budget is exhausted. Operational
* failures throw a classified TransportFailure; only a proven non-fast-forward
* returns false.
*/
export async function verifyFastForward(req: {
repoUrl: string;
ancestorSha: string;
descendantSha: string;
token?: string | null;
timeoutMs?: number;
workspaceRoot: string;
maxBytes: number;
}): Promise<boolean> {
const ancestor = req.ancestorSha.toLowerCase();
const descendant = req.descendantSha.toLowerCase();
if (ancestor === descendant) return true;
const hasToken = Boolean(req.token);
await ensureBinaryReady(hasToken);
const url = assertValidRepoUrl(req.repoUrl, hasToken);
const timeoutMs = req.timeoutMs ?? DEFAULT_TIMEOUT_MS;
const deadline = Date.now() + timeoutMs;
const remainingMs = (): number => Math.max(1, deadline - Date.now());
const assertTimeBudget = (): void => {
if (Date.now() >= deadline) {
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
}
};
const { env, baseArgs } = await prepareInvocation(req.workspaceRoot, req.token);
const repoDir = path.join(req.workspaceRoot, 'ff-check');
await fs.mkdir(repoDir, { recursive: true });
let sizeExceeded = false;
let activeChild: ChildProcess | undefined;
let breachKill: Promise<void> | undefined;
const watchdog = startSizeWatchdog(req.workspaceRoot, req.maxBytes, () => {
sizeExceeded = true;
breachKill = killTree(activeChild);
});
const throwIfSizeExceeded = (): void => {
if (sizeExceeded) {
throw { transportFailure: true as const, reason: 'size', maxBytes: req.maxBytes, host: url.host, hasToken } satisfies TransportFailure;
}
};
try {
const materialize = async (args: string[]): Promise<RunResult> => {
assertTimeBudget();
let res: RunResult;
try {
res = await runGit(args, {
cwd: repoDir,
env,
timeoutMs: remainingMs(),
onSpawn: (child) => { activeChild = child; },
});
} catch (e) {
throwIfSizeExceeded();
if (isTimeoutError(e)) {
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
}
throw { transportFailure: true as const, reason: 'exit', stderr: e instanceof Error ? e.message : String(e), argv: args, host: url.host, hasToken } satisfies TransportFailure;
}
throwIfSizeExceeded();
if (res.exitCode !== 0) {
throw { transportFailure: true as const, reason: 'exit', stderr: res.stderr, exitCode: res.exitCode, argv: args, host: url.host, hasToken } satisfies TransportFailure;
}
return res;
};
const probeFailure = (res: RunResult, argv: string[]): never => {
throw {
transportFailure: true as const,
reason: 'exit',
stderr: res.stderr,
exitCode: res.exitCode,
argv,
host: url.host,
hasToken,
} satisfies TransportFailure;
};
const runProbe = async (args: string[]): Promise<RunResult> => {
assertTimeBudget();
try {
const res = await runGit(args, {
cwd: repoDir,
env,
timeoutMs: Math.min(remainingMs(), LS_REMOTE_MAX_MS),
});
throwIfSizeExceeded();
return res;
} catch (e) {
throwIfSizeExceeded();
if (isTimeoutError(e)) {
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
}
throw {
transportFailure: true as const,
reason: 'exit',
stderr: e instanceof Error ? e.message : String(e),
argv: args,
host: url.host,
hasToken,
} satisfies TransportFailure;
}
};
const isMissingObjectProbe = (res: RunResult): boolean => {
if (res.exitCode === 0) return false;
if (res.exitCode === 1) return true;
const err = res.stderr.toLowerCase();
return err.includes('not a valid object name')
|| err.includes('bad object')
|| err.includes('could not get');
};
await materialize([...baseArgs, 'init']);
await materialize([...baseArgs, 'fetch', '--depth=1', url.href, descendant]);
const countReachable = async (): Promise<number> => {
const argv = [...baseArgs, 'rev-list', '--count', descendant];
const listed = await runProbe(argv);
if (listed.exitCode !== 0) {
return probeFailure(listed, argv);
}
const parsed = Number.parseInt(listed.stdout.trim(), 10);
if (!Number.isFinite(parsed) || parsed < 0) {
throw {
transportFailure: true as const,
reason: 'exit',
stderr: `unexpected rev-list output: ${listed.stdout}`,
exitCode: listed.exitCode,
argv,
host: url.host,
hasToken,
} satisfies TransportFailure;
}
return parsed;
};
const isShallowRepository = async (): Promise<boolean> => {
const argv = [...baseArgs, 'rev-parse', '--is-shallow-repository'];
const shallow = await runProbe(argv);
if (shallow.exitCode !== 0) {
return probeFailure(shallow, argv);
}
const flag = shallow.stdout.trim();
if (flag === 'true') return true;
if (flag === 'false') return false;
throw {
transportFailure: true as const,
reason: 'exit',
stderr: `unexpected shallow-repository output: ${shallow.stdout}`,
exitCode: shallow.exitCode,
argv,
host: url.host,
hasToken,
} satisfies TransportFailure;
};
const isProvenAncestor = async (): Promise<boolean> => {
const argv = [...baseArgs, 'merge-base', '--is-ancestor', ancestor, descendant];
const ancestry = await runProbe(argv);
if (ancestry.exitCode === 0) return true;
if (ancestry.exitCode === 1) return false;
return probeFailure(ancestry, argv);
};
let reachableCount = await countReachable();
let fetchRounds = 1;
let deepenStep = 1;
const assertWithinSizeBudget = async (): Promise<void> => {
const finalSize = await treeSize(req.workspaceRoot).catch((e: unknown) => {
console.warn(`[GitSource:transport] final size measurement failed for ${req.workspaceRoot}, failing closed: ${e instanceof Error ? e.message : String(e)}`);
return -1;
});
if (sizeExceeded || finalSize < 0 || finalSize > req.maxBytes) {
throw { transportFailure: true as const, reason: 'size', maxBytes: req.maxBytes, host: url.host, hasToken } satisfies TransportFailure;
}
};
while (true) {
assertTimeBudget();
const ancestorArgv = [...baseArgs, 'cat-file', '-e', `${ancestor}^{commit}`];
const hasAncestor = await runProbe(ancestorArgv);
if (hasAncestor.exitCode === 0) {
await assertWithinSizeBudget();
return await isProvenAncestor();
}
if (!isMissingObjectProbe(hasAncestor)) {
return probeFailure(hasAncestor, ancestorArgv);
}
if (!(await isShallowRepository())) {
await assertWithinSizeBudget();
return false;
}
if (fetchRounds >= MAX_FF_FETCH_ROUNDS) {
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
}
const previousCount = reachableCount;
await materialize([...baseArgs, 'fetch', `--deepen=${deepenStep}`, url.href, descendant]);
fetchRounds += 1;
reachableCount = await countReachable();
if (reachableCount <= previousCount) {
if (!(await isShallowRepository())) {
await assertWithinSizeBudget();
return false;
}
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
}
deepenStep = Math.min(deepenStep * 2, MAX_FF_DEEPEN_STEP);
}
} finally {
watchdog.stop();
await awaitKillConfirmed(breachKill, `size-breach kill for ${url.host}`);
await fs.rm(repoDir, { recursive: true, force: true }).catch((e: unknown) => {
console.warn(`[GitSource:transport] failed to remove fast-forward scratch repo ${repoDir}: ${e instanceof Error ? e.message : String(e)}`);
});
}
throw { transportFailure: true as const, reason: 'ref-not-found', host: url.host, hasToken } satisfies TransportFailure;
}
export const nativeGitTransport: GitTransport = {
async resolveRef(req: ResolveRequest): Promise<{ commitSha: string }> {
async resolveRef(req: ResolveRequest): Promise<ResolveResult> {
const hasToken = Boolean(req.token);
await ensureBinaryReady(hasToken);
const url = assertValidRepoUrl(req.repoUrl, hasToken);
assertValidRef(req.ref, url.host, hasToken);
if (SHA_PATTERN.test(req.ref)) {
// A full SHA is self-resolving: the immutable identity IS the
// value, so there is nothing to look up. Reachability is verified
// at fetch, where the host either serves the object or refuses it
// (classified as UNSUPPORTED_REF).
return { commitSha: req.ref.toLowerCase(), kind: 'sha' };
}
assertValidRef(req.ref, url.host, hasToken);
const { env, baseArgs } = await prepareInvocation(req.workspaceRoot, req.token);
const commitSha = await lsRemoteHead(
const found = await lsRemoteRefs(
url, req.ref, env, baseArgs,
req.timeoutMs ?? DEFAULT_TIMEOUT_MS, hasToken,
);
return { commitSha };
if (found.branchSha) return { commitSha: found.branchSha, kind: 'branch' };
if (found.tagSha) return { commitSha: found.tagSha, kind: 'tag' };
throw { transportFailure: true as const, reason: 'ref-not-found', host: url.host, hasToken } satisfies TransportFailure;
},
async fetchAtCommit(req: FetchRequest): Promise<FetchResult> {
@@ -670,49 +937,63 @@ export const nativeGitTransport: GitTransport = {
});
try {
let cloneResult: RunResult;
try {
cloneResult = await runGit(
[
...baseArgs, 'clone',
'--depth=1', '--single-branch', '--no-tags', '--no-recurse-submodules',
'--branch', req.ref, url.href, checkout,
],
{ cwd: layout.homeDir, env, timeoutMs, onSpawn: (child) => { activeChild = child; } },
);
} catch (e) {
// A size breach wins over the timeout wording: both kills are
// ours, but the operator guidance differs.
// Run each materialization step through one failure mapper. A size
// breach wins over the timeout wording: both kills are ours, but
// the operator guidance differs. runGit resolves on any exit code,
// so a non-zero exit is classified here by its real stderr rather
// than leaking a generic GIT_ERROR upstream.
const materialize = async (args: string[]): Promise<RunResult> => {
let res: RunResult;
try {
res = await runGit(args, {
cwd: layout.homeDir, env, timeoutMs,
onSpawn: (child) => { activeChild = child; },
});
} catch (e) {
// A size breach wins over the timeout wording.
if (sizeExceeded) {
throw { transportFailure: true as const, reason: 'size', maxBytes: req.maxBytes, host: url.host, hasToken } satisfies TransportFailure;
}
if (isTimeoutError(e)) {
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
}
throw { transportFailure: true as const, reason: 'exit', stderr: e instanceof Error ? e.message : String(e), argv: args, host: url.host, hasToken } satisfies TransportFailure;
}
// A watchdog-triggered SIGKILL settles runGit's promise via the
// child's normal 'close' event (code null -> exitCode -1), not
// a rejection, so this is the common path for an in-flight
// breach and must check sizeExceeded before the generic mapping.
if (sizeExceeded) {
throw { transportFailure: true as const, reason: 'size', maxBytes: req.maxBytes, host: url.host, hasToken } satisfies TransportFailure;
}
if (isTimeoutError(e)) {
throw { transportFailure: true as const, reason: 'timeout', host: url.host, hasToken } satisfies TransportFailure;
if (res.exitCode !== 0) {
throw { transportFailure: true as const, reason: 'exit', stderr: res.stderr, exitCode: res.exitCode, argv: args, host: url.host, hasToken } satisfies TransportFailure;
}
throw { transportFailure: true as const, reason: 'exit', stderr: e instanceof Error ? e.message : String(e), argv: [...baseArgs, 'clone'], host: url.host, hasToken } satisfies TransportFailure;
}
return res;
};
// A watchdog-triggered SIGKILL settles runGit's promise via the
// child's normal 'close' event (code null -> exitCode -1), not a
// rejection, so this branch is the common path for an in-flight
// breach and must check sizeExceeded before the generic mapping.
if (sizeExceeded) {
throw { transportFailure: true as const, reason: 'size', maxBytes: req.maxBytes, host: url.host, hasToken } satisfies TransportFailure;
}
// runGit resolves on any exit code: a failed clone (auth, missing
// repo, TLS) must classify by its real stderr here rather than
// fall through to rev-parse and surface as a generic GIT_ERROR.
if (cloneResult.exitCode !== 0) {
throw {
transportFailure: true as const,
reason: 'exit',
stderr: cloneResult.stderr,
exitCode: cloneResult.exitCode,
argv: [...baseArgs, 'clone'],
host: url.host,
hasToken,
} satisfies TransportFailure;
if (req.refKind === 'sha') {
// `--branch` cannot take a bare SHA, so a pinned commit uses a
// third strategy: init a repo, fetch exactly that object, and
// check it out detached. The host must allow fetching a direct
// SHA (GitHub does by default); a refusal surfaces as a
// non-zero `git fetch` here and classifies as UNSUPPORTED_REF.
await materialize([...baseArgs, 'init', checkout]);
await materialize([...baseArgs, '-C', checkout, 'fetch', '--depth=1', url.href, req.ref]);
await materialize([...baseArgs, '-C', checkout, 'checkout', '--detach', req.ref]);
} else {
// A bare name works for both branches and tags: `--branch`
// detaches at the named ref's commit either way, and passing a
// fully-qualified `refs/tags/<ref>` is rejected by git
// (`Remote branch ... not found`). The resolved kind is
// already pinned by ls-remote, and the rev-parse HEAD
// verification below confirms the checkout matched it.
const branchArg = req.ref;
await materialize([
...baseArgs, 'clone',
'--depth=1', '--single-branch', '--no-tags', '--no-recurse-submodules',
'--branch', branchArg, url.href, checkout,
]);
}
let actual: string;
+28 -3
View File
@@ -5,13 +5,22 @@
* configured ref to an immutable commit BEFORE any content is downloaded,
* and the fetch verifies it landed on exactly that commit. That makes
* immutable resolution structural rather than a convention callers have to
* remember. The ref field carries branch names today; widening it to tags and
* pinned SHAs later does not change either method's shape.
* remember.
*
* The configured ref is a free string: a branch name, a tag name, or a full
* commit SHA. Only a full 40/64-hex SHA is unambiguous on its own, so the
* transport resolves a bare name by asking the remote which namespace it
* lives in (branch, then tag) and returns the concrete kind it resolved
* through. That resolved kind is what callers record next to the immutable
* SHA, so "tag v1 -> <sha>" and "branch v1 -> <sha>" stay distinguishable in
* persisted revision state.
*/
export type RefKind = 'branch' | 'tag' | 'sha';
export interface ResolveRequest {
repoUrl: string;
/** Branch names today; tags and pinned SHAs may widen this later. */
/** Configured ref: a branch name, a tag name, or a full 40/64-hex commit SHA. */
ref: string;
token?: string | null;
/**
@@ -37,6 +46,14 @@ export interface FetchRequest extends ResolveRequest {
* `commitSha` pins what may be trusted.
*/
commitSha: string;
/**
* The kind resolveRef resolved `ref` through. It drives the fetch
* strategy: a branch or tag both ride `--branch <ref>` (git detaches at the
* named ref's commit either way), and a pinned SHA needs a third path
* (`git init` + `git fetch <sha>` + detached checkout), because `--branch`
* cannot take a bare SHA.
*/
refKind: RefKind;
/** Ceiling for the on-disk clone; enforced by the size watchdog. */
maxBytes: number;
}
@@ -48,7 +65,15 @@ export interface FetchResult {
}
export interface ResolveResult {
/** The immutable commit the configured ref resolved to. */
commitSha: string;
/**
* The namespace the configured ref resolved through. A bare name may be a
* branch or a tag, so this is resolved by the remote, not guessed. A full
* 40/64-hex SHA self-resolves with no network round-trip, so it always
* reports `sha`.
*/
kind: RefKind;
}
export interface GitTransport {
@@ -359,6 +359,7 @@ export function blankInlineApplication(id: string, blueprintId: number, at: numb
materialization_fingerprint: null,
desired_commit_sha: null,
fetched_commit_sha: null,
fetched_resolved_ref_kind: null,
candidate_generation_id: null,
accepted_generation_id: null,
candidate_plan_blocked: 0,
@@ -5,6 +5,7 @@ import { MANAGED_ROOT_NAME } from './managedPaths';
import { encodeGitOpsJson } from './json';
import { materializationFingerprint } from './fingerprint';
import { parseHttpsRepoUrl, parseLegacyRepoUrl, secretFreeRepoUrl, serializeRepoIdentity, type RepoIdentity } from './repoIdentity';
import type { RefKind } from '../git/types';
import type {
GitOpsApplicationRow,
GitOpsCreateCheckpointRow,
@@ -123,6 +124,7 @@ export function buildDirectApplicationRow(args: {
materialization_fingerprint: args.identity.fingerprint,
desired_commit_sha: null,
fetched_commit_sha: null,
fetched_resolved_ref_kind: null,
candidate_generation_id: null,
accepted_generation_id: null,
candidate_plan_blocked: 0,
@@ -170,6 +172,7 @@ export function buildGenerationRow(args: {
commitSha: string;
identity: DirectSourceIdentity;
configuredRef: string;
resolvedRefKind: RefKind;
candidateRelPath: string;
appliedRelPath: string;
manifestVersion: number;
@@ -188,6 +191,7 @@ export function buildGenerationRow(args: {
commit_sha: args.commitSha,
repo_url: args.identity.repoUrl,
configured_ref: args.configuredRef,
resolved_ref_kind: args.resolvedRefKind,
repo_identity_json: encodeGitOpsJson(args.identity.identity),
manifest_version: args.manifestVersion,
candidate_dir: args.candidateRelPath,
+2
View File
@@ -195,6 +195,7 @@ function migrateAccepted(
application.desired_commit_sha = trust.commitSha;
application.fetched_commit_sha = trust.commitSha;
application.fetched_resolved_ref_kind = 'branch';
application.accepted_generation_id = generationId;
application.artifact_set_id = artifactSetId;
application.latest_artifact_set_id = artifactSetId;
@@ -207,6 +208,7 @@ function migrateAccepted(
application_id: application.id,
commit_sha: trust.commitSha,
repo_url: application.configured_repo_url ?? '',
resolved_ref_kind: 'branch',
configured_ref: source.branch,
repo_identity_json: application.repo_identity_json ?? '{}',
manifest_version: trust.manifestVersion,
+6
View File
@@ -63,6 +63,9 @@ CREATE TABLE IF NOT EXISTS gitops_applications (
materialization_fingerprint TEXT NULL,
desired_commit_sha TEXT NULL,
fetched_commit_sha TEXT NULL,
fetched_resolved_ref_kind TEXT NULL CHECK (
fetched_resolved_ref_kind IS NULL OR fetched_resolved_ref_kind IN ('branch','tag','sha')
),
candidate_generation_id TEXT NULL,
accepted_generation_id TEXT NULL,
candidate_plan_blocked INTEGER NOT NULL DEFAULT 0,
@@ -148,6 +151,9 @@ CREATE TABLE IF NOT EXISTS gitops_generations (
commit_sha TEXT NOT NULL,
repo_url TEXT NOT NULL,
configured_ref TEXT NOT NULL,
resolved_ref_kind TEXT NULL CHECK (
resolved_ref_kind IS NULL OR resolved_ref_kind IN ('branch','tag','sha')
),
repo_identity_json TEXT NOT NULL,
manifest_version INTEGER NOT NULL,
candidate_dir TEXT NOT NULL,
+6 -6
View File
@@ -356,7 +356,7 @@ export class GitOpsStore {
id, lifecycle_key, lifecycle_status, target_mode, stack_name, blueprint_id,
configured_repo_url, repo_identity_json, configured_ref, compose_paths_json,
context_dir, sync_env, env_path, materialization_fingerprint, desired_commit_sha,
fetched_commit_sha, candidate_generation_id, accepted_generation_id,
fetched_commit_sha, fetched_resolved_ref_kind, candidate_generation_id, accepted_generation_id,
candidate_plan_blocked, review_required, artifact_set_id, latest_artifact_set_id,
intent_revision_id, rollout_candidate_id, rollout_generation_id, source_acceptance_ref,
placement_approval_ref, rollout_authorization_ref, legacy_combined_approval_ref,
@@ -366,12 +366,12 @@ export class GitOpsStore {
recovery_ref, recovery_phase, interruption_stage, interruption_at,
interruption_operation_id, interruption_generation_id, evidence_fresh_at,
evidence_limitations_json, created_at, updated_at
) VALUES (${Array(54).fill('?').join(', ')})`,
) VALUES (${Array(55).fill('?').join(', ')})`,
).run(
row.id, row.lifecycle_key, row.lifecycle_status, row.target_mode, row.stack_name, row.blueprint_id,
row.configured_repo_url, row.repo_identity_json, row.configured_ref, row.compose_paths_json,
row.context_dir, row.sync_env, row.env_path, row.materialization_fingerprint, row.desired_commit_sha,
row.fetched_commit_sha, row.candidate_generation_id, row.accepted_generation_id,
row.fetched_commit_sha, row.fetched_resolved_ref_kind, row.candidate_generation_id, row.accepted_generation_id,
row.candidate_plan_blocked, row.review_required, row.artifact_set_id, row.latest_artifact_set_id,
row.intent_revision_id, row.rollout_candidate_id, row.rollout_generation_id, row.source_acceptance_ref,
row.placement_approval_ref, row.rollout_authorization_ref, row.legacy_combined_approval_ref,
@@ -387,13 +387,13 @@ export class GitOpsStore {
insertGeneration(row: GitOpsGenerationRow): void {
this.db().prepare(
`INSERT INTO gitops_generations (
id, application_id, commit_sha, repo_url, configured_ref, repo_identity_json,
id, application_id, commit_sha, repo_url, configured_ref, resolved_ref_kind, repo_identity_json,
manifest_version, candidate_dir, applied_dir, expected_invocation_json,
materialization_fingerprint, validation_ok, plan_blocked, change_plan_fingerprint,
operation_id, trigger, actor, previous_generation_id, redacted_limitations_json, created_at
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
).run(
row.id, row.application_id, row.commit_sha, row.repo_url, row.configured_ref, row.repo_identity_json,
row.id, row.application_id, row.commit_sha, row.repo_url, row.configured_ref, row.resolved_ref_kind, row.repo_identity_json,
row.manifest_version, row.candidate_dir, row.applied_dir, row.expected_invocation_json,
row.materialization_fingerprint, row.validation_ok, row.plan_blocked, row.change_plan_fingerprint,
row.operation_id, row.trigger, row.actor, row.previous_generation_id, row.redacted_limitations_json,
+9 -4
View File
@@ -1,3 +1,4 @@
import type { RefKind } from '../git/types';
import { DatabaseService } from '../DatabaseService';
import {
decodeArtifactEvidenceJson,
@@ -145,12 +146,13 @@ export class GitOpsTransitions {
})();
}
fetched(applicationId: string, commitSha: string, envelope: EventEnvelope): TransitionResult {
fetched(applicationId: string, commitSha: string, envelope: EventEnvelope, resolvedRefKind: RefKind | null = null): TransitionResult {
return this.mutateApp(applicationId, envelope, 'fetched', 'committed', (app) => {
this.requireMatchingFetch(app, envelope);
this.clearActive(app);
app.desired_commit_sha = commitSha;
app.fetched_commit_sha = commitSha;
app.fetched_resolved_ref_kind = resolvedRefKind;
app.retry_count = 0;
this.clearAppFailure(app, ['fetch', 'validation']);
this.clearInterruption(app, 'fetch_started');
@@ -193,12 +195,13 @@ export class GitOpsTransitions {
* from it. Retry count is deliberately not reset, because nothing about this
* outcome suggests the next attempt will differ.
*/
fetchedInvalid(applicationId: string, commitSha: string, envelope: EventEnvelope): TransitionResult {
fetchedInvalid(applicationId: string, commitSha: string, envelope: EventEnvelope, resolvedRefKind: RefKind | null = null): TransitionResult {
return this.mutateApp(applicationId, envelope, 'fetched_invalid', 'failed', (app) => {
this.requireMatchingFetch(app, envelope);
this.clearActive(app);
app.desired_commit_sha = commitSha;
app.fetched_commit_sha = commitSha;
app.fetched_resolved_ref_kind = resolvedRefKind;
app.failure_stage = 'validation';
app.failure_class = 'validation';
app.failure_at = envelope.at;
@@ -295,6 +298,7 @@ export class GitOpsTransitions {
app.materialization_fingerprint = args.material.fingerprint;
app.desired_commit_sha = null;
app.fetched_commit_sha = null;
app.fetched_resolved_ref_kind = null;
app.candidate_generation_id = null;
app.candidate_plan_blocked = 0;
app.review_required = 0;
@@ -456,6 +460,7 @@ export class GitOpsTransitions {
app.desired_commit_sha = args.commitSha;
app.fetched_commit_sha = args.commitSha;
app.fetched_resolved_ref_kind = args.generation.resolved_ref_kind;
app.retry_count = 0;
pushHistory(this.history(app, args.envelope, {
stage: 'fetched',
@@ -2235,7 +2240,7 @@ export class GitOpsTransitions {
lifecycle_status=?, configured_repo_url=?, repo_identity_json=?, configured_ref=?,
compose_paths_json=?, context_dir=?, sync_env=?, env_path=?,
materialization_fingerprint=?, desired_commit_sha=?, fetched_commit_sha=?,
candidate_generation_id=?, accepted_generation_id=?, candidate_plan_blocked=?,
fetched_resolved_ref_kind=?, candidate_generation_id=?, accepted_generation_id=?, candidate_plan_blocked=?,
review_required=?, artifact_set_id=?, latest_artifact_set_id=?,
intent_revision_id=?, rollout_candidate_id=?, rollout_generation_id=?,
source_acceptance_ref=?, placement_approval_ref=?, rollout_authorization_ref=?,
@@ -2252,7 +2257,7 @@ export class GitOpsTransitions {
app.lifecycle_status, app.configured_repo_url, app.repo_identity_json, app.configured_ref,
app.compose_paths_json, app.context_dir, app.sync_env, app.env_path,
app.materialization_fingerprint, app.desired_commit_sha, app.fetched_commit_sha,
app.candidate_generation_id, app.accepted_generation_id, app.candidate_plan_blocked,
app.fetched_resolved_ref_kind, app.candidate_generation_id, app.accepted_generation_id, app.candidate_plan_blocked,
app.review_required, app.artifact_set_id, app.latest_artifact_set_id,
app.intent_revision_id, app.rollout_candidate_id, app.rollout_generation_id,
app.source_acceptance_ref, app.placement_approval_ref, app.rollout_authorization_ref,
+4
View File
@@ -1,5 +1,6 @@
import type { ArtifactEvidenceJson, ObservedArtifactIdentity } from './json';
import type { RepoIdentity } from './repoIdentity';
import type { RefKind } from '../git/types';
export type GitOpsTargetMode = 'direct' | 'inline_blueprint' | 'blueprint';
export type GitOpsLifecycleStatus = 'active' | 'creating' | 'detached' | 'deleted';
@@ -48,6 +49,7 @@ export type GitOpsApplicationRow = {
materialization_fingerprint: string | null;
desired_commit_sha: string | null;
fetched_commit_sha: string | null;
fetched_resolved_ref_kind: RefKind | null;
candidate_generation_id: string | null;
accepted_generation_id: string | null;
candidate_plan_blocked: number;
@@ -137,6 +139,8 @@ export type GitOpsGenerationRow = {
commit_sha: string;
repo_url: string;
configured_ref: string;
/** The namespace (branch | tag | sha) the configured ref resolved through. */
resolved_ref_kind: RefKind | null;
repo_identity_json: string;
manifest_version: number;
candidate_dir: string;