mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-09-03 06:07:58 +00:00
feat(git): swap isomorphic-git for native git transport behind clone seam (#1849)
* feat(git): swap isomorphic-git for native git transport behind clone seam Replace the isomorphic-git engine (HTTP-only, single importer) with the native git CLI behind the existing withClonedRepo seam, so SSH deploy keys, ref semantics, and private CAs become reachable in later PRs. - resolve-before-fetch: ls-remote pins the branch to an immutable SHA, then rev-parse verifies the checkout against it; tip races refuse - hardened spawns: argv arrays only, protocol allowlist (https only), neutralized hooks, isolated HOME and all config channels, no prompts - token reaches git only via a credential helper reading SENCHO_GIT_TOKEN from the child env; never argv or URL - size cap becomes a workspace watchdog (on-disk measure) keeping the same knob and breach message; deterministic final gate added - Windows: pin http.sslBackend=openssl (schannel ignores sslCAInfo) and anchor to Git's bundled CA; NODE_EXTRA_CA_CERTS combines with platform defaults instead of replacing them - error classification retargets to exit code + stderr while preserving the contractual mappings (AUTH_FAILED maps to 400, never 401; unauthenticated refusals mask as REPO_NOT_FOUND) - runtime image installs git; tests re-pointed at the transport boundary plus a new engine suite (classifier corpus, argv hardening, watchdog) Zero externally visible behavior change except two edge cases: an empty branch now surfaces BRANCH_NOT_FOUND, and a mid-fetch force push refuses instead of materializing the moved tip. * fix(git): unblock CI on linux kill-path test and codeql log warning Two CI-only findings from the first pipeline run: - The scripted spawn child in the transport tests lacked the kill method that killTree's POSIX fallback reaches when a fake process group does not exist; Linux runs crashed inside the timeout tests while Windows (taskkill branch) could not reproduce it. Give the fixture the method the real ChildProcess always has. - CodeQL flagged the workspace-removal warning that interpolated the NODE_EXTRA_CA_CERTS path (environment-sourced values are treated as sensitive at log sinks). Reword the warning to name the variable instead of its value; operators know their own environment. * fix(git): collapse remaining duplicated test setup so the shared helper is used * fix(git): close watchdog, size-gate, ref-validator, and kill-ordering gaps in native transport Resolves the release-blocking findings from an independent pre-merge audit of the native git transport swap: - A watchdog-triggered kill mid-clone was misclassified as a generic exit failure instead of a size breach, because runGit resolves (not rejects) when the child is killed via SIGKILL. - The final on-disk size measurement failed open when it could not be read (workspace removed mid-walk, permissions), letting an unmeasured clone through as a success. Now fails closed and logs the real cause. - The ref-name validator was an overly restrictive allow-list that rejected valid branch names (leading underscore, non-ASCII, '#'). Replaced with a deny-list matching real `git check-ref-format --branch` semantics, verified against the git binary, including a per-path-segment `.lock` check the first pass missed. - runGit's timeout handler settled as soon as a kill was issued rather than confirmed, racing workspace cleanup against a still-alive child tree. It now waits for the child's close event, with a bounded fallback if termination is never confirmed, and preserves the timeout classification if 'error' fires after the kill. - Windows killTree now also falls back to child.kill() when taskkill itself exits non-zero, not just when it fails to spawn. - Added a real, non-mocked integration test that drives the credential helper through the actual git binary against a local HTTPS server with Basic Auth checking. It caught a genuine bug the mocked suite could not see: the credential.helper config value was quoted in a way that broke git's own absolute-path helper detection, failing every authenticated clone. Fixed by removing the quotes. - Migrated a separately developed test file's mocks off the deleted isomorphic-git module onto the native transport seam, matching the pattern already used elsewhere, after merging with main pulled in that feature. Also updates two stale comments left over from the isomorphic-git era and adds a git version check to the Docker runtime image smoke tests. * fix(git): make credential-helper path safe, unify ref length, and fix Windows kill ordering Addresses three PR 1 correction items from pre-merge audit: - credential.helper is a shell string, not argv: interpolating the helper's workspace-relative path broke authenticated fetches whenever the workspace sat under a directory with a space in its name. The config value is now a fixed string that names an environment variable instead, so no workspace path character can affect how git's shell parses it. - The transport rejected branch names over 200 characters while the route accepted up to 256 and real git has no comparable limit. REF_MAX_LEN is now a single exported constant shared by the transport and both routes. - On Windows, taskkill runs as a separate process and could still be walking a killed process tree after the direct git child reported closed, letting the caller delete the workspace early. Kill operations are now awaited to completion (bounded by a timeout) before a timed-out or size-breached run settles, on both the close and error event paths. Verified against a real authenticated git server inside the built runtime image: public HTTPS, private HTTPS with a valid PAT, invalid PAT, a deleted branch, an oversized repository, and the awkward workspace-path case, including from a workspace path containing spaces and shell metacharacters. * fix(git): reap killed helpers and classify curl refusals
This commit is contained in:
Anso
@@ -0,0 +1,280 @@
|
||||
/**
|
||||
* Real end-to-end coverage for authenticated native-git transport.
|
||||
*
|
||||
* Every other transport test mocks `child_process` or bypasses the
|
||||
* transport module entirely, so nothing proves the credential helper, the
|
||||
* `x-access-token` username convention, argv quoting, and env-var handoff
|
||||
* actually work against a real git binary talking to a server that checks
|
||||
* Basic Auth. This file does: a local HTTPS smart-HTTP server that requires
|
||||
* a token and rejects everything else, driven through the real
|
||||
* `nativeGitTransport` with nothing mocked.
|
||||
*
|
||||
* Reuses the committed dev-only TLS fixture from the Git Sources E2E specs
|
||||
* (e2e/fixtures/git-ca.pem / git-server.pem|key) via direct file reads
|
||||
* rather than importing e2e/gitServer.helper.ts: backend's tsconfig pins
|
||||
* rootDir to backend/src, so a cross-directory import would fail `tsc
|
||||
* --noEmit`.
|
||||
*
|
||||
* Soft-skips when the system git binary is unavailable, mirroring the E2E
|
||||
* fixture server's own skip.
|
||||
*/
|
||||
import { spawn, spawnSync } from 'child_process';
|
||||
import { promises as fs, mkdtempSync, readFileSync, writeFileSync } from 'fs';
|
||||
import https from 'https';
|
||||
import os from 'os';
|
||||
import path from 'path';
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest';
|
||||
import { classifyGitFailure, isTransportFailure } from '../services/git/errors';
|
||||
import { nativeGitTransport } from '../services/git/nativeGitTransport';
|
||||
|
||||
function gitAvailable(): boolean {
|
||||
return spawnSync('git', ['--version'], { stdio: 'ignore' }).status === 0;
|
||||
}
|
||||
|
||||
const FIXTURES_DIR = path.resolve(__dirname, '..', '..', '..', 'e2e', 'fixtures');
|
||||
const VALID_TOKEN = 'sencho-integration-test-token-do-not-leak';
|
||||
const FILE_CONTENT = 'hello from the authenticated fixture repo\n';
|
||||
|
||||
/**
|
||||
* Build a bare repo with one committed file. Mirrors e2e/gitServer.helper.ts's
|
||||
* fixture builder. Returns both the served bare dir and every scratch
|
||||
* directory created along the way, so the caller can remove them all.
|
||||
*/
|
||||
function buildBareFixtureRepo(): { bareDir: string; scratchDirs: string[] } {
|
||||
const srcDir = mkdtempSync(path.join(os.tmpdir(), 'sencho-git-auth-src-'));
|
||||
writeFileSync(path.join(srcDir, 'hello.txt'), FILE_CONTENT);
|
||||
const run = (args: string[]) => {
|
||||
const r = spawnSync('git', args, { cwd: srcDir, encoding: 'utf8' });
|
||||
if (r.status !== 0) throw new Error(`git ${args[0]} failed: ${r.stderr}`);
|
||||
};
|
||||
run(['init', '-b', 'main']);
|
||||
run(['config', 'user.email', 'integration-test@sencho.test']);
|
||||
run(['config', 'user.name', 'Sencho Integration Test']);
|
||||
run(['add', '-A']);
|
||||
// Explicitly off: a developer machine or CI runner with commit.gpgsign=true
|
||||
// in its global gitconfig would otherwise fail this fixture commit.
|
||||
run(['-c', 'commit.gpgsign=false', 'commit', '-m', 'fixture']);
|
||||
|
||||
const bareRoot = mkdtempSync(path.join(os.tmpdir(), 'sencho-git-auth-bare-'));
|
||||
const bareDir = path.join(bareRoot, 'repo.git');
|
||||
const clone = spawnSync('git', ['clone', '--bare', '--quiet', srcDir, bareDir], { encoding: 'utf8' });
|
||||
if (clone.status !== 0) throw new Error(`git clone --bare failed: ${clone.stderr}`);
|
||||
return { bareDir, scratchDirs: [srcDir, bareRoot] };
|
||||
}
|
||||
|
||||
/** Serve one bare repo over HTTPS smart-HTTP, rejecting any request without a valid Basic Auth token. */
|
||||
function serveAuthedRepo(bareDir: string): Promise<{ url: string; close: () => void }> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const expectedAuth = `Basic ${Buffer.from(`x-access-token:${VALID_TOKEN}`).toString('base64')}`;
|
||||
const server = https.createServer(
|
||||
{
|
||||
cert: readFileSync(path.join(FIXTURES_DIR, 'git-server.pem')),
|
||||
key: readFileSync(path.join(FIXTURES_DIR, 'git-server.key')),
|
||||
},
|
||||
(req, res) => {
|
||||
if (req.headers.authorization !== expectedAuth) {
|
||||
res.statusCode = 401;
|
||||
res.setHeader('WWW-Authenticate', 'Basic realm="sencho-integration-test"');
|
||||
res.end('authentication required');
|
||||
return;
|
||||
}
|
||||
const url = req.url ?? '/';
|
||||
if (!url.startsWith('/repo.git/')) {
|
||||
res.statusCode = 404;
|
||||
res.end('unknown repo');
|
||||
return;
|
||||
}
|
||||
const pathname = url.slice('/repo.git'.length).split('?')[0];
|
||||
if (pathname === '/info/refs' && (req.method === 'GET' || req.method === 'POST')) {
|
||||
const ps = spawn('git', ['upload-pack', '--stateless-rpc', '--advertise-refs', bareDir]);
|
||||
let out = Buffer.alloc(0);
|
||||
ps.stdout.on('data', (d: Buffer) => {
|
||||
out = Buffer.concat([out, d]);
|
||||
});
|
||||
ps.on('close', (code) => {
|
||||
if (code !== 0) {
|
||||
res.statusCode = 500;
|
||||
res.end('git upload-pack failed');
|
||||
return;
|
||||
}
|
||||
res.setHeader('content-type', 'application/x-git-upload-pack-advertisement');
|
||||
res.end(Buffer.concat([Buffer.from('001e# service=git-upload-pack\n0000'), out]));
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (pathname === '/git-upload-pack' && req.method === 'POST') {
|
||||
const ps = spawn('git', ['upload-pack', '--stateless-rpc', bareDir]);
|
||||
res.setHeader('content-type', 'application/x-git-upload-pack-result');
|
||||
ps.stdout.pipe(res);
|
||||
ps.stdin.on('error', (err) => {
|
||||
// EPIPE/ECONNRESET: the client aborted mid-stream.
|
||||
// Anything else is a real bug in this fixture server.
|
||||
const code = (err as NodeJS.ErrnoException).code;
|
||||
if (code !== 'EPIPE' && code !== 'ECONNRESET') throw err;
|
||||
});
|
||||
req.pipe(ps.stdin);
|
||||
return;
|
||||
}
|
||||
res.statusCode = 404;
|
||||
res.end('unsupported git endpoint');
|
||||
},
|
||||
);
|
||||
server.on('error', reject);
|
||||
server.listen(0, '127.0.0.1', () => {
|
||||
const address = server.address();
|
||||
if (address === null || typeof address === 'string') {
|
||||
reject(new Error('server did not bind'));
|
||||
return;
|
||||
}
|
||||
resolve({ url: `https://127.0.0.1:${address.port}/repo.git`, close: () => server.close() });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
describe.skipIf(!gitAvailable())('authenticated native git transport (real git, real TLS, real auth)', () => {
|
||||
let repoUrl: string;
|
||||
let closeServer: () => void;
|
||||
let prevExtraCaCerts: string | undefined;
|
||||
let fixtureScratchDirs: string[] = [];
|
||||
const workspaces: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
const { bareDir, scratchDirs } = buildBareFixtureRepo();
|
||||
fixtureScratchDirs = scratchDirs;
|
||||
const served = await serveAuthedRepo(bareDir);
|
||||
repoUrl = served.url;
|
||||
closeServer = served.close;
|
||||
prevExtraCaCerts = process.env.NODE_EXTRA_CA_CERTS;
|
||||
process.env.NODE_EXTRA_CA_CERTS = path.join(FIXTURES_DIR, 'git-ca.pem');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
closeServer?.();
|
||||
if (prevExtraCaCerts === undefined) delete process.env.NODE_EXTRA_CA_CERTS;
|
||||
else process.env.NODE_EXTRA_CA_CERTS = prevExtraCaCerts;
|
||||
await Promise.all(fixtureScratchDirs.map((d) => fs.rm(d, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(workspaces.splice(0).map((w) => fs.rm(w, { recursive: true, force: true })));
|
||||
});
|
||||
|
||||
async function makeWorkspace(): Promise<string> {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'sencho-git-auth-ws-'));
|
||||
workspaces.push(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
/**
|
||||
* A workspace nested under a directory whose name contains a space, plus
|
||||
* the other characters git's shell treats specially. Git reads
|
||||
* `credential.helper` as a shell string, so a transport that interpolates
|
||||
* the helper's path into it breaks here (and on any host whose temp dir
|
||||
* sits under something like `C:/Users/Ada Lovelace/...`) while passing
|
||||
* every normal-path test.
|
||||
*/
|
||||
async function makeAwkwardWorkspace(): Promise<string> {
|
||||
const parent = await fs.mkdtemp(path.join(os.tmpdir(), 'sencho-git-auth-odd-'));
|
||||
workspaces.push(parent);
|
||||
const dir = path.join(parent, "a dir with spaces & 'quotes' $dollar");
|
||||
await fs.mkdir(dir);
|
||||
return dir;
|
||||
}
|
||||
|
||||
it('clones a private repo end-to-end with a valid token', async () => {
|
||||
const workspaceRoot = await makeWorkspace();
|
||||
const resolved = await nativeGitTransport.resolveRef({
|
||||
repoUrl,
|
||||
ref: 'main',
|
||||
token: VALID_TOKEN,
|
||||
timeoutMs: 15_000,
|
||||
workspaceRoot,
|
||||
});
|
||||
expect(resolved.commitSha).toMatch(/^[0-9a-f]{40}$/);
|
||||
|
||||
const fetchWorkspace = await makeWorkspace();
|
||||
const fetched = await nativeGitTransport.fetchAtCommit({
|
||||
repoUrl,
|
||||
ref: 'main',
|
||||
token: VALID_TOKEN,
|
||||
commitSha: resolved.commitSha,
|
||||
timeoutMs: 15_000,
|
||||
workspaceRoot: fetchWorkspace,
|
||||
maxBytes: 10 * 1024 * 1024,
|
||||
});
|
||||
expect(fetched.commitSha).toBe(resolved.commitSha);
|
||||
const content = await fs.readFile(path.join(fetched.dir, 'hello.txt'), 'utf8');
|
||||
expect(content).toBe(FILE_CONTENT);
|
||||
});
|
||||
|
||||
it('clones a private repo end-to-end from a workspace path containing spaces and shell metacharacters', async () => {
|
||||
const workspaceRoot = await makeAwkwardWorkspace();
|
||||
const resolved = await nativeGitTransport.resolveRef({
|
||||
repoUrl,
|
||||
ref: 'main',
|
||||
token: VALID_TOKEN,
|
||||
timeoutMs: 15_000,
|
||||
workspaceRoot,
|
||||
});
|
||||
expect(resolved.commitSha).toMatch(/^[0-9a-f]{40}$/);
|
||||
|
||||
const fetchWorkspace = await makeAwkwardWorkspace();
|
||||
const fetched = await nativeGitTransport.fetchAtCommit({
|
||||
repoUrl,
|
||||
ref: 'main',
|
||||
token: VALID_TOKEN,
|
||||
commitSha: resolved.commitSha,
|
||||
timeoutMs: 15_000,
|
||||
workspaceRoot: fetchWorkspace,
|
||||
maxBytes: 10 * 1024 * 1024,
|
||||
});
|
||||
expect(fetched.commitSha).toBe(resolved.commitSha);
|
||||
expect(await fs.readFile(path.join(fetched.dir, 'hello.txt'), 'utf8')).toBe(FILE_CONTENT);
|
||||
});
|
||||
|
||||
it('still classifies a wrong token as AUTH_FAILED from an awkward workspace path', async () => {
|
||||
// Guards the subtler half of the same defect: when the helper cannot
|
||||
// execute, git sends no credentials at all and the server's 401 reads
|
||||
// like an anonymous request, so the failure silently downgrades to the
|
||||
// private-repo masking classification instead of AUTH_FAILED.
|
||||
const workspaceRoot = await makeAwkwardWorkspace();
|
||||
const failure = await nativeGitTransport
|
||||
.resolveRef({ repoUrl, ref: 'main', token: 'wrong-token', timeoutMs: 15_000, workspaceRoot })
|
||||
.then(() => null, (e: unknown) => e);
|
||||
|
||||
expect(isTransportFailure(failure)).toBe(true);
|
||||
if (!isTransportFailure(failure)) throw new Error('unreachable');
|
||||
expect(failure.hasToken).toBe(true);
|
||||
expect(classifyGitFailure(failure).code).toBe('AUTH_FAILED');
|
||||
});
|
||||
|
||||
it('fails with the private-repo masking classification when no token is supplied', async () => {
|
||||
const workspaceRoot = await makeWorkspace();
|
||||
const failure = await nativeGitTransport
|
||||
.resolveRef({ repoUrl, ref: 'main', timeoutMs: 15_000, workspaceRoot })
|
||||
.then(() => null, (e: unknown) => e);
|
||||
|
||||
expect(isTransportFailure(failure)).toBe(true);
|
||||
if (!isTransportFailure(failure)) throw new Error('unreachable');
|
||||
expect(failure.hasToken).toBe(false);
|
||||
expect(classifyGitFailure(failure).code).toBe('REPO_NOT_FOUND');
|
||||
});
|
||||
|
||||
it('fails with AUTH_FAILED when an invalid token is supplied, and never leaks it', async () => {
|
||||
const wrongToken = 'this-token-is-wrong-and-must-never-appear-in-output';
|
||||
const workspaceRoot = await makeWorkspace();
|
||||
const failure = await nativeGitTransport
|
||||
.resolveRef({ repoUrl, ref: 'main', token: wrongToken, timeoutMs: 15_000, workspaceRoot })
|
||||
.then(() => null, (e: unknown) => e);
|
||||
|
||||
expect(isTransportFailure(failure)).toBe(true);
|
||||
if (!isTransportFailure(failure)) throw new Error('unreachable');
|
||||
expect(failure.hasToken).toBe(true);
|
||||
const classified = classifyGitFailure(failure);
|
||||
expect(classified.code).toBe('AUTH_FAILED');
|
||||
|
||||
const serialized = JSON.stringify(failure) + classified.message;
|
||||
expect(serialized).not.toContain(wrongToken);
|
||||
expect(serialized).not.toContain(VALID_TOKEN);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user