Files
sencho/e2e/gitServer.helper.ts
T
Anso d5ef403f67 feat(git): per-source private CA bundles and redirect credential guard (#1870)
* feat(git): add per-source private CA bundles and redirect credential guard

Let operators trust self-hosted HTTPS git servers by storing an encrypted
per-source CA PEM that is combined with system anchors at fetch time, and
block smart-HTTP redirects plus credential helper host scoping so PATs
cannot follow a cross-host Location header.

* fix(git): support removing a stored custom CA bundle

The custom CA bundle field in the Git source edit panel could be
replaced but not removed. The textarea starts empty after load, and
the save body omitted ca_bundle whenever the field was empty, which
the backend interpreted as "keep existing." An operator who retired
or no longer trusted a private CA had no way to revoke the stored
trust anchor.

Add an explicit remove_ca_bundle: true flag the UI sends alongside
the empty ca_bundle when the operator clicks "Remove stored CA."
The backend treats the flag as a clear, even when the field is
omitted, so saved revisions can revoke trust. Round-trip tests at
the service and route layers store, revoke, reload, and confirm
has_ca_bundle is false and the encrypted column is null.

In the same change, address three follow-on gaps in the same surface:

* Extract the per-fetch PEM-file write to
  backend/src/services/git/gitCaBundleSink.ts and add the file to
  paths-ignore in .github/codeql/codeql-config.yml with a comment
  explaining the trust boundary. The sink validates every PEM it
  writes and refuses non-PEM material; the path is always under the
  caller's per-fetch workspace.
* Add e2e/git-source-ca.spec.ts, which drives the full chain
  (API PUT with ca_bundle, API GET, real HTTPS pull, API PUT with
  remove_ca_bundle, API GET) against a local TLS fixture server.
* Drop http.followRedirects=false for HTTPS. Cross-host credential
  safety is already enforced by the host-scoped credential helper,
  which refuses to emit credentials to a host that does not match
  the configured repository. Same-host redirects now continue to
  work, and a new live integration test proves a cross-host
  redirect receives no credentials and the fetch fails closed
  (the redirected host records no Authorization header).

Extract the buildBareRepo helper into a shared test fixture so the
two git integration tests no longer duplicate the bootstrap.

* chore(git): clean up test surfaces on the private-CA branch

Two small follow-ups on the per-source custom CA bundle work:

* Drop the unused Page import in e2e/git-source-ca.spec.ts that the
  code-quality review surfaced. The test body never referenced the
  type, so the import is dead weight.
* Tighten the file header in
  backend/src/__tests__/git-redirect.integration.test.ts so it
  describes what the test pins (cross-host credential refusal, with
  same-host redirects preserved) instead of how it came to be
  written. No behavior change; the assertion set is unchanged.

* fix(git): restore additive platform CA trust and redirect-scope validation

* fix(git): redirect protection, CA bundle fixtures, docs accuracy

* fix(git): redirect enforcement, fixtures, docs, E2E, packet

* fix(git): validate redirect destinations before contacting them

Git ran with http.followRedirects=false and the code that was meant to
recover legitimate redirects keyed off a `Location:` header in git's
stderr. git-remote-http never prints one: it reports only
"The requested URL returned error: 302" when following is disabled, and
prints the destination only on the path where it has already followed
the redirect. The parser therefore never matched, the same-host retry
never fired, and the policy collapsed into deny-all, so every same-host
redirect failed with exit 128 across resolve, fetch, and fast-forward
verification. The retry itself was also malformed: it dropped the config
value while leaving its preceding `-c`.

Redirect policy now lives in redirectPreflight.ts. When git refuses a
redirect, the chain is walked here with an unauthenticated request and
every hop is validated before it is followed: HTTPS only, no loopback,
RFC 1918 or link-local destination, and no host outside the credential
scope. Only an approved chain yields a URL git is re-run against, and it
is applied consistently to resolveRef, fetchAtCommit and
verifyFastForward. A rejected destination is never contacted at all,
which is what keeps the internal-range guard preventive rather than
after the fact.

* test(git): prove redirect policy and per-source CA trust from observed behaviour

The redirect tests asserted only that a fetch rejected, which any failure
satisfied, including one where git never reached the fixture at all. They
are now a matrix over the cases that actually differ: a same-host
redirect resolves the ref both anonymously and with a token, a wrong
token behind that redirect still reports an authentication failure rather
than a redirect failure, and a cross-host redirect is refused against a
destination proven in the same run to serve the ref. Each fixture records
the requests it received, so "never contacted" and "never offered the
token" are read off the server rather than inferred. A probe detects
environments where a spawned git cannot reach loopback and skips there
instead of passing without asserting anything.

The per-source CA E2E ran against a fixture whose certificate the backend
also trusted process-wide, so it passed whether or not the stored bundle
ever reached git, and its closing assertion accepted 200, 500 or 404. The
fixture now presents a certificate from a separate CA that nothing else
trusts, which makes the stored bundle the only thing that can authorise
the fetch, and removing it is required to produce the classified TLS
trust failure.

* fix(git): report why a redirect preflight declined instead of failing quietly

Review of the redirect work found two fail-closed paths that were correct
but undiagnosable. A probe that could not complete was swallowed by a bare
catch, so a private CA that fails to validate looked exactly like a server
that does not redirect. A CA bundle that could not be read fell back to
default trust, which would then validate the operator's private-CA host
against the wrong anchors and fail for a reason nothing reported.

Both now say what happened. An unreadable bundle also stops authorising a
retry rather than probing with trust the operator did not configure, since
that file was written moments earlier by the same invocation and failing
to read it back is a fault rather than a missing option.

Also pins the stderr wording the redirect detector matches, so a git
upgrade that rephrases it fails a test instead of quietly making relocated
repositories unreachable, covers the absolute-Location branch of the
chain walker, and makes the real-git matrix a hard failure in CI when git
cannot reach a loopback fixture. Skipping is right on a workstation that
cannot do this, but in CI it would retire the whole matrix and leave a
green run with nothing exercised.

Documents the redirect behaviour operators can now rely on: a relocation
that stays on the same server keeps working, and one that points
elsewhere is refused without that server being contacted.

* fix(git): run the redirect matrix instead of skipping it, and sanitize its logs

The reachability probe added with the matrix used spawnSync, which blocks
the event loop, so the in-process TLS fixture could never answer it. The
probe timed out and concluded git could not reach loopback, which was
wrong: the cases themselves drive git through the non-blocking spawn path
and work fine. Locally that silently skipped all five, and in CI the guard
turned the mistake into a failure. Removed, so the matrix runs everywhere:
all five now execute in well under a second each.

The two warnings added for declined preflights interpolated a host and an
error message straight into the log line. Both now go through the
sanitizer the repository already registers as a log-injection barrier.

The preflight's outbound request is reported as request forgery because
the URL derives from the configured repository. The first request goes to
that same URL git fetches from anyway, and every later hop is checked
against its origin before being requested, so the walk cannot reach a host
the operator did not configure. Recorded as a scoped exclusion for that one
query, alongside the existing entries that settle the same trust model, so
every other query still analyzes this file.

* fix(git): route every preflight request through one origin check

The redirect preflight necessarily sends the operator's configured
repository URL to an outbound request, which reads as request forgery. The
guarantee the module provides is narrower than the URL being trusted:
nothing is requested that has not first been checked against the
configured origin. That was true of the loop but only as a property of its
shape, so it is now a single function every URL passes through, the seed
included, leaving no path to the network that skips the check.

Declaring that function a barrier states the property to the analysis
instead of excluding the file, so every other query keeps analyzing the
one module whose job is preventing this class of bug. Same mechanism the
repository already uses for log sanitization.

Also sanitizes the kill-confirmation log line, which interpolates a
repository host label supplied by configuration.

* fix(git): fall back to excluding the redirect preflight from CodeQL JS analysis

The barrier model on approvedUrl did not clear the request-forgery alert:
js/request-forgery does not consult the general dataflow barrierModel the
way js/log-injection does, so declaring the origin check's return value
clean had no effect on this query. Falling back to the paths-ignore
mechanism already proven for the two credential sink modules, with the
same trust-model rationale recorded inline: every URL requested, the seed
included, is checked against the configured repository's origin first,
so the walk cannot reach a host the operator did not configure.

The origin-check refactor itself stays; it is a real improvement (one
inspectable choke point instead of a property of the loop's shape) whether
or not the analysis can see it.

* fix(git): allow explicit CA removal to save even when the server currently needs it

Every save runs a dry-run reachability fetch before persisting, including
a revocation. Resolving the stored CA bundle for that fetch already
returns null once removeCaBundle is set, so removing a CA that the
server actually needs to be reached makes the dry-run fail on certificate
trust, and the removal itself gets refused with the same TLS error the
operator was trying to get past. Retiring a certificate that is expiring,
rotated, or no longer trusted was blocked by exactly the unreachability
that retiring it causes.

The dry-run now runs only when a CA bundle is not being explicitly
removed. Every other save path (add or replace a CA, change the
repository or branch) keeps the check unchanged; only remove_ca_bundle
skips it, and only for that one field. Removal always persists, and the
next pull reports the real reachability state.

This surfaced from the E2E hardening in the previous commit: isolating
the CA fixture so the stored bundle is actually load-bearing exposed a
save-time check that the old, globally-trusted fixture had always masked.

* fix(git-source): classify IP-SAN TLS mismatches, fix redirect probe URL, show CA-removal armed state

Live fleet QA against this branch surfaced three defects introduced by
this PR:

- classifyGitFailure's hostname-mismatch regex missed curl's actual
  wording for an IP-address SAN mismatch, so the raw stderr leaked
  through instead of the classified TLS message.
- resolveRedirectedRepoUrl built its initial ref-advertise probe URL by
  string concatenation, corrupting the URL when the source repo URL
  already carried a query string.
- Clicking "Remove stored CA" armed a revocation flag with no visible
  feedback, so an operator could not tell whether the click registered
  or whether typing in the textarea had silently un-armed it.

Adds regression tests for all three.
2026-09-01 09:33:27 -04:00

189 lines
7.0 KiB
TypeScript

/**
* Local smart-HTTP Git server for the Git Sources E2E specs.
*
* Builds fixture repositories with the system git binary and serves them over
* HTTPS, so the full clone -> pull -> apply pipeline runs without network
* egress. Implements the two smart-HTTP endpoints the git CLI needs
* (GET info/refs advertise + POST upload-pack) directly; git-http-backend's
* stream internals break on modern Node.
*
* Git Sources requires HTTPS URLs, so the server speaks TLS with the committed
* dev-only CA (e2e/fixtures/git-ca.pem). The backend must trust that CA via
* NODE_EXTRA_CA_CERTS (wired in CI and in the local validation lifecycle).
* The key is a throwaway test certificate with no security value.
*
* Soft-skips when the system git binary is unavailable.
*/
import { spawn, spawnSync } from 'child_process';
import fs from 'fs';
import https from 'https';
import os from 'os';
import path from 'path';
export function gitAvailable(): boolean {
const probe = spawnSync('git', ['--version'], { stdio: 'ignore' });
return probe.status === 0;
}
/** Build a git repository with the given files on `branch`, returns the repo dir. */
export function buildFixtureRepo(files: Record<string, string>, branch = 'main'): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-e2e-repo-'));
for (const [rel, content] of Object.entries(files)) {
const abs = path.join(dir, rel);
fs.mkdirSync(path.dirname(abs), { recursive: true });
fs.writeFileSync(abs, content);
}
const run = (args: string[]) => {
const r = spawnSync('git', args, { cwd: dir, encoding: 'utf8' });
if (r.status !== 0) throw new Error(`git ${args[0]} failed: ${r.stderr}`);
};
run(['init', '-b', branch]);
run(['config', 'user.email', 'e2e@sencho.test']);
run(['config', 'user.name', 'Sencho E2E']);
run(['add', '-A']);
run(['commit', '-m', 'fixture']);
return dir;
}
/**
* Serve the given repos (keyed by served name) over smart HTTPS. Returns the
* base URL; repos are reachable at `<url>/<name>.git`.
*/
export function serveRepos(
repoDirs: Record<string, string>,
/**
* Basename (without extension) of the certificate pair under e2e/fixtures to
* present. Defaults to the shared dev CA that the app also trusts globally.
* The per-source CA spec passes a pair signed by a CA that is deliberately
* absent from process-wide trust, so that only a stored per-source bundle
* can make its fetch succeed.
*/
certBasename = 'git-server',
): Promise<{ url: string; close: () => void }> {
return new Promise((resolve, reject) => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'sencho-e2e-git-'));
for (const [name, dir] of Object.entries(repoDirs)) {
const bare = path.join(root, `${name}.git`);
const r = spawnSync('git', ['clone', '--bare', '--quiet', dir, bare], { encoding: 'utf8' });
if (r.status !== 0) throw new Error(`git clone --bare failed: ${r.stderr}`);
repoDirs[`${name}.git`] = bare;
}
const fixtures = path.join(process.cwd(), 'e2e', 'fixtures');
const server = https.createServer(
{
cert: fs.readFileSync(path.join(fixtures, `${certBasename}.pem`)),
key: fs.readFileSync(path.join(fixtures, `${certBasename}.key`)),
},
(req, res) => {
const url = req.url ?? '/';
const repoName = url.split('/')[1] ?? '';
const bare = repoDirs[repoName];
if (!bare) {
res.statusCode = 404;
res.end('unknown repo');
return;
}
const pathname = url.slice(url.indexOf(repoName) + repoName.length).split('?')[0];
if (pathname === '/info/refs' && (req.method === 'GET' || req.method === 'POST')) {
const ps = spawn('git', ['upload-pack', '--stateless-rpc', '--advertise-refs', bare]);
let out = Buffer.alloc(0);
let err = '';
ps.stdout.on('data', (d: Buffer) => {
out = Buffer.concat([out, d]);
});
ps.stderr.on('data', (d: Buffer) => {
err += d.toString();
});
ps.on('error', (e) => {
console.error('[gitServer.helper] upload-pack spawn failed:', e.message);
res.statusCode = 500;
res.end('git upload-pack failed to start');
});
ps.on('close', (code) => {
if (code !== 0) {
console.error('[gitServer.helper] upload-pack exited', code, err);
res.statusCode = 500;
res.end(err || 'git upload-pack failed');
return;
}
res.setHeader('content-type', 'application/x-git-upload-pack-advertisement');
res.end(Buffer.concat([Buffer.from('001e# service=git-upload-pack\n0000'), out]));
});
return;
}
if (pathname === '/git-upload-pack' && req.method === 'POST') {
const ps = spawn('git', ['upload-pack', '--stateless-rpc', bare]);
res.setHeader('content-type', 'application/x-git-upload-pack-result');
ps.stdout.pipe(res);
ps.on('error', (e) => {
console.error('[gitServer.helper] upload-pack spawn failed:', e.message);
if (!res.headersSent) {
res.statusCode = 500;
res.end('git upload-pack failed to start');
}
});
ps.stdin.on('error', () => {
// client aborted mid-stream; the response is already ending
});
req.pipe(ps.stdin);
ps.stderr.on('data', (d: Buffer) => console.error('[gitServer.helper] upload-pack stderr:', d.toString()));
return;
}
res.statusCode = 404;
res.end('unsupported git endpoint');
},
);
server.on('error', reject);
server.listen(0, '127.0.0.1', () => {
const address = server.address();
if (address === null || typeof address === 'string') {
reject(new Error('server did not bind'));
return;
}
resolve({
url: `https://127.0.0.1:${address.port}`,
close: () => server.close(),
});
});
});
}
/** The full-project fixture: compose + env file + config + build context. */
export function fullProjectFiles(): Record<string, string> {
return {
'compose.yaml': `services:
web:
image: nginx
env_file: web.env
configs: [app-conf]
build:
context: web
configs:
app-conf:
file: config/app.conf
`,
'web.env': 'FOO=bar\n',
'config/app.conf': 'server {}\n',
'web/.dockerignore': 'node_modules\n',
'web/Dockerfile': 'FROM nginx\n',
'web/index.html': '<h1>fixture</h1>\n',
};
}
/** Multi-file fixture: base + override under a project dir. */
export function multiFileFiles(): Record<string, string> {
return {
'deploy/base.yaml': 'services:\n web:\n image: nginx\n env_file: web.env\n',
'deploy/prod.yaml': 'services:\n web:\n environment:\n - MODE=prod\n',
'deploy/web.env': 'FOO=bar\n',
};
}
/** Refusal fixture: an include that escapes the repository. */
export function refusalFiles(): Record<string, string> {
return {
'compose.yaml': 'include:\n - ../outside.yaml\nservices: {}\n',
};
}