Files
sencho/backend/src/routes/gitSources.ts
T
Anso 69e76090cc feat(gitops): durable reconcile-attempt reservation, coalescing, and recovery (#1893)
* feat(gitops): add reconcile trigger normalization and coalescing keys

The source controller needs a single normalized shape for every trigger
that can request evaluation (manual, API, webhook, poll, retry, config
change, startup, resume, and future provider/schedule/binding-change
producers), and a way to decide whether two concurrent submissions
describe the same work.

Add ReconcileTrigger, the discriminated ReconcileRequest (fetch vs apply),
and coalesceKey/deliveryKey. A fetch coalesces by application alone, since
it has one live outcome regardless of trigger; an apply coalesces only
when the commit, plan fingerprint, and deploy choice all match, so two
applies that differ in any of those can never be joined and have one
silently receive the other's result.

* feat(gitops): add exhaustive failure classification and retry backoff

The controller needs to tell a transient network condition from a
permanent configuration one from a broad range of Git-source, policy, and
target failures, since several distinct causes collapse onto the same
public error code and a wrong call either retries forever on a bad URL or
gives up on a DNS blip.

Add classifyFailure, built on two total lookup records (one keyed by
TransportFailureReason, one by GitSourceErrorCode) so adding a new value to
either source union fails the build until this classifier accounts for it,
rather than silently defaulting. A tip-changed race classifies as
supersession, not backoff; an unrecognized exit-coded git error gets a
lower retry ceiling than a plain network timeout; a deploy or health
failure after a successful apply is its own class that must never refetch
or reapply.

Add nextRetryAt: bounded exponential backoff (60s doubling, capped at one
hour) with +-10% jitter, honoring a provider-supplied retry floor when it
is larger than the computed delay.

* feat(gitops): derive normalized reconcile outcomes from the source facet

Silence is not an acceptable GitOps result: every reconcile attempt needs
to settle into one named outcome an operator can act on, not a bare
success/failure. Deriving that outcome from the existing source-facet
projection (rather than a second, parallel status source) is what keeps
"no source change" from silently collapsing into "converged" the way a
bare commit-SHA comparison would.

Add outcomeFromSourceFacet, exhaustive over all 17 SourceFacet statuses,
each mapped to a ReconcileOutcome, a human reason, and a next action
(review, resume, retry, resolve_conflict, configure_credentials,
view_target_results, or none). converged is deliberately never produced
here: it requires target and health evidence this source-only projection
does not carry, so a later composition over source + target + health is
the only thing allowed to report it.

* feat(gitops): add the portable accepted-generation contract

Direct and Blueprint dispatch need to consume the exact same description
of what an accepted generation contains, without either side inventing a
node id, local candidate path, target project name, or placement field
into it, since that is exactly the kind of drift that would let a stale
acceptance authorize a routing decision made after it.

Add six additive, nullable columns to gitops_generations (portable
manifest, Compose inputs, source/security policy evidence, support and
compatibility requirements), all decoded honestly: a legacy row missing one
records an explicit limitation rather than inventing evidence, and a
legacy pending candidate lacking the new contract must be re-evaluated
before it can be accepted or dispatched.

Add gitops/handoff.ts: the AcceptedGeneration type built from a generation
row via buildAcceptedGeneration, a compile-time assertion that the
contract cannot carry a target-mode-specific field, and the TargetAdapter
boundary with BlueprintTargetAdapter failing closed until Blueprint
rollout orchestration exists. Current target mode and binding travel
separately in DispatchContext, re-read under the dispatch lock rather than
carried on the generation itself.

* feat(gitops): add controller-owned bookkeeping columns to gitops_applications

The source controller needs somewhere to persist source policy, poll
cadence, and a durable attempt sequence per application, and it needs to
work identically for Direct and Blueprint applications. stack_git_sources
(the existing home for auto_apply_on_webhook/auto_deploy_on_apply) is
Direct-only and keyed by stack name, so it cannot represent a Blueprint
application at all.

Add source_policy (manual|review|automatic, default manual),
poll_interval_secs (NULL inherits the global default, 0 disables),
next_poll_at (the durable scheduling cursor), and attempt_seq to
gitops_applications instead. All default to values that start no
unattended work: an upgraded installation begins polling nothing and stays
on manual policy until explicitly migrated or configured.

* feat(gitops): add durable attempt reservation and poll/retry queries

History dedupe alone is not execution idempotency: mutateApp writes the
application row and only then inserts history, so a dedupe conflict still
commits the row write. The controller needs a reservation that runs before
any side effect and touches nothing else, so a duplicate or restarted
submission can be told apart from new work without repeating it.

Add reserveReconcileAttempt/settleReconcileAttempt: a bare history insert
in its own transaction, deliberately not through mutateApp, using the
existing history dedupe index as the reservation/idempotency check itself.
A repeated reservation or settlement for the same operation is a no-op,
never overwriting the first settled result.

Add the store queries a controller needs to drive this: getSettledAttempt
and latestSettledAttempt (exact vs. most-recent lookup, the latter
tie-broken by rowid since the id column is a random UUID unrelated to
recency), listUnsettledReconcileAttempts (a reservation with no matching
settled row, for startup recovery), and listSourcesDueForPoll /
listApplicationsDueForRetry (excluding suspended, in-flight, and
Blueprint-mode applications).

* feat(gitops): add controller-facing reconcile entry point to GitSourceService

Adds GitSourceService.reconcile(), a single normalized entry point that
takes a fetch or apply request and returns a normalized outcome, next
action, and reason instead of a thrown error or a raw boolean. It owns
the git mutex for the whole evaluation and calls the same private
fetch/apply bodies the existing pull()/apply() routes use, so it never
nests locks.

Resolves the live application for the stack first and fails closed,
without attempting any work, when the request's application id no
longer matches it or when no application exists at all. A fetch or
apply failure that the underlying transition never persisted (missing
config, a stale commit, lock contention) is classified through the
existing retry disposition table so an unretryable failure is never
reported as retryable, and a stale-looking success is never reported
in place of a real failure.

* feat(gitops): dispatch accepted generations to their target

Adds GitSourceService.dispatchAcceptedGeneration(), which routes a
portable accepted-generation contract to its target: Blueprint mode
delegates to the existing BlueprintTargetAdapter (rollout orchestration
is not built yet, so it always blocks), and Direct mode dispatches by
driving reconcile() with the generation's commit, since there is no
separate generation-based promotion pipeline yet. The deploy flag is
read from the stack's own auto_deploy_on_apply setting, matching how
every other producer decides it.

Also fixes a case reconcile() got wrong: when a promotion succeeds but
the following deploy fails, the source itself has genuinely changed,
so reporting it as an unchanged failure was as untruthful as reporting
it a plain success would have been. It now reports a result that
neither claims, pointing at the target instead of asking for a source
retry that could never succeed.

* feat(gitops): add background poll and retry driver for source reconciliation

Adds SourceController, a self-rescheduling background timer that finds
sources whose poll interval or retry time has arrived and evaluates
each through the existing reconcile() entry point. A per-application
in-flight set is what keeps one slow evaluation from blocking the rest
of the fleet; the tick never waits on any evaluation before scheduling
the next one, and a scan that throws still reschedules rather than
stopping the driver permanently.

This delivery only covers detection: a tick issues a fetch, the same
step a manual pull performs, so a new candidate is staged for review
but not automatically accepted or dispatched, and a retry re-issues a
plain fetch rather than resuming whatever stage previously failed.
Both are documented as open follow-on work rather than silent gaps.

Not yet wired into the process lifecycle; that follows separately.

* feat(gitops): start and stop the source reconciliation driver with the process

Wires SourceController into the background-service lifecycle: it starts
after the existing GitOps recovery and orphan-sweep steps, so it never
scans an application still carrying a stale in-progress marker from a
killed process, and stops alongside every other background timer on
shutdown.

* feat(gitops): add suspend, resume, and explicit retry for a source

Adds GitSourceService.suspend()/resume()/retry(), the service-layer
methods behind an upcoming suspend/resume/retry control surface.

Suspending a source now genuinely stops it: pullLocked and
applyLockedBody both check suspension before doing any work, not just
inside the transition bookkeeping, which used to reject but then let
the fetch or apply proceed anyway. A refused suspend surfaces as a
real error rather than a silent no-op, since an operator believing a
source is suspended when it isn't is the exact failure this exists to
prevent; a refused resume stays silent, since the row read back after
the attempt already reports the true state either way. A webhook
delivery to a suspended source is reported as skipped rather than a
failed pull, so a long suspension does not read to the Git host as a
broken webhook.

* feat(gitops): add suspend, resume, and retry routes for a git source

Adds POST endpoints for suspending a source (with an optional, length-
capped reason), resuming it, and explicitly retrying it, wired to the
existing service-layer methods behind the same stack:edit permission
as pull and apply. Neither route can trigger a deploy today, so unlike
apply and webhook-pull they need no conditional stack:deploy check.

* fix(gitops): close path-injection gaps at two candidate-file sinks

Adds the inline resolve-and-prefix-check barrier this codebase already
uses at other filesystem sinks derived from a stack name or a stored
candidate path, closing two sinks that lacked it: the manifest write
in GitProjectManifestService, and the pending-candidate access check
before promoting an apply. The apply-side check now shares the same
strict candidate-path validator the registry-delivery path already
uses on the identical stored field, rather than a looser check, so
both call sites treat a tampered candidate reference the same way.

Fixed two test fixtures that had never matched the shape a real
candidate path takes, which the stricter check would otherwise have
rejected.

* fix(gitops): confine the manifest write directory to the managed area

The prior fix checked the manifest write/rename targets against the
resolved managed directory, but left the mkdir call on that directory
itself unconfined and checked it against the data root rather than
the narrower managed area every sibling barrier in this file uses.
Aligns it with the established convention and the actual boundary
that matters.

* feat(gitops): wire durable attempt reservation and coalescing into reconcile()

reconcile() now reserves a durable attempt before doing any fetch or
apply and settles it with the normalized result once execution
finishes, using the reservation and settlement primitives that already
existed but had no caller. A request carrying a stable external
delivery id reserves under a producer- and intent-namespaced key, so a
redelivery reuses the same attempt instead of minting a second one; a
request with no such identity gets a freshly allocated attemptSeq-
based id, bumped atomically with its reservation.

Concurrent submissions that would do the same work now coalesce: the
first becomes the leader and actually runs, and any submission that
joins while it is still in flight awaits the leader's real result
instead of running a duplicate fetch or apply. Each still gets its own
durable attempt and its own settled row, including a concurrent
redelivery of the same external event, which joins the in-flight
leader rather than falling back to a snapshot of the row from before
the leader's work landed.

Startup gains an attempt-recovery phase, run before the managed-area
sweep and before SourceController starts: every attempt reserved but
never settled, most likely from a crash between the two, is resolved
from durable state without re-executing anything. One row failing to
recover no longer blocks the rest; it is skipped and logged, and
recovery keeps paging until nothing unsettled remains.

A settlement failure is caught and logged rather than turning an
already-successful fetch or apply into a thrown error for the caller,
and a settled attempt's stored result is now decoded through a
validated outcome/next-action check instead of a blind cast, logging
rather than silently reporting unknown when a stored row is corrupt or
unreadable.

* fix(gitops): remove unused store variable from coalescing test

* fix(gitops): make reconcile-attempt recovery leader-aware and cursor-paginated

A fresh audit found real gaps in the reservation/coalescing wiring from
the previous commit: recovery paged by "still unsettled" status rather
than a cursor, so once a permanently unrecoverable row occupied every
slot in a page, every genuinely recoverable row beyond it was silently
never reached; a follower's outcome was reconstructed independently
from row state rather than from its leader's actual stored result, so
a leader and its follower could durably disagree; and an attempt
resolved on the live path was returned but never actually settled,
leaving it open indefinitely until the next restart.

listUnsettledReconcileAttempts now takes an optional (created_at, id)
cursor, matching the pagination shape queryHistoryRows already uses,
so paging always advances regardless of which rows settle. Recovery
is now two-pass: independent attempts settle first from row state,
followers are deferred, then each deferred follower settles from its
leader's now-settled result. A follower whose leader is a real but
still-unresolved reservation is left unsettled rather than guessed at
independently, since the leader could still settle to something else
later, including when the leader simply failed to settle in this same
pass rather than "never will". The same leader-aware resolution now
backs the live reconcile() path too, via a shared helper, so an
already-reserved attempt is durably settled instead of merely
returning a value.

Also fixes a narrower race in reconcile() itself: two submissions can
share an operation id (a stable external delivery id) while running
under different coalesce keys, since an apply's coalesce key includes
its commit sha, plan fingerprint, and deploy flag, which the delivery
id does not carry. reconcile() now checks in-process executions by
operation id directly before falling back to a durable-state
resolution, so such a submission joins the real in-flight leader
instead of settling a stale pre-execution snapshot ahead of it.

* feat(gitops): route pull, apply, and webhook producers through reservation

A fresh audit found the same "primitive built, real caller does not use
it" pattern one layer deeper than the previous commit fixed: reconcile(),
the controller-facing entry point, reserved and coalesced durable
attempts correctly, but the actual production producers, the manual
pull button, the manual apply button, and the webhook route, all called
pullLocked/applyWithSharedLock directly, bypassing reservation entirely.
pullLocked also minted its own independent operation id rather than
using a reserved attempt's, breaking the "one operation id spans an
attempt and its stage evidence" invariant.

pullLocked, applyWithSharedLock, applyLocked, and applyLockedBody now
accept an optional operation id, using it in place of their own default
when a caller supplies one. pull() and apply() reserve a durable
attempt and coalesce with a concurrent call to themselves whenever a
real GitOps application exists for the stack, the same definition
pullLocked itself already used to decide whether it has any GitOps
bookkeeping to do at all. handleWebhookPull()'s fetch step and its
conditional auto-apply step reserve too, without a coalescing map:
the route's own debounce window plus its single per-stack lock
acquisition already prevent a concurrent duplicate from reaching that
point, so there is nothing to coalesce there.

Fixed along the way: apply()'s coalesce key computed its deploy flag
differently than the code that actually executed the apply, so two
concurrent applies that genuinely differed in deploy behavior could
share a key and one could silently receive the other's result; deploy
is now resolved once, the same way applyLockedBody itself resolves it,
before it drives either the key or the execution. A policy-bypassing
apply is now routed through a fresh, never-shared coalescing map,
since bypassPolicy changes behavior but was not part of the key. A
coalesced follower's own reservation is now settled in a finally
rather than only after a successful await, so a rejecting leader
(the ordinary path for a producer that preserves its own throw
contract, unlike reconcile()'s internal error handling) no longer
leaves the follower's attempt open until the next restart. A
reservation bookkeeping failure no longer turns a manual pull or apply
that would otherwise have succeeded into a hard failure; it logs and
falls through to unreserved execution instead.

* fix(gitops): make the boot sweep consult claimant pointers before reaping a candidate

The boot-time managed-area sweep decided whether to delete a staged
candidate directory purely from file age and a completeness marker,
with no awareness of the database. A fresh audit gave the concrete
failure: a reconcile stages a candidate, the process crashes before
settlement, the installation stays down more than a day, startup
settles the attempt from a snapshot rather than real stage evidence,
and the sweep then deletes the still-needed candidate out from under
it.

sweepManagedArea now takes the set of candidate directory basenames
still referenced by the stack, and never reaps one of them regardless
of age or completeness. GitSourceService computes that set from three
independent sources: the live application's current candidate
generation, its accepted-but-not-yet-promoted generation (the
sourceAccepted-committed, targetApplied-not-yet-committed window; that
path has no production caller yet, so this is forward-looking coverage
for it), and the pending fetch record's own candidate reference, which
is written outside the transaction that mints a generation and can
therefore be the only claimant for a candidate that failed validation
or was staged while no live application existed to read a pointer
from.

* fix(gitops): fail closed on reservation failure and a torn-down application

A fresh audit found that a reservation-bookkeeping failure (a transient
DB error, an application torn down in the window between resolving it
and reserving against it) fell through to unreserved execution. That
directly defeated this delivery's own purpose: a manual apply could
still promote Compose files and deploy with zero durable record of it
happening. Reservation failure now fails closed for pull, apply, and
the webhook route: the operation is refused, logged, and recorded to
the stack's own activity history, distinguishing a torn-down
application (never retryable) from a transient failure (worth
retrying).

Closing that hole surfaced a second, related gap: pull, apply, and the
webhook route only checked for a live (active) application before
deciding whether to reserve at all, so a stack whose GitOps tracking
was explicitly torn down while its Git source configuration survived
could still run fully untracked, the same failure mode reached a
different way. Refusing this case took two attempts to get right,
both caught by review before landing: the first version refused on
any tombstoned state, which would have permanently and unrecoverably
blocked pull and apply for an application deliberately tombstoned as
deleted while its config survives for a future rebuild, a state two
existing production paths produce on purpose; narrowing to detached
only still misfired on a routine, fully completed detach, since that
same operation deletes the source row in the same transaction, so the
refusal must also confirm the source row actually survived before
firing. Both are covered by regression tests now, alongside the
original reservation-failure fix.

* fix(gitops): unify fetch-intent coalescing across pull and reconcile

A manual pull and a concurrently poll-triggered reconcile for the same
application previously ran their own separate in-flight maps and could
each start a clone for the same fetch. They now share one coalescing
map, with reconcile's fetch path classifying its own outcome instead of
relying on generic row-state derivation, so a pre-transition failure the
row does not yet reflect is never durably recorded as a plain success.

Apply-intent coalescing stays producer-local for now; unifying it needs
deploy-failure awareness threaded into the shared settlement path first,
which is a separately scoped follow-up.

* feat(gitops): recognize a webhook delivery id for traceability

The real webhook trigger endpoint is generic and HMAC-signed, with no
delivery identity of its own. It now extracts one from a recognized
provider header (GitHub, GitLab, Bitbucket, or a generic fallback) when
the caller sends one, and threads it through to the git-pull execution
path as a plain traceability breadcrumb on failure logs.

Redelivery dedup is deliberately not implemented here: an earlier attempt
routed the delivery id through the durable attempt reservation itself,
which silently dropped a redelivery's history through the existing
dedupe index instead of recording it. Building real dedup needs settlement
to reflect classified outcomes rather than generic row-state derivation
for both a settled and a crashed-mid-flight prior attempt, which is a
wider, separately scoped change shared with the same gap already deferred
for apply-intent reconcile.

* fix(gitops): thread the reserved attempt's operation id into the pending fetch record

The pending fetch record stamped its own independent random operation id
instead of the reserved attempt's real one, so an apply falling back to it
(when it holds no reservation of its own) inherited an identity unrelated
to the fetch that actually produced the candidate. One id now spans
reservation, fetch, generation, and the pending record.

Also fixes the short "op" token rendered in activity log lines: a fixed
prefix stopped discriminating between attempts once operation ids became
structured (<applicationId>:attempt:<seq>), since the prefix is now the
same applicationId every time. A shared helper renders the actual
attempt-discriminating suffix instead, applied consistently across pull,
apply, and create so each event's logged identity matches what its own
durable history actually recorded.

* test(gitops): cover the reconcile-recovery-then-sweep startup ordering

Reconcile-attempt recovery and the managed-area sweep must run in that
fixed order before the source controller's own poll loop starts, but the
guarantee lived only in a comment inside startServer, a function with
roughly two dozen unrelated service initializations that makes it
impractical to exercise end to end in a test.

Extracted the two steps into their own function so they're directly
testable in isolation, without moving the source controller's own start
call: that stays exactly where it was, since pulling it earlier would
have crossed a separate, already-documented ordering requirement for
registry delivery recovery. A structural test guards the one property
that can't be covered by driving the function directly: that the real
startServer body still calls the extracted function before starting the
controller.

* fix(gitops): complete durable reconciliation execution

* fix(gitops): resolve static analysis findings
2026-09-08 12:41:30 -04:00

710 lines
29 KiB
TypeScript

import { Router, type Request, type Response } from 'express';
import { GitSourceService, type PublicGitSource } from '../services/GitSourceService';
import type { GitOpsRevisionProjection } from '../services/gitops/types';
import { GitProjectManifestService } from '../services/GitProjectManifestService';
import { FileSystemService } from '../services/FileSystemService';
import { DatabaseService } from '../services/DatabaseService';
import { CryptoService } from '../services/CryptoService';
import { checkPermission, requirePermission } from '../middleware/permissions';
import { classifySourceRow, satisfiesGitOpsRead } from '../services/gitops/readAuth';
import { NOT_APPLICABLE_REVISION, projectStackRevision, stackResourceSet } from '../helpers/gitopsResponse';
import { respondWithHistory } from '../helpers/gitopsHistoryPage';
import { invalidateNodeCaches } from '../helpers/cacheInvalidation';
import { parseComposeSelection, defaultEnvPath } from '../helpers/gitSourceSelection';
import { isValidGitSourcePath, isValidStackName } from '../utils/validation';
import { sendGitSourceError, webhookPullStatus } from '../utils/gitSourceHttp';
import { sanitizeForLog } from '../utils/safeLog';
import { parseStorableRepoUrl, repoUrlRejectionMessage } from '../services/gitops/repoIdentity';
import { REF_MAX_LEN } from '../services/git/nativeGitTransport';
import { validateCaBundlePem } from '../services/git/caBundle';
import { auditActorUsername } from '../helpers/auditActor';
import { assertSafeOutboundHostname, resolveSafeOutboundHostname, UnsafeOutboundTargetError } from '../utils/outboundTarget';
// Reasonable upper bounds so a caller cannot flood the service with huge
// payloads. Generous compared to anything a real Git provider emits.
// The branch bound comes from the transport that ultimately fetches the ref,
// so a branch this route accepts can never be refused later as too long.
const MAX_BRANCH_LENGTH = REF_MAX_LEN;
const MAX_ENV_PATH_LENGTH = 1024;
const MAX_TOKEN_LENGTH = 8192;
const MAX_SUSPEND_REASON_LENGTH = 512;
const MAX_WEBHOOK_DELIVERY_ID_LENGTH = 512;
/**
* Shared handler for the "browse repository" compose-file picker: validate the
* repo target, clone it, and list its files. `storedToken` (already decrypted)
* is reused when the request omits a token, so the edit-mode flow does not force
* re-entering a stored PAT.
*/
const MAX_DEPLOY_KEY_LENGTH = 16384;
const MAX_CA_BUNDLE_LENGTH = 65536;
async function handleBrowse(
req: Request,
res: Response,
storedToken: string | null,
storedDeployKey: string | null,
storedKnownHosts: string | null,
storedCaBundle: string | null,
): Promise<void> {
const { repo_url, branch, auth_type, token, deploy_key, ssh_known_hosts_entry, ca_bundle } = req.body ?? {};
if (typeof repo_url !== 'string' || !repo_url.trim()) {
res.status(400).json({ error: 'repo_url is required' });
return;
}
if (typeof branch !== 'string' || !branch.trim()) {
res.status(400).json({ error: 'A branch, tag, or commit SHA is required.' });
return;
}
const repoUrlError = repoUrlRejectionMessage(repo_url);
if (repoUrlError) {
res.status(400).json({ error: repoUrlError });
return;
}
const parsedRepo = parseStorableRepoUrl(repo_url);
if (!parsedRepo.ok) {
res.status(400).json({ error: 'Repository URL is invalid' });
return;
}
const repoHostname = parsedRepo.kind === 'https' ? parsedRepo.url.hostname : parsedRepo.ssh.host;
try {
await assertSafeOutboundHostname(repoHostname);
} catch (error: unknown) {
if (error instanceof UnsafeOutboundTargetError) {
res.status(400).json({
error: error.reason === 'blocked'
? 'Repository host is not allowed'
: 'Repository host could not be resolved',
});
return;
}
throw error;
}
if (branch.length > MAX_BRANCH_LENGTH) {
res.status(400).json({ error: 'The branch, tag, or commit SHA is too long.' });
return;
}
if (auth_type !== undefined && auth_type !== 'none' && auth_type !== 'token' && auth_type !== 'deploy_key') {
res.status(400).json({ error: 'auth_type must be "none", "token", or "deploy_key"' });
return;
}
if (typeof token === 'string' && token.length > MAX_TOKEN_LENGTH) {
res.status(400).json({ error: 'token is too long' });
return;
}
if (typeof deploy_key === 'string' && deploy_key.length > MAX_DEPLOY_KEY_LENGTH) {
res.status(400).json({ error: 'deploy_key is too long' });
return;
}
if (typeof ca_bundle === 'string' && ca_bundle.length > MAX_CA_BUNDLE_LENGTH) {
res.status(400).json({ error: 'ca_bundle is too long' });
return;
}
const explicitToken = typeof token === 'string' && token.trim() ? token : null;
const effectiveToken = auth_type === 'token' ? (explicitToken ?? storedToken) : null;
const explicitDeployKey = typeof deploy_key === 'string' && deploy_key.trim() ? deploy_key : null;
const effectiveDeployKey = auth_type === 'deploy_key' ? (explicitDeployKey ?? storedDeployKey) : null;
const effectiveKnownHosts = auth_type === 'deploy_key'
? (typeof ssh_known_hosts_entry === 'string' && ssh_known_hosts_entry.trim()
? ssh_known_hosts_entry.trim()
: storedKnownHosts)
: null;
const explicitCaBundle = typeof ca_bundle === 'string' && ca_bundle.trim() ? ca_bundle.trim() : null;
const effectiveCaBundle = explicitCaBundle ?? storedCaBundle;
if (explicitCaBundle && !validateCaBundlePem(explicitCaBundle)) {
res.status(400).json({ error: 'ca_bundle must contain one or more PEM certificates' });
return;
}
const listParams: {
repoUrl: string;
branch: string;
token?: string | null;
sshAuth?: { privateKey: string; knownHostsEntry: string };
caBundlePem?: string | null;
} = {
repoUrl: repo_url.trim(),
branch: branch.trim(),
};
if (auth_type === 'token') {
listParams.token = effectiveToken;
} else if (auth_type === 'deploy_key' && effectiveDeployKey && effectiveKnownHosts) {
listParams.sshAuth = { privateKey: effectiveDeployKey, knownHostsEntry: effectiveKnownHosts };
}
if (effectiveCaBundle) {
listParams.caBundlePem = effectiveCaBundle;
}
try {
const result = await GitSourceService.getInstance().listRepoTree(listParams);
res.json(result);
} catch (error) {
sendGitSourceError(res, error);
}
}
/** Router for listing git-source configuration: `GET /api/git-sources`. */
export const gitSourcesRouter = Router();
gitSourcesRouter.post('/ssh-host-key', async (req: Request, res: Response): Promise<void> => {
const { repo_url, stack_name } = req.body ?? {};
if (typeof stack_name === 'string' && stack_name.trim()) {
if (!isValidStackName(stack_name.trim())) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stack_name.trim())) return;
} else if (!requirePermission(req, res, 'stack:create')) {
return;
}
if (typeof repo_url !== 'string' || !repo_url.trim()) {
res.status(400).json({ error: 'repo_url is required' });
return;
}
const repoUrlError = repoUrlRejectionMessage(repo_url);
if (repoUrlError) {
res.status(400).json({ error: repoUrlError });
return;
}
try {
const { parseSshUrl, scanHostKeys } = await import('../services/git/sshTrust');
const parsed = parseSshUrl(repo_url.trim());
if (!parsed) {
res.status(400).json({ error: 'Host key probe requires an SSH repository URL' });
return;
}
const [{ address }] = await resolveSafeOutboundHostname(parsed.host);
const keys = await scanHostKeys(parsed.host, parsed.port, address);
res.json({
host: parsed.host,
port: parsed.port,
keys,
});
} catch (error) {
if (error instanceof UnsafeOutboundTargetError) {
res.status(400).json({
error: error.reason === 'blocked'
? 'Repository host is not allowed'
: 'Repository host could not be resolved',
});
return;
}
sendGitSourceError(res, error);
}
});
gitSourcesRouter.get('/', async (req: Request, res: Response): Promise<void> => {
try {
const all = GitSourceService.getInstance().list();
const present = await stackResourceSet(req.nodeId);
// Filter to the subset of stacks the caller can read. Keeps scoped
// roles from discovering git config for stacks outside their grant.
// A row we cannot tie to a live, on-disk stack falls back to Admin, so a
// source whose application is missing or half-created is never authorized
// by a stack grant that may since have been reassigned.
const visible: Array<PublicGitSource & {
gitopsRevision: GitOpsRevisionProjection;
stackResourcePresent: boolean;
}> = [];
for (const src of all) {
const gitopsRevision = projectStackRevision(src.stack_name);
const stackResourcePresent = present.has(src.stack_name);
const requirement = classifySourceRow({
stackName: src.stack_name,
gitopsRevision,
stackResourcePresent,
});
if (!satisfiesGitOpsRead(req, requirement)) continue;
visible.push({ ...src, gitopsRevision, stackResourcePresent });
}
res.json(visible);
} catch (error) {
sendGitSourceError(res, error);
}
});
/**
* Cross-stack GitOps history for this instance.
*
* Every row is authorized on its own, so this returns the operator's own
* stacks for a scoped role and every row on this instance for an Admin.
* History is instance-local: a remote node's rows are read by proxying this
* same route to that node.
*/
gitSourcesRouter.get('/history', async (req: Request, res: Response): Promise<void> => {
try {
await respondWithHistory(req, res, { kind: 'per_row' });
} catch (error) {
sendGitSourceError(res, error);
}
});
// Create-mode repo browse (no stack yet): gated by the same permission as
// creating a stack from Git.
gitSourcesRouter.post('/browse', async (req: Request, res: Response): Promise<void> => {
if (!requirePermission(req, res, 'stack:create')) return;
await handleBrowse(req, res, null, null, null, null);
});
/**
* Router for per-stack git-source endpoints. Mount at `/api/stacks` so the
* `/:stackName/git-source*` paths work alongside other stack-scoped routes
* (such as the label-assignments router extracted in Phase 4A-1).
*/
export const stackGitSourceRouter = Router();
stackGitSourceRouter.get('/:stackName/git-source', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
try {
const gitSources = GitSourceService.getInstance();
const source = gitSources.get(stackName);
// Only this instance can say whether the stack's directory is really here,
// so the answer travels with the response rather than being inferred by a
// hub that has never seen the filesystem.
const stackResourcePresent = (await stackResourceSet(req.nodeId)).has(stackName);
if (source) {
// The managed-project manifest summary rides the source branch; the
// unlinked {linked:false} shape below is unchanged. Heal-on-read may
// rewrite the DB cache, so re-read the flat row for same-response parity.
const manifest = await gitSources.getManifestSummary(stackName);
const refreshed = gitSources.get(stackName) ?? source;
res.json({
...refreshed,
manifest_state: manifest?.state ?? refreshed.manifest_state,
manifest,
gitopsRevision: projectStackRevision(stackName),
stackResourcePresent,
});
return;
}
// No source row. A non-existent stack is a genuine 404, but an existing
// stack with no Git source attached is a normal, non-error state. The
// dashboard probes this endpoint for every stack, so returning 404 here
// would paint a console error for every unlinked stack; answer 200 with
// a discriminator instead and reserve 404 for the stack-not-found case.
if (!stackResourcePresent) {
res.status(404).json({ error: 'Stack not found' });
return;
}
res.json({
linked: false,
gitopsRevision: NOT_APPLICABLE_REVISION,
stackResourcePresent,
});
} catch (error) {
sendGitSourceError(res, error);
}
});
/**
* GitOps history for one stack.
*
* The stack read below covers the application holding this name now. Rows from
* an application that held it earlier are a different resource and are
* authorized per row, so a reused stack name cannot expose its predecessor.
*/
stackGitSourceRouter.get('/:stackName/git-source/history', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
try {
await respondWithHistory(req, res, { kind: 'authorized_stack', stackName });
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.put('/:stackName/git-source', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
const {
repo_url,
branch,
sync_env,
env_path,
auth_type,
token,
deploy_key,
ssh_known_hosts_entry,
ssh_host_key_fingerprint,
ca_bundle,
remove_ca_bundle,
auto_apply_on_webhook,
auto_deploy_on_apply,
} = req.body ?? {};
if (typeof repo_url !== 'string' || !repo_url.trim()) {
res.status(400).json({ error: 'repo_url is required' });
return;
}
if (typeof branch !== 'string' || !branch.trim()) {
res.status(400).json({ error: 'A branch, tag, or commit SHA is required.' });
return;
}
const selection = parseComposeSelection(req.body);
if (!selection.ok) {
res.status(400).json({ error: selection.error });
return;
}
if (auth_type !== 'none' && auth_type !== 'token' && auth_type !== 'deploy_key') {
res.status(400).json({ error: 'auth_type must be "none", "token", or "deploy_key"' });
return;
}
if (auto_apply_on_webhook !== undefined && typeof auto_apply_on_webhook !== 'boolean') {
res.status(400).json({ error: 'auto_apply_on_webhook must be a boolean' });
return;
}
if (auto_deploy_on_apply !== undefined && typeof auto_deploy_on_apply !== 'boolean') {
res.status(400).json({ error: 'auto_deploy_on_apply must be a boolean' });
return;
}
const repoUrlError = repoUrlRejectionMessage(repo_url);
if (repoUrlError) {
res.status(400).json({ error: repoUrlError });
return;
}
if (branch.length > MAX_BRANCH_LENGTH) {
res.status(400).json({ error: 'The branch, tag, or commit SHA is too long.' });
return;
}
if (typeof env_path === 'string' && env_path.length > MAX_ENV_PATH_LENGTH) {
res.status(400).json({ error: 'env_path is too long' });
return;
}
if (typeof env_path === 'string' && env_path.trim() && !isValidGitSourcePath(env_path.trim())) {
res.status(400).json({ error: 'env_path must be a relative repository file path' });
return;
}
if (typeof token === 'string' && token.length > MAX_TOKEN_LENGTH) {
res.status(400).json({ error: 'token is too long' });
return;
}
if (typeof deploy_key === 'string' && deploy_key.length > MAX_DEPLOY_KEY_LENGTH) {
res.status(400).json({ error: 'deploy_key is too long' });
return;
}
if (typeof ca_bundle === 'string' && ca_bundle.length > MAX_CA_BUNDLE_LENGTH) {
res.status(400).json({ error: 'ca_bundle is too long' });
return;
}
if (typeof ca_bundle === 'string' && ca_bundle.trim() && !validateCaBundlePem(ca_bundle)) {
res.status(400).json({ error: 'ca_bundle must contain one or more PEM certificates' });
return;
}
if (remove_ca_bundle !== undefined && typeof remove_ca_bundle !== 'boolean') {
res.status(400).json({ error: 'remove_ca_bundle must be a boolean' });
return;
}
const autoApplyOnWebhook = auto_apply_on_webhook === true;
const autoDeployOnApply = auto_deploy_on_apply === true;
if (autoDeployOnApply && !requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
// Confirm the stack actually exists on the active node. Without this guard
// a caller could stash a git-source row for a name that does not exist
// yet and have it auto-link when a stack with that name is later created.
const stacks = await FileSystemService.getInstance(req.nodeId).getStacks();
if (!stacks.includes(stackName)) {
res.status(404).json({ error: 'Stack not found' });
return;
}
const syncEnv = Boolean(sync_env);
const resolvedEnvPath = syncEnv
? defaultEnvPath(selection.value.composePaths[0], env_path)
: null;
const source = await GitSourceService.getInstance().upsert({
stackName,
repoUrl: repo_url.trim(),
branch: branch.trim(),
composePaths: selection.value.composePaths,
contextDir: selection.value.contextDir,
syncEnv,
envPath: resolvedEnvPath,
authType: auth_type,
token: typeof token === 'string' ? token : undefined,
deployKey: typeof deploy_key === 'string' ? deploy_key : undefined,
sshKnownHostsEntry: typeof ssh_known_hosts_entry === 'string' ? ssh_known_hosts_entry : undefined,
sshHostKeyFingerprint: typeof ssh_host_key_fingerprint === 'string' ? ssh_host_key_fingerprint : undefined,
caBundle: typeof ca_bundle === 'string' ? ca_bundle : undefined,
removeCaBundle: remove_ca_bundle === true,
autoApplyOnWebhook,
autoDeployOnApply,
auditContext: {
username: auditActorUsername(req),
method: req.method,
path: req.originalUrl,
ipAddress: req.ip || 'unknown',
},
});
// The cached /stacks/statuses payload carries the source label; drop it
// before responding so a client refetch on this response recomputes. The
// full invalidateNodeCaches helper is deliberate here (matching every
// other mutation route): link/unlink is a low-frequency user action, so
// dropping the project-name map and file-root allowlists alongside is
// harmless, unlike the high-frequency container-event path.
invalidateNodeCaches(req.nodeId);
console.log(`[GitSource] Configured git source for ${stackName}`);
res.json(source);
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.delete('/:stackName/git-source', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
// Detach with the export contract: the effective compose model is rendered
// into a single compose.yaml and the materialized files are kept, so a
// multi-file / project-directory stack stays deployable after unlinking.
// A render failure aborts with 409 and the row is left intact.
await GitSourceService.getInstance().detach(stackName);
// The cached /stacks/statuses payload carries the source label; drop it
// before responding so a client refetch on this response recomputes. The
// full invalidateNodeCaches helper is deliberate here (matching every
// other mutation route): link/unlink is a low-frequency user action, so
// dropping the project-name map and file-root allowlists alongside is
// harmless, unlike the high-frequency container-event path.
invalidateNodeCaches(req.nodeId);
console.log(`[GitSource] Detached git source for ${stackName}`);
res.json({ success: true });
} catch (error) {
const code = (error as { code?: string } | null)?.code;
if (code === 'RENDER_FAILED') {
res.status(409).json({ error: (error as Error).message });
return;
}
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.get('/:stackName/git-source/manifest', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:read', 'stack', stackName)) return;
try {
const manifest = await GitSourceService.getInstance().getManifest(stackName);
if (!manifest) {
res.status(404).json({ error: 'No managed-project manifest for this stack' });
return;
}
// The public projection: no content hashes, size metadata, provenance, or
// deletion authority, and high-sensitivity input paths are redacted.
res.json({ manifest: GitProjectManifestService.getInstance().toPublicManifest(manifest) });
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/pull', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
const result = await GitSourceService.getInstance().pull(stackName, {
actor: req.user?.username ?? 'unknown',
});
res.json(result);
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/apply', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
const { commitSha, deploy, planFingerprint } = req.body ?? {};
if (typeof commitSha !== 'string' || !commitSha.trim()) {
res.status(400).json({ error: 'commitSha is required' });
return;
}
if (typeof planFingerprint !== 'string' || !planFingerprint.trim()) {
res.status(400).json({ error: 'planFingerprint is required', code: 'PLAN_FINGERPRINT_REQUIRED' });
return;
}
const source = DatabaseService.getInstance().getGitSource(stackName);
const willDeploy = typeof deploy === 'boolean' ? deploy : source?.auto_deploy_on_apply === true;
if (willDeploy && !requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
const result = await GitSourceService.getInstance().apply(
stackName,
commitSha.trim(),
{
deploy: typeof deploy === 'boolean' ? deploy : undefined,
actor: req.user?.username ?? 'unknown',
bypassPolicy: req.query.ignorePolicy === 'true' && req.user?.role === 'admin',
planFingerprint: planFingerprint.trim(),
requirePlanFingerprint: true,
},
);
// Cache invalidation and the post-deploy scan now run inside GitSourceService.apply() itself.
const shortSha = commitSha.trim().slice(0, 7);
if (result.deployed) {
console.log('[GitSource] Applied commit %s to %s (deployed)', sanitizeForLog(shortSha), sanitizeForLog(stackName));
} else if (result.deployError) {
console.warn('[GitSource] Applied commit %s to %s, deploy failed: %s', sanitizeForLog(shortSha), sanitizeForLog(stackName), sanitizeForLog(result.deployError));
} else {
console.log('[GitSource] Applied commit %s to %s', sanitizeForLog(shortSha), sanitizeForLog(stackName));
}
res.json(result);
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/webhook-pull', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
const deliveryId = req.body?.deliveryId;
if (
deliveryId !== undefined
&& (typeof deliveryId !== 'string' || !deliveryId.trim() || deliveryId.length > MAX_WEBHOOK_DELIVERY_ID_LENGTH)
) {
res.status(400).json({ error: 'deliveryId must be a non-empty string of at most 512 characters' });
return;
}
try {
const service = GitSourceService.getInstance();
const source = service.get(stackName);
if (!source) {
res.status(404).json({ error: 'No Git source configured for this stack', status: 'error' });
return;
}
const normalizedDeliveryId = deliveryId?.trim();
const deployAuthorized = checkPermission(req, 'stack:deploy', 'stack', stackName);
if (service.webhookDeliveryRequiresDeploy(stackName, normalizedDeliveryId) && !deployAuthorized) {
requirePermission(req, res, 'stack:deploy', 'stack', stackName);
return;
}
const result = await service.handleWebhookPull(stackName, deployAuthorized, normalizedDeliveryId);
// Map the outcome to a real HTTP status so a Git provider sees a 4xx on
// failure instead of a 200 with an error body (which it would read as
// "delivered fine, stop retrying").
res.status(webhookPullStatus(result.status)).json(result);
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/dismiss-pending', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
GitSourceService.getInstance().dismissPending(stackName, req.user?.username ?? 'unknown');
res.json({ success: true });
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/suspend', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
const { reason: rawReason } = req.body ?? {};
const reason = typeof rawReason === 'string' ? rawReason : undefined;
if (reason !== undefined && reason.length > MAX_SUSPEND_REASON_LENGTH) {
res.status(400).json({ error: 'reason is too long' });
return;
}
try {
const result = await GitSourceService.getInstance().suspend(stackName, {
actor: req.user?.username ?? 'unknown',
reason,
});
res.json(result);
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/resume', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
const result = await GitSourceService.getInstance().resume(stackName, {
actor: req.user?.username ?? 'unknown',
});
res.json(result);
} catch (error) {
sendGitSourceError(res, error);
}
});
stackGitSourceRouter.post('/:stackName/git-source/retry', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
try {
const result = await GitSourceService.getInstance().retry(stackName, {
actor: req.user?.username ?? 'unknown',
});
res.json(result);
} catch (error) {
sendGitSourceError(res, error);
}
});
// Edit-mode repo browse for an existing stack: gated by stack:edit so a user who
// can edit (but not create) stacks can re-pick files, and reuses the stored token
// when the request omits one.
stackGitSourceRouter.post('/:stackName/git-source/browse', async (req: Request, res: Response): Promise<void> => {
const stackName = req.params.stackName as string;
if (!isValidStackName(stackName)) {
res.status(400).json({ error: 'Invalid stack name' });
return;
}
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
const src = DatabaseService.getInstance().getGitSource(stackName);
const storedToken = src?.encrypted_token ? CryptoService.getInstance().decrypt(src.encrypted_token) : null;
const storedDeployKey = src?.encrypted_deploy_key ? CryptoService.getInstance().decrypt(src.encrypted_deploy_key) : null;
const storedKnownHosts = src?.ssh_known_hosts_entry ?? null;
const storedCaBundle = src?.encrypted_ca_bundle ? CryptoService.getInstance().decrypt(src.encrypted_ca_bundle) : null;
await handleBrowse(req, res, storedToken, storedDeployKey, storedKnownHosts, storedCaBundle);
});