mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-09-09 17:05:51 +00:00
feat(gitops): durable reconcile-attempt reservation, coalescing, and recovery (#1893)
* feat(gitops): add reconcile trigger normalization and coalescing keys The source controller needs a single normalized shape for every trigger that can request evaluation (manual, API, webhook, poll, retry, config change, startup, resume, and future provider/schedule/binding-change producers), and a way to decide whether two concurrent submissions describe the same work. Add ReconcileTrigger, the discriminated ReconcileRequest (fetch vs apply), and coalesceKey/deliveryKey. A fetch coalesces by application alone, since it has one live outcome regardless of trigger; an apply coalesces only when the commit, plan fingerprint, and deploy choice all match, so two applies that differ in any of those can never be joined and have one silently receive the other's result. * feat(gitops): add exhaustive failure classification and retry backoff The controller needs to tell a transient network condition from a permanent configuration one from a broad range of Git-source, policy, and target failures, since several distinct causes collapse onto the same public error code and a wrong call either retries forever on a bad URL or gives up on a DNS blip. Add classifyFailure, built on two total lookup records (one keyed by TransportFailureReason, one by GitSourceErrorCode) so adding a new value to either source union fails the build until this classifier accounts for it, rather than silently defaulting. A tip-changed race classifies as supersession, not backoff; an unrecognized exit-coded git error gets a lower retry ceiling than a plain network timeout; a deploy or health failure after a successful apply is its own class that must never refetch or reapply. Add nextRetryAt: bounded exponential backoff (60s doubling, capped at one hour) with +-10% jitter, honoring a provider-supplied retry floor when it is larger than the computed delay. * feat(gitops): derive normalized reconcile outcomes from the source facet Silence is not an acceptable GitOps result: every reconcile attempt needs to settle into one named outcome an operator can act on, not a bare success/failure. Deriving that outcome from the existing source-facet projection (rather than a second, parallel status source) is what keeps "no source change" from silently collapsing into "converged" the way a bare commit-SHA comparison would. Add outcomeFromSourceFacet, exhaustive over all 17 SourceFacet statuses, each mapped to a ReconcileOutcome, a human reason, and a next action (review, resume, retry, resolve_conflict, configure_credentials, view_target_results, or none). converged is deliberately never produced here: it requires target and health evidence this source-only projection does not carry, so a later composition over source + target + health is the only thing allowed to report it. * feat(gitops): add the portable accepted-generation contract Direct and Blueprint dispatch need to consume the exact same description of what an accepted generation contains, without either side inventing a node id, local candidate path, target project name, or placement field into it, since that is exactly the kind of drift that would let a stale acceptance authorize a routing decision made after it. Add six additive, nullable columns to gitops_generations (portable manifest, Compose inputs, source/security policy evidence, support and compatibility requirements), all decoded honestly: a legacy row missing one records an explicit limitation rather than inventing evidence, and a legacy pending candidate lacking the new contract must be re-evaluated before it can be accepted or dispatched. Add gitops/handoff.ts: the AcceptedGeneration type built from a generation row via buildAcceptedGeneration, a compile-time assertion that the contract cannot carry a target-mode-specific field, and the TargetAdapter boundary with BlueprintTargetAdapter failing closed until Blueprint rollout orchestration exists. Current target mode and binding travel separately in DispatchContext, re-read under the dispatch lock rather than carried on the generation itself. * feat(gitops): add controller-owned bookkeeping columns to gitops_applications The source controller needs somewhere to persist source policy, poll cadence, and a durable attempt sequence per application, and it needs to work identically for Direct and Blueprint applications. stack_git_sources (the existing home for auto_apply_on_webhook/auto_deploy_on_apply) is Direct-only and keyed by stack name, so it cannot represent a Blueprint application at all. Add source_policy (manual|review|automatic, default manual), poll_interval_secs (NULL inherits the global default, 0 disables), next_poll_at (the durable scheduling cursor), and attempt_seq to gitops_applications instead. All default to values that start no unattended work: an upgraded installation begins polling nothing and stays on manual policy until explicitly migrated or configured. * feat(gitops): add durable attempt reservation and poll/retry queries History dedupe alone is not execution idempotency: mutateApp writes the application row and only then inserts history, so a dedupe conflict still commits the row write. The controller needs a reservation that runs before any side effect and touches nothing else, so a duplicate or restarted submission can be told apart from new work without repeating it. Add reserveReconcileAttempt/settleReconcileAttempt: a bare history insert in its own transaction, deliberately not through mutateApp, using the existing history dedupe index as the reservation/idempotency check itself. A repeated reservation or settlement for the same operation is a no-op, never overwriting the first settled result. Add the store queries a controller needs to drive this: getSettledAttempt and latestSettledAttempt (exact vs. most-recent lookup, the latter tie-broken by rowid since the id column is a random UUID unrelated to recency), listUnsettledReconcileAttempts (a reservation with no matching settled row, for startup recovery), and listSourcesDueForPoll / listApplicationsDueForRetry (excluding suspended, in-flight, and Blueprint-mode applications). * feat(gitops): add controller-facing reconcile entry point to GitSourceService Adds GitSourceService.reconcile(), a single normalized entry point that takes a fetch or apply request and returns a normalized outcome, next action, and reason instead of a thrown error or a raw boolean. It owns the git mutex for the whole evaluation and calls the same private fetch/apply bodies the existing pull()/apply() routes use, so it never nests locks. Resolves the live application for the stack first and fails closed, without attempting any work, when the request's application id no longer matches it or when no application exists at all. A fetch or apply failure that the underlying transition never persisted (missing config, a stale commit, lock contention) is classified through the existing retry disposition table so an unretryable failure is never reported as retryable, and a stale-looking success is never reported in place of a real failure. * feat(gitops): dispatch accepted generations to their target Adds GitSourceService.dispatchAcceptedGeneration(), which routes a portable accepted-generation contract to its target: Blueprint mode delegates to the existing BlueprintTargetAdapter (rollout orchestration is not built yet, so it always blocks), and Direct mode dispatches by driving reconcile() with the generation's commit, since there is no separate generation-based promotion pipeline yet. The deploy flag is read from the stack's own auto_deploy_on_apply setting, matching how every other producer decides it. Also fixes a case reconcile() got wrong: when a promotion succeeds but the following deploy fails, the source itself has genuinely changed, so reporting it as an unchanged failure was as untruthful as reporting it a plain success would have been. It now reports a result that neither claims, pointing at the target instead of asking for a source retry that could never succeed. * feat(gitops): add background poll and retry driver for source reconciliation Adds SourceController, a self-rescheduling background timer that finds sources whose poll interval or retry time has arrived and evaluates each through the existing reconcile() entry point. A per-application in-flight set is what keeps one slow evaluation from blocking the rest of the fleet; the tick never waits on any evaluation before scheduling the next one, and a scan that throws still reschedules rather than stopping the driver permanently. This delivery only covers detection: a tick issues a fetch, the same step a manual pull performs, so a new candidate is staged for review but not automatically accepted or dispatched, and a retry re-issues a plain fetch rather than resuming whatever stage previously failed. Both are documented as open follow-on work rather than silent gaps. Not yet wired into the process lifecycle; that follows separately. * feat(gitops): start and stop the source reconciliation driver with the process Wires SourceController into the background-service lifecycle: it starts after the existing GitOps recovery and orphan-sweep steps, so it never scans an application still carrying a stale in-progress marker from a killed process, and stops alongside every other background timer on shutdown. * feat(gitops): add suspend, resume, and explicit retry for a source Adds GitSourceService.suspend()/resume()/retry(), the service-layer methods behind an upcoming suspend/resume/retry control surface. Suspending a source now genuinely stops it: pullLocked and applyLockedBody both check suspension before doing any work, not just inside the transition bookkeeping, which used to reject but then let the fetch or apply proceed anyway. A refused suspend surfaces as a real error rather than a silent no-op, since an operator believing a source is suspended when it isn't is the exact failure this exists to prevent; a refused resume stays silent, since the row read back after the attempt already reports the true state either way. A webhook delivery to a suspended source is reported as skipped rather than a failed pull, so a long suspension does not read to the Git host as a broken webhook. * feat(gitops): add suspend, resume, and retry routes for a git source Adds POST endpoints for suspending a source (with an optional, length- capped reason), resuming it, and explicitly retrying it, wired to the existing service-layer methods behind the same stack:edit permission as pull and apply. Neither route can trigger a deploy today, so unlike apply and webhook-pull they need no conditional stack:deploy check. * fix(gitops): close path-injection gaps at two candidate-file sinks Adds the inline resolve-and-prefix-check barrier this codebase already uses at other filesystem sinks derived from a stack name or a stored candidate path, closing two sinks that lacked it: the manifest write in GitProjectManifestService, and the pending-candidate access check before promoting an apply. The apply-side check now shares the same strict candidate-path validator the registry-delivery path already uses on the identical stored field, rather than a looser check, so both call sites treat a tampered candidate reference the same way. Fixed two test fixtures that had never matched the shape a real candidate path takes, which the stricter check would otherwise have rejected. * fix(gitops): confine the manifest write directory to the managed area The prior fix checked the manifest write/rename targets against the resolved managed directory, but left the mkdir call on that directory itself unconfined and checked it against the data root rather than the narrower managed area every sibling barrier in this file uses. Aligns it with the established convention and the actual boundary that matters. * feat(gitops): wire durable attempt reservation and coalescing into reconcile() reconcile() now reserves a durable attempt before doing any fetch or apply and settles it with the normalized result once execution finishes, using the reservation and settlement primitives that already existed but had no caller. A request carrying a stable external delivery id reserves under a producer- and intent-namespaced key, so a redelivery reuses the same attempt instead of minting a second one; a request with no such identity gets a freshly allocated attemptSeq- based id, bumped atomically with its reservation. Concurrent submissions that would do the same work now coalesce: the first becomes the leader and actually runs, and any submission that joins while it is still in flight awaits the leader's real result instead of running a duplicate fetch or apply. Each still gets its own durable attempt and its own settled row, including a concurrent redelivery of the same external event, which joins the in-flight leader rather than falling back to a snapshot of the row from before the leader's work landed. Startup gains an attempt-recovery phase, run before the managed-area sweep and before SourceController starts: every attempt reserved but never settled, most likely from a crash between the two, is resolved from durable state without re-executing anything. One row failing to recover no longer blocks the rest; it is skipped and logged, and recovery keeps paging until nothing unsettled remains. A settlement failure is caught and logged rather than turning an already-successful fetch or apply into a thrown error for the caller, and a settled attempt's stored result is now decoded through a validated outcome/next-action check instead of a blind cast, logging rather than silently reporting unknown when a stored row is corrupt or unreadable. * fix(gitops): remove unused store variable from coalescing test * fix(gitops): make reconcile-attempt recovery leader-aware and cursor-paginated A fresh audit found real gaps in the reservation/coalescing wiring from the previous commit: recovery paged by "still unsettled" status rather than a cursor, so once a permanently unrecoverable row occupied every slot in a page, every genuinely recoverable row beyond it was silently never reached; a follower's outcome was reconstructed independently from row state rather than from its leader's actual stored result, so a leader and its follower could durably disagree; and an attempt resolved on the live path was returned but never actually settled, leaving it open indefinitely until the next restart. listUnsettledReconcileAttempts now takes an optional (created_at, id) cursor, matching the pagination shape queryHistoryRows already uses, so paging always advances regardless of which rows settle. Recovery is now two-pass: independent attempts settle first from row state, followers are deferred, then each deferred follower settles from its leader's now-settled result. A follower whose leader is a real but still-unresolved reservation is left unsettled rather than guessed at independently, since the leader could still settle to something else later, including when the leader simply failed to settle in this same pass rather than "never will". The same leader-aware resolution now backs the live reconcile() path too, via a shared helper, so an already-reserved attempt is durably settled instead of merely returning a value. Also fixes a narrower race in reconcile() itself: two submissions can share an operation id (a stable external delivery id) while running under different coalesce keys, since an apply's coalesce key includes its commit sha, plan fingerprint, and deploy flag, which the delivery id does not carry. reconcile() now checks in-process executions by operation id directly before falling back to a durable-state resolution, so such a submission joins the real in-flight leader instead of settling a stale pre-execution snapshot ahead of it. * feat(gitops): route pull, apply, and webhook producers through reservation A fresh audit found the same "primitive built, real caller does not use it" pattern one layer deeper than the previous commit fixed: reconcile(), the controller-facing entry point, reserved and coalesced durable attempts correctly, but the actual production producers, the manual pull button, the manual apply button, and the webhook route, all called pullLocked/applyWithSharedLock directly, bypassing reservation entirely. pullLocked also minted its own independent operation id rather than using a reserved attempt's, breaking the "one operation id spans an attempt and its stage evidence" invariant. pullLocked, applyWithSharedLock, applyLocked, and applyLockedBody now accept an optional operation id, using it in place of their own default when a caller supplies one. pull() and apply() reserve a durable attempt and coalesce with a concurrent call to themselves whenever a real GitOps application exists for the stack, the same definition pullLocked itself already used to decide whether it has any GitOps bookkeeping to do at all. handleWebhookPull()'s fetch step and its conditional auto-apply step reserve too, without a coalescing map: the route's own debounce window plus its single per-stack lock acquisition already prevent a concurrent duplicate from reaching that point, so there is nothing to coalesce there. Fixed along the way: apply()'s coalesce key computed its deploy flag differently than the code that actually executed the apply, so two concurrent applies that genuinely differed in deploy behavior could share a key and one could silently receive the other's result; deploy is now resolved once, the same way applyLockedBody itself resolves it, before it drives either the key or the execution. A policy-bypassing apply is now routed through a fresh, never-shared coalescing map, since bypassPolicy changes behavior but was not part of the key. A coalesced follower's own reservation is now settled in a finally rather than only after a successful await, so a rejecting leader (the ordinary path for a producer that preserves its own throw contract, unlike reconcile()'s internal error handling) no longer leaves the follower's attempt open until the next restart. A reservation bookkeeping failure no longer turns a manual pull or apply that would otherwise have succeeded into a hard failure; it logs and falls through to unreserved execution instead. * fix(gitops): make the boot sweep consult claimant pointers before reaping a candidate The boot-time managed-area sweep decided whether to delete a staged candidate directory purely from file age and a completeness marker, with no awareness of the database. A fresh audit gave the concrete failure: a reconcile stages a candidate, the process crashes before settlement, the installation stays down more than a day, startup settles the attempt from a snapshot rather than real stage evidence, and the sweep then deletes the still-needed candidate out from under it. sweepManagedArea now takes the set of candidate directory basenames still referenced by the stack, and never reaps one of them regardless of age or completeness. GitSourceService computes that set from three independent sources: the live application's current candidate generation, its accepted-but-not-yet-promoted generation (the sourceAccepted-committed, targetApplied-not-yet-committed window; that path has no production caller yet, so this is forward-looking coverage for it), and the pending fetch record's own candidate reference, which is written outside the transaction that mints a generation and can therefore be the only claimant for a candidate that failed validation or was staged while no live application existed to read a pointer from. * fix(gitops): fail closed on reservation failure and a torn-down application A fresh audit found that a reservation-bookkeeping failure (a transient DB error, an application torn down in the window between resolving it and reserving against it) fell through to unreserved execution. That directly defeated this delivery's own purpose: a manual apply could still promote Compose files and deploy with zero durable record of it happening. Reservation failure now fails closed for pull, apply, and the webhook route: the operation is refused, logged, and recorded to the stack's own activity history, distinguishing a torn-down application (never retryable) from a transient failure (worth retrying). Closing that hole surfaced a second, related gap: pull, apply, and the webhook route only checked for a live (active) application before deciding whether to reserve at all, so a stack whose GitOps tracking was explicitly torn down while its Git source configuration survived could still run fully untracked, the same failure mode reached a different way. Refusing this case took two attempts to get right, both caught by review before landing: the first version refused on any tombstoned state, which would have permanently and unrecoverably blocked pull and apply for an application deliberately tombstoned as deleted while its config survives for a future rebuild, a state two existing production paths produce on purpose; narrowing to detached only still misfired on a routine, fully completed detach, since that same operation deletes the source row in the same transaction, so the refusal must also confirm the source row actually survived before firing. Both are covered by regression tests now, alongside the original reservation-failure fix. * fix(gitops): unify fetch-intent coalescing across pull and reconcile A manual pull and a concurrently poll-triggered reconcile for the same application previously ran their own separate in-flight maps and could each start a clone for the same fetch. They now share one coalescing map, with reconcile's fetch path classifying its own outcome instead of relying on generic row-state derivation, so a pre-transition failure the row does not yet reflect is never durably recorded as a plain success. Apply-intent coalescing stays producer-local for now; unifying it needs deploy-failure awareness threaded into the shared settlement path first, which is a separately scoped follow-up. * feat(gitops): recognize a webhook delivery id for traceability The real webhook trigger endpoint is generic and HMAC-signed, with no delivery identity of its own. It now extracts one from a recognized provider header (GitHub, GitLab, Bitbucket, or a generic fallback) when the caller sends one, and threads it through to the git-pull execution path as a plain traceability breadcrumb on failure logs. Redelivery dedup is deliberately not implemented here: an earlier attempt routed the delivery id through the durable attempt reservation itself, which silently dropped a redelivery's history through the existing dedupe index instead of recording it. Building real dedup needs settlement to reflect classified outcomes rather than generic row-state derivation for both a settled and a crashed-mid-flight prior attempt, which is a wider, separately scoped change shared with the same gap already deferred for apply-intent reconcile. * fix(gitops): thread the reserved attempt's operation id into the pending fetch record The pending fetch record stamped its own independent random operation id instead of the reserved attempt's real one, so an apply falling back to it (when it holds no reservation of its own) inherited an identity unrelated to the fetch that actually produced the candidate. One id now spans reservation, fetch, generation, and the pending record. Also fixes the short "op" token rendered in activity log lines: a fixed prefix stopped discriminating between attempts once operation ids became structured (<applicationId>:attempt:<seq>), since the prefix is now the same applicationId every time. A shared helper renders the actual attempt-discriminating suffix instead, applied consistently across pull, apply, and create so each event's logged identity matches what its own durable history actually recorded. * test(gitops): cover the reconcile-recovery-then-sweep startup ordering Reconcile-attempt recovery and the managed-area sweep must run in that fixed order before the source controller's own poll loop starts, but the guarantee lived only in a comment inside startServer, a function with roughly two dozen unrelated service initializations that makes it impractical to exercise end to end in a test. Extracted the two steps into their own function so they're directly testable in isolation, without moving the source controller's own start call: that stays exactly where it was, since pulling it earlier would have crossed a separate, already-documented ordering requirement for registry delivery recovery. A structural test guards the one property that can't be covered by driving the function directly: that the real startServer body still calls the extracted function before starting the controller. * fix(gitops): complete durable reconciliation execution * fix(gitops): resolve static analysis findings
This commit is contained in:
Anso
@@ -0,0 +1,99 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, afterEach, vi } from 'vitest';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
|
||||
let tmpDir: string;
|
||||
let runGitOpsSourceRecovery: typeof import('../bootstrap/startup').runGitOpsSourceRecovery;
|
||||
let GitSourceService: typeof import('../services/GitSourceService').GitSourceService;
|
||||
let gitSourceServiceModule: typeof import('../services/GitSourceService');
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ runGitOpsSourceRecovery } = await import('../bootstrap/startup'));
|
||||
gitSourceServiceModule = await import('../services/GitSourceService');
|
||||
({ GitSourceService } = gitSourceServiceModule);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
describe('runGitOpsSourceRecovery', () => {
|
||||
it('recovers unsettled reconcile attempts before sweeping the managed area', async () => {
|
||||
const order: string[] = [];
|
||||
// Recovery yields before recording itself: if the sweep were ever
|
||||
// started concurrently instead of strictly after recovery resolves,
|
||||
// the sweep's own synchronous push would land first and this would
|
||||
// catch it, rather than merely proving call order at invocation
|
||||
// time.
|
||||
vi.spyOn(GitSourceService.getInstance(), 'recoverUnsettledReconcileAttempts')
|
||||
.mockImplementation(async () => {
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
order.push('recover');
|
||||
});
|
||||
vi.spyOn(gitSourceServiceModule, 'sweepGitManifestOrphans')
|
||||
.mockImplementation(async () => { order.push('sweep'); });
|
||||
|
||||
await runGitOpsSourceRecovery();
|
||||
|
||||
expect(order).toEqual(['recover', 'sweep']);
|
||||
});
|
||||
|
||||
it('still runs the sweep when recovery itself throws, tolerating the failure', async () => {
|
||||
const order: string[] = [];
|
||||
vi.spyOn(GitSourceService.getInstance(), 'recoverUnsettledReconcileAttempts')
|
||||
.mockImplementation(async () => { throw new Error('simulated recovery failure'); });
|
||||
vi.spyOn(gitSourceServiceModule, 'sweepGitManifestOrphans')
|
||||
.mockImplementation(async () => { order.push('sweep'); });
|
||||
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
await expect(runGitOpsSourceRecovery()).resolves.toBeUndefined();
|
||||
|
||||
expect(order).toEqual(['sweep']);
|
||||
expect(errorSpy).toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('still resolves when the sweep itself throws, tolerating the failure rather than aborting startup', async () => {
|
||||
const recoverSpy = vi.spyOn(GitSourceService.getInstance(), 'recoverUnsettledReconcileAttempts')
|
||||
.mockImplementation(async () => {});
|
||||
vi.spyOn(gitSourceServiceModule, 'sweepGitManifestOrphans')
|
||||
.mockImplementation(async () => { throw new Error('simulated sweep failure'); });
|
||||
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
|
||||
await expect(runGitOpsSourceRecovery()).resolves.toBeUndefined();
|
||||
|
||||
expect(recoverSpy).toHaveBeenCalled();
|
||||
expect(warnSpy).toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
describe('startServer source ordering', () => {
|
||||
it('calls runGitOpsSourceRecovery before starting SourceController, so recovery and the sweep always precede the controller poll loop', () => {
|
||||
// A structural check, not a behavioral one: startServer drives ~25
|
||||
// unrelated services and isn't practical to run end to end in a
|
||||
// test. What matters here is the one property a future edit could
|
||||
// silently break: SourceController must never start before
|
||||
// recovery and the sweep have both awaited to completion. Comments
|
||||
// are stripped first so a mention of either symbol in prose can't
|
||||
// satisfy the match, and matching is whitespace/chaining-tolerant
|
||||
// so a harmless reformat (line-wrapped method chain, a `const`
|
||||
// extracted for the controller instance) doesn't false-fail this.
|
||||
const source = fs.readFileSync(path.join(__dirname, '../bootstrap/startup.ts'), 'utf-8');
|
||||
const startServerStart = source.indexOf('export async function startServer');
|
||||
// startServer is the last top-level declaration in this file today;
|
||||
// if that ever changes, bound this slice to its closing brace
|
||||
// instead of running to end of file.
|
||||
const startServerBody = source.slice(startServerStart);
|
||||
const code = startServerBody.replace(/\/\*[\s\S]*?\*\//g, '').replace(/\/\/[^\n]*/g, '');
|
||||
const reconcileCallIndex = code.search(/await\s+runGitOpsSourceRecovery\s*\(/);
|
||||
const controllerStartIndex = code.search(/SourceController\s*\.\s*getInstance\s*\(\s*\)\s*\.\s*start\s*\(/);
|
||||
expect(reconcileCallIndex).toBeGreaterThan(-1);
|
||||
expect(controllerStartIndex).toBeGreaterThan(-1);
|
||||
expect(reconcileCallIndex).toBeLessThan(controllerStartIndex);
|
||||
});
|
||||
});
|
||||
@@ -92,6 +92,7 @@ function makeClone(files: Record<string, string>): string {
|
||||
}
|
||||
|
||||
const REPO = { repo_url: 'https://github.com/example/repo.git', branch: 'main' };
|
||||
const NO_CANDIDATE_CLAIMS = { complete: true as const, dirs: new Set<string>() };
|
||||
|
||||
function seedGitSource(stackName: string): void {
|
||||
DatabaseService.getInstance().upsertGitSource({
|
||||
@@ -731,6 +732,59 @@ describe('promoteGeneration', () => {
|
||||
});
|
||||
|
||||
describe('sweepManagedArea (crash recovery)', () => {
|
||||
it('does not delete a candidate when its completion marker cannot be inspected', async () => {
|
||||
const svc = GitProjectManifestService.getInstance();
|
||||
const stackName = 'sweep-candidate-marker-io';
|
||||
const candidateAbs = path.join(tmpDir, 'git-managed', '1', stackName, 'generations', 'candidate-marker-io');
|
||||
const markerPath = path.join(candidateAbs, CANDIDATE_COMPLETE_MARKER);
|
||||
fs.mkdirSync(candidateAbs, { recursive: true });
|
||||
fs.writeFileSync(markerPath, 'complete');
|
||||
const originalAccess = fs.promises.access.bind(fs.promises);
|
||||
const accessSpy = vi.spyOn(fs.promises, 'access').mockImplementation(async (...args: Parameters<typeof fs.promises.access>) => {
|
||||
if (String(args[0]) === markerPath) {
|
||||
throw Object.assign(new Error('candidate marker permission denied'), { code: 'EACCES' });
|
||||
}
|
||||
return originalAccess(...args);
|
||||
});
|
||||
|
||||
try {
|
||||
await expect(svc.sweepManagedArea(stackName, {
|
||||
repoUrl: REPO.repo_url,
|
||||
branch: REPO.branch,
|
||||
stackExists: true,
|
||||
candidateClaims: NO_CANDIDATE_CLAIMS,
|
||||
})).rejects.toThrow(/candidate marker permission denied/);
|
||||
expect(fs.existsSync(candidateAbs)).toBe(true);
|
||||
} finally {
|
||||
accessSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('surfaces a generations-directory read failure', async () => {
|
||||
const svc = GitProjectManifestService.getInstance();
|
||||
const stackName = 'sweep-generations-read-io';
|
||||
const generationsDir = path.join(tmpDir, 'git-managed', '1', stackName, 'generations');
|
||||
fs.mkdirSync(generationsDir, { recursive: true });
|
||||
const originalReaddir = fs.promises.readdir.bind(fs.promises);
|
||||
const readdirSpy = vi.spyOn(fs.promises, 'readdir').mockImplementation(async (...args: Parameters<typeof fs.promises.readdir>) => {
|
||||
if (String(args[0]) === generationsDir) {
|
||||
throw Object.assign(new Error('generations directory unavailable'), { code: 'EIO' });
|
||||
}
|
||||
return originalReaddir(...args);
|
||||
});
|
||||
|
||||
try {
|
||||
await expect(svc.sweepManagedArea(stackName, {
|
||||
repoUrl: REPO.repo_url,
|
||||
branch: REPO.branch,
|
||||
stackExists: true,
|
||||
candidateClaims: NO_CANDIDATE_CLAIMS,
|
||||
})).rejects.toThrow(/generations directory unavailable/);
|
||||
} finally {
|
||||
readdirSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('restores the previous applied generation when the marker matches the stack dir', async () => {
|
||||
const svc = GitProjectManifestService.getInstance();
|
||||
const stackName = 'sweep-restore';
|
||||
@@ -763,7 +817,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
affected: ['app.env', 'compose.yaml'],
|
||||
});
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n');
|
||||
expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false);
|
||||
@@ -817,7 +871,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
affected: ['compose.yaml'],
|
||||
});
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe('OPERATOR FIXED ME\n');
|
||||
expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false);
|
||||
@@ -852,7 +906,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
affected: ['compose.yaml'],
|
||||
});
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n');
|
||||
expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false);
|
||||
@@ -887,7 +941,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
affected: ['compose.yaml'],
|
||||
});
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe('NEW\n');
|
||||
expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false);
|
||||
@@ -927,7 +981,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
affected: ['compose.yaml'],
|
||||
});
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe('PRIOR\n');
|
||||
const row = DatabaseService.getInstance().getGitSource(stackName);
|
||||
@@ -969,7 +1023,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
affected: ['app.env', 'compose.yaml'],
|
||||
});
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe('NEW\n');
|
||||
expect(readStackFile(stackName, 'app.env')).toBe('OPERATOR\n');
|
||||
@@ -981,7 +1035,7 @@ describe('sweepManagedArea (crash recovery)', () => {
|
||||
const svc = GitProjectManifestService.getInstance();
|
||||
const stackName = 'sweep-orphan';
|
||||
await svc.writeManifest(stackName, buildManifest(stackName, [managedEntry({ materializedPath: 'compose.yaml' })]));
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: false });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: false, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
expect(await svc.readManifest(stackName, REPO.repo_url, REPO.branch)).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -1059,7 +1113,7 @@ describe('detach crash recovery', () => {
|
||||
writeStackFile(stackName, 'compose.yaml', 'services:\n web:\n image: nginx:new\n');
|
||||
expect(await svc.stageManagedAreaForDetach(stackName)).toBe(true);
|
||||
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
|
||||
expect(readStackFile(stackName, 'compose.yaml')).toBe(original.toString('utf8'));
|
||||
const restored = await svc.readManifest(stackName, REPO.repo_url, REPO.branch);
|
||||
@@ -1303,7 +1357,7 @@ describe('promoteGeneration mid-write failure recovery', () => {
|
||||
});
|
||||
fs.mkdirSync(path.join(tmpDir, 'git-managed', '1', stackName), { recursive: true });
|
||||
fs.writeFileSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER), '{"v":3 torn', 'utf8');
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true });
|
||||
await svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS });
|
||||
expect(fs.existsSync(path.join(tmpDir, 'git-managed', '1', stackName, PROMOTION_MARKER))).toBe(false);
|
||||
expect(DatabaseService.getInstance().getGitSource(stackName)?.manifest_state).toBe('migration_required');
|
||||
});
|
||||
@@ -1325,7 +1379,7 @@ describe('promoteGeneration mid-write failure recovery', () => {
|
||||
}), 'utf8');
|
||||
const stateSpy = vi.spyOn(DatabaseService.getInstance(), 'setGitSourceManifestState');
|
||||
try {
|
||||
await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true })).resolves.toBeUndefined();
|
||||
await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS })).resolves.toBeUndefined();
|
||||
expect(stateSpy).toHaveBeenCalledWith(stackName, null, 'migration_required', null);
|
||||
} finally {
|
||||
stateSpy.mockRestore();
|
||||
@@ -1343,7 +1397,7 @@ describe('promoteGeneration mid-write failure recovery', () => {
|
||||
throw new Error('database unavailable');
|
||||
});
|
||||
try {
|
||||
await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true })).rejects.toThrow(/database unavailable/);
|
||||
await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS })).rejects.toThrow(/database unavailable/);
|
||||
expect(fs.existsSync(markerPath)).toBe(true);
|
||||
} finally {
|
||||
stateSpy.mockRestore();
|
||||
@@ -1371,7 +1425,7 @@ describe('promoteGeneration mid-write failure recovery', () => {
|
||||
return originalAccess(...args);
|
||||
});
|
||||
try {
|
||||
await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true })).rejects.toThrow(/permission denied/);
|
||||
await expect(svc.sweepManagedArea(stackName, { repoUrl: REPO.repo_url, branch: REPO.branch, stackExists: true, candidateClaims: NO_CANDIDATE_CLAIMS })).rejects.toThrow(/permission denied/);
|
||||
expect(fs.existsSync(markerPath)).toBe(true);
|
||||
} finally {
|
||||
accessSpy.mockRestore();
|
||||
|
||||
@@ -13,6 +13,38 @@ const mockMarkReconciling = vi.fn().mockReturnValue(true);
|
||||
const mockMarkImmediateVerified = vi.fn().mockReturnValue(true);
|
||||
const mockGet = vi.fn();
|
||||
const mockCompensate = vi.fn();
|
||||
const mockGitOpsApplication = {
|
||||
id: 'gitops-app',
|
||||
lifecycle_status: 'active',
|
||||
stack_name: 'app',
|
||||
candidate_generation_id: null,
|
||||
};
|
||||
const mockGitOpsStore = {
|
||||
getLiveDirectApplication: vi.fn().mockReturnValue(mockGitOpsApplication),
|
||||
getApplication: vi.fn().mockReturnValue(mockGitOpsApplication),
|
||||
getGeneration: vi.fn().mockReturnValue(undefined),
|
||||
getSettledAttempt: vi.fn().mockReturnValue(undefined),
|
||||
};
|
||||
const mockGitOpsTransitions = {
|
||||
allocateReconcileAttempt: vi.fn().mockReturnValue({ operationId: 'gitops-app:attempt:1', reserved: true }),
|
||||
settleReconcileAttempt: vi.fn().mockReturnValue({ settled: true }),
|
||||
};
|
||||
|
||||
vi.mock('../services/gitops/store', () => ({
|
||||
GitOpsStore: {
|
||||
getInstance: () => mockGitOpsStore,
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock('../services/gitops/transitions', async () => {
|
||||
const actual = await vi.importActual<typeof import('../services/gitops/transitions')>('../services/gitops/transitions');
|
||||
return {
|
||||
...actual,
|
||||
GitOpsTransitions: {
|
||||
getInstance: () => mockGitOpsTransitions,
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
vi.mock('../services/StackUpdateRecoveryService', () => ({
|
||||
StackUpdateRecoveryService: {
|
||||
@@ -129,10 +161,6 @@ vi.mock('../services/DatabaseService', () => ({
|
||||
setGitSourceLastPlan: mockSetGitSourceLastPlan,
|
||||
addNotificationHistory: mockAddNotificationHistory,
|
||||
getStackProjectEnvFiles: vi.fn().mockReturnValue([]),
|
||||
// The apply path now asks whether this stack has a GitOps application.
|
||||
// These fixtures predate the revision-state model, so the lookup finds
|
||||
// nothing and every GitOps producer stays a no-op, which is exactly the
|
||||
// behavior an install with pre-existing Git stacks gets.
|
||||
getDb: () => ({
|
||||
prepare: () => ({ get: () => undefined, all: () => [], run: () => ({ changes: 0 }) }),
|
||||
transaction: (fn: () => unknown) => () => fn(),
|
||||
@@ -197,7 +225,7 @@ describe('git-source apply recovery (R1)', () => {
|
||||
v: 4,
|
||||
files: { 'compose.yaml': 'services:\n web:\n image: nginx\n' },
|
||||
contextDir: null,
|
||||
candidateRelPath: 'generations/cand',
|
||||
candidateRelPath: 'generations/candidate-abc1234deadbeef',
|
||||
inventory: {
|
||||
inputs: [],
|
||||
refusals: [],
|
||||
@@ -246,7 +274,7 @@ describe('git-source apply recovery (R1)', () => {
|
||||
version: 4,
|
||||
files: [{ path: 'compose.yaml', content: 'services:\n web:\n image: nginx\n' }],
|
||||
contextDir: null,
|
||||
candidateRelPath: 'generations/cand',
|
||||
candidateRelPath: 'generations/candidate-abc1234deadbeef',
|
||||
inventory: { inputs: [], refusals: [], buildContexts: [] },
|
||||
planFingerprint: 'fp-test',
|
||||
planSchemaVersion: GIT_CHANGE_PLAN_SCHEMA_VERSION,
|
||||
|
||||
@@ -26,11 +26,13 @@ import { ComposeService } from '../services/ComposeService';
|
||||
import { GitSourceService, GitSourceError } from '../services/GitSourceService';
|
||||
import { GitOpsStore } from '../services/gitops/store';
|
||||
import { GitOpsTransitions } from '../services/gitops/transitions';
|
||||
import { deliveryKey } from '../services/gitops/triggers';
|
||||
import { insertHistory } from '../services/gitops/history';
|
||||
import type { GitOpsApplicationRow } from '../services/gitops/types';
|
||||
import { PROXY_DEPLOY_ACTOR_HEADER, PROXY_DEPLOY_SOURCE_HEADER } from '../services/license-headers';
|
||||
import { withLoopbackTargetProtection } from './helpers/allowLoopbackTargets';
|
||||
import { directApplicationFixture } from './helpers/gitopsFixtures';
|
||||
import { ROLE_PERMISSIONS } from '../middleware/permissions';
|
||||
|
||||
// ── Hoisted mocks (must come before importing the app) ─────────────────
|
||||
|
||||
@@ -75,8 +77,17 @@ function adminToken(): string {
|
||||
return jwt.sign({ username: TEST_USERNAME, role: 'admin' }, TEST_JWT_SECRET, { expiresIn: '1m' });
|
||||
}
|
||||
|
||||
function viewerToken(): string {
|
||||
return jwt.sign({ username: 'viewer', role: 'viewer' }, TEST_JWT_SECRET, { expiresIn: '1m' });
|
||||
}
|
||||
|
||||
function nodeAdminToken(): string {
|
||||
return jwt.sign({ username: 'node-admin', role: 'node-admin' }, TEST_JWT_SECRET, { expiresIn: '1m' });
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
DatabaseService.getInstance().addUser({ username: 'node-admin', password_hash: 'test', role: 'node-admin' });
|
||||
({ app } = await import('../index'));
|
||||
|
||||
// Seed a real stack directory so the PUT handler's existence guard is satisfied
|
||||
@@ -646,6 +657,253 @@ describe('POST /api/stacks/:stackName/git-source/webhook-pull status codes', ()
|
||||
expect(res.status).toBe(200);
|
||||
pullSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('passes a remote webhook delivery id into the durable pull path', async () => {
|
||||
seedGitSource('webhook-delivery-id');
|
||||
const pullSpy = vi.spyOn(GitSourceService.getInstance(), 'handleWebhookPull')
|
||||
.mockResolvedValue({ status: 'success', message: 'Pending update ready at abc1234.' });
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/webhook-delivery-id/git-source/webhook-pull')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({ deliveryId: 'webhook:42:provider-delivery-1' });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(pullSpy).toHaveBeenCalledWith('webhook-delivery-id', true, 'webhook:42:provider-delivery-1');
|
||||
pullSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('requires stack:deploy when a redelivery carries a persisted deploy intent', async () => {
|
||||
const stackName = 'webhook-delivery-deploy-auth';
|
||||
const applicationId = 'webhook-delivery-deploy-auth-app';
|
||||
const deliveryId = 'webhook:control:42:provider-delivery-deploy';
|
||||
seedGitSource(stackName);
|
||||
GitOpsStore.getInstance().insertApplication(directApplicationFixture(applicationId, stackName));
|
||||
GitOpsTransitions.getInstance().reserveReconcileAttempt(
|
||||
applicationId,
|
||||
{
|
||||
operationId: deliveryKey('webhook', 'fetch', deliveryId),
|
||||
actor: 'system:webhook',
|
||||
trigger: 'webhook',
|
||||
at: Date.now(),
|
||||
},
|
||||
undefined,
|
||||
{ autoApply: true, deploy: true },
|
||||
);
|
||||
const pullSpy = vi.spyOn(GitSourceService.getInstance(), 'handleWebhookPull');
|
||||
const originalPermissions = ROLE_PERMISSIONS['node-admin'];
|
||||
ROLE_PERMISSIONS['node-admin'] = originalPermissions.filter((permission) => permission !== 'stack:deploy');
|
||||
|
||||
try {
|
||||
const res = await request(app)
|
||||
.post(`/api/stacks/${stackName}/git-source/webhook-pull`)
|
||||
.set('Authorization', `Bearer ${nodeAdminToken()}`)
|
||||
.send({ deliveryId });
|
||||
|
||||
expect(res.status).toBe(403);
|
||||
expect(res.body.code).toBe('PERMISSION_DENIED');
|
||||
expect(pullSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
ROLE_PERMISSIONS['node-admin'] = originalPermissions;
|
||||
pullSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('requires stack:deploy when a first delivery is configured to auto-apply and deploy', async () => {
|
||||
const stackName = 'webhook-first-delivery-deploy-auth';
|
||||
seedGitSource(stackName);
|
||||
DatabaseService.getInstance().getDb()
|
||||
.prepare('UPDATE stack_git_sources SET auto_apply_on_webhook = 1, auto_deploy_on_apply = 1 WHERE stack_name = ?')
|
||||
.run(stackName);
|
||||
const pullSpy = vi.spyOn(GitSourceService.getInstance(), 'handleWebhookPull');
|
||||
const originalPermissions = ROLE_PERMISSIONS['node-admin'];
|
||||
ROLE_PERMISSIONS['node-admin'] = originalPermissions.filter((permission) => permission !== 'stack:deploy');
|
||||
|
||||
try {
|
||||
const res = await request(app)
|
||||
.post(`/api/stacks/${stackName}/git-source/webhook-pull`)
|
||||
.set('Authorization', `Bearer ${nodeAdminToken()}`);
|
||||
|
||||
expect(res.status).toBe(403);
|
||||
expect(res.body.code).toBe('PERMISSION_DENIED');
|
||||
expect(pullSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
ROLE_PERMISSIONS['node-admin'] = originalPermissions;
|
||||
pullSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it.each([
|
||||
['an object', { nested: true }],
|
||||
['a blank string', ' '],
|
||||
['a string over 512 characters', 'x'.repeat(513)],
|
||||
])('rejects %s as a remote webhook delivery id', async (_caseName, deliveryId) => {
|
||||
seedGitSource('webhook-delivery-id-invalid');
|
||||
const pullSpy = vi.spyOn(GitSourceService.getInstance(), 'handleWebhookPull');
|
||||
|
||||
try {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/webhook-delivery-id-invalid/git-source/webhook-pull')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({ deliveryId });
|
||||
|
||||
expect(res.status).toBe(400);
|
||||
expect(pullSpy).not.toHaveBeenCalled();
|
||||
} finally {
|
||||
pullSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('POST /api/stacks/:stackName/git-source/suspend', () => {
|
||||
it('returns 401 without auth', async () => {
|
||||
const res = await request(app).post('/api/stacks/existing-stack/git-source/suspend');
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
it('returns 400 for an invalid stack name', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/..%2fescape/git-source/suspend')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect([400, 404]).toContain(res.status);
|
||||
});
|
||||
|
||||
it('passes the reason through and returns the normalized result', async () => {
|
||||
const suspendSpy = vi.spyOn(GitSourceService.getInstance(), 'suspend')
|
||||
.mockResolvedValue({ outcome: 'suspended', reason: 'Reconciliation is suspended: maintenance', nextAction: 'resume' });
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/suspend')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({ reason: 'maintenance' });
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.outcome).toBe('suspended');
|
||||
expect(suspendSpy).toHaveBeenCalledWith('existing-stack', expect.objectContaining({ reason: 'maintenance' }));
|
||||
suspendSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('omits reason when none is given in the body', async () => {
|
||||
const suspendSpy = vi.spyOn(GitSourceService.getInstance(), 'suspend')
|
||||
.mockResolvedValue({ outcome: 'suspended', reason: 'Reconciliation is suspended.', nextAction: 'resume' });
|
||||
await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/suspend')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect(suspendSpy).toHaveBeenCalledWith('existing-stack', expect.objectContaining({ reason: undefined }));
|
||||
suspendSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('maps a refused suspend to 409', async () => {
|
||||
const suspendSpy = vi.spyOn(GitSourceService.getInstance(), 'suspend')
|
||||
.mockRejectedValue(new GitSourceError('OPERATION_IN_FLIGHT', 'Cannot suspend existing-stack: source is not live'));
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/suspend')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect(res.status).toBe(409);
|
||||
suspendSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('denies without the stack:edit permission', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/suspend')
|
||||
.set('Authorization', `Bearer ${viewerToken()}`)
|
||||
.send({});
|
||||
expect([401, 403]).toContain(res.status);
|
||||
});
|
||||
|
||||
it('rejects an oversized reason with 400', async () => {
|
||||
const suspendSpy = vi.spyOn(GitSourceService.getInstance(), 'suspend');
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/suspend')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({ reason: 'x'.repeat(513) });
|
||||
expect(res.status).toBe(400);
|
||||
expect(suspendSpy).not.toHaveBeenCalled();
|
||||
suspendSpy.mockRestore();
|
||||
});
|
||||
});
|
||||
|
||||
describe('POST /api/stacks/:stackName/git-source/resume', () => {
|
||||
it('returns 401 without auth', async () => {
|
||||
const res = await request(app).post('/api/stacks/existing-stack/git-source/resume');
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
it('returns 400 for an invalid stack name', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/..%2fescape/git-source/resume')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect([400, 404]).toContain(res.status);
|
||||
});
|
||||
|
||||
it('returns the normalized result', async () => {
|
||||
const resumeSpy = vi.spyOn(GitSourceService.getInstance(), 'resume')
|
||||
.mockResolvedValue({ outcome: 'no_source_change', reason: 'ok', nextAction: 'none' });
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/resume')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.outcome).toBe('no_source_change');
|
||||
expect(resumeSpy).toHaveBeenCalledWith('existing-stack', expect.objectContaining({ actor: expect.any(String) }));
|
||||
resumeSpy.mockRestore();
|
||||
});
|
||||
|
||||
it('denies without the stack:edit permission', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/resume')
|
||||
.set('Authorization', `Bearer ${viewerToken()}`)
|
||||
.send({});
|
||||
expect([401, 403]).toContain(res.status);
|
||||
});
|
||||
});
|
||||
|
||||
describe('POST /api/stacks/:stackName/git-source/retry', () => {
|
||||
it('returns 401 without auth', async () => {
|
||||
const res = await request(app).post('/api/stacks/existing-stack/git-source/retry');
|
||||
expect(res.status).toBe(401);
|
||||
});
|
||||
|
||||
it('returns 400 for an invalid stack name', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/..%2fescape/git-source/retry')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect([400, 404]).toContain(res.status);
|
||||
});
|
||||
|
||||
it('returns the normalized result', async () => {
|
||||
const retrySpy = vi.spyOn(GitSourceService.getInstance(), 'retry')
|
||||
.mockResolvedValue({ outcome: 'candidate_already_fetched', reason: 'ok', nextAction: 'none' });
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/retry')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.outcome).toBe('candidate_already_fetched');
|
||||
expect(retrySpy).toHaveBeenCalledWith('existing-stack', expect.objectContaining({ actor: expect.any(String) }));
|
||||
retrySpy.mockRestore();
|
||||
});
|
||||
|
||||
it('maps an unexpected failure to 500', async () => {
|
||||
const retrySpy = vi.spyOn(GitSourceService.getInstance(), 'retry')
|
||||
.mockRejectedValue(new Error('unexpected'));
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/retry')
|
||||
.set('Authorization', `Bearer ${adminToken()}`)
|
||||
.send({});
|
||||
expect(res.status).toBe(500);
|
||||
retrySpy.mockRestore();
|
||||
});
|
||||
|
||||
it('denies without the stack:edit permission', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/stacks/existing-stack/git-source/retry')
|
||||
.set('Authorization', `Bearer ${viewerToken()}`)
|
||||
.send({});
|
||||
expect([401, 403]).toContain(res.status);
|
||||
});
|
||||
});
|
||||
|
||||
describe('DELETE /api/stacks/:stackName/git-source, detach/export contract', () => {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -309,6 +309,10 @@ function directApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -351,6 +355,12 @@ function generation(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
/**
|
||||
* Failure classification and retry-delay coverage for the GitOps source
|
||||
* controller. classifyFailure is a compile-enforced total map: adding a new
|
||||
* TransportFailureReason or GitSourceErrorCode without updating the lookup
|
||||
* tables here fails the build, not just these tests.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
classifyFailure,
|
||||
nextRetryAt,
|
||||
DEFAULT_TRANSIENT_CEILING,
|
||||
LOW_TRANSIENT_CEILING,
|
||||
type FailureEvidence,
|
||||
} from '../services/gitops/backoff';
|
||||
|
||||
function gitSourceError(code: string, transportReason?: string): FailureEvidence {
|
||||
return { kind: 'git_source_error', code: code as never, transportReason: transportReason as never };
|
||||
}
|
||||
|
||||
describe('classifyFailure', () => {
|
||||
it('classifies a tip-changed race as supersession, not backoff', () => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', 'tip-changed'))).toEqual({ class: 'supersession' });
|
||||
});
|
||||
|
||||
it('classifies a standalone timeout reason as transient', () => {
|
||||
expect(classifyFailure(gitSourceError('NETWORK_TIMEOUT', 'timeout')))
|
||||
.toEqual({ class: 'transient', retryCeiling: DEFAULT_TRANSIENT_CEILING });
|
||||
});
|
||||
|
||||
it('classifies DNS resolution failure (target-unresolved) as transient', () => {
|
||||
expect(classifyFailure(gitSourceError('NETWORK_TIMEOUT', 'target-unresolved')))
|
||||
.toEqual({ class: 'transient', retryCeiling: DEFAULT_TRANSIENT_CEILING });
|
||||
});
|
||||
|
||||
it('classifies an exit-coded network timeout as transient', () => {
|
||||
expect(classifyFailure(gitSourceError('NETWORK_TIMEOUT', 'exit')))
|
||||
.toEqual({ class: 'transient', retryCeiling: DEFAULT_TRANSIENT_CEILING });
|
||||
});
|
||||
|
||||
it('classifies an exit-coded rate limit as transient', () => {
|
||||
expect(classifyFailure(gitSourceError('RATE_LIMITED', 'exit')))
|
||||
.toEqual({ class: 'transient', retryCeiling: DEFAULT_TRANSIENT_CEILING });
|
||||
});
|
||||
|
||||
it('classifies an exit-coded unrecognized git error with a low retry ceiling', () => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', 'exit')))
|
||||
.toEqual({ class: 'transient', retryCeiling: LOW_TRANSIENT_CEILING });
|
||||
});
|
||||
|
||||
it.each(['invalid-url', 'unsafe-target', 'invalid-ref', 'redirect-scope'])(
|
||||
'classifies %s as permanent configuration',
|
||||
(reason) => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', reason))).toEqual({ class: 'permanent' });
|
||||
},
|
||||
);
|
||||
|
||||
it.each(['git-missing', 'git-old'])('classifies %s as permanent environment', (reason) => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', reason))).toEqual({ class: 'permanent' });
|
||||
});
|
||||
|
||||
it('classifies a repository over the size cap as permanent', () => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', 'size'))).toEqual({ class: 'permanent' });
|
||||
});
|
||||
|
||||
it.each(['ssh-auth-required'])('classifies %s as permanent authorization', (reason) => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', reason))).toEqual({ class: 'permanent' });
|
||||
});
|
||||
|
||||
it.each(['AUTH_FAILED', 'SSH_HOST_KEY_FAILED'])('classifies %s (no transport reason) as permanent', (code) => {
|
||||
expect(classifyFailure(gitSourceError(code))).toEqual({ class: 'permanent' });
|
||||
});
|
||||
|
||||
it.each(['ref-not-found', 'unsupported-ref'])('classifies %s as permanent configuration', (reason) => {
|
||||
expect(classifyFailure(gitSourceError('GIT_ERROR', reason))).toEqual({ class: 'permanent' });
|
||||
});
|
||||
|
||||
it.each(['REPO_NOT_FOUND', 'REF_NOT_FOUND', 'REF_DELETED', 'UNSUPPORTED_REF'])(
|
||||
'classifies %s (no transport reason) as permanent',
|
||||
(code) => {
|
||||
expect(classifyFailure(gitSourceError(code))).toEqual({ class: 'permanent' });
|
||||
},
|
||||
);
|
||||
|
||||
it.each(['STALE_PLAN', 'PLAN_BLOCKED', 'PLAN_FINGERPRINT_REQUIRED', 'LEGACY_PENDING', 'PLAN_UNAVAILABLE', 'FILE_NOT_FOUND'])(
|
||||
'classifies %s as requiring operator action',
|
||||
(code) => {
|
||||
expect(classifyFailure(gitSourceError(code))).toEqual({ class: 'operator_action_required' });
|
||||
},
|
||||
);
|
||||
|
||||
it('classifies a conflicting in-flight operation for reconciliation, not blind retry', () => {
|
||||
expect(classifyFailure(gitSourceError('OPERATION_IN_FLIGHT'))).toEqual({ class: 'reconcile' });
|
||||
});
|
||||
|
||||
it('classifies an unavailable policy scanner as degraded', () => {
|
||||
expect(classifyFailure({ kind: 'policy_unavailable' })).toEqual({ class: 'degraded' });
|
||||
});
|
||||
|
||||
it('classifies unavailable persistence as transient with no source-stage progress', () => {
|
||||
expect(classifyFailure({ kind: 'persistence_unavailable' }))
|
||||
.toEqual({ class: 'transient', retryCeiling: DEFAULT_TRANSIENT_CEILING });
|
||||
});
|
||||
|
||||
it('classifies an invalid target binding as permanent at the target level', () => {
|
||||
expect(classifyFailure({ kind: 'target_binding_invalid' })).toEqual({ class: 'target_permanent' });
|
||||
});
|
||||
|
||||
it('classifies a temporarily unavailable target as transient at the target level', () => {
|
||||
expect(classifyFailure({ kind: 'target_unavailable' })).toEqual({ class: 'target_transient' });
|
||||
});
|
||||
|
||||
it('classifies a deploy/health failure after a successful apply as its own class, never refetch or reapply', () => {
|
||||
expect(classifyFailure({ kind: 'target_mutation_failed' })).toEqual({ class: 'target_mutation_failed' });
|
||||
});
|
||||
|
||||
it('classifies unavailable Blueprint evaluation as blocked, not retried', () => {
|
||||
expect(classifyFailure({ kind: 'blueprint_unavailable' })).toEqual({ class: 'blocked' });
|
||||
});
|
||||
|
||||
it('classifies an interrupted or unknown-completion operation for reconciliation', () => {
|
||||
expect(classifyFailure({ kind: 'interrupted' })).toEqual({ class: 'reconcile' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('nextRetryAt', () => {
|
||||
it('computes the base delay with up to +-10% jitter on the first attempt', () => {
|
||||
const now = 1_000_000;
|
||||
const at = nextRetryAt(now, 0);
|
||||
expect(at).toBeGreaterThanOrEqual(now + 54_000);
|
||||
expect(at).toBeLessThanOrEqual(now + 66_000);
|
||||
});
|
||||
|
||||
it('doubles the delay per retry count', () => {
|
||||
const now = 1_000_000;
|
||||
const at = nextRetryAt(now, 3); // 60s * 2^3 = 480s
|
||||
expect(at).toBeGreaterThanOrEqual(now + 432_000);
|
||||
expect(at).toBeLessThanOrEqual(now + 528_000);
|
||||
});
|
||||
|
||||
it('caps the delay at one hour regardless of retry count', () => {
|
||||
const now = 1_000_000;
|
||||
const at = nextRetryAt(now, 20);
|
||||
expect(at).toBeLessThanOrEqual(now + 3_600_000 * 1.1);
|
||||
});
|
||||
|
||||
it('honors a provider retry floor larger than the computed delay', () => {
|
||||
const now = 1_000_000;
|
||||
const at = nextRetryAt(now, 0, 10_000_000);
|
||||
expect(at).toBe(now + 10_000_000);
|
||||
});
|
||||
|
||||
it('ignores a provider retry floor smaller than the computed delay', () => {
|
||||
const now = 1_000_000;
|
||||
const at = nextRetryAt(now, 5, 1_000);
|
||||
expect(at).toBeGreaterThan(now + 1_000);
|
||||
});
|
||||
});
|
||||
@@ -609,6 +609,10 @@ function inlineApp(id: string, blueprintId: number): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
|
||||
@@ -441,6 +441,10 @@ function creatingApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -483,6 +487,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -707,6 +707,10 @@ function creatingApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -749,6 +753,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -340,6 +340,10 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -382,6 +386,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -963,6 +963,10 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -1010,6 +1014,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -644,7 +644,7 @@ describe('Direct Git producers drive the revision state', () => {
|
||||
expect(recovered.active_operation_stage).toBeNull();
|
||||
});
|
||||
|
||||
it('leaves a stack with no GitOps application untouched', async () => {
|
||||
it('refuses to fetch when a configured stack has no GitOps application', async () => {
|
||||
const svc = GitSourceService.getInstance();
|
||||
const store = GitOpsStore.getInstance();
|
||||
const stackName = 'producers-legacy';
|
||||
@@ -681,9 +681,12 @@ describe('Direct Git producers drive the revision state', () => {
|
||||
expect(store.getLiveDirectApplication(stackName)).toBeUndefined();
|
||||
|
||||
stageRepo(COMPOSE_V2, 'fffffff6');
|
||||
await svc.pull(stackName, { actor: 'tester' });
|
||||
await expect(svc.pull(stackName, { actor: 'tester' })).rejects.toMatchObject({
|
||||
code: 'GIT_ERROR',
|
||||
message: expect.stringContaining('GitOps tracking is unavailable'),
|
||||
});
|
||||
|
||||
// The pull succeeded operationally and wrote no GitOps rows.
|
||||
// No untracked fetch or GitOps history was written.
|
||||
expect(store.getLiveDirectApplication(stackName)).toBeUndefined();
|
||||
const historyRows = (await import('../services/DatabaseService')).DatabaseService
|
||||
.getInstance().getDb()
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* The accepted-generation contract: a portable, content-only projection of
|
||||
* a gitops_generations row, plus the target-dispatch boundary.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
buildAcceptedGeneration,
|
||||
BlueprintTargetAdapter,
|
||||
type AcceptedGeneration,
|
||||
} from '../services/gitops/handoff';
|
||||
import type { GitOpsGenerationRow } from '../services/gitops/types';
|
||||
|
||||
function baseRow(overrides: Partial<GitOpsGenerationRow> = {}): GitOpsGenerationRow {
|
||||
return {
|
||||
id: 'gen-1',
|
||||
application_id: 'app-1',
|
||||
commit_sha: 'a'.repeat(40),
|
||||
repo_url: 'https://github.com/example/repo.git',
|
||||
configured_ref: 'main',
|
||||
resolved_ref_kind: 'branch',
|
||||
repo_identity_json: '{"host":"github.com","pathname":"/example/repo.git"}',
|
||||
manifest_version: 4,
|
||||
candidate_dir: 'generations/candidate-a',
|
||||
applied_dir: 'generations/applied-a-0',
|
||||
expected_invocation_json: '{}',
|
||||
materialization_fingerprint: 'f'.repeat(64),
|
||||
validation_ok: 1,
|
||||
plan_blocked: 0,
|
||||
change_plan_fingerprint: 'fp-1',
|
||||
operation_id: 'op-1',
|
||||
trigger: 'manual',
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('buildAcceptedGeneration', () => {
|
||||
it('decodes identity and lineage fields directly from the row', () => {
|
||||
const gen = buildAcceptedGeneration(baseRow());
|
||||
expect(gen.contractVersion).toBe(1);
|
||||
expect(gen.generationId).toBe('gen-1');
|
||||
expect(gen.applicationId).toBe('app-1');
|
||||
expect(gen.repoIdentity).toEqual({ host: 'github.com', pathname: '/example/repo.git' });
|
||||
expect(gen.configuredRef).toBe('main');
|
||||
expect(gen.commitSha).toBe('a'.repeat(40));
|
||||
expect(gen.resolvedRefKind).toBe('branch');
|
||||
expect(gen.validationOk).toBe(true);
|
||||
expect(gen.trigger).toBe('manual');
|
||||
expect(gen.operationId).toBe('op-1');
|
||||
});
|
||||
|
||||
it('records an explicit limitation for each missing portable field on a legacy row, never inventing evidence', () => {
|
||||
const gen = buildAcceptedGeneration(baseRow());
|
||||
expect(gen.portableManifest).toBeNull();
|
||||
expect(gen.composeInputs).toBeNull();
|
||||
expect(gen.sourcePolicyEvidence).toBeNull();
|
||||
expect(gen.limitations).toEqual(expect.arrayContaining([
|
||||
'portable_manifest_missing',
|
||||
'compose_inputs_missing',
|
||||
'source_policy_evidence_missing',
|
||||
'security_policy_evidence_missing',
|
||||
'support_requirements_missing',
|
||||
'compatibility_requirements_missing',
|
||||
]));
|
||||
});
|
||||
|
||||
it('decodes real evidence when the row carries it, recording no limitation for that field', () => {
|
||||
const gen = buildAcceptedGeneration(baseRow({
|
||||
portable_manifest_json: '{"files":[]}',
|
||||
compose_inputs_json: '{"composeFileOrder":["compose.yaml"]}',
|
||||
}));
|
||||
expect(gen.portableManifest).toEqual({ files: [] });
|
||||
expect(gen.composeInputs).toEqual({ composeFileOrder: ['compose.yaml'] });
|
||||
expect(gen.limitations).not.toContain('portable_manifest_missing');
|
||||
expect(gen.limitations).not.toContain('compose_inputs_missing');
|
||||
});
|
||||
|
||||
it('refuses to build a contract from an unparseable repo identity', () => {
|
||||
expect(() => buildAcceptedGeneration(baseRow({ repo_identity_json: 'not json' }))).toThrow();
|
||||
});
|
||||
|
||||
it('never populates secretCapability with a value, only its absence as capability metadata', () => {
|
||||
const gen = buildAcceptedGeneration(baseRow());
|
||||
expect(gen.secretCapability).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
// A future field on AcceptedGeneration named like a target-mode concept
|
||||
// (selector, frozen target set, node id, rollout batch, target project
|
||||
// name, local path, or secret value) must fail this compile, not merely
|
||||
// this test run: the contract stays structurally content-only.
|
||||
type AssertNoTargetModeFields<T> = T extends Record<
|
||||
'selector' | 'targetSet' | 'nodeId' | 'nodeIds' | 'rolloutBatch' | 'projectName' | 'candidateDir' | 'secretValue',
|
||||
unknown
|
||||
> ? never : true;
|
||||
const _structurallyContentOnly: AssertNoTargetModeFields<AcceptedGeneration> = true;
|
||||
void _structurallyContentOnly;
|
||||
|
||||
describe('BlueprintTargetAdapter', () => {
|
||||
it('always returns a durable blocked result, never inspecting selectors or placement', async () => {
|
||||
const adapter = new BlueprintTargetAdapter();
|
||||
const gen = buildAcceptedGeneration(baseRow());
|
||||
const result = await adapter.dispatch(gen, { targetMode: 'blueprint', nodeId: null, bindingRevision: null });
|
||||
expect(result.status).toBe('blocked');
|
||||
});
|
||||
});
|
||||
@@ -531,6 +531,10 @@ function application(): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
|
||||
@@ -167,6 +167,10 @@ function creatingApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
/**
|
||||
* Normalized reconcile-outcome coverage. outcomeFromSourceFacet derives from
|
||||
* the existing SourceFacet projection rather than inventing a second status
|
||||
* source, so "no source change" and "converged" cannot silently collapse
|
||||
* into the same result.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { outcomeFromSourceFacet } from '../services/gitops/outcomes';
|
||||
import type { SourceFacet } from '../services/gitops/types';
|
||||
|
||||
const identity = {
|
||||
configuredRepoUrl: 'https://github.com/example/repo.git',
|
||||
repoIdentity: { host: 'github.com', pathname: '/example/repo.git' },
|
||||
configuredRef: 'main',
|
||||
desiredCommitSha: 'a'.repeat(40),
|
||||
fetchedCommitSha: 'a'.repeat(40),
|
||||
candidateGenerationId: null,
|
||||
acceptedGenerationId: 'gen-1',
|
||||
};
|
||||
|
||||
describe('outcomeFromSourceFacet', () => {
|
||||
it('reports no_source_change for an accepted generation, never converged on SHA alone', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'application_generation_accepted' };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('no_source_change');
|
||||
expect(result.commitSha).toBe('a'.repeat(40));
|
||||
});
|
||||
|
||||
it('reports candidate_already_fetched for a ready candidate', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'candidate_ready' };
|
||||
expect(outcomeFromSourceFacet(facet).outcome).toBe('candidate_already_fetched');
|
||||
});
|
||||
|
||||
it('reports pending_review with a review next action', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'source_review_pending' };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('pending_review');
|
||||
expect(result.nextAction).toBe('review');
|
||||
});
|
||||
|
||||
it('reports blocked with a resolve_conflict next action for a source conflict', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'source_conflict_blocker' };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('blocked');
|
||||
expect(result.nextAction).toBe('resolve_conflict');
|
||||
});
|
||||
|
||||
it('reports superseded for a candidate a newer revision replaced', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'source_superseded', supersededGenerationId: 'gen-old' };
|
||||
expect(outcomeFromSourceFacet(facet).outcome).toBe('superseded');
|
||||
});
|
||||
|
||||
it('reports retry_scheduled with the retry time surfaced', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'source_retry_scheduled', retryAt: 12345, retryCount: 2 };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('retry_scheduled');
|
||||
expect(result.retryAt).toBe(12345);
|
||||
expect(result.nextAction).toBe('none');
|
||||
});
|
||||
|
||||
it('reports suspended with a resume next action', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'source_suspended', suspendedAt: 999, suspendedReason: 'operator paused sync' };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('suspended');
|
||||
expect(result.nextAction).toBe('resume');
|
||||
expect(result.reason).toContain('operator paused sync');
|
||||
});
|
||||
|
||||
it('reports failed_previous_intact for a source failure, with a retry next action when a retry is scheduled', () => {
|
||||
const facet: SourceFacet = {
|
||||
...identity,
|
||||
status: 'source_failed',
|
||||
failureStage: 'fetch',
|
||||
failureClass: 'permanent',
|
||||
failureAt: 100,
|
||||
retryAt: 200,
|
||||
retryCount: 1,
|
||||
};
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('failed_previous_intact');
|
||||
expect(result.nextAction).toBe('retry');
|
||||
expect(result.retryAt).toBe(200);
|
||||
});
|
||||
|
||||
it('reports failed_previous_intact with no retry next action when no retry is scheduled', () => {
|
||||
const facet: SourceFacet = {
|
||||
...identity,
|
||||
status: 'source_failed',
|
||||
failureStage: 'fetch',
|
||||
failureClass: 'permanent',
|
||||
failureAt: 100,
|
||||
retryAt: null,
|
||||
retryCount: 0,
|
||||
};
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('failed_previous_intact');
|
||||
expect(result.nextAction).toBe('configure_credentials');
|
||||
});
|
||||
|
||||
it('reports recovery_required for an interrupted operation', () => {
|
||||
const facet: SourceFacet = {
|
||||
...identity,
|
||||
status: 'source_unknown',
|
||||
interruptedStage: 'fetch_started',
|
||||
interruptedAt: 100,
|
||||
interruptedOperationId: 'op-1',
|
||||
interruptedGenerationId: null,
|
||||
};
|
||||
expect(outcomeFromSourceFacet(facet).outcome).toBe('recovery_required');
|
||||
});
|
||||
|
||||
it('reports recovery_required with a view_target_results next action when recovery is outstanding', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'recovery_required', recoveryRef: 'rec-1', recoveryGenerationId: 'gen-1' };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('recovery_required');
|
||||
expect(result.nextAction).toBe('view_target_results');
|
||||
});
|
||||
|
||||
it('reports recovery_required when recovery itself failed, distinguishing that in the reason', () => {
|
||||
const facet: SourceFacet = {
|
||||
...identity,
|
||||
status: 'recovery_failed',
|
||||
recoveryRef: 'rec-1',
|
||||
recoveryGenerationId: 'gen-1',
|
||||
failureClass: 'io_error',
|
||||
failureAt: 100,
|
||||
};
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('recovery_required');
|
||||
expect(result.reason).toMatch(/recovery/i);
|
||||
});
|
||||
|
||||
it('reports unknown for an application that is no longer live', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'not_live', lifecycleStatus: 'detached' };
|
||||
expect(outcomeFromSourceFacet(facet).outcome).toBe('unknown');
|
||||
});
|
||||
|
||||
it.each(['not_applicable', 'never_reconciled', 'checking_fetching', 'source_reconcile_required'] as const)(
|
||||
'reports unknown for %s, which has no settled outcome yet',
|
||||
(status) => {
|
||||
const facet = status === 'not_applicable'
|
||||
? ({ status } as SourceFacet)
|
||||
: ({ ...identity, status } as SourceFacet);
|
||||
expect(outcomeFromSourceFacet(facet).outcome).toBe('unknown');
|
||||
},
|
||||
);
|
||||
|
||||
it('reports unknown while an operation is in flight (applying)', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'applying', activeOperationId: 'op-1', activeGenerationId: 'gen-1' };
|
||||
expect(outcomeFromSourceFacet(facet).outcome).toBe('unknown');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,387 @@
|
||||
/**
|
||||
* Durable reconcile-attempt reservation and settlement: a bare history
|
||||
* insert in its own transaction, never through mutateApp, so a reservation
|
||||
* writes no application-row state and can be safely repeated.
|
||||
*/
|
||||
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { GitOpsStore } from '../services/gitops/store';
|
||||
import { GitOpsTransitions, type EventEnvelope } from '../services/gitops/transitions';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
import type { GitOpsApplicationRow } from '../services/gitops/types';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
GitOpsStore.resetForTests();
|
||||
GitOpsTransitions.resetForTests();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
describe('reconcile attempt reservation and settlement', () => {
|
||||
it('reserves an attempt without touching application state', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-res', 'res-web'), nodeId: 1, envelope: env('op-act-res') });
|
||||
const before = store.getApplication('app-res')!;
|
||||
|
||||
const result = tx.reserveReconcileAttempt('app-res', env('op-res-1'));
|
||||
|
||||
expect(result.reserved).toBe(true);
|
||||
const after = store.getApplication('app-res')!;
|
||||
expect(after.updated_at).toBe(before.updated_at);
|
||||
expect(after.desired_commit_sha).toBe(before.desired_commit_sha);
|
||||
});
|
||||
|
||||
it('returns reserved: false on a repeated reservation for the same operation', () => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-res2', 'res2-web'), nodeId: 1, envelope: env('op-act-res2') });
|
||||
|
||||
const first = tx.reserveReconcileAttempt('app-res2', env('op-res2-1'));
|
||||
const second = tx.reserveReconcileAttempt('app-res2', env('op-res2-1'));
|
||||
|
||||
expect(first.reserved).toBe(true);
|
||||
expect(second.reserved).toBe(false);
|
||||
});
|
||||
|
||||
it('allows two different operations to each reserve their own attempt', () => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-res3', 'res3-web'), nodeId: 1, envelope: env('op-act-res3') });
|
||||
|
||||
const first = tx.reserveReconcileAttempt('app-res3', env('op-res3-a'));
|
||||
const second = tx.reserveReconcileAttempt('app-res3', env('op-res3-b'));
|
||||
|
||||
expect(first.reserved).toBe(true);
|
||||
expect(second.reserved).toBe(true);
|
||||
});
|
||||
|
||||
it('settles a reserved attempt and finds it by operation id afterward', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-settle', 'settle-web'), nodeId: 1, envelope: env('op-act-settle') });
|
||||
tx.reserveReconcileAttempt('app-settle', env('op-settle-1'));
|
||||
|
||||
const settleResult = tx.settleReconcileAttempt('app-settle', env('op-settle-1'), {
|
||||
outcome: 'no_source_change',
|
||||
reason: 'Nothing new to fetch.',
|
||||
nextAction: 'none',
|
||||
});
|
||||
|
||||
expect(settleResult.settled).toBe(true);
|
||||
const settled = store.getSettledAttempt('app-settle', 'op-settle-1');
|
||||
expect(settled).toBeDefined();
|
||||
});
|
||||
|
||||
it('settling twice for the same operation is a no-op the second time', () => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-settle2', 'settle2-web'), nodeId: 1, envelope: env('op-act-settle2') });
|
||||
tx.reserveReconcileAttempt('app-settle2', env('op-settle2-1'));
|
||||
|
||||
const first = tx.settleReconcileAttempt('app-settle2', env('op-settle2-1'), {
|
||||
outcome: 'no_source_change',
|
||||
reason: 'first',
|
||||
nextAction: 'none',
|
||||
});
|
||||
const second = tx.settleReconcileAttempt('app-settle2', env('op-settle2-1'), {
|
||||
outcome: 'no_source_change',
|
||||
reason: 'second, must not overwrite',
|
||||
nextAction: 'none',
|
||||
});
|
||||
|
||||
expect(first.settled).toBe(true);
|
||||
expect(second.settled).toBe(false);
|
||||
});
|
||||
|
||||
it('has no settled attempt for a reservation that was never settled', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-orphan', 'orphan-web'), nodeId: 1, envelope: env('op-act-orphan') });
|
||||
tx.reserveReconcileAttempt('app-orphan', env('op-orphan-1'));
|
||||
|
||||
expect(store.getSettledAttempt('app-orphan', 'op-orphan-1')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('lists an unsettled reservation but not one that has settled', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-unsettled', 'unsettled-web'), nodeId: 1, envelope: env('op-act-unsettled') });
|
||||
tx.reserveReconcileAttempt('app-unsettled', env('op-unsettled-orphan'));
|
||||
tx.reserveReconcileAttempt('app-unsettled', env('op-unsettled-done'));
|
||||
tx.settleReconcileAttempt('app-unsettled', env('op-unsettled-done'), {
|
||||
outcome: 'no_source_change',
|
||||
reason: 'done',
|
||||
nextAction: 'none',
|
||||
});
|
||||
|
||||
const unsettled = store.listUnsettledReconcileAttempts();
|
||||
const operationIds = unsettled.filter((r) => r.application_id === 'app-unsettled').map((r) => r.operation_id);
|
||||
expect(operationIds).toContain('op-unsettled-orphan');
|
||||
expect(operationIds).not.toContain('op-unsettled-done');
|
||||
});
|
||||
|
||||
it('gets the started row for one exact attempt, or undefined when it was never reserved', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-started', 'started-web'), nodeId: 1, envelope: env('op-act-started') });
|
||||
tx.reserveReconcileAttempt('app-started', env('op-started-1'));
|
||||
|
||||
expect(store.getStartedAttempt('app-started', 'op-started-1')).toBeDefined();
|
||||
expect(store.getStartedAttempt('app-started', 'op-never-reserved')).toBeUndefined();
|
||||
});
|
||||
|
||||
it('pages past a permanently unsettled row instead of returning it forever on every call with the same cursor', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-cursor', 'cursor-web'), nodeId: 1, envelope: env('op-act-cursor') });
|
||||
tx.reserveReconcileAttempt('app-cursor', { operationId: 'op-cursor-stuck', actor: 'tester', trigger: 'manual', at: 1_000 });
|
||||
tx.reserveReconcileAttempt('app-cursor', { operationId: 'op-cursor-next', actor: 'tester', trigger: 'manual', at: 2_000 });
|
||||
|
||||
const firstPage = store.listUnsettledReconcileAttempts(1);
|
||||
expect(firstPage.map((r) => r.operation_id)).toEqual(['op-cursor-stuck']);
|
||||
const cursor = { createdAt: firstPage[0].created_at, id: firstPage[0].id };
|
||||
// Simulate op-cursor-stuck being permanently unrecoverable: it is never
|
||||
// settled, so a caller must page past it using the cursor rather than
|
||||
// seeing it again on the next call.
|
||||
const secondPage = store.listUnsettledReconcileAttempts(1, cursor);
|
||||
expect(secondPage.map((r) => r.operation_id)).toEqual(['op-cursor-next']);
|
||||
});
|
||||
|
||||
it('allocates a fresh attemptSeq-derived operation id and reserves it in one transaction', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-alloc', 'alloc-web'), nodeId: 1, envelope: env('op-act-alloc') });
|
||||
const before = store.getApplication('app-alloc')!;
|
||||
|
||||
const first = tx.allocateReconcileAttempt('app-alloc', 'tester', 'manual', Date.now());
|
||||
const second = tx.allocateReconcileAttempt('app-alloc', 'tester', 'manual', Date.now());
|
||||
|
||||
expect(first.reserved).toBe(true);
|
||||
expect(second.reserved).toBe(true);
|
||||
expect(first.operationId).not.toBe(second.operationId);
|
||||
const after = store.getApplication('app-alloc')!;
|
||||
expect(after.attempt_seq).toBe(before.attempt_seq + 2);
|
||||
});
|
||||
|
||||
it('rolls back the allocated sequence when reservation insertion fails', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-alloc-rollback', 'alloc-rollback-web'), nodeId: 1, envelope: env('op-act-alloc-rollback') });
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const before = store.getApplication('app-alloc-rollback')!;
|
||||
db.exec(`
|
||||
CREATE TRIGGER fail_reconcile_reservation
|
||||
BEFORE INSERT ON gitops_history
|
||||
WHEN NEW.stage = 'source_reconcile_started'
|
||||
BEGIN
|
||||
SELECT RAISE(ABORT, 'simulated reservation insert failure');
|
||||
END
|
||||
`);
|
||||
|
||||
try {
|
||||
expect(() => tx.allocateReconcileAttempt('app-alloc-rollback', 'tester', 'manual', Date.now()))
|
||||
.toThrow('simulated reservation insert failure');
|
||||
expect(store.getApplication('app-alloc-rollback')!.attempt_seq).toBe(before.attempt_seq);
|
||||
expect(store.listUnsettledReconcileAttempts().some((row) => row.application_id === 'app-alloc-rollback')).toBe(false);
|
||||
} finally {
|
||||
db.exec('DROP TRIGGER fail_reconcile_reservation');
|
||||
}
|
||||
});
|
||||
|
||||
it('records a follower link on a reservation made on behalf of a coalesced request', () => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-follower', 'follower-web'), nodeId: 1, envelope: env('op-act-follower') });
|
||||
|
||||
const leader = tx.allocateReconcileAttempt('app-follower', 'tester', 'manual', Date.now());
|
||||
const follower = tx.allocateReconcileAttempt('app-follower', 'tester', 'manual', Date.now(), leader.operationId);
|
||||
|
||||
expect(follower.reserved).toBe(true);
|
||||
const started = DatabaseService.getInstance().getDb()
|
||||
.prepare("SELECT after_json FROM gitops_history WHERE application_id = ? AND operation_id = ? AND stage = 'source_reconcile_started'")
|
||||
.get('app-follower', follower.operationId) as { after_json: string };
|
||||
expect(JSON.parse(started.after_json)).toEqual({ followerOf: leader.operationId });
|
||||
});
|
||||
|
||||
it('records the original webhook delivery intent on its stable reservation', () => {
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-delivery-intent', 'delivery-intent-web'), nodeId: 1, envelope: env('op-act-delivery-intent') });
|
||||
|
||||
tx.reserveReconcileAttempt(
|
||||
'app-delivery-intent',
|
||||
env('webhook:fetch:delivery-intent'),
|
||||
undefined,
|
||||
{ autoApply: true, deploy: false },
|
||||
);
|
||||
|
||||
const started = GitOpsStore.getInstance().getStartedAttempt('app-delivery-intent', 'webhook:fetch:delivery-intent')!;
|
||||
expect(JSON.parse(started.after_json)).toEqual({
|
||||
deliveryIntent: { autoApply: true, deploy: false },
|
||||
});
|
||||
});
|
||||
|
||||
it('reports the most recently settled attempt even when both share the same millisecond timestamp', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
const tx = GitOpsTransitions.getInstance();
|
||||
tx.activateDirect({ application: app('app-latest', 'latest-web'), nodeId: 1, envelope: env('op-act-latest') });
|
||||
// Same `at` on purpose: created_at alone cannot tell these two apart,
|
||||
// so the query must break the tie by insertion order (rowid), not the
|
||||
// id column, which is a random UUID unrelated to recency.
|
||||
const sameInstant = 5_000;
|
||||
tx.reserveReconcileAttempt('app-latest', { operationId: 'op-latest-1', actor: 'tester', trigger: 'manual', at: sameInstant });
|
||||
tx.settleReconcileAttempt(
|
||||
'app-latest',
|
||||
{ operationId: 'op-latest-1', actor: 'tester', trigger: 'manual', at: sameInstant },
|
||||
{ outcome: 'no_source_change', reason: 'first', nextAction: 'none' },
|
||||
);
|
||||
tx.reserveReconcileAttempt('app-latest', { operationId: 'op-latest-2', actor: 'tester', trigger: 'manual', at: sameInstant });
|
||||
tx.settleReconcileAttempt(
|
||||
'app-latest',
|
||||
{ operationId: 'op-latest-2', actor: 'tester', trigger: 'manual', at: sameInstant },
|
||||
{ outcome: 'retry_scheduled', reason: 'second', nextAction: 'none' },
|
||||
);
|
||||
|
||||
const latest = store.latestSettledAttempt('app-latest');
|
||||
expect(latest?.operation_id).toBe('op-latest-2');
|
||||
});
|
||||
});
|
||||
|
||||
describe('poll and retry eligibility queries', () => {
|
||||
it('lists a source whose next_poll_at has arrived', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({ ...app('app-poll-due', 'poll-due-web'), next_poll_at: 1_000 });
|
||||
const due = store.listSourcesDueForPoll(1_000);
|
||||
expect(due.map((a) => a.id)).toContain('app-poll-due');
|
||||
});
|
||||
|
||||
it('excludes a source whose next_poll_at has not arrived yet', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({ ...app('app-poll-future', 'poll-future-web'), next_poll_at: 5_000 });
|
||||
const due = store.listSourcesDueForPoll(1_000);
|
||||
expect(due.map((a) => a.id)).not.toContain('app-poll-future');
|
||||
});
|
||||
|
||||
it('excludes a suspended source even when its poll time has arrived', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({ ...app('app-poll-susp', 'poll-susp-web'), next_poll_at: 1_000, suspended_at: 500 });
|
||||
const due = store.listSourcesDueForPoll(1_000);
|
||||
expect(due.map((a) => a.id)).not.toContain('app-poll-susp');
|
||||
});
|
||||
|
||||
it('excludes a source with an operation already in flight', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({
|
||||
...app('app-poll-busy', 'poll-busy-web'),
|
||||
next_poll_at: 1_000,
|
||||
active_operation_stage: 'fetch_started',
|
||||
});
|
||||
const due = store.listSourcesDueForPoll(1_000);
|
||||
expect(due.map((a) => a.id)).not.toContain('app-poll-busy');
|
||||
});
|
||||
|
||||
it('excludes a Blueprint-mode application from polling', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({
|
||||
...app('app-poll-bp', 'unused-bp'),
|
||||
stack_name: null,
|
||||
blueprint_id: 42,
|
||||
target_mode: 'blueprint',
|
||||
configured_repo_url: 'https://github.com/org/repo.git',
|
||||
next_poll_at: 1_000,
|
||||
});
|
||||
const due = store.listSourcesDueForPoll(1_000);
|
||||
expect(due.map((a) => a.id)).not.toContain('app-poll-bp');
|
||||
});
|
||||
|
||||
it('lists an application whose retry_at has arrived', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({ ...app('app-retry-due', 'retry-due-web'), retry_at: 1_000 });
|
||||
const due = store.listApplicationsDueForRetry(1_000);
|
||||
expect(due.map((a) => a.id)).toContain('app-retry-due');
|
||||
});
|
||||
|
||||
it('excludes an application with no retry scheduled', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication(app('app-retry-none', 'retry-none-web'));
|
||||
const due = store.listApplicationsDueForRetry(1_000);
|
||||
expect(due.map((a) => a.id)).not.toContain('app-retry-none');
|
||||
});
|
||||
|
||||
it('excludes a suspended application even when its retry time has arrived', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({ ...app('app-retry-susp', 'retry-susp-web'), retry_at: 1_000, suspended_at: 500 });
|
||||
const due = store.listApplicationsDueForRetry(1_000);
|
||||
expect(due.map((a) => a.id)).not.toContain('app-retry-susp');
|
||||
});
|
||||
});
|
||||
|
||||
function env(operationId: string): EventEnvelope {
|
||||
return { operationId, actor: 'tester', trigger: 'manual', at: Date.now() };
|
||||
}
|
||||
|
||||
function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
return {
|
||||
id,
|
||||
lifecycle_key: `direct:${stackName}`,
|
||||
lifecycle_status: 'active',
|
||||
target_mode: 'direct',
|
||||
stack_name: stackName,
|
||||
blueprint_id: null,
|
||||
configured_repo_url: 'https://github.com/org/repo.git',
|
||||
repo_identity_json: '{"host":"github.com","pathname":"/org/repo.git"}',
|
||||
configured_ref: 'main',
|
||||
compose_paths_json: '["compose.yml"]',
|
||||
context_dir: null,
|
||||
sync_env: 0,
|
||||
env_path: null,
|
||||
materialization_fingerprint: 'a'.repeat(64),
|
||||
desired_commit_sha: null,
|
||||
fetched_commit_sha: null,
|
||||
fetched_resolved_ref_kind: null,
|
||||
candidate_generation_id: null,
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
rollout_candidate_id: null,
|
||||
rollout_generation_id: null,
|
||||
source_acceptance_ref: null,
|
||||
placement_approval_ref: null,
|
||||
rollout_authorization_ref: null,
|
||||
legacy_combined_approval_ref: null,
|
||||
preflight_fingerprint: null,
|
||||
latest_operation_id: null,
|
||||
active_operation_id: null,
|
||||
active_operation_stage: null,
|
||||
active_operation_at: null,
|
||||
active_generation_id: null,
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
failure_at: null,
|
||||
retry_at: null,
|
||||
retry_count: 0,
|
||||
suspended_at: null,
|
||||
recovery_ref: null,
|
||||
recovery_phase: null,
|
||||
interruption_stage: null,
|
||||
interruption_at: null,
|
||||
interruption_operation_id: null,
|
||||
interruption_generation_id: null,
|
||||
evidence_fresh_at: null,
|
||||
evidence_limitations_json: null,
|
||||
created_at: 1,
|
||||
updated_at: 1,
|
||||
};
|
||||
}
|
||||
@@ -147,6 +147,10 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -189,6 +193,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -461,6 +461,10 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -503,6 +507,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -239,6 +239,61 @@ describe('gitops schema', () => {
|
||||
expect(row?.configured_ref).toBe('v1');
|
||||
});
|
||||
|
||||
it('round-trips the portable generation contract fields, defaulting to null for legacy rows', async () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication(directApp('app-portable', 'portable-web'));
|
||||
store.insertGeneration({
|
||||
...generation('gen-portable-legacy', 'app-portable'),
|
||||
});
|
||||
const legacy = store.getGeneration('gen-portable-legacy');
|
||||
expect(legacy?.portable_manifest_json).toBeNull();
|
||||
expect(legacy?.compose_inputs_json).toBeNull();
|
||||
expect(legacy?.source_policy_evidence_json).toBeNull();
|
||||
expect(legacy?.security_policy_evidence_json).toBeNull();
|
||||
expect(legacy?.support_requirements_json).toBeNull();
|
||||
expect(legacy?.compatibility_requirements_json).toBeNull();
|
||||
|
||||
store.insertGeneration({
|
||||
...generation('gen-portable-new', 'app-portable'),
|
||||
portable_manifest_json: '{"files":[]}',
|
||||
compose_inputs_json: '{"composeFileOrder":["compose.yaml"]}',
|
||||
source_policy_evidence_json: '{"policy":"manual"}',
|
||||
security_policy_evidence_json: '{"status":"allowed"}',
|
||||
support_requirements_json: '{}',
|
||||
compatibility_requirements_json: '{}',
|
||||
});
|
||||
const populated = store.getGeneration('gen-portable-new');
|
||||
expect(populated?.portable_manifest_json).toBe('{"files":[]}');
|
||||
expect(populated?.compose_inputs_json).toBe('{"composeFileOrder":["compose.yaml"]}');
|
||||
expect(populated?.source_policy_evidence_json).toBe('{"policy":"manual"}');
|
||||
});
|
||||
|
||||
it('defaults controller-owned columns to manual, off, and zero on a fresh application', async () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication(directApp('app-ctrl', 'ctrl-web'));
|
||||
const app = store.getApplication('app-ctrl');
|
||||
expect(app?.source_policy).toBe('manual');
|
||||
expect(app?.poll_interval_secs).toBeNull();
|
||||
expect(app?.next_poll_at).toBeNull();
|
||||
expect(app?.attempt_seq).toBe(0);
|
||||
});
|
||||
|
||||
it('round-trips a configured poll interval and policy', async () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication({
|
||||
...directApp('app-ctrl2', 'ctrl2-web'),
|
||||
source_policy: 'automatic',
|
||||
poll_interval_secs: 120,
|
||||
next_poll_at: 5000,
|
||||
attempt_seq: 3,
|
||||
});
|
||||
const app = store.getApplication('app-ctrl2');
|
||||
expect(app?.source_policy).toBe('automatic');
|
||||
expect(app?.poll_interval_secs).toBe(120);
|
||||
expect(app?.next_poll_at).toBe(5000);
|
||||
expect(app?.attempt_seq).toBe(3);
|
||||
});
|
||||
|
||||
it('round-trips fetched_resolved_ref_kind on application fetch transitions', async () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
store.insertApplication(directApp('app-fetch-kind', 'fetch-web'));
|
||||
@@ -294,6 +349,10 @@ function directApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -362,6 +421,12 @@ function generation(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -813,6 +813,10 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -855,6 +859,12 @@ function gen(id: string, applicationId: string): GitOpsGenerationRow {
|
||||
actor: 'tester',
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: 1,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* Pure trigger normalization and coalescing-key coverage for the GitOps
|
||||
* source controller. No DB, no store: these are the identity/joining rules
|
||||
* a controller submission goes through before anything durable happens.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { coalesceKey, deliveryKey, type ReconcileRequest } from '../services/gitops/triggers';
|
||||
|
||||
function fetchRequest(overrides: Partial<Extract<ReconcileRequest, { intent: 'fetch' }>> = {}): ReconcileRequest {
|
||||
return {
|
||||
intent: 'fetch',
|
||||
applicationId: 'app-1',
|
||||
stackName: 'web',
|
||||
trigger: 'manual',
|
||||
actor: 'tester',
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function applyRequest(overrides: Partial<Extract<ReconcileRequest, { intent: 'apply' }>> = {}): ReconcileRequest {
|
||||
return {
|
||||
intent: 'apply',
|
||||
applicationId: 'app-1',
|
||||
stackName: 'web',
|
||||
trigger: 'manual',
|
||||
actor: 'tester',
|
||||
commitSha: 'a'.repeat(40),
|
||||
planFingerprint: 'fp-1',
|
||||
deploy: false,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('coalesceKey', () => {
|
||||
it('joins two fetch requests for the same application', () => {
|
||||
expect(coalesceKey(fetchRequest())).toBe(coalesceKey(fetchRequest({ trigger: 'poll' })));
|
||||
});
|
||||
|
||||
it('does not join fetch requests for different applications', () => {
|
||||
expect(coalesceKey(fetchRequest({ applicationId: 'app-1' })))
|
||||
.not.toBe(coalesceKey(fetchRequest({ applicationId: 'app-2' })));
|
||||
});
|
||||
|
||||
it('joins two apply requests with identical commit, fingerprint, and deploy flag', () => {
|
||||
expect(coalesceKey(applyRequest())).toBe(coalesceKey(applyRequest({ trigger: 'webhook' })));
|
||||
});
|
||||
|
||||
it('does not join two applies with different plan fingerprints', () => {
|
||||
expect(coalesceKey(applyRequest({ planFingerprint: 'fp-1' })))
|
||||
.not.toBe(coalesceKey(applyRequest({ planFingerprint: 'fp-2' })));
|
||||
});
|
||||
|
||||
it('does not join two applies with different commits', () => {
|
||||
expect(coalesceKey(applyRequest({ commitSha: 'a'.repeat(40) })))
|
||||
.not.toBe(coalesceKey(applyRequest({ commitSha: 'b'.repeat(40) })));
|
||||
});
|
||||
|
||||
it('does not join two applies that differ only in deploy', () => {
|
||||
expect(coalesceKey(applyRequest({ deploy: false })))
|
||||
.not.toBe(coalesceKey(applyRequest({ deploy: true })));
|
||||
});
|
||||
|
||||
it('never joins a fetch and an apply for the same application', () => {
|
||||
expect(coalesceKey(fetchRequest())).not.toBe(coalesceKey(applyRequest()));
|
||||
});
|
||||
|
||||
it('does not join two fetches for the same applicationId but different stack names', () => {
|
||||
expect(coalesceKey(fetchRequest({ stackName: 'web' })))
|
||||
.not.toBe(coalesceKey(fetchRequest({ stackName: 'other-stack' })));
|
||||
});
|
||||
});
|
||||
|
||||
describe('deliveryKey', () => {
|
||||
it('namespaces the same delivery id differently per trigger', () => {
|
||||
expect(deliveryKey('webhook', 'fetch', 'delivery-1')).not.toBe(deliveryKey('api', 'fetch', 'delivery-1'));
|
||||
});
|
||||
|
||||
it('namespaces the same delivery id differently per intent', () => {
|
||||
expect(deliveryKey('webhook', 'fetch', 'delivery-1')).not.toBe(deliveryKey('webhook', 'apply', 'delivery-1'));
|
||||
});
|
||||
|
||||
it('is stable for the same trigger, intent, and delivery id', () => {
|
||||
expect(deliveryKey('webhook', 'fetch', 'delivery-1')).toBe(deliveryKey('webhook', 'fetch', 'delivery-1'));
|
||||
});
|
||||
});
|
||||
@@ -57,6 +57,10 @@ export function directApplicationFixture(id: string, stackName: string): GitOpsA
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
/**
|
||||
* SourceController: the background driver for unattended reconciliation.
|
||||
* GitOpsStore's due-queries and GitSourceService.reconcile() are mocked so
|
||||
* these tests exercise only the timer/coalescing behavior, not real fetch
|
||||
* or apply mechanics (already covered by git-source-service.test.ts).
|
||||
*/
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { directApplicationFixture } from './helpers/gitopsFixtures';
|
||||
import { GitOpsStore } from '../services/gitops/store';
|
||||
import { GitSourceService } from '../services/GitSourceService';
|
||||
import { SourceController } from '../services/gitops/SourceController';
|
||||
import type { GitOpsApplicationRow } from '../services/gitops/types';
|
||||
import type { ReconcileResult } from '../services/gitops/outcomes';
|
||||
|
||||
const TICK_MS = 60_000;
|
||||
const okResult: ReconcileResult = { outcome: 'no_source_change', reason: 'ok', nextAction: 'none' };
|
||||
|
||||
let tmpDir: string;
|
||||
let controller: SourceController;
|
||||
|
||||
/** Point both due-queries at fixed rows; the scan reads nothing else. */
|
||||
function mockDue(duePoll: GitOpsApplicationRow[], dueRetry: GitOpsApplicationRow[] = []): void {
|
||||
vi.spyOn(GitOpsStore.getInstance(), 'listSourcesDueForPoll').mockReturnValue(duePoll);
|
||||
vi.spyOn(GitOpsStore.getInstance(), 'listApplicationsDueForRetry').mockReturnValue(dueRetry);
|
||||
}
|
||||
|
||||
function spyOnReconcile() {
|
||||
return vi.spyOn(GitSourceService.getInstance(), 'reconcile');
|
||||
}
|
||||
|
||||
/** Run the next scheduled tick and let the evaluations it fires settle. */
|
||||
async function advanceOneTick(): Promise<void> {
|
||||
await vi.advanceTimersByTimeAsync(TICK_MS);
|
||||
}
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
GitOpsStore.resetForTests();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
cleanupTestDb(tmpDir);
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
vi.useFakeTimers();
|
||||
SourceController.resetForTests();
|
||||
controller = SourceController.getInstance();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
controller.stop();
|
||||
vi.restoreAllMocks();
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
describe('SourceController', () => {
|
||||
it('evaluates a source whose poll interval is due', async () => {
|
||||
mockDue([directApplicationFixture('app-poll', 'poll-web')]);
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).toHaveBeenCalledWith(expect.objectContaining({
|
||||
intent: 'fetch',
|
||||
applicationId: 'app-poll',
|
||||
stackName: 'poll-web',
|
||||
trigger: 'poll',
|
||||
}));
|
||||
});
|
||||
|
||||
it('evaluates an application whose retry_at has arrived, tagged as a retry trigger', async () => {
|
||||
const app = { ...directApplicationFixture('app-retry', 'retry-web'), retry_at: Date.now() - 1_000 };
|
||||
mockDue([], [app]);
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).toHaveBeenCalledWith(expect.objectContaining({
|
||||
applicationId: 'app-retry',
|
||||
trigger: 'retry',
|
||||
}));
|
||||
});
|
||||
|
||||
it('evaluates an application due for both poll and retry exactly once', async () => {
|
||||
const app = { ...directApplicationFixture('app-both', 'both-web'), retry_at: Date.now() - 1_000 };
|
||||
mockDue([app], [app]);
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('does not re-evaluate an application still in flight from a previous tick', async () => {
|
||||
mockDue([directApplicationFixture('app-slow', 'slow-web')]);
|
||||
let settleFirstCall!: (result: ReconcileResult) => void;
|
||||
const firstCall = new Promise<ReconcileResult>((resolve) => { settleFirstCall = resolve; });
|
||||
const reconcile = spyOnReconcile().mockReturnValue(firstCall);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
expect(reconcile).toHaveBeenCalledTimes(1);
|
||||
|
||||
// A second tick fires while the first evaluation is still pending.
|
||||
await advanceOneTick();
|
||||
expect(reconcile).toHaveBeenCalledTimes(1);
|
||||
|
||||
settleFirstCall(okResult);
|
||||
await Promise.resolve();
|
||||
await Promise.resolve();
|
||||
|
||||
// Now that the first evaluation has settled, a later tick may pick it up again.
|
||||
await advanceOneTick();
|
||||
expect(reconcile).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('recovers on the next tick after a store query throws, rather than dying permanently', async () => {
|
||||
mockDue([directApplicationFixture('app-recovers', 'recovers-web')]);
|
||||
vi.spyOn(GitOpsStore.getInstance(), 'listSourcesDueForPoll').mockImplementationOnce(() => {
|
||||
throw new Error('database is locked');
|
||||
});
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
expect(reconcile).not.toHaveBeenCalled();
|
||||
|
||||
await advanceOneTick();
|
||||
expect(reconcile).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('releases the in-flight slot for an application whose reconcile rejects', async () => {
|
||||
mockDue([directApplicationFixture('app-rejects', 'rejects-web')]);
|
||||
const reconcile = spyOnReconcile().mockRejectedValue(new Error('boom'));
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('does not evaluate anything after stop', async () => {
|
||||
mockDue([directApplicationFixture('app-stopped', 'stopped-web')]);
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
|
||||
controller.start();
|
||||
controller.stop();
|
||||
await advanceOneTick();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('does not double-arm when start() is called reentrantly from within an in-flight evaluation', async () => {
|
||||
mockDue([directApplicationFixture('app-reentrant-start', 'reentrant-start-web')]);
|
||||
// tick() nulls `timer` before scanning, so a start() call landing
|
||||
// synchronously during that scan must not see a false "not running"
|
||||
// reading and arm a second timer.
|
||||
spyOnReconcile().mockImplementation(() => {
|
||||
controller.start();
|
||||
return Promise.resolve(okResult);
|
||||
});
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(vi.getTimerCount()).toBe(1);
|
||||
});
|
||||
|
||||
it('logs rather than silently skipping an application with no stack_name', async () => {
|
||||
mockDue([{ ...directApplicationFixture('app-no-stack', 'no-stack-web'), stack_name: null }]);
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).not.toHaveBeenCalled();
|
||||
expect(warn).toHaveBeenCalledWith(expect.stringContaining('app-no-stack'));
|
||||
});
|
||||
|
||||
it('restartPolling never leaves two timers running', () => {
|
||||
controller.start();
|
||||
controller.restartPolling();
|
||||
controller.restartPolling();
|
||||
|
||||
expect(vi.getTimerCount()).toBe(1);
|
||||
});
|
||||
|
||||
it('start is a no-op when already running', async () => {
|
||||
mockDue([directApplicationFixture('app-double-start', 'double-start-web')]);
|
||||
const reconcile = spyOnReconcile().mockResolvedValue(okResult);
|
||||
|
||||
controller.start();
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(reconcile).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,7 @@
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest';
|
||||
import request from 'supertest';
|
||||
import jwt from 'jsonwebtoken';
|
||||
import crypto from 'crypto';
|
||||
import { setupTestDb, cleanupTestDb, TEST_USERNAME, TEST_JWT_SECRET } from './helpers/setupTestDb';
|
||||
|
||||
let tmpDir: string;
|
||||
@@ -237,4 +238,142 @@ describe('node-aware Git source webhooks', () => {
|
||||
expect(history[0].status).toBe('success');
|
||||
expect(history[0].error).toMatch(/debounced/i);
|
||||
});
|
||||
|
||||
it('forwards a provider delivery id to a remote node under the webhook namespace', async () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
const remoteNodeId = db.addNode({
|
||||
name: 'remote-delivery-id-webhook',
|
||||
type: 'remote',
|
||||
compose_dir: '/tmp',
|
||||
is_default: false,
|
||||
api_url: 'http://remote-delivery.example',
|
||||
api_token: 'remote-token',
|
||||
});
|
||||
const webhookId = db.addWebhook({
|
||||
node_id: remoteNodeId,
|
||||
name: 'delivery id remote git',
|
||||
stack_name: 'remote-stack',
|
||||
action: 'git-pull',
|
||||
secret: WebhookService.getInstance().generateSecret(),
|
||||
enabled: true,
|
||||
});
|
||||
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue(
|
||||
new Response(JSON.stringify({ status: 'success', message: 'Fetched.' }), { status: 200 }),
|
||||
);
|
||||
|
||||
const webhook = db.getWebhook(webhookId)!;
|
||||
const deliverySourceId = db.getGlobalSettings().delivery_source_id;
|
||||
const result = await WebhookService.getInstance().execute(
|
||||
webhook,
|
||||
'git-pull',
|
||||
'test',
|
||||
undefined,
|
||||
'provider-delivery-1',
|
||||
);
|
||||
|
||||
expect(result.success).toBe(true);
|
||||
expect(fetchSpy).toHaveBeenCalledWith(
|
||||
'http://remote-delivery.example/api/stacks/remote-stack/git-source/webhook-pull',
|
||||
expect.objectContaining({
|
||||
method: 'POST',
|
||||
body: expect.stringContaining(`"deliveryId":"webhook:${deliverySourceId}:${webhookId}:provider-delivery-1"`),
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it('passes the same producer-scoped delivery identity to a local Git source', async () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
const nodeId = db.getDefaultNode()!.id;
|
||||
const webhookId = db.addWebhook({
|
||||
node_id: nodeId,
|
||||
name: 'delivery id local git',
|
||||
stack_name: 'local-delivery-stack',
|
||||
action: 'git-pull',
|
||||
secret: WebhookService.getInstance().generateSecret(),
|
||||
enabled: true,
|
||||
});
|
||||
const webhook = db.getWebhook(webhookId)!;
|
||||
const deliverySourceId = db.getGlobalSettings().delivery_source_id;
|
||||
const { FileSystemService } = await import('../services/FileSystemService');
|
||||
const { GitSourceService } = await import('../services/GitSourceService');
|
||||
const stacksSpy = vi.spyOn(FileSystemService.prototype, 'getStacks')
|
||||
.mockResolvedValue(['local-delivery-stack']);
|
||||
const pullSpy = vi.spyOn(GitSourceService.getInstance(), 'handleWebhookPull')
|
||||
.mockResolvedValue({ status: 'success', message: 'Fetched.' });
|
||||
|
||||
try {
|
||||
const result = await WebhookService.getInstance().execute(
|
||||
webhook,
|
||||
'git-pull',
|
||||
'test',
|
||||
undefined,
|
||||
'provider-delivery-2',
|
||||
);
|
||||
|
||||
expect(result).toEqual({ success: true, duration_ms: expect.any(Number) });
|
||||
expect(pullSpy).toHaveBeenCalledWith(
|
||||
'local-delivery-stack',
|
||||
true,
|
||||
`webhook:${deliverySourceId}:${webhookId}:provider-delivery-2`,
|
||||
);
|
||||
|
||||
const secondWebhookId = db.addWebhook({
|
||||
node_id: nodeId,
|
||||
name: 'second delivery id local git',
|
||||
stack_name: 'local-delivery-stack',
|
||||
action: 'git-pull',
|
||||
secret: webhook.secret,
|
||||
enabled: true,
|
||||
});
|
||||
pullSpy.mockClear();
|
||||
await WebhookService.getInstance().execute(
|
||||
db.getWebhook(secondWebhookId)!,
|
||||
'git-pull',
|
||||
'test',
|
||||
undefined,
|
||||
'provider-delivery-2',
|
||||
);
|
||||
expect(pullSpy).toHaveBeenCalledWith(
|
||||
'local-delivery-stack',
|
||||
true,
|
||||
`webhook:${deliverySourceId}:${secondWebhookId}:provider-delivery-2`,
|
||||
);
|
||||
|
||||
db.updateGlobalSetting('delivery_source_id', 'second-control-source');
|
||||
pullSpy.mockClear();
|
||||
await WebhookService.getInstance().execute(
|
||||
webhook,
|
||||
'git-pull',
|
||||
'test',
|
||||
undefined,
|
||||
'provider-delivery-2',
|
||||
);
|
||||
expect(pullSpy).toHaveBeenCalledWith(
|
||||
'local-delivery-stack',
|
||||
true,
|
||||
`webhook:second-control-source:${webhookId}:provider-delivery-2`,
|
||||
);
|
||||
db.updateGlobalSetting('delivery_source_id', deliverySourceId!);
|
||||
|
||||
const oversizedDeliveryId = 'x'.repeat(300);
|
||||
const boundedId = crypto.createHash('sha256').update(oversizedDeliveryId).digest('hex');
|
||||
pullSpy.mockClear();
|
||||
await WebhookService.getInstance().execute(
|
||||
webhook,
|
||||
'git-pull',
|
||||
'test',
|
||||
undefined,
|
||||
oversizedDeliveryId,
|
||||
);
|
||||
expect(pullSpy).toHaveBeenCalledWith(
|
||||
'local-delivery-stack',
|
||||
true,
|
||||
`webhook:${deliverySourceId}:${webhookId}:sha256:${boundedId}`,
|
||||
);
|
||||
} finally {
|
||||
if (deliverySourceId) db.updateGlobalSetting('delivery_source_id', deliverySourceId);
|
||||
stacksSpy.mockRestore();
|
||||
pullSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -251,6 +251,55 @@ describe('POST /api/webhooks/:id/trigger: authenticated happy path', () => {
|
||||
expect(res.body).toMatchObject({ action: 'start' });
|
||||
});
|
||||
|
||||
it('extracts a recognized provider delivery header and passes it through to execute', async () => {
|
||||
const { id, secret } = createWebhook({ action: 'stop' });
|
||||
const body = '{}';
|
||||
const executeSpy = vi.spyOn(WebhookService.getInstance(), 'execute').mockResolvedValue({ success: true, duration_ms: 0 });
|
||||
|
||||
try {
|
||||
await request(app)
|
||||
.post(`/api/webhooks/${id}/trigger`)
|
||||
.set('Content-Type', 'application/json')
|
||||
.set('X-Webhook-Signature', sign(body, secret))
|
||||
.set('X-GitHub-Delivery', 'gh-delivery-123')
|
||||
.send(body);
|
||||
|
||||
expect(executeSpy).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id }),
|
||||
'stop',
|
||||
expect.anything(),
|
||||
true,
|
||||
'gh-delivery-123',
|
||||
);
|
||||
} finally {
|
||||
executeSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('passes no delivery id through when the caller sends no recognized header', async () => {
|
||||
const { id, secret } = createWebhook({ action: 'stop' });
|
||||
const body = '{}';
|
||||
const executeSpy = vi.spyOn(WebhookService.getInstance(), 'execute').mockResolvedValue({ success: true, duration_ms: 0 });
|
||||
|
||||
try {
|
||||
await request(app)
|
||||
.post(`/api/webhooks/${id}/trigger`)
|
||||
.set('Content-Type', 'application/json')
|
||||
.set('X-Webhook-Signature', sign(body, secret))
|
||||
.send(body);
|
||||
|
||||
expect(executeSpy).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ id }),
|
||||
'stop',
|
||||
expect.anything(),
|
||||
true,
|
||||
undefined,
|
||||
);
|
||||
} finally {
|
||||
executeSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('rejects an unknown action override with 400 after the signature passes (L2)', async () => {
|
||||
const { id, secret } = createWebhook();
|
||||
const body = '{"action":"nuke-the-cluster"}';
|
||||
|
||||
@@ -9,6 +9,7 @@ import { FleetSyncRetryService } from '../services/FleetSyncRetryService';
|
||||
import { SuppressionRetractionRetryService } from '../services/SuppressionRetractionRetryService';
|
||||
import { DockerEventManager } from '../services/DockerEventManager';
|
||||
import { ImageUpdateService } from '../services/ImageUpdateService';
|
||||
import { SourceController } from '../services/gitops/SourceController';
|
||||
import { SchedulerService } from '../services/SchedulerService';
|
||||
import { MfaService } from '../services/MfaService';
|
||||
import { MeshService } from '../services/MeshService';
|
||||
@@ -49,6 +50,9 @@ export function installShutdownHandlers(server: Server): void {
|
||||
try { ImageUpdateService.getInstance().stop(); } catch (e) {
|
||||
console.warn('[Shutdown] ImageUpdateService cleanup failed:', (e as Error).message);
|
||||
}
|
||||
try { SourceController.getInstance().stop(); } catch (e) {
|
||||
console.warn('[Shutdown] SourceController cleanup failed:', (e as Error).message);
|
||||
}
|
||||
try { SchedulerService.getInstance().stop(); } catch (e) {
|
||||
console.warn('[Shutdown] SchedulerService cleanup failed:', (e as Error).message);
|
||||
}
|
||||
|
||||
@@ -28,10 +28,11 @@ import { applyPilotModeCapabilityFilter } from '../services/CapabilityRegistry';
|
||||
import { PilotTunnelManager } from '../services/PilotTunnelManager';
|
||||
import { PilotMetrics } from '../services/PilotMetrics';
|
||||
import { invalidateRemoteMetaCache } from '../helpers/cacheInvalidation';
|
||||
import { sweepStaleTempDirs as sweepStaleGitTempDirs, sweepGitManifestOrphans } from '../services/GitSourceService';
|
||||
import { GitSourceService, sweepStaleTempDirs as sweepStaleGitTempDirs, sweepGitManifestOrphans } from '../services/GitSourceService';
|
||||
import { assertCreatesSettled, reclassifyInterruptedOperations, resolveInterruptedCreates } from '../services/gitops/createRecovery';
|
||||
import { loadMigrationManifests, migrateDirectGitStacks, migrateInlineBlueprints } from '../services/gitops/migrate';
|
||||
import { setGitOpsEventSink } from '../services/gitops/publish';
|
||||
import { SourceController } from '../services/gitops/SourceController';
|
||||
import { NotificationService } from '../services/NotificationService';
|
||||
import { sanitizeForLog } from '../utils/safeLog';
|
||||
import { PORT } from '../helpers/constants';
|
||||
@@ -98,6 +99,33 @@ function clearSelfContainerNotificationRouting(): void {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconcile-attempt recovery, then the managed-area sweep, in that fixed
|
||||
* order: an attempt reserved but never settled (a crash between the two)
|
||||
* must be resolved from durable state before the sweep or
|
||||
* SourceController's own timer (started later in startServer, after
|
||||
* registry delivery recovery settles per AUD-36) can race a recovery pass
|
||||
* over the same attempts. Exported so this ordering is directly testable
|
||||
* without driving the rest of startServer's unrelated service
|
||||
* initialization.
|
||||
*/
|
||||
export async function runGitOpsSourceRecovery(): Promise<void> {
|
||||
try {
|
||||
await GitSourceService.getInstance().recoverUnsettledReconcileAttempts();
|
||||
} catch (err) {
|
||||
console.error('[GitSource] Reconcile-attempt recovery failed:', err instanceof Error ? err.stack ?? err.message : String(err));
|
||||
}
|
||||
|
||||
// The managed-area sweep follows. It preserves anything whose ownership it
|
||||
// cannot prove, so a failure here can only leave files behind, never remove
|
||||
// the wrong ones, and retrying next boot is safe.
|
||||
try {
|
||||
await sweepGitManifestOrphans();
|
||||
} catch (err) {
|
||||
console.warn('[GitManifest] Managed-area sweep failed:', err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the startup sequence: stack-directory migration, service initialization,
|
||||
* background watchdogs, then bind the HTTP server. The caller passes the
|
||||
@@ -233,14 +261,9 @@ export async function startServer(server: Server): Promise<void> {
|
||||
console.error('[GitOps] Migration of pre-existing blueprints failed:', err instanceof Error ? err.stack ?? err.message : String(err));
|
||||
}
|
||||
|
||||
// The managed-area sweep follows. It preserves anything whose ownership it
|
||||
// cannot prove, so a failure here can only leave files behind, never remove
|
||||
// the wrong ones, and retrying next boot is safe.
|
||||
try {
|
||||
await sweepGitManifestOrphans();
|
||||
} catch (err) {
|
||||
console.warn('[GitManifest] Managed-area sweep failed:', err instanceof Error ? err.message : String(err));
|
||||
}
|
||||
// Both steps must settle before SourceController starts below; see that
|
||||
// function's own doc comment for why.
|
||||
await runGitOpsSourceRecovery();
|
||||
|
||||
// Registry delivery recovery sweeps must settle before any mutation-capable
|
||||
// producer starts (AUD-36).
|
||||
@@ -287,6 +310,7 @@ export async function startServer(server: Server): Promise<void> {
|
||||
FleetSyncRetryService.getInstance().start();
|
||||
SuppressionRetractionRetryService.getInstance().start();
|
||||
ImageUpdateService.getInstance().start();
|
||||
SourceController.getInstance().start();
|
||||
SchedulerService.getInstance().start();
|
||||
MfaService.getInstance().start();
|
||||
MeshService.getInstance().start().catch((err) => {
|
||||
|
||||
@@ -5,7 +5,7 @@ import { GitProjectManifestService } from '../services/GitProjectManifestService
|
||||
import { FileSystemService } from '../services/FileSystemService';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
import { CryptoService } from '../services/CryptoService';
|
||||
import { requirePermission } from '../middleware/permissions';
|
||||
import { checkPermission, requirePermission } from '../middleware/permissions';
|
||||
import { classifySourceRow, satisfiesGitOpsRead } from '../services/gitops/readAuth';
|
||||
import { NOT_APPLICABLE_REVISION, projectStackRevision, stackResourceSet } from '../helpers/gitopsResponse';
|
||||
import { respondWithHistory } from '../helpers/gitopsHistoryPage';
|
||||
@@ -27,6 +27,8 @@ import { assertSafeOutboundHostname, resolveSafeOutboundHostname, UnsafeOutbound
|
||||
const MAX_BRANCH_LENGTH = REF_MAX_LEN;
|
||||
const MAX_ENV_PATH_LENGTH = 1024;
|
||||
const MAX_TOKEN_LENGTH = 8192;
|
||||
const MAX_SUSPEND_REASON_LENGTH = 512;
|
||||
const MAX_WEBHOOK_DELIVERY_ID_LENGTH = 512;
|
||||
|
||||
/**
|
||||
* Shared handler for the "browse repository" compose-file picker: validate the
|
||||
@@ -584,14 +586,28 @@ stackGitSourceRouter.post('/:stackName/git-source/webhook-pull', async (req: Req
|
||||
return;
|
||||
}
|
||||
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
||||
const deliveryId = req.body?.deliveryId;
|
||||
if (
|
||||
deliveryId !== undefined
|
||||
&& (typeof deliveryId !== 'string' || !deliveryId.trim() || deliveryId.length > MAX_WEBHOOK_DELIVERY_ID_LENGTH)
|
||||
) {
|
||||
res.status(400).json({ error: 'deliveryId must be a non-empty string of at most 512 characters' });
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const source = GitSourceService.getInstance().get(stackName);
|
||||
const service = GitSourceService.getInstance();
|
||||
const source = service.get(stackName);
|
||||
if (!source) {
|
||||
res.status(404).json({ error: 'No Git source configured for this stack', status: 'error' });
|
||||
return;
|
||||
}
|
||||
if (source.auto_apply_on_webhook && source.auto_deploy_on_apply && !requirePermission(req, res, 'stack:deploy', 'stack', stackName)) return;
|
||||
const result = await GitSourceService.getInstance().handleWebhookPull(stackName);
|
||||
const normalizedDeliveryId = deliveryId?.trim();
|
||||
const deployAuthorized = checkPermission(req, 'stack:deploy', 'stack', stackName);
|
||||
if (service.webhookDeliveryRequiresDeploy(stackName, normalizedDeliveryId) && !deployAuthorized) {
|
||||
requirePermission(req, res, 'stack:deploy', 'stack', stackName);
|
||||
return;
|
||||
}
|
||||
const result = await service.handleWebhookPull(stackName, deployAuthorized, normalizedDeliveryId);
|
||||
// Map the outcome to a real HTTP status so a Git provider sees a 4xx on
|
||||
// failure instead of a 200 with an error body (which it would read as
|
||||
// "delivered fine, stop retrying").
|
||||
@@ -616,6 +632,64 @@ stackGitSourceRouter.post('/:stackName/git-source/dismiss-pending', async (req:
|
||||
}
|
||||
});
|
||||
|
||||
stackGitSourceRouter.post('/:stackName/git-source/suspend', async (req: Request, res: Response): Promise<void> => {
|
||||
const stackName = req.params.stackName as string;
|
||||
if (!isValidStackName(stackName)) {
|
||||
res.status(400).json({ error: 'Invalid stack name' });
|
||||
return;
|
||||
}
|
||||
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
||||
const { reason: rawReason } = req.body ?? {};
|
||||
const reason = typeof rawReason === 'string' ? rawReason : undefined;
|
||||
if (reason !== undefined && reason.length > MAX_SUSPEND_REASON_LENGTH) {
|
||||
res.status(400).json({ error: 'reason is too long' });
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const result = await GitSourceService.getInstance().suspend(stackName, {
|
||||
actor: req.user?.username ?? 'unknown',
|
||||
reason,
|
||||
});
|
||||
res.json(result);
|
||||
} catch (error) {
|
||||
sendGitSourceError(res, error);
|
||||
}
|
||||
});
|
||||
|
||||
stackGitSourceRouter.post('/:stackName/git-source/resume', async (req: Request, res: Response): Promise<void> => {
|
||||
const stackName = req.params.stackName as string;
|
||||
if (!isValidStackName(stackName)) {
|
||||
res.status(400).json({ error: 'Invalid stack name' });
|
||||
return;
|
||||
}
|
||||
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
||||
try {
|
||||
const result = await GitSourceService.getInstance().resume(stackName, {
|
||||
actor: req.user?.username ?? 'unknown',
|
||||
});
|
||||
res.json(result);
|
||||
} catch (error) {
|
||||
sendGitSourceError(res, error);
|
||||
}
|
||||
});
|
||||
|
||||
stackGitSourceRouter.post('/:stackName/git-source/retry', async (req: Request, res: Response): Promise<void> => {
|
||||
const stackName = req.params.stackName as string;
|
||||
if (!isValidStackName(stackName)) {
|
||||
res.status(400).json({ error: 'Invalid stack name' });
|
||||
return;
|
||||
}
|
||||
if (!requirePermission(req, res, 'stack:edit', 'stack', stackName)) return;
|
||||
try {
|
||||
const result = await GitSourceService.getInstance().retry(stackName, {
|
||||
actor: req.user?.username ?? 'unknown',
|
||||
});
|
||||
res.json(result);
|
||||
} catch (error) {
|
||||
sendGitSourceError(res, error);
|
||||
}
|
||||
});
|
||||
|
||||
// Edit-mode repo browse for an existing stack: gated by stack:edit so a user who
|
||||
// can edit (but not create) stacks can re-pick files, and reuses the stored token
|
||||
// when the request omits one.
|
||||
|
||||
@@ -12,6 +12,34 @@ function isWebhookAction(value: unknown): value is WebhookAction {
|
||||
return typeof value === 'string' && (VALID_WEBHOOK_ACTIONS as readonly string[]).includes(value);
|
||||
}
|
||||
|
||||
// Recognized per-delivery identity headers, in priority order. This
|
||||
// endpoint is a generic HMAC-signed trigger, not a provider-specific
|
||||
// receiver, so a well-known provider header is the only delivery identity
|
||||
// available, and only when the caller happens to send one. The value is a
|
||||
// stable delivery identity available. WebhookService namespaces the value by
|
||||
// control instance and configured webhook before it reaches GitSourceService,
|
||||
// so two producers cannot collide on the same provider-assigned id.
|
||||
//
|
||||
// Each header is still the provider's actual per-delivery identity rather
|
||||
// than a webhook- or connection-level id that stays constant across every
|
||||
// delivery from that source: GitHub's X-GitHub-Delivery GUID changes per
|
||||
// delivery (it is stable only across redeliveries of the same delivery);
|
||||
// GitLab's is Webhook-ID, the modern name for its Idempotency-Key, not
|
||||
// X-Gitlab-Event-UUID, which tracks recursive-trigger chains and can
|
||||
// repeat across genuinely distinct events; Bitbucket's is X-Request-UUID,
|
||||
// not X-Hook-UUID, which identifies the webhook configuration itself.
|
||||
// Picking the wrong one would deduplicate genuinely distinct pushes.
|
||||
const DELIVERY_ID_HEADERS = ['x-github-delivery', 'webhook-id', 'idempotency-key', 'x-request-uuid', 'x-webhook-delivery-id'] as const;
|
||||
|
||||
function deliveryIdFromHeaders(headers: Request['headers']): string | undefined {
|
||||
for (const name of DELIVERY_ID_HEADERS) {
|
||||
const value = headers[name];
|
||||
const first = Array.isArray(value) ? value[0] : value;
|
||||
if (first) return first;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
export const webhooksRouter = Router();
|
||||
|
||||
webhooksRouter.get('/', authMiddleware, async (req: Request, res: Response): Promise<void> => {
|
||||
@@ -193,6 +221,7 @@ webhooksRouter.post('/:id/trigger', webhookTriggerLimiter, async (req: Request,
|
||||
action = overrideAction;
|
||||
}
|
||||
const triggerSource = req.headers['user-agent'] || req.ip || null;
|
||||
const deliveryId = deliveryIdFromHeaders(req.headers);
|
||||
|
||||
// Execute asynchronously; return 202 immediately.
|
||||
res.status(202).json({ message: 'Webhook accepted', action });
|
||||
@@ -202,7 +231,7 @@ webhooksRouter.post('/:id/trigger', webhookTriggerLimiter, async (req: Request,
|
||||
// dispatch the action still completes and recordExecution swallows the
|
||||
// FK error from the CASCADE. atomic is unconditionally true, so the
|
||||
// deploy/pull paths always run in atomic mode here.
|
||||
svc.execute(webhook, action, triggerSource, true).catch(err => {
|
||||
svc.execute(webhook, action, triggerSource, true, deliveryId).catch(err => {
|
||||
console.error(`[Webhooks] Execution error for webhook ${id}:`, err);
|
||||
});
|
||||
} catch (error) {
|
||||
|
||||
@@ -1976,6 +1976,24 @@ export class DatabaseService {
|
||||
// existing installs already have. New installs get it from the
|
||||
// CREATE TABLE; older DBs need the additive column here.
|
||||
maybeAddCol('gitops_applications', 'source_suspended_reason', 'TEXT NULL');
|
||||
// Portable accepted-generation contract fields. Additive and
|
||||
// nullable: existing generation rows decode these as an explicit
|
||||
// limitation rather than invented evidence.
|
||||
maybeAddCol('gitops_generations', 'portable_manifest_json', 'TEXT NULL');
|
||||
maybeAddCol('gitops_generations', 'compose_inputs_json', 'TEXT NULL');
|
||||
maybeAddCol('gitops_generations', 'source_policy_evidence_json', 'TEXT NULL');
|
||||
maybeAddCol('gitops_generations', 'security_policy_evidence_json', 'TEXT NULL');
|
||||
maybeAddCol('gitops_generations', 'support_requirements_json', 'TEXT NULL');
|
||||
maybeAddCol('gitops_generations', 'compatibility_requirements_json', 'TEXT NULL');
|
||||
// Controller-owned bookkeeping (source policy, poll cadence, attempt
|
||||
// sequence). New installs get these from the CREATE TABLE; older DBs
|
||||
// need the additive columns here. Existing installations must not
|
||||
// start unattended polling, so poll_interval_secs and next_poll_at
|
||||
// stay NULL until an operator (or the migration below) sets one.
|
||||
maybeAddCol('gitops_applications', 'source_policy', "TEXT NOT NULL DEFAULT 'manual' CHECK (source_policy IN ('manual','review','automatic'))");
|
||||
maybeAddCol('gitops_applications', 'poll_interval_secs', 'INTEGER NULL');
|
||||
maybeAddCol('gitops_applications', 'next_poll_at', 'INTEGER NULL');
|
||||
maybeAddCol('gitops_applications', 'attempt_seq', 'INTEGER NOT NULL DEFAULT 0');
|
||||
|
||||
// Distributed API model columns
|
||||
maybeAddCol('nodes', 'api_url', "TEXT DEFAULT ''");
|
||||
|
||||
@@ -51,6 +51,10 @@ export const MANAGED_ROOT_NAME = 'git-managed';
|
||||
export const MANIFEST_FILENAME = 'manifest.v1.json';
|
||||
export const PROMOTION_MARKER = 'promotion.json';
|
||||
export const CANDIDATE_COMPLETE_MARKER = '.candidate-complete';
|
||||
|
||||
type CandidateClaims =
|
||||
| { complete: true; dirs: ReadonlySet<string> }
|
||||
| { complete: false };
|
||||
export const GENERATIONS_DIR = 'generations';
|
||||
const DETACH_RECOVERY_MARKER = 'detach-recovery.v1.json';
|
||||
|
||||
@@ -421,10 +425,19 @@ export class GitProjectManifestService {
|
||||
|
||||
/** Atomic manifest write (tmp + rename). */
|
||||
async writeManifest(stackName: string, manifest: GitProjectManifest): Promise<void> {
|
||||
const dir = this.managedRoot(stackName);
|
||||
await fs.promises.mkdir(dir, { recursive: true });
|
||||
const target = path.join(dir, MANIFEST_FILENAME);
|
||||
const tmp = path.join(dir, `${MANIFEST_FILENAME}.tmp`);
|
||||
// Inline barrier at the mkdir/write/rename sinks (CodeQL path-injection):
|
||||
// confine the resolved managed directory to the managed area before any
|
||||
// filesystem call touches it, then confine the filenames joined onto it.
|
||||
const root = path.resolve(this.managedRoot(stackName));
|
||||
if (!root.startsWith(managedAreaBase() + path.sep)) {
|
||||
throw Object.assign(new Error('Path escapes the managed area'), { code: 'INVALID_PATH' });
|
||||
}
|
||||
const target = path.resolve(root, MANIFEST_FILENAME);
|
||||
const tmp = path.resolve(root, `${MANIFEST_FILENAME}.tmp`);
|
||||
if (!target.startsWith(root + path.sep) || !tmp.startsWith(root + path.sep)) {
|
||||
throw Object.assign(new Error('Path escapes managed project directory'), { code: 'INVALID_PATH' });
|
||||
}
|
||||
await fs.promises.mkdir(root, { recursive: true });
|
||||
await fs.promises.writeFile(tmp, JSON.stringify(manifest, null, 2), 'utf8');
|
||||
await fs.promises.rename(tmp, target);
|
||||
}
|
||||
@@ -1268,12 +1281,20 @@ export class GitProjectManifestService {
|
||||
* is finalized; an uncommitted promotion restores the prior generation.
|
||||
* A third state is treated as an operator edit, so recovery declines and
|
||||
* flags migration_required. Interrupted detach snapshots are restored first.
|
||||
* Complete candidate claims hold the directory basenames that durable state
|
||||
* still references. Incomplete claims preserve every candidate because
|
||||
* ownership is uncertain.
|
||||
*/
|
||||
async sweepManagedArea(
|
||||
stackName: string,
|
||||
opts: { repoUrl: string; branch: string; stackExists: boolean },
|
||||
opts: {
|
||||
repoUrl: string;
|
||||
branch: string;
|
||||
stackExists: boolean;
|
||||
candidateClaims: CandidateClaims;
|
||||
},
|
||||
): Promise<void> {
|
||||
const { repoUrl, branch, stackExists } = opts;
|
||||
const { repoUrl, branch, stackExists, candidateClaims } = opts;
|
||||
if (!stackExists) {
|
||||
await this.deleteManagedArea(stackName);
|
||||
return;
|
||||
@@ -1359,44 +1380,64 @@ export class GitProjectManifestService {
|
||||
}
|
||||
}
|
||||
|
||||
// Orphan candidates: incomplete or stale.
|
||||
if (!candidateClaims.complete) {
|
||||
await this.flagRecoveryRequired(
|
||||
stackName,
|
||||
`candidate ownership for ${sanitizeForLog(stackName)} could not be established`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// With a complete claim inventory, reap candidates that are incomplete
|
||||
// or stale and unclaimed.
|
||||
const dir = this.generationsDir(stackName);
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
const entries = await fs.promises.readdir(dir, { withFileTypes: true });
|
||||
const now = Date.now();
|
||||
const areaBase = managedAreaBase();
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !entry.name.startsWith('candidate-')) continue;
|
||||
const abs = path.resolve(dir, entry.name);
|
||||
// Inline containment barrier at the removal sink (see
|
||||
// `managedAreaBase`): the analyzer credits this literal
|
||||
// comparison, not the positional check below it.
|
||||
if (!abs.startsWith(areaBase + path.sep)) {
|
||||
console.warn(`[GitManifest] refusing to reap orphan candidate ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it resolves outside the managed area`);
|
||||
continue;
|
||||
}
|
||||
// Same positional barrier as generation pruning: the boot sweep
|
||||
// reaps candidate directories nobody claims, which is precisely
|
||||
// the kind of unattended delete a planted link would steer.
|
||||
if (!await isRealPathAtManagedLocation(abs)) {
|
||||
console.warn(`[GitManifest] refusing to reap orphan candidate ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it is not at its own location in the managed area`);
|
||||
continue;
|
||||
}
|
||||
const complete = await fs.promises
|
||||
.access(path.join(abs, CANDIDATE_COMPLETE_MARKER))
|
||||
.then(() => true)
|
||||
.catch(() => false);
|
||||
if (!complete) {
|
||||
await fs.promises.rm(abs, { recursive: true, force: true });
|
||||
continue;
|
||||
}
|
||||
const st = await fs.promises.stat(abs);
|
||||
if (now - st.mtimeMs > ORPHAN_CANDIDATE_AGE_MS) {
|
||||
await fs.promises.rm(abs, { recursive: true, force: true });
|
||||
}
|
||||
entries = await fs.promises.readdir(dir, { withFileTypes: true });
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException).code === 'ENOENT') return;
|
||||
throw e;
|
||||
}
|
||||
const now = Date.now();
|
||||
const areaBase = managedAreaBase();
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !entry.name.startsWith('candidate-')) continue;
|
||||
// A claimed candidate is still needed regardless of age or
|
||||
// completeness. Claims come from application pointers, the
|
||||
// source pending record, and generation rows linked to
|
||||
// unsettled attempts. Together they are the durable ownership
|
||||
// record that makes recovery before cleanup safe.
|
||||
if (candidateClaims.dirs.has(entry.name)) continue;
|
||||
const abs = path.resolve(dir, entry.name);
|
||||
// Inline containment barrier at the removal sink (see
|
||||
// `managedAreaBase`): the analyzer credits this literal
|
||||
// comparison, not the positional check below it.
|
||||
if (!abs.startsWith(areaBase + path.sep)) {
|
||||
console.warn(`[GitManifest] refusing to reap orphan candidate ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it resolves outside the managed area`);
|
||||
continue;
|
||||
}
|
||||
// Same positional barrier as generation pruning: the boot sweep
|
||||
// reaps candidate directories nobody claims, which is precisely
|
||||
// the kind of unattended delete a planted link would steer.
|
||||
if (!await isRealPathAtManagedLocation(abs)) {
|
||||
console.warn(`[GitManifest] refusing to reap orphan candidate ${sanitizeForLog(entry.name)} for ${sanitizeForLog(stackName)}: it is not at its own location in the managed area`);
|
||||
continue;
|
||||
}
|
||||
let complete = true;
|
||||
try {
|
||||
await fs.promises.access(path.join(abs, CANDIDATE_COMPLETE_MARKER));
|
||||
} catch (e) {
|
||||
if ((e as NodeJS.ErrnoException).code !== 'ENOENT') throw e;
|
||||
complete = false;
|
||||
}
|
||||
if (!complete) {
|
||||
await fs.promises.rm(abs, { recursive: true, force: true });
|
||||
continue;
|
||||
}
|
||||
const st = await fs.promises.stat(abs);
|
||||
if (now - st.mtimeMs > ORPHAN_CANDIDATE_AGE_MS) {
|
||||
await fs.promises.rm(abs, { recursive: true, force: true });
|
||||
}
|
||||
} catch {
|
||||
// no generations dir yet
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -20,6 +20,7 @@ type ExecutionResult = { success: boolean; error?: string; duration_ms: number }
|
||||
type ExecutionStatus = 'success' | 'failure';
|
||||
|
||||
const REMOTE_WEBHOOK_REQUEST_TIMEOUT_MS = 30_000;
|
||||
const MAX_PROVIDER_DELIVERY_ID_LENGTH = 256;
|
||||
|
||||
// Maps a webhook lifecycle action to the per-stack lock action. 'pull' updates,
|
||||
// so it locks as 'update'; 'git-pull' is excluded (it locks inside GitSourceService).
|
||||
@@ -96,6 +97,7 @@ export class WebhookService {
|
||||
action: string,
|
||||
triggerSource: string | null,
|
||||
atomic?: boolean,
|
||||
deliveryId?: string,
|
||||
): Promise<ExecutionResult> {
|
||||
if (webhook.id === undefined) {
|
||||
throw new Error('Webhook must be loaded from the database before execution');
|
||||
@@ -110,11 +112,34 @@ export class WebhookService {
|
||||
return { success: false, error, duration_ms: 0 };
|
||||
}
|
||||
|
||||
const scopedDeliveryId = action === 'git-pull'
|
||||
? WebhookService.scopedDeliveryId(
|
||||
DatabaseService.getInstance().getGlobalSettings().delivery_source_id,
|
||||
webhookId,
|
||||
deliveryId,
|
||||
)
|
||||
: undefined;
|
||||
|
||||
if (node.type === 'remote') {
|
||||
return this.executeRemote(webhookId, nodeId, webhook.stack_name, action, triggerSource, atomic);
|
||||
return this.executeRemote(webhookId, nodeId, webhook.stack_name, action, triggerSource, atomic, scopedDeliveryId);
|
||||
}
|
||||
|
||||
return this.executeLocal(webhookId, nodeId, webhook.stack_name, action, triggerSource, atomic);
|
||||
return this.executeLocal(webhookId, nodeId, webhook.stack_name, action, triggerSource, atomic, scopedDeliveryId);
|
||||
}
|
||||
|
||||
/** Stable external identity, isolated to one configured webhook producer. */
|
||||
private static scopedDeliveryId(
|
||||
deliverySourceId: string | undefined,
|
||||
webhookId: number,
|
||||
deliveryId: string | undefined,
|
||||
): string | undefined {
|
||||
const normalized = deliveryId?.trim();
|
||||
if (!normalized) return undefined;
|
||||
if (!deliverySourceId) throw new Error('Webhook delivery source identity is not configured');
|
||||
const bounded = normalized.length <= MAX_PROVIDER_DELIVERY_ID_LENGTH
|
||||
? normalized
|
||||
: `sha256:${crypto.createHash('sha256').update(normalized).digest('hex')}`;
|
||||
return `webhook:${deliverySourceId}:${webhookId}:${bounded}`;
|
||||
}
|
||||
|
||||
public maskSecret(secret: string): string {
|
||||
@@ -129,6 +154,7 @@ export class WebhookService {
|
||||
action: string,
|
||||
triggerSource: string | null,
|
||||
atomic?: boolean,
|
||||
deliveryId?: string,
|
||||
): Promise<ExecutionResult> {
|
||||
const stacks = await FileSystemService.getInstance(nodeId).getStacks();
|
||||
if (!stacks.includes(stackName)) {
|
||||
@@ -142,7 +168,7 @@ export class WebhookService {
|
||||
// git-pull pulls then deploys through GitSourceService, which holds
|
||||
// the per-stack lock itself; locking here too would self-conflict.
|
||||
if (action === 'git-pull') {
|
||||
return this.executeLocalGitPull(webhookId, stackName, action, triggerSource, startTime);
|
||||
return this.executeLocalGitPull(webhookId, stackName, action, triggerSource, startTime, deliveryId);
|
||||
}
|
||||
const lockAction = WEBHOOK_LOCK_ACTION[action];
|
||||
if (!lockAction) throw new Error(`Unknown action: ${action}`);
|
||||
@@ -225,8 +251,9 @@ export class WebhookService {
|
||||
action: string,
|
||||
triggerSource: string | null,
|
||||
startTime: number,
|
||||
deliveryId?: string,
|
||||
): Promise<ExecutionResult> {
|
||||
const result = await GitSourceService.getInstance().handleWebhookPull(stackName);
|
||||
const result = await GitSourceService.getInstance().handleWebhookPull(stackName, true, deliveryId);
|
||||
const durationMs = Date.now() - startTime;
|
||||
if (result.status === 'error') {
|
||||
this.recordExecution(webhookId, action, 'failure', triggerSource, durationMs, result.message);
|
||||
@@ -255,6 +282,7 @@ export class WebhookService {
|
||||
action: string,
|
||||
triggerSource: string | null,
|
||||
atomic?: boolean,
|
||||
deliveryId?: string,
|
||||
): Promise<ExecutionResult> {
|
||||
const startTime = Date.now();
|
||||
try {
|
||||
@@ -263,7 +291,9 @@ export class WebhookService {
|
||||
: action === 'pull'
|
||||
? 'update'
|
||||
: action;
|
||||
const body = atomic === undefined ? undefined : { atomic };
|
||||
const body = action === 'git-pull'
|
||||
? { ...(atomic === undefined ? {} : { atomic }), ...(deliveryId ? { deliveryId } : {}) }
|
||||
: atomic === undefined ? undefined : { atomic };
|
||||
const response = await this.remoteStackRequest(nodeId, stackName, endpoint, 'POST', body);
|
||||
const durationMs = Date.now() - startTime;
|
||||
const payload = await response.json().catch(() => ({})) as { error?: string; message?: string; status?: string };
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
import { GitOpsStore } from './store';
|
||||
import { GitSourceService } from '../GitSourceService';
|
||||
import type { GitOpsApplicationRow } from './types';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
|
||||
/**
|
||||
* Background driver for unattended GitOps reconciliation: polls sources on
|
||||
* their configured interval and re-evaluates applications whose retry_at
|
||||
* has arrived, driving each through GitSourceService.reconcile().
|
||||
*
|
||||
* Scope for this delivery: a tick only issues a fetch-intent reconcile, the
|
||||
* same "detect and stage a candidate" step a manual pull performs. It does
|
||||
* not evaluate source policy or drive automatic acceptance/dispatch for a
|
||||
* newly staged candidate, and it does not yet resume a failure at the
|
||||
* specific stage it failed at (fetch vs. dispatch); every retry re-issues
|
||||
* a fetch. Both are real gaps against the source policy design, not silent
|
||||
* omissions: automatic-policy acceptance and stage-aware retry are follow-on
|
||||
* work once dispatchAcceptedGeneration() has a caller that decides when a
|
||||
* staged candidate should be accepted.
|
||||
*
|
||||
* One self-rescheduling timer drives the scan, matching ImageUpdateService.
|
||||
* The re-arm always runs, even when a scan throws, so one bad tick (a
|
||||
* locked database, a transient store error) never permanently stops the
|
||||
* driver. The per-application in-flight set, not the timer, is what keeps
|
||||
* one busy application from blocking another: the tick never awaits any
|
||||
* evaluation before rescheduling, so a slow application only pauses itself.
|
||||
*/
|
||||
export class SourceController {
|
||||
private static instance: SourceController;
|
||||
|
||||
private static readonly TICK_INTERVAL_MS = 60_000;
|
||||
|
||||
private timer: NodeJS.Timeout | null = null;
|
||||
private polling = false;
|
||||
// Bumped by cancelPending(), so by stop() and restartPolling(). tick() has
|
||||
// no internal await point today, so nothing can currently call either one
|
||||
// mid-tick; this is a second, currently-redundant line of defense against a
|
||||
// stale timer firing, kept cheap on purpose for when stage-aware retry (see
|
||||
// above) gives evaluate() a real yield point.
|
||||
private scheduleGeneration = 0;
|
||||
private readonly inFlight = new Set<string>();
|
||||
|
||||
private constructor() { }
|
||||
|
||||
static getInstance(): SourceController {
|
||||
if (!SourceController.instance) {
|
||||
SourceController.instance = new SourceController();
|
||||
}
|
||||
return SourceController.instance;
|
||||
}
|
||||
|
||||
/** Test-only: replace the singleton so timer/in-flight state never leaks between tests. */
|
||||
static resetForTests(): void {
|
||||
SourceController.instance = new SourceController();
|
||||
}
|
||||
|
||||
start(): void {
|
||||
// Guards on `polling`, not `timer`: tick() nulls `timer` before it
|
||||
// scans (so a stale timer reference can never block a restart), which
|
||||
// would otherwise let a start() call landing during that scan see a
|
||||
// false "not running" reading and arm a second timer.
|
||||
if (this.polling) return;
|
||||
this.polling = true;
|
||||
this.armNext();
|
||||
}
|
||||
|
||||
stop(): void {
|
||||
this.cancelPending();
|
||||
this.polling = false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reschedule the next tick without restarting: always clears any pending
|
||||
* timer first, so calling this any number of times in a row never leaves
|
||||
* more than one timer armed.
|
||||
*/
|
||||
restartPolling(): void {
|
||||
this.cancelPending();
|
||||
if (this.polling) {
|
||||
this.armNext();
|
||||
}
|
||||
}
|
||||
|
||||
isPolling(): boolean {
|
||||
return this.polling;
|
||||
}
|
||||
|
||||
/** Clear any armed timer and invalidate the tick it would have run. */
|
||||
private cancelPending(): void {
|
||||
this.scheduleGeneration++;
|
||||
if (this.timer) {
|
||||
clearTimeout(this.timer);
|
||||
this.timer = null;
|
||||
}
|
||||
}
|
||||
|
||||
private armNext(): void {
|
||||
const gen = this.scheduleGeneration;
|
||||
this.timer = setTimeout(() => { void this.tick(gen); }, SourceController.TICK_INTERVAL_MS);
|
||||
this.timer.unref();
|
||||
}
|
||||
|
||||
private async tick(gen: number): Promise<void> {
|
||||
if (!this.polling || gen !== this.scheduleGeneration) return;
|
||||
this.timer = null;
|
||||
try {
|
||||
this.scan();
|
||||
} catch (e) {
|
||||
console.error('[SourceController] scan failed:', e instanceof Error ? e.message : String(e));
|
||||
} finally {
|
||||
if (this.polling && gen === this.scheduleGeneration) {
|
||||
this.armNext();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Fire an evaluation for every due application without waiting for any
|
||||
* of them. An application already in the in-flight set is left for a
|
||||
* later tick instead of being queued behind its own still-running
|
||||
* evaluation. A row due for both poll and retry is evaluated once.
|
||||
*/
|
||||
private scan(): void {
|
||||
const now = Date.now();
|
||||
const store = GitOpsStore.getInstance();
|
||||
const due = new Map<string, GitOpsApplicationRow>();
|
||||
for (const app of store.listSourcesDueForPoll(now)) due.set(app.id, app);
|
||||
for (const app of store.listApplicationsDueForRetry(now)) due.set(app.id, app);
|
||||
|
||||
for (const app of due.values()) {
|
||||
if (this.inFlight.has(app.id)) continue;
|
||||
this.inFlight.add(app.id);
|
||||
this.evaluate(app).finally(() => this.inFlight.delete(app.id));
|
||||
}
|
||||
}
|
||||
|
||||
private async evaluate(app: GitOpsApplicationRow): Promise<void> {
|
||||
if (!app.stack_name) {
|
||||
console.warn(`[SourceController] Skipping ${sanitizeForLog(app.id)}: direct-mode application has no stack_name.`);
|
||||
return;
|
||||
}
|
||||
const isRetry = app.retry_at !== null && app.retry_at <= Date.now();
|
||||
try {
|
||||
await GitSourceService.getInstance().reconcile({
|
||||
intent: 'fetch',
|
||||
applicationId: app.id,
|
||||
stackName: app.stack_name,
|
||||
trigger: isRetry ? 'retry' : 'poll',
|
||||
actor: 'system:source-controller',
|
||||
});
|
||||
} catch (e) {
|
||||
console.error(
|
||||
`[SourceController] evaluation failed for ${sanitizeForLog(app.id)}:`,
|
||||
e instanceof Error ? e.message : String(e),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
import type { GitSourceErrorCode } from '../GitSourceService';
|
||||
import type { TransportFailureReason } from '../git/errors';
|
||||
|
||||
/**
|
||||
* Everything a controller attempt can fail with. `git_source_error` covers
|
||||
* every GitSourceError the fetch/apply path can throw, transport-classified
|
||||
* or not. The remaining kinds cover dispatch-stage failures that have no
|
||||
* GitSourceError at all (target binding, target availability, a deploy or
|
||||
* health failure after a successful apply, Blueprint evaluation, and an
|
||||
* interrupted or unknown-completion operation).
|
||||
*/
|
||||
export type FailureEvidence =
|
||||
| { kind: 'git_source_error'; code: GitSourceErrorCode; transportReason?: TransportFailureReason }
|
||||
| { kind: 'policy_unavailable' }
|
||||
| { kind: 'persistence_unavailable' }
|
||||
| { kind: 'target_binding_invalid' }
|
||||
| { kind: 'target_unavailable' }
|
||||
| { kind: 'target_mutation_failed' }
|
||||
| { kind: 'blueprint_unavailable' }
|
||||
| { kind: 'interrupted' };
|
||||
|
||||
export type FailureDisposition =
|
||||
/** A newer revision replaced the one being worked on; re-resolve, not backoff. */
|
||||
| { class: 'supersession' }
|
||||
/** Retryable. retryCeiling bounds how many attempts before it escalates to permanent. */
|
||||
| { class: 'transient'; retryCeiling: number }
|
||||
/** Will never succeed by retrying; needs a configuration, environment, or credential change. */
|
||||
| { class: 'permanent' }
|
||||
/** A human decision is required (review a plan, resolve a conflict); not retried automatically. */
|
||||
| { class: 'operator_action_required' }
|
||||
/** Evidence could not be produced (e.g. scanner unavailable); held for review, not retried blind. */
|
||||
| { class: 'degraded' }
|
||||
/** The target itself will never accept this generation without a configuration change. */
|
||||
| { class: 'target_permanent' }
|
||||
/** The target is temporarily unreachable; retryable at the target/dispatch stage only. */
|
||||
| { class: 'target_transient' }
|
||||
/** Applied but deploy or health failed: never refetch or reapply, only redeploy. */
|
||||
| { class: 'target_mutation_failed' }
|
||||
/** Blocked pending a capability this program does not yet provide (e.g. Blueprint rollout). */
|
||||
| { class: 'blocked' }
|
||||
/** Ambiguous or interrupted; reconcile from durable state, never blind retry. */
|
||||
| { class: 'reconcile' };
|
||||
|
||||
export const DEFAULT_TRANSIENT_CEILING = 8;
|
||||
export const LOW_TRANSIENT_CEILING = 3;
|
||||
|
||||
const TRANSIENT_DEFAULT: FailureDisposition = { class: 'transient', retryCeiling: DEFAULT_TRANSIENT_CEILING };
|
||||
const TRANSIENT_LOW: FailureDisposition = { class: 'transient', retryCeiling: LOW_TRANSIENT_CEILING };
|
||||
const PERMANENT: FailureDisposition = { class: 'permanent' };
|
||||
|
||||
/**
|
||||
* Total over every TransportFailureReason except `exit`, which is generic
|
||||
* and needs the classified GitSourceErrorCode (see CODE_DISPOSITION) to
|
||||
* tell a transient network condition from a rate limit from an
|
||||
* unrecognized error. Adding a new reason to the source union without
|
||||
* adding it here fails the build.
|
||||
*/
|
||||
const REASON_DISPOSITION: Record<Exclude<TransportFailureReason, 'exit'>, FailureDisposition> = {
|
||||
'tip-changed': { class: 'supersession' },
|
||||
timeout: TRANSIENT_DEFAULT,
|
||||
'target-unresolved': TRANSIENT_DEFAULT,
|
||||
'invalid-url': PERMANENT,
|
||||
'unsafe-target': PERMANENT,
|
||||
'invalid-ref': PERMANENT,
|
||||
'redirect-scope': PERMANENT,
|
||||
'git-missing': PERMANENT,
|
||||
'git-old': PERMANENT,
|
||||
size: PERMANENT,
|
||||
'ssh-auth-required': PERMANENT,
|
||||
'ref-not-found': PERMANENT,
|
||||
'unsupported-ref': PERMANENT,
|
||||
};
|
||||
|
||||
/**
|
||||
* Total over every GitSourceErrorCode. Used directly when there is no
|
||||
* transport reason (a plan/validation/file/operation-conflict error), and
|
||||
* as the exit-reason fallback (RATE_LIMITED, NETWORK_TIMEOUT, and GIT_ERROR
|
||||
* only ever arise from an `exit` transport reason). Adding a new code
|
||||
* without adding it here fails the build.
|
||||
*/
|
||||
const CODE_DISPOSITION: Record<GitSourceErrorCode, FailureDisposition> = {
|
||||
REPO_NOT_FOUND: PERMANENT,
|
||||
AUTH_FAILED: PERMANENT,
|
||||
REF_NOT_FOUND: PERMANENT,
|
||||
REF_DELETED: PERMANENT,
|
||||
UNSUPPORTED_REF: PERMANENT,
|
||||
SSH_HOST_KEY_FAILED: PERMANENT,
|
||||
FILE_NOT_FOUND: { class: 'operator_action_required' },
|
||||
RATE_LIMITED: TRANSIENT_DEFAULT,
|
||||
NETWORK_TIMEOUT: TRANSIENT_DEFAULT,
|
||||
GIT_ERROR: TRANSIENT_LOW,
|
||||
STALE_PLAN: { class: 'operator_action_required' },
|
||||
PLAN_FINGERPRINT_REQUIRED: { class: 'operator_action_required' },
|
||||
PLAN_BLOCKED: { class: 'operator_action_required' },
|
||||
LEGACY_PENDING: { class: 'operator_action_required' },
|
||||
PLAN_UNAVAILABLE: { class: 'operator_action_required' },
|
||||
OPERATION_IN_FLIGHT: { class: 'reconcile' },
|
||||
};
|
||||
|
||||
export function classifyFailure(evidence: FailureEvidence): FailureDisposition {
|
||||
switch (evidence.kind) {
|
||||
case 'git_source_error':
|
||||
if (evidence.transportReason && evidence.transportReason !== 'exit') {
|
||||
return REASON_DISPOSITION[evidence.transportReason];
|
||||
}
|
||||
return CODE_DISPOSITION[evidence.code];
|
||||
case 'policy_unavailable':
|
||||
return { class: 'degraded' };
|
||||
case 'persistence_unavailable':
|
||||
return TRANSIENT_DEFAULT;
|
||||
case 'target_binding_invalid':
|
||||
return { class: 'target_permanent' };
|
||||
case 'target_unavailable':
|
||||
return { class: 'target_transient' };
|
||||
case 'target_mutation_failed':
|
||||
return { class: 'target_mutation_failed' };
|
||||
case 'blueprint_unavailable':
|
||||
return { class: 'blocked' };
|
||||
case 'interrupted':
|
||||
return { class: 'reconcile' };
|
||||
}
|
||||
}
|
||||
|
||||
const BASE_DELAY_MS = 60_000;
|
||||
const MAX_DELAY_MS = 3_600_000;
|
||||
const JITTER_RATIO = 0.1;
|
||||
|
||||
/**
|
||||
* Bounded exponential backoff with jitter: 60s * 2^retryCount, capped at one
|
||||
* hour, with up to +-10% jitter so many sources retrying at once do not
|
||||
* all land on the same second. A provider-supplied retry floor (e.g. a
|
||||
* rate-limit Retry-After) takes precedence whenever it is larger than the
|
||||
* computed delay.
|
||||
*/
|
||||
export function nextRetryAt(now: number, retryCount: number, providerFloorMs?: number): number {
|
||||
const capped = Math.min(BASE_DELAY_MS * 2 ** retryCount, MAX_DELAY_MS);
|
||||
const jittered = capped + capped * JITTER_RATIO * (Math.random() * 2 - 1);
|
||||
const delay = providerFloorMs !== undefined ? Math.max(jittered, providerFloorMs) : jittered;
|
||||
return now + delay;
|
||||
}
|
||||
@@ -341,7 +341,7 @@ export function commitBlueprintDelete(blueprintId: number, actor: string | null)
|
||||
}
|
||||
|
||||
/** A Blueprint application before anything has been asked of it. */
|
||||
export function blankInlineApplication(id: string, blueprintId: number, at: number) {
|
||||
export function blankInlineApplication(id: string, blueprintId: number, at: number): GitOpsApplicationRow {
|
||||
return {
|
||||
id,
|
||||
lifecycle_key: `blueprint:${blueprintId}`,
|
||||
@@ -382,6 +382,10 @@ export function blankInlineApplication(id: string, blueprintId: number, at: numb
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
|
||||
@@ -157,6 +157,10 @@ export function buildDirectApplicationRow(args: {
|
||||
pause_at: null,
|
||||
pause_reason: null,
|
||||
source_suspended_reason: null,
|
||||
source_policy: 'manual',
|
||||
poll_interval_secs: null,
|
||||
next_poll_at: null,
|
||||
attempt_seq: 0,
|
||||
partial_json: null,
|
||||
failure_stage: null,
|
||||
failure_class: null,
|
||||
@@ -217,6 +221,12 @@ export function buildGenerationRow(args: {
|
||||
actor: args.actor,
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: args.at,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
import type { RepoIdentity } from './repoIdentity';
|
||||
import type { RefKind } from '../git/types';
|
||||
import type { GitOpsGenerationRow } from './types';
|
||||
|
||||
/**
|
||||
* A portable projection of the managed-project manifest: authored file set
|
||||
* and per-file content digests, without node id, stack name, or generation
|
||||
* directory. The real GitProjectManifest carries those identity fields;
|
||||
* this is deliberately narrower.
|
||||
*/
|
||||
export type PortableManifest = {
|
||||
files: Array<{ path: string; role: string; contentSha256?: string | null }>;
|
||||
};
|
||||
|
||||
/** Authored Compose invocation shape, without a target project name. */
|
||||
export type ComposeInputs = {
|
||||
composeFileOrder: string[];
|
||||
profiles?: string[];
|
||||
contextDir?: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* One accepted generation, described purely by what it contains. Direct and
|
||||
* Blueprint dispatch consume the exact same shape; current target mode,
|
||||
* binding revision, and any execution-local path travel separately in
|
||||
* DispatchContext, re-read under the dispatch lock rather than carried
|
||||
* here, so a stale acceptance can never authorize a routing decision made
|
||||
* after it.
|
||||
*/
|
||||
export type AcceptedGeneration = {
|
||||
contractVersion: 1;
|
||||
generationId: string;
|
||||
applicationId: string;
|
||||
repoIdentity: RepoIdentity;
|
||||
configuredRef: string;
|
||||
commitSha: string;
|
||||
resolvedRefKind: RefKind | null;
|
||||
manifestVersion: number;
|
||||
portableManifest: PortableManifest | null;
|
||||
composeInputs: ComposeInputs | null;
|
||||
materializationFingerprint: string;
|
||||
changePlanFingerprint: string | null;
|
||||
validationOk: boolean;
|
||||
sourcePolicyEvidence: unknown | null;
|
||||
securityPolicyEvidence: unknown | null;
|
||||
supportRequirements: unknown | null;
|
||||
compatibilityRequirements: unknown | null;
|
||||
/** Capability metadata only; never a secret value. Not yet populated by any producer. */
|
||||
secretCapability: unknown | null;
|
||||
trigger: string;
|
||||
actor: string | null;
|
||||
operationId: string;
|
||||
previousGenerationId: string | null;
|
||||
/** Why some field above could not be proven, recorded honestly rather than guessed. */
|
||||
limitations: string[];
|
||||
};
|
||||
|
||||
function parseOptionalJson<T>(raw: string | null, limitationLabel: string, limitations: string[]): T | null {
|
||||
if (raw === null) {
|
||||
limitations.push(limitationLabel);
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return JSON.parse(raw) as T;
|
||||
} catch {
|
||||
limitations.push(`${limitationLabel}_unparseable`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the portable accepted-generation contract from a persisted
|
||||
* generation row. A legacy row predating one of the portable-contract
|
||||
* columns decodes that field as null with an explicit limitation recorded,
|
||||
* never as invented evidence.
|
||||
*/
|
||||
export function buildAcceptedGeneration(row: GitOpsGenerationRow): AcceptedGeneration {
|
||||
let repoIdentity: RepoIdentity;
|
||||
try {
|
||||
repoIdentity = JSON.parse(row.repo_identity_json) as RepoIdentity;
|
||||
} catch {
|
||||
throw new Error(`Generation ${row.id} has an unparseable repo_identity_json; refusing to build an accepted-generation contract from corrupt evidence.`);
|
||||
}
|
||||
|
||||
const limitations: string[] = JSON.parse(row.redacted_limitations_json) as string[];
|
||||
const portableManifest = parseOptionalJson<PortableManifest>(row.portable_manifest_json, 'portable_manifest_missing', limitations);
|
||||
const composeInputs = parseOptionalJson<ComposeInputs>(row.compose_inputs_json, 'compose_inputs_missing', limitations);
|
||||
const sourcePolicyEvidence = parseOptionalJson<unknown>(row.source_policy_evidence_json, 'source_policy_evidence_missing', limitations);
|
||||
const securityPolicyEvidence = parseOptionalJson<unknown>(row.security_policy_evidence_json, 'security_policy_evidence_missing', limitations);
|
||||
const supportRequirements = parseOptionalJson<unknown>(row.support_requirements_json, 'support_requirements_missing', limitations);
|
||||
const compatibilityRequirements = parseOptionalJson<unknown>(row.compatibility_requirements_json, 'compatibility_requirements_missing', limitations);
|
||||
|
||||
return {
|
||||
contractVersion: 1,
|
||||
generationId: row.id,
|
||||
applicationId: row.application_id,
|
||||
repoIdentity,
|
||||
configuredRef: row.configured_ref,
|
||||
commitSha: row.commit_sha,
|
||||
resolvedRefKind: row.resolved_ref_kind,
|
||||
manifestVersion: row.manifest_version,
|
||||
portableManifest,
|
||||
composeInputs,
|
||||
materializationFingerprint: row.materialization_fingerprint,
|
||||
changePlanFingerprint: row.change_plan_fingerprint,
|
||||
validationOk: row.validation_ok === 1,
|
||||
sourcePolicyEvidence,
|
||||
securityPolicyEvidence,
|
||||
supportRequirements,
|
||||
compatibilityRequirements,
|
||||
secretCapability: null,
|
||||
trigger: row.trigger,
|
||||
actor: row.actor,
|
||||
operationId: row.operation_id,
|
||||
previousGenerationId: row.previous_generation_id,
|
||||
limitations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Current target mode and binding, re-read under the dispatch lock rather
|
||||
* than carried on AcceptedGeneration, so a routing decision is always made
|
||||
* from the current state, never a value an earlier acceptance froze.
|
||||
*/
|
||||
export type DispatchContext = {
|
||||
targetMode: 'direct' | 'blueprint';
|
||||
nodeId: number | null;
|
||||
bindingRevision: string | null;
|
||||
};
|
||||
|
||||
export type DispatchResult =
|
||||
| { status: 'dispatched' }
|
||||
| { status: 'blocked'; reason: string };
|
||||
|
||||
export interface TargetAdapter {
|
||||
dispatch(generation: AcceptedGeneration, context: DispatchContext): Promise<DispatchResult>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fails closed until Blueprint rollout orchestration exists. Never
|
||||
* inspects selectors, target sets, or placement: an accepted generation
|
||||
* for a Blueprint-mode application is evaluated the same as Direct, but
|
||||
* dispatch stops here.
|
||||
*/
|
||||
export class BlueprintTargetAdapter implements TargetAdapter {
|
||||
async dispatch(_generation: AcceptedGeneration, _context: DispatchContext): Promise<DispatchResult> {
|
||||
return { status: 'blocked', reason: 'Blueprint rollout orchestration is not yet implemented.' };
|
||||
}
|
||||
}
|
||||
@@ -70,6 +70,8 @@ export type GitOpsHistoryStage =
|
||||
| 'rollout_unpaused'
|
||||
| 'source_accepted'
|
||||
| 'source_conflict_blocker'
|
||||
| 'source_reconcile_started'
|
||||
| 'source_reconcile_settled'
|
||||
| 'source_retry_scheduled'
|
||||
| 'source_suspended'
|
||||
| 'source_unsuspended'
|
||||
|
||||
@@ -226,6 +226,12 @@ function migrateAccepted(
|
||||
actor: envelope.actor,
|
||||
previous_generation_id: null,
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: null,
|
||||
source_policy_evidence_json: null,
|
||||
security_policy_evidence_json: null,
|
||||
support_requirements_json: null,
|
||||
compatibility_requirements_json: null,
|
||||
created_at: envelope.at,
|
||||
};
|
||||
store.insertGeneration(generation);
|
||||
|
||||
@@ -0,0 +1,211 @@
|
||||
import type { SourceFacet } from './types';
|
||||
|
||||
/**
|
||||
* Normalized reconcile outcomes. Silence is not an acceptable GitOps
|
||||
* result: every attempt settles into exactly one of these, never a bare
|
||||
* success/failure boolean.
|
||||
*
|
||||
* `converged` is deliberately never produced by outcomeFromSourceFacet: it
|
||||
* requires target and health evidence this source-only projection does not
|
||||
* have, and "no source change" is not proof of full convergence. A later
|
||||
* composition over source + target + health facets is what may report it.
|
||||
*/
|
||||
export type ReconcileOutcome =
|
||||
| 'converged'
|
||||
| 'no_source_change'
|
||||
| 'candidate_already_fetched'
|
||||
| 'pending_review'
|
||||
| 'suspended'
|
||||
| 'retry_scheduled'
|
||||
| 'blocked'
|
||||
| 'superseded'
|
||||
| 'failed_previous_intact'
|
||||
| 'recovery_required'
|
||||
| 'unknown';
|
||||
|
||||
export type NextAction =
|
||||
| 'none'
|
||||
| 'review'
|
||||
| 'resume'
|
||||
| 'retry'
|
||||
| 'resolve_conflict'
|
||||
| 'configure_credentials'
|
||||
| 'view_target_results';
|
||||
|
||||
export type ReconcileResult = {
|
||||
outcome: ReconcileOutcome;
|
||||
reason: string;
|
||||
nextAction: NextAction;
|
||||
retryAt?: number;
|
||||
commitSha?: string;
|
||||
};
|
||||
|
||||
/** Every ReconcileOutcome member, for runtime validation of a value read back from storage. */
|
||||
const RECONCILE_OUTCOMES: ReadonlySet<string> = new Set<ReconcileOutcome>([
|
||||
'converged',
|
||||
'no_source_change',
|
||||
'candidate_already_fetched',
|
||||
'pending_review',
|
||||
'suspended',
|
||||
'retry_scheduled',
|
||||
'blocked',
|
||||
'superseded',
|
||||
'failed_previous_intact',
|
||||
'recovery_required',
|
||||
'unknown',
|
||||
]);
|
||||
|
||||
/** Every NextAction member, for runtime validation of a value read back from storage. */
|
||||
const NEXT_ACTIONS: ReadonlySet<string> = new Set<NextAction>([
|
||||
'none',
|
||||
'review',
|
||||
'resume',
|
||||
'retry',
|
||||
'resolve_conflict',
|
||||
'configure_credentials',
|
||||
'view_target_results',
|
||||
]);
|
||||
|
||||
export function isReconcileOutcome(value: unknown): value is ReconcileOutcome {
|
||||
return typeof value === 'string' && RECONCILE_OUTCOMES.has(value);
|
||||
}
|
||||
|
||||
export function isNextAction(value: unknown): value is NextAction {
|
||||
return typeof value === 'string' && NEXT_ACTIONS.has(value);
|
||||
}
|
||||
|
||||
function commitShaOf(facet: Extract<SourceFacet, { desiredCommitSha: unknown }>): string | undefined {
|
||||
return facet.desiredCommitSha ?? facet.fetchedCommitSha ?? undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive the normalized outcome of a settled source reconcile attempt from
|
||||
* the existing source-facet projection, rather than re-deriving status from
|
||||
* raw application-row fields. Keeps the outcome vocabulary and the
|
||||
* projection's own status vocabulary from silently drifting apart.
|
||||
*/
|
||||
export function outcomeFromSourceFacet(facet: SourceFacet): ReconcileResult {
|
||||
switch (facet.status) {
|
||||
case 'not_applicable':
|
||||
return { outcome: 'unknown', reason: 'No GitOps application exists for this stack.', nextAction: 'none' };
|
||||
|
||||
case 'never_reconciled':
|
||||
return { outcome: 'unknown', reason: 'The source has never been reconciled.', nextAction: 'none' };
|
||||
|
||||
case 'checking_fetching':
|
||||
case 'applying':
|
||||
return {
|
||||
outcome: 'unknown',
|
||||
reason: 'A reconcile operation is currently in flight; no settled result yet.',
|
||||
nextAction: 'none',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_reconcile_required':
|
||||
return {
|
||||
outcome: 'unknown',
|
||||
reason: 'The source has advanced but reconciliation has not evaluated it yet.',
|
||||
nextAction: 'none',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'application_generation_accepted':
|
||||
return {
|
||||
outcome: 'no_source_change',
|
||||
reason: 'The configured ref still resolves to the accepted generation. This is not proof of full convergence.',
|
||||
nextAction: 'none',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'candidate_ready':
|
||||
return {
|
||||
outcome: 'candidate_already_fetched',
|
||||
reason: 'A candidate generation is already staged and awaiting acceptance.',
|
||||
nextAction: 'none',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_review_pending':
|
||||
return {
|
||||
outcome: 'pending_review',
|
||||
reason: 'A candidate is staged and requires explicit review before acceptance.',
|
||||
nextAction: 'review',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_conflict_blocker':
|
||||
return {
|
||||
outcome: 'blocked',
|
||||
reason: 'A local conflict is blocking the candidate from being accepted.',
|
||||
nextAction: 'resolve_conflict',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_superseded':
|
||||
return {
|
||||
outcome: 'superseded',
|
||||
reason: 'A newer revision superseded this candidate before it was accepted.',
|
||||
nextAction: 'none',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_retry_scheduled':
|
||||
return {
|
||||
outcome: 'retry_scheduled',
|
||||
reason: `A previous attempt failed transiently; retry ${facet.retryCount + 1} is scheduled.`,
|
||||
nextAction: 'none',
|
||||
retryAt: facet.retryAt,
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_suspended':
|
||||
return {
|
||||
outcome: 'suspended',
|
||||
reason: facet.suspendedReason
|
||||
? `Reconciliation is suspended: ${facet.suspendedReason}`
|
||||
: 'Reconciliation is suspended.',
|
||||
nextAction: 'resume',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_failed':
|
||||
return {
|
||||
outcome: 'failed_previous_intact',
|
||||
reason: `The ${facet.failureStage} stage failed (${facet.failureClass}). The previously accepted generation is unchanged.`,
|
||||
nextAction: facet.retryAt !== null ? 'retry' : 'configure_credentials',
|
||||
retryAt: facet.retryAt ?? undefined,
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'source_unknown':
|
||||
return {
|
||||
outcome: 'recovery_required',
|
||||
reason: `An operation was interrupted at ${facet.interruptedStage} and its outcome is unproven.`,
|
||||
nextAction: 'view_target_results',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'recovery_required':
|
||||
return {
|
||||
outcome: 'recovery_required',
|
||||
reason: 'Recovery from an earlier failed mutation is still outstanding.',
|
||||
nextAction: 'view_target_results',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'recovery_failed':
|
||||
return {
|
||||
outcome: 'recovery_required',
|
||||
reason: `Recovery itself failed (${facet.failureClass}); this needs operator attention.`,
|
||||
nextAction: 'view_target_results',
|
||||
commitSha: commitShaOf(facet),
|
||||
};
|
||||
|
||||
case 'not_live':
|
||||
return {
|
||||
outcome: 'unknown',
|
||||
reason: `The application is ${facet.lifecycleStatus}, not live; there is nothing to reconcile.`,
|
||||
nextAction: 'none',
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -100,6 +100,16 @@ CREATE TABLE IF NOT EXISTS gitops_applications (
|
||||
-- target rows, so suspending a source can never clobber an unrelated
|
||||
-- rollout pause reason (or the reverse).
|
||||
source_suspended_reason TEXT NULL,
|
||||
-- Controller-owned bookkeeping. NULL poll_interval_secs inherits the
|
||||
-- global default; 0 disables polling for this application. next_poll_at
|
||||
-- is the durable scheduling cursor. attempt_seq is allocated
|
||||
-- transactionally per submission lacking a stable external delivery id.
|
||||
source_policy TEXT NOT NULL DEFAULT 'manual' CHECK (
|
||||
source_policy IN ('manual','review','automatic')
|
||||
),
|
||||
poll_interval_secs INTEGER NULL,
|
||||
next_poll_at INTEGER NULL,
|
||||
attempt_seq INTEGER NOT NULL DEFAULT 0,
|
||||
partial_json TEXT NULL,
|
||||
failure_stage TEXT NULL CHECK (
|
||||
failure_stage IS NULL OR failure_stage IN (
|
||||
@@ -177,6 +187,17 @@ CREATE TABLE IF NOT EXISTS gitops_generations (
|
||||
actor TEXT NULL,
|
||||
previous_generation_id TEXT NULL,
|
||||
redacted_limitations_json TEXT NOT NULL DEFAULT '[]',
|
||||
-- Portable accepted-generation contract (content only: no node id, local
|
||||
-- path, target mode, or secret value). Additive and nullable so existing
|
||||
-- rows decode as an explicit limitation rather than invented evidence; a
|
||||
-- legacy pending candidate lacking these must be re-evaluated before it
|
||||
-- can be accepted or dispatched.
|
||||
portable_manifest_json TEXT NULL,
|
||||
compose_inputs_json TEXT NULL,
|
||||
source_policy_evidence_json TEXT NULL,
|
||||
security_policy_evidence_json TEXT NULL,
|
||||
support_requirements_json TEXT NULL,
|
||||
compatibility_requirements_json TEXT NULL,
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_gen_app_created
|
||||
@@ -453,6 +474,10 @@ CREATE INDEX IF NOT EXISTS idx_gitops_history_node ON gitops_history(node_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_trigger ON gitops_history(trigger);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_actor ON gitops_history(actor);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_outcome ON gitops_history(outcome);
|
||||
-- listUnsettledReconcileAttempts filters on stage and orders by created_at;
|
||||
-- without this, that query (run on every startup, ahead of the server
|
||||
-- listening) scans and sorts the whole table.
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_stage_created ON gitops_history(stage, created_at);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_repo_ref
|
||||
ON gitops_history(repo_url, configured_ref);
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_history_stack_created
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import type Database from 'better-sqlite3';
|
||||
import { DatabaseService } from '../DatabaseService';
|
||||
import type { GitOpsHistoryCursor } from './history';
|
||||
import {
|
||||
decodeArtifactEvidenceJson,
|
||||
decodeGitOpsApprovedTargetEffectJson,
|
||||
@@ -16,6 +17,7 @@ import type {
|
||||
GitOpsCreateCheckpointRow,
|
||||
GitOpsCreatePhase,
|
||||
GitOpsGenerationRow,
|
||||
GitOpsHistoryRow,
|
||||
GitOpsIntentRevisionRow,
|
||||
GitOpsRolloutCandidateRow,
|
||||
GitOpsTargetCurrentRow,
|
||||
@@ -131,6 +133,29 @@ export class GitOpsStore {
|
||||
).get(stackName) as GitOpsApplicationRow | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a detached Direct application exists for this stack name,
|
||||
* distinct from "no application was ever created" (the legitimate
|
||||
* pre-migration case, where a stack's git-source config predates
|
||||
* GitOps tracking entirely). `detached` only, matching
|
||||
* getDetachedDirectApplication above and for the same reason:
|
||||
* `deleted` is not a safe signal here. Two production paths tombstone
|
||||
* an application as `deleted` while deliberately preserving its
|
||||
* git-source row so a future upsert or migration can rebuild from it
|
||||
* (`gitops/createRecovery.ts`'s checkpointless-create sweep, and
|
||||
* `gitops/migrate.ts`'s `tombstoned_missing_stack` outcome) --
|
||||
* treating that state as a refusal would be permanent and
|
||||
* unrecoverable, since neither upsert() nor migration currently mints
|
||||
* a fresh application once a matching migration checkpoint exists.
|
||||
* `detach()` itself deletes the git-source row in the same transaction
|
||||
* as tombstoning (`detached`), so the window this method exists to
|
||||
* catch (tracking removed, config surviving) is a crash between those
|
||||
* two writes, not routine operation.
|
||||
*/
|
||||
hasDetachedDirectApplication(stackName: string): boolean {
|
||||
return this.getDetachedDirectApplication(stackName) !== undefined;
|
||||
}
|
||||
|
||||
/** Direct applications that never reached their success boundary. */
|
||||
listCreatingDirectApplications(): GitOpsApplicationRow[] {
|
||||
return this.db().prepare(
|
||||
@@ -144,6 +169,25 @@ export class GitOpsStore {
|
||||
return this.db().prepare('SELECT * FROM gitops_generations WHERE id = ?').get(id) as GitOpsGenerationRow | undefined;
|
||||
}
|
||||
|
||||
/** Generations whose creating reconcile attempt has not durably settled. */
|
||||
listGenerationsClaimedByUnsettledAttempts(applicationId: string): GitOpsGenerationRow[] {
|
||||
return this.db().prepare(
|
||||
`SELECT DISTINCT generation.*
|
||||
FROM gitops_generations generation
|
||||
JOIN gitops_history started
|
||||
ON started.application_id = generation.application_id
|
||||
AND started.operation_id = generation.operation_id
|
||||
AND started.stage = 'source_reconcile_started'
|
||||
WHERE generation.application_id = ?
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM gitops_history settled
|
||||
WHERE settled.application_id = started.application_id
|
||||
AND settled.operation_id = started.operation_id
|
||||
AND settled.stage = 'source_reconcile_settled'
|
||||
)`,
|
||||
).all(applicationId) as GitOpsGenerationRow[];
|
||||
}
|
||||
|
||||
getArtifactSet(id: string): GitOpsArtifactSetRow | undefined {
|
||||
return this.db().prepare('SELECT * FROM gitops_artifact_sets WHERE id = ?').get(id) as GitOpsArtifactSetRow | undefined;
|
||||
}
|
||||
@@ -195,6 +239,126 @@ export class GitOpsStore {
|
||||
).all() as GitOpsApplicationRow[];
|
||||
}
|
||||
|
||||
/**
|
||||
* The settled result for one exact reconcile attempt, or undefined when
|
||||
* that attempt has not settled (or never existed). Used to recover a
|
||||
* reservation that already completed rather than repeating the work.
|
||||
*/
|
||||
getSettledAttempt(applicationId: string, operationId: string): GitOpsHistoryRow | undefined {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_history
|
||||
WHERE application_id = ? AND operation_id = ? AND stage = 'source_reconcile_settled'
|
||||
LIMIT 1`,
|
||||
).get(applicationId, operationId) as GitOpsHistoryRow | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* The reservation row for one exact reconcile attempt, or undefined when
|
||||
* it was never reserved. Used to recover this attempt's own recorded
|
||||
* follower link (if any), so a follower can be settled from its
|
||||
* leader's actual result rather than derived independently of it.
|
||||
*/
|
||||
getStartedAttempt(applicationId: string, operationId: string): GitOpsHistoryRow | undefined {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_history
|
||||
WHERE application_id = ? AND operation_id = ? AND stage = 'source_reconcile_started'
|
||||
LIMIT 1`,
|
||||
).get(applicationId, operationId) as GitOpsHistoryRow | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Every reservation with no matching settled row, oldest first: an
|
||||
* attempt that started but never recorded a result, most likely because
|
||||
* the process crashed between reservation and settlement. Startup
|
||||
* recovery reconciles these from durable stage evidence rather than
|
||||
* leaving them silently open forever.
|
||||
*
|
||||
* `after` pages strictly forward by (created_at, id), the same cursor
|
||||
* shape `queryHistoryRows` uses, and is load-bearing rather than
|
||||
* cosmetic: a row a caller cannot settle (its application vanished, a DB
|
||||
* error) stays unsettled forever by definition, so without a cursor it
|
||||
* would occupy the same "oldest N" window on every future call and hide
|
||||
* every genuinely recoverable row behind it once the backlog exceeds one
|
||||
* page.
|
||||
*/
|
||||
listUnsettledReconcileAttempts(limit = 200, after?: GitOpsHistoryCursor): GitOpsHistoryRow[] {
|
||||
const clauses = ["started.stage = 'source_reconcile_started'"];
|
||||
const params: Array<string | number> = [];
|
||||
if (after) {
|
||||
clauses.push('(started.created_at > ? OR (started.created_at = ? AND started.id > ?))');
|
||||
params.push(after.createdAt, after.createdAt, after.id);
|
||||
}
|
||||
params.push(limit);
|
||||
return this.db().prepare(
|
||||
`SELECT started.* FROM gitops_history started
|
||||
WHERE ${clauses.join(' AND ')}
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM gitops_history settled
|
||||
WHERE settled.application_id = started.application_id
|
||||
AND settled.operation_id = started.operation_id
|
||||
AND settled.stage = 'source_reconcile_settled'
|
||||
)
|
||||
ORDER BY started.created_at ASC, started.id ASC
|
||||
LIMIT ?`,
|
||||
).all(...params) as GitOpsHistoryRow[];
|
||||
}
|
||||
|
||||
/**
|
||||
* The most recently settled attempt for an application, for API and UI
|
||||
* projection. Distinct from getSettledAttempt, which looks up one exact
|
||||
* operation rather than the newest one.
|
||||
*/
|
||||
latestSettledAttempt(applicationId: string): GitOpsHistoryRow | undefined {
|
||||
// rowid (SQLite's implicit insertion-order key), not the id column: id
|
||||
// is a random UUID and does not sort by recency the way rowid does, so
|
||||
// it cannot break a created_at tie between two attempts settled within
|
||||
// the same millisecond.
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_history
|
||||
WHERE application_id = ? AND stage = 'source_reconcile_settled'
|
||||
ORDER BY created_at DESC, rowid DESC
|
||||
LIMIT 1`,
|
||||
).get(applicationId) as GitOpsHistoryRow | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Direct sources whose poll time has arrived: active, not suspended, no
|
||||
* operation in flight. Blueprint-mode applications are never polled here
|
||||
* -- source evaluation for them is blocked at the evaluation boundary
|
||||
* until an application-keyed source engine exists for that mode.
|
||||
*/
|
||||
listSourcesDueForPoll(now: number, limit = 200): GitOpsApplicationRow[] {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_applications
|
||||
WHERE target_mode = 'direct'
|
||||
AND lifecycle_status = 'active'
|
||||
AND suspended_at IS NULL
|
||||
AND active_operation_stage IS NULL
|
||||
AND next_poll_at IS NOT NULL
|
||||
AND next_poll_at <= ?
|
||||
ORDER BY next_poll_at ASC
|
||||
LIMIT ?`,
|
||||
).all(now, limit) as GitOpsApplicationRow[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Applications with a scheduled retry that has come due: not suspended,
|
||||
* no operation in flight. Poll eligibility and retry eligibility are
|
||||
* deliberately separate queries, since a retry can be due on an
|
||||
* application whose poll cadence would not otherwise select it yet.
|
||||
*/
|
||||
listApplicationsDueForRetry(now: number, limit = 200): GitOpsApplicationRow[] {
|
||||
return this.db().prepare(
|
||||
`SELECT * FROM gitops_applications
|
||||
WHERE retry_at IS NOT NULL
|
||||
AND retry_at <= ?
|
||||
AND suspended_at IS NULL
|
||||
AND active_operation_stage IS NULL
|
||||
ORDER BY retry_at ASC
|
||||
LIMIT ?`,
|
||||
).all(now, limit) as GitOpsApplicationRow[];
|
||||
}
|
||||
|
||||
/** Every live target on one node, across all applications. */
|
||||
listActiveTargetsForNode(nodeId: number): GitOpsTargetCurrentRow[] {
|
||||
return this.db().prepare(
|
||||
@@ -365,12 +529,13 @@ export class GitOpsStore {
|
||||
intent_revision_id, rollout_candidate_id, rollout_generation_id, source_acceptance_ref,
|
||||
placement_approval_ref, rollout_authorization_ref, legacy_combined_approval_ref,
|
||||
preflight_fingerprint, latest_operation_id, active_operation_id, active_operation_stage,
|
||||
active_operation_at, active_generation_id, pause_at, pause_reason, source_suspended_reason, partial_json,
|
||||
active_operation_at, active_generation_id, pause_at, pause_reason, source_suspended_reason,
|
||||
source_policy, poll_interval_secs, next_poll_at, attempt_seq, partial_json,
|
||||
failure_stage, failure_class, failure_at, retry_at, retry_count, suspended_at,
|
||||
recovery_ref, recovery_phase, interruption_stage, interruption_at,
|
||||
interruption_operation_id, interruption_generation_id, evidence_fresh_at,
|
||||
evidence_limitations_json, created_at, updated_at
|
||||
) VALUES (${Array(56).fill('?').join(', ')})`,
|
||||
) VALUES (${Array(60).fill('?').join(', ')})`,
|
||||
).run(
|
||||
row.id, row.lifecycle_key, row.lifecycle_status, row.target_mode, row.stack_name, row.blueprint_id,
|
||||
row.configured_repo_url, row.repo_identity_json, row.configured_ref, row.compose_paths_json,
|
||||
@@ -380,7 +545,8 @@ export class GitOpsStore {
|
||||
row.intent_revision_id, row.rollout_candidate_id, row.rollout_generation_id, row.source_acceptance_ref,
|
||||
row.placement_approval_ref, row.rollout_authorization_ref, row.legacy_combined_approval_ref,
|
||||
row.preflight_fingerprint, row.latest_operation_id, row.active_operation_id, row.active_operation_stage,
|
||||
row.active_operation_at, row.active_generation_id, row.pause_at, row.pause_reason, row.source_suspended_reason, row.partial_json,
|
||||
row.active_operation_at, row.active_generation_id, row.pause_at, row.pause_reason, row.source_suspended_reason,
|
||||
row.source_policy, row.poll_interval_secs, row.next_poll_at, row.attempt_seq, row.partial_json,
|
||||
row.failure_stage, row.failure_class, row.failure_at, row.retry_at, row.retry_count, row.suspended_at,
|
||||
row.recovery_ref, row.recovery_phase, row.interruption_stage, row.interruption_at,
|
||||
row.interruption_operation_id, row.interruption_generation_id, row.evidence_fresh_at,
|
||||
@@ -394,13 +560,18 @@ export class GitOpsStore {
|
||||
id, application_id, commit_sha, repo_url, configured_ref, resolved_ref_kind, repo_identity_json,
|
||||
manifest_version, candidate_dir, applied_dir, expected_invocation_json,
|
||||
materialization_fingerprint, validation_ok, plan_blocked, change_plan_fingerprint,
|
||||
operation_id, trigger, actor, previous_generation_id, redacted_limitations_json, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
operation_id, trigger, actor, previous_generation_id, redacted_limitations_json,
|
||||
portable_manifest_json, compose_inputs_json, source_policy_evidence_json,
|
||||
security_policy_evidence_json, support_requirements_json, compatibility_requirements_json,
|
||||
created_at
|
||||
) VALUES (${Array(27).fill('?').join(', ')})`,
|
||||
).run(
|
||||
row.id, row.application_id, row.commit_sha, row.repo_url, row.configured_ref, row.resolved_ref_kind, row.repo_identity_json,
|
||||
row.manifest_version, row.candidate_dir, row.applied_dir, row.expected_invocation_json,
|
||||
row.materialization_fingerprint, row.validation_ok, row.plan_blocked, row.change_plan_fingerprint,
|
||||
row.operation_id, row.trigger, row.actor, row.previous_generation_id, row.redacted_limitations_json,
|
||||
row.portable_manifest_json, row.compose_inputs_json, row.source_policy_evidence_json,
|
||||
row.security_policy_evidence_json, row.support_requirements_json, row.compatibility_requirements_json,
|
||||
row.created_at,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -28,6 +28,10 @@ export type EventEnvelope = {
|
||||
at: number;
|
||||
};
|
||||
|
||||
export type ReconcileDeliveryIntent =
|
||||
| { autoApply: false; deploy: false }
|
||||
| { autoApply: true; deploy: boolean };
|
||||
|
||||
export type AppliedArgs = {
|
||||
applicationId: string;
|
||||
generationId: string;
|
||||
@@ -893,6 +897,102 @@ export class GitOpsTransitions {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Reserve a durable attempt before any side effect: a bare history
|
||||
* insert in its own transaction, deliberately not through mutateApp, so
|
||||
* nothing about the application row changes. A `reserved: false` return
|
||||
* means this exact (application, operation) already reserved -- the
|
||||
* caller reconstructs from durable state rather than repeating work.
|
||||
*
|
||||
* `followerOf` records that this reservation joined another running
|
||||
* attempt. Recovery uses the link to settle the follower from its leader's
|
||||
* result. `deliveryIntent` preserves the original webhook apply and deploy
|
||||
* decision so redelivery cannot change behavior with later settings.
|
||||
*/
|
||||
reserveReconcileAttempt(
|
||||
applicationId: string,
|
||||
envelope: EventEnvelope,
|
||||
followerOf?: string,
|
||||
deliveryIntent?: ReconcileDeliveryIntent,
|
||||
): { reserved: boolean } {
|
||||
return this.raw().transaction(() => ({
|
||||
reserved: this.insertReconcileReservation(this.requireApp(applicationId), envelope, followerOf, deliveryIntent),
|
||||
}))();
|
||||
}
|
||||
|
||||
/**
|
||||
* Allocate the next attemptSeq for a submission with no stable external
|
||||
* delivery identity, and reserve its durable attempt in the same
|
||||
* transaction, so two concurrent submissions can never mint the same
|
||||
* operation id. Unlike reserveReconcileAttempt, this does write one
|
||||
* column of application state (attempt_seq) -- allocation is the one
|
||||
* thing here that is not a bare history insert, since a fresh id has to
|
||||
* come from somewhere durable. Only the allocated id's uniqueness is
|
||||
* load-bearing; its embedded sequence number is for traceability.
|
||||
*/
|
||||
allocateReconcileAttempt(
|
||||
applicationId: string,
|
||||
actor: string | null,
|
||||
trigger: string,
|
||||
at: number,
|
||||
followerOf?: string,
|
||||
): { operationId: string; reserved: boolean } {
|
||||
return this.raw().transaction(() => {
|
||||
const app = this.requireApp(applicationId);
|
||||
const seq = app.attempt_seq + 1;
|
||||
this.raw().prepare('UPDATE gitops_applications SET attempt_seq = ? WHERE id = ?').run(seq, applicationId);
|
||||
const operationId = `${applicationId}:attempt:${seq}`;
|
||||
const envelope: EventEnvelope = { operationId, actor, trigger, at };
|
||||
return { operationId, reserved: this.insertReconcileReservation(app, envelope, followerOf) };
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* The reservation row itself, shared by reserveReconcileAttempt and
|
||||
* allocateReconcileAttempt: a bare history insert whose dedupe index is
|
||||
* what makes a repeat reservation report false rather than recording a
|
||||
* second attempt.
|
||||
*/
|
||||
private insertReconcileReservation(
|
||||
app: GitOpsApplicationRow,
|
||||
envelope: EventEnvelope,
|
||||
followerOf: string | undefined,
|
||||
deliveryIntent?: ReconcileDeliveryIntent,
|
||||
): boolean {
|
||||
return this.history(app, envelope, {
|
||||
stage: 'source_reconcile_started',
|
||||
outcome: 'committed',
|
||||
before: {},
|
||||
after: {
|
||||
...(followerOf ? { followerOf } : {}),
|
||||
...(deliveryIntent ? { deliveryIntent } : {}),
|
||||
},
|
||||
}) !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle a reserved attempt with its normalized outcome, the same way:
|
||||
* a bare history insert, not a state mutation. Safe to call more than
|
||||
* once for the same operation; a repeat is a no-op via the history
|
||||
* dedupe index, so the first settled result is never overwritten.
|
||||
*/
|
||||
settleReconcileAttempt(
|
||||
applicationId: string,
|
||||
envelope: EventEnvelope,
|
||||
result: { outcome: string; reason: string; nextAction: string; retryAt?: number; commitSha?: string },
|
||||
): { settled: boolean } {
|
||||
return this.raw().transaction(() => {
|
||||
const app = this.requireApp(applicationId);
|
||||
const historyId = this.history(app, envelope, {
|
||||
stage: 'source_reconcile_settled',
|
||||
outcome: 'committed',
|
||||
before: {},
|
||||
after: { ...result },
|
||||
});
|
||||
return { settled: historyId !== null };
|
||||
})();
|
||||
}
|
||||
|
||||
/**
|
||||
* Pause a rollout, application-wide or on one target.
|
||||
*
|
||||
@@ -2319,7 +2419,8 @@ export class GitOpsTransitions {
|
||||
legacy_combined_approval_ref=?, preflight_fingerprint=?,
|
||||
latest_operation_id=?, active_operation_id=?,
|
||||
active_operation_stage=?, active_operation_at=?, active_generation_id=?,
|
||||
pause_at=?, pause_reason=?, source_suspended_reason=?, partial_json=?,
|
||||
pause_at=?, pause_reason=?, source_suspended_reason=?,
|
||||
source_policy=?, poll_interval_secs=?, next_poll_at=?, attempt_seq=?, partial_json=?,
|
||||
failure_stage=?, failure_class=?, failure_at=?, retry_at=?, retry_count=?,
|
||||
suspended_at=?, recovery_ref=?, recovery_phase=?,
|
||||
interruption_stage=?, interruption_at=?, interruption_operation_id=?,
|
||||
@@ -2336,7 +2437,8 @@ export class GitOpsTransitions {
|
||||
app.legacy_combined_approval_ref, app.preflight_fingerprint,
|
||||
app.latest_operation_id, app.active_operation_id,
|
||||
app.active_operation_stage, app.active_operation_at, app.active_generation_id,
|
||||
app.pause_at, app.pause_reason, app.source_suspended_reason, app.partial_json,
|
||||
app.pause_at, app.pause_reason, app.source_suspended_reason,
|
||||
app.source_policy, app.poll_interval_secs, app.next_poll_at, app.attempt_seq, app.partial_json,
|
||||
app.failure_stage, app.failure_class, app.failure_at, app.retry_at, app.retry_count,
|
||||
app.suspended_at, app.recovery_ref, app.recovery_phase,
|
||||
app.interruption_stage, app.interruption_at, app.interruption_operation_id,
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* Normalized reconciliation triggers for the GitOps source controller.
|
||||
*
|
||||
* A trigger only authorizes evaluation; it is not proof anything changed.
|
||||
* `manual`, `webhook`, `poll`, and `retry` have current execution producers.
|
||||
* The remaining values are typed ahead of later deliveries so those callers
|
||||
* extend this union instead of inventing a parallel one.
|
||||
*/
|
||||
export type ReconcileTrigger =
|
||||
| 'manual'
|
||||
| 'api'
|
||||
| 'webhook'
|
||||
| 'poll'
|
||||
| 'retry'
|
||||
| 'config_change'
|
||||
| 'startup'
|
||||
| 'resume'
|
||||
| 'provider_event'
|
||||
| 'schedule'
|
||||
| 'binding_change';
|
||||
|
||||
/**
|
||||
* One normalized submission to the controller. `dismiss` is deliberately
|
||||
* not a reconcile intent: it changes candidate state but does not
|
||||
* authorize source evaluation.
|
||||
*/
|
||||
export type ReconcileRequest =
|
||||
| {
|
||||
intent: 'fetch';
|
||||
applicationId: string;
|
||||
stackName: string;
|
||||
trigger: ReconcileTrigger;
|
||||
actor: string;
|
||||
deliveryId?: string;
|
||||
}
|
||||
| {
|
||||
intent: 'apply';
|
||||
applicationId: string;
|
||||
stackName: string;
|
||||
trigger: ReconcileTrigger;
|
||||
actor: string;
|
||||
commitSha: string;
|
||||
planFingerprint: string;
|
||||
deploy: boolean;
|
||||
deliveryId?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* The in-process joining key for concurrent evaluations of the same work.
|
||||
* A fetch has only one live outcome per application regardless of trigger,
|
||||
* so any two fetch submissions for the same application and stack join. An
|
||||
* apply is identified by exactly what it would do: two applies join only
|
||||
* when they target the same commit, the same plan fingerprint, and the
|
||||
* same deploy choice. Two applies that differ in any of those must never
|
||||
* join, or one request could silently receive another request's result.
|
||||
*
|
||||
* Both the fetch and the apply form carry the stack name alongside the
|
||||
* applicationId, so a caller that pairs a live applicationId with the
|
||||
* wrong stackName can never join a leader evaluating the right one.
|
||||
*/
|
||||
export function coalesceKey(request: ReconcileRequest): string {
|
||||
if (request.intent === 'fetch') {
|
||||
return `${request.applicationId}:${request.stackName}:fetch`;
|
||||
}
|
||||
return `${request.applicationId}:${request.stackName}:apply:${request.commitSha}:${request.planFingerprint}:${request.deploy}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* A producer-namespaced key for an external delivery, so the same delivery
|
||||
* id from two different trigger sources is never treated as one delivery.
|
||||
* Also namespaced by intent: a webhook that both fetches and applies under
|
||||
* one delivery id must reserve two distinct attempts, not have the apply's
|
||||
* reservation collide with the fetch's and silently never run.
|
||||
*/
|
||||
export function deliveryKey(trigger: ReconcileTrigger, intent: ReconcileRequest['intent'], deliveryId: string): string {
|
||||
return `${trigger}:${intent}:${deliveryId}`;
|
||||
}
|
||||
@@ -73,6 +73,11 @@ export type GitOpsApplicationRow = {
|
||||
pause_reason: string | null;
|
||||
/** sourceSuspended/sourceUnsuspended's own reason field; independent of pause_reason. */
|
||||
source_suspended_reason: string | null;
|
||||
/** Controller-owned. See gitops/SourceController.ts. */
|
||||
source_policy: 'manual' | 'review' | 'automatic';
|
||||
poll_interval_secs: number | null;
|
||||
next_poll_at: number | null;
|
||||
attempt_seq: number;
|
||||
partial_json: string | null;
|
||||
failure_stage: ApplicationFailureStage | null;
|
||||
failure_class: string | null;
|
||||
@@ -161,6 +166,13 @@ export type GitOpsGenerationRow = {
|
||||
actor: string | null;
|
||||
previous_generation_id: string | null;
|
||||
redacted_limitations_json: string;
|
||||
/** Portable accepted-generation contract fields. See gitops/handoff.ts. */
|
||||
portable_manifest_json: string | null;
|
||||
compose_inputs_json: string | null;
|
||||
source_policy_evidence_json: string | null;
|
||||
security_policy_evidence_json: string | null;
|
||||
support_requirements_json: string | null;
|
||||
compatibility_requirements_json: string | null;
|
||||
created_at: number;
|
||||
};
|
||||
|
||||
|
||||
Reference in New Issue
Block a user