mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-26 05:30:18 +00:00
fix(gitops): guard automatic acceptance against stateful withdrawal (#1975)
* fix(gitops): guard automatic acceptance against stateful withdrawal Automatic GitOps acceptance had no stateful check: a candidate that removed, renamed, or stripped the volumes of a stateful service could be accepted and applied unattended, destroying or orphaning that service's data. The poll acceptance path and the automatic webhook apply path now share one guard that diffs the candidate's stateful service set against the generation in force, reading managed evidence only. When the candidate withdraws a stateful service, or when that cannot be proven, the candidate is held for review: nothing is accepted or applied, a durable block reason is recorded, and the projection names the block as its own attention reason, so the workplace shows it instead of an unexplained review. Webhook auto-apply reports the hold in its delivery result. Repeat pulls of the same held commit reuse the staged candidate, so the hold survives polling instead of churning generations, and a stored redelivery keeps its automated authority even if the policy moved since. First acceptance under a source with no accepted generation still defines the baseline, since there is no managed state to compare against. Validation: backend tsc, 9479 backend tests, frontend tsc, lint, and tests, with new coverage for the poll hold, rename, non-withdrawing acceptance, no re-evaluation, unreadable evidence, the webhook hold, a stored redelivery after a policy change, and repeat-pull reuse. The full backend run has one pre-existing environment failure in the SSH integration suite because port 22 is already in use on this machine. * security(gitops): contain generation reads under the managed root CodeQL flagged three path-injection sinks in the new stateful guard: the staged compose reads, the staged .env read, and the generation-in-force compose reads each joined a row- or user-derived relative path into a read without a containment check the analyzer can see. Each read now resolves its target and checks it inline at the sink, the pattern the repository already uses where CodeQL credits the guard, and every generation directory is contained under the managed root before anything inside it is read. Escapes are treated as unreadable evidence, so the guard holds rather than reads outside the managed area. No behavior change for legitimate paths. * test(gitops): cover stateful guard edge cases and review block lifecycle Pin the guard's hold on stripped volumes and unparseable candidates, the first-acceptance baseline, and every exit from the durable review block, including that automatic acceptance resumes once a newer candidate clears it.
This commit is contained in:
@@ -1919,6 +1919,99 @@ describe('GitSourceService.handleWebhookPull debounce', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('holds a webhook auto-apply that would withdraw a stateful service', async () => {
|
||||
const stackName = 'webhook-stateful-hold';
|
||||
const svc = GitSourceService.getInstance();
|
||||
const validateSpy = vi.spyOn(svc, 'validateCompose').mockResolvedValue({ ok: true });
|
||||
const statefulCompose = [
|
||||
'services:',
|
||||
' db:',
|
||||
' image: postgres:16',
|
||||
' volumes:',
|
||||
' - pgdata:/var/lib/postgresql/data',
|
||||
'volumes:',
|
||||
' pgdata:',
|
||||
'',
|
||||
].join('\n');
|
||||
const withdrawnCompose = 'services:\n web:\n image: nginx:1.27\n';
|
||||
try {
|
||||
// The promotion writes the stack's compose file for real, exactly
|
||||
// as a linked stack would have one, so the second delivery's
|
||||
// change plan compares against live state rather than a missing
|
||||
// file (which would read as a local conflict).
|
||||
fs.mkdirSync(path.join(process.env.COMPOSE_DIR!, stackName), { recursive: true });
|
||||
mockSuccessfulClone({ compose: statefulCompose, sha: 'b1'.repeat(20) });
|
||||
await svc.upsert({
|
||||
stackName,
|
||||
repoUrl: 'https://github.com/example/repo.git',
|
||||
branch: 'main',
|
||||
composePaths: ['compose.yaml'],
|
||||
contextDir: null,
|
||||
syncEnv: false,
|
||||
envPath: null,
|
||||
authType: 'none',
|
||||
autoApplyOnWebhook: true,
|
||||
autoDeployOnApply: false,
|
||||
});
|
||||
|
||||
// The first delivery applies a stateful generation; that defines
|
||||
// the managed baseline the guard compares later deliveries to.
|
||||
const first = await svc.handleWebhookPull(stackName, true, 'delivery-stateful-first');
|
||||
expect(first.status).toBe('success');
|
||||
const baseline = GitOpsStore.getInstance().getLiveDirectApplication(stackName)!;
|
||||
const acceptedId = baseline.accepted_generation_id;
|
||||
expect(acceptedId).toBeTruthy();
|
||||
|
||||
// The next commit withdraws that service. An automatic webhook
|
||||
// delivery must hold it for review instead of applying it, and the
|
||||
// hold must be visible as a canonical attention reason.
|
||||
DatabaseService.getInstance().getDb()
|
||||
.prepare('UPDATE stack_git_sources SET last_debounce_at = ? WHERE stack_name = ?')
|
||||
.run(Date.now() - 999_999, stackName);
|
||||
mockSuccessfulClone({ compose: withdrawnCompose, sha: 'b2'.repeat(20) });
|
||||
const held = await svc.handleWebhookPull(stackName, true, 'delivery-stateful-second');
|
||||
|
||||
expect(held.status).toBe('success');
|
||||
expect(held.message).toMatch(/Held for review/i);
|
||||
const after = GitOpsStore.getInstance().getLiveDirectApplication(stackName)!;
|
||||
expect(after.accepted_generation_id).toBe(acceptedId);
|
||||
expect(after.review_required).toBe(1);
|
||||
expect(after.review_block_reason).toBe('stateful_withdrawal');
|
||||
// The withdrawing candidate is still staged, not applied.
|
||||
expect(after.candidate_generation_id).toBeTruthy();
|
||||
expect(GitOpsStore.getInstance().getGeneration(after.candidate_generation_id!)?.commit_sha)
|
||||
.toBe('b2'.repeat(20));
|
||||
|
||||
// A stored redelivery keeps its automated intent even if the policy
|
||||
// moved to review in the meantime. The guard must still hold the
|
||||
// withdrawal rather than let the stored intent apply it.
|
||||
GitOpsTransitions.getInstance().sourcePolicyChanged(after.id, 'review', {
|
||||
operationId: 'policy-to-review',
|
||||
actor: 'operator',
|
||||
trigger: 'manual',
|
||||
at: Date.now(),
|
||||
});
|
||||
const redeliveryId = 'delivery-stateful-redelivery';
|
||||
GitOpsTransitions.getInstance().reserveReconcileAttempt(after.id, {
|
||||
operationId: deliveryKey('webhook', 'fetch', redeliveryId),
|
||||
actor: 'system:webhook',
|
||||
trigger: 'webhook',
|
||||
at: Date.now(),
|
||||
}, undefined, { autoApply: true, deploy: false });
|
||||
mockSuccessfulClone({ compose: withdrawnCompose, sha: 'b3'.repeat(20) });
|
||||
const redelivered = await svc.handleWebhookPull(stackName, false, redeliveryId);
|
||||
|
||||
expect(redelivered.status).toBe('success');
|
||||
expect(redelivered.message).toMatch(/Held for review/i);
|
||||
const afterRedelivery = GitOpsStore.getInstance().getLiveDirectApplication(stackName)!;
|
||||
expect(afterRedelivery.accepted_generation_id).toBe(acceptedId);
|
||||
expect(afterRedelivery.review_block_reason).toBe('stateful_withdrawal');
|
||||
} finally {
|
||||
validateSpy.mockRestore();
|
||||
await cleanupStackDir(stackName);
|
||||
}
|
||||
});
|
||||
|
||||
it('reserves and durably settles an attempt for a successful webhook fetch', async () => {
|
||||
mockSuccessfulClone({ sha: '8'.repeat(40) });
|
||||
const svc = GitSourceService.getInstance();
|
||||
@@ -2988,6 +3081,46 @@ describe('GitSourceService.pull', () => {
|
||||
await cleanupStackDir('pull-repeat');
|
||||
});
|
||||
|
||||
it('repeat pulls of a safety-held candidate keep one candidate and its block', async () => {
|
||||
const svc = GitSourceService.getInstance();
|
||||
await createFromGit('pull-held', '4444444444444444444444444444444444444444', true);
|
||||
const base = generationCount('pull-held');
|
||||
|
||||
const updatedSha = '5555555555555555555555555555555555555555';
|
||||
mockSuccessfulClone({
|
||||
compose: 'services:\n web:\n image: nginx:1.29\n',
|
||||
sha: updatedSha,
|
||||
});
|
||||
await svc.pull('pull-held');
|
||||
const app = GitOpsStore.getInstance().getLiveDirectApplication('pull-held')!;
|
||||
const stagedId = app.candidate_generation_id;
|
||||
expect(stagedId).toBeTruthy();
|
||||
expect(generationCount('pull-held')).toBe(base + 1);
|
||||
|
||||
// The automatic path refused the candidate for stateful safety. The
|
||||
// hold must survive a repeat pull of the same commit: minting a
|
||||
// lookalike would clear the reason and re-run the guard every poll.
|
||||
GitOpsTransitions.getInstance().sourceReviewBlocked({
|
||||
applicationId: app.id,
|
||||
generationId: stagedId!,
|
||||
reason: 'stateful_withdrawal',
|
||||
envelope: { operationId: 'hold-pull-held', actor: 'system:source-controller', trigger: 'poll', at: Date.now() },
|
||||
});
|
||||
|
||||
mockSuccessfulClone({
|
||||
compose: 'services:\n web:\n image: nginx:1.29\n',
|
||||
sha: updatedSha,
|
||||
});
|
||||
await svc.pull('pull-held');
|
||||
|
||||
expect(generationCount('pull-held')).toBe(base + 1);
|
||||
const after = GitOpsStore.getInstance().getLiveDirectApplication('pull-held')!;
|
||||
expect(after.candidate_generation_id).toBe(stagedId);
|
||||
expect(after.review_required).toBe(1);
|
||||
expect(after.review_block_reason).toBe('stateful_withdrawal');
|
||||
await cleanupStackDir('pull-held');
|
||||
});
|
||||
|
||||
it('a pull against a staged candidate missing its content digest mints anew', async () => {
|
||||
const svc = GitSourceService.getInstance();
|
||||
await createFromGit('pull-nohash', '8888888888888888888888888888888888888888');
|
||||
|
||||
@@ -441,6 +441,7 @@ function directApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -413,6 +413,7 @@ function app(
|
||||
accepted_generation_id: generationId,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: artifactSetId,
|
||||
latest_artifact_set_id: artifactSetId,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -175,8 +175,9 @@ describe('attentionReasons', () => {
|
||||
});
|
||||
|
||||
it('flags review, conflict, reconcile, retry, and suspension source states', () => {
|
||||
expect(attentionReasons(liveProjection({ source: { ...RESTING_SOURCE, status: 'source_review_pending' } })))
|
||||
.toContain('source_review_pending');
|
||||
expect(attentionReasons(liveProjection({
|
||||
source: { ...RESTING_SOURCE, status: 'source_review_pending', reviewBlockReason: null },
|
||||
}))).toContain('source_review_pending');
|
||||
expect(attentionReasons(liveProjection({ source: { ...RESTING_SOURCE, status: 'source_conflict_blocker' } })))
|
||||
.toContain('source_conflict_blocker');
|
||||
expect(attentionReasons(liveProjection({ source: { ...RESTING_SOURCE, status: 'source_reconcile_required' } })))
|
||||
@@ -189,6 +190,20 @@ describe('attentionReasons', () => {
|
||||
}))).toContain('source_suspended');
|
||||
});
|
||||
|
||||
it('names a stateful-withdrawal block instead of the generic review state', () => {
|
||||
const blocked = attentionReasons(liveProjection({
|
||||
source: {
|
||||
...RESTING_SOURCE,
|
||||
status: 'source_review_pending',
|
||||
reviewBlockReason: 'stateful_withdrawal',
|
||||
},
|
||||
}));
|
||||
expect(blocked).toEqual(['stateful_withdrawal_blocked']);
|
||||
// The block is a pending operator decision, not damage: the posture stays
|
||||
// `attention`, never `failed`.
|
||||
expect(hasFailureReason(blocked)).toBe(false);
|
||||
});
|
||||
|
||||
it('does not flag a pending-but-normal candidate as attention', () => {
|
||||
// candidate_ready means the state machine is waiting for the operator to
|
||||
// apply an update by design; the issue's attention set intentionally does
|
||||
@@ -363,6 +378,7 @@ describe('attentionReasons', () => {
|
||||
'source_suspended',
|
||||
'source_unknown_outcome',
|
||||
'stateful_confirmation_required',
|
||||
'stateful_withdrawal_blocked',
|
||||
'target_stale',
|
||||
'target_unreachable',
|
||||
]);
|
||||
|
||||
@@ -767,6 +767,7 @@ function inlineApp(id: string, blueprintId: number): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -472,6 +472,7 @@ function creatingApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -690,6 +690,7 @@ function creatingApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -326,6 +326,7 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -1390,6 +1390,7 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -514,6 +514,7 @@ function application(): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -150,6 +150,7 @@ function creatingApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -32,7 +32,7 @@ describe('outcomeFromSourceFacet', () => {
|
||||
});
|
||||
|
||||
it('reports pending_review with a review next action', () => {
|
||||
const facet: SourceFacet = { ...identity, status: 'source_review_pending' };
|
||||
const facet: SourceFacet = { ...identity, status: 'source_review_pending', reviewBlockReason: null };
|
||||
const result = outcomeFromSourceFacet(facet);
|
||||
expect(result.outcome).toBe('pending_review');
|
||||
expect(result.nextAction).toBe('review');
|
||||
|
||||
@@ -468,6 +468,7 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -263,6 +263,7 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -444,6 +444,7 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -165,6 +165,7 @@ function directApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -204,6 +204,7 @@ function seedGitManagedBlueprint(opts: { nodeCount?: number } = {}): Seeded {
|
||||
latest_artifact_set_id: artifactId,
|
||||
source_acceptance_ref: acceptanceId,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
source_policy: 'manual',
|
||||
};
|
||||
store.insertApplication(row);
|
||||
|
||||
@@ -529,6 +529,7 @@ function directApp(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -1117,6 +1117,7 @@ function app(id: string, stackName: string): GitOpsApplicationRow {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -40,6 +40,7 @@ export function directApplicationFixture(id: string, stackName: string): GitOpsA
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -280,6 +280,7 @@ function app(
|
||||
accepted_generation_id: generationId,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: artifactSetId,
|
||||
latest_artifact_set_id: artifactSetId,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -23,12 +23,15 @@ import { stackManagedRoot } from '../services/gitops/directApplication';
|
||||
import { candidateContentFingerprint } from '../services/gitops/fingerprint';
|
||||
import { GitSourceService } from '../services/GitSourceService';
|
||||
import { SourceController } from '../services/gitops/SourceController';
|
||||
import { attentionReasons } from '../services/gitops/attention';
|
||||
import { projectApplication } from '../services/gitops/derive';
|
||||
import { DatabaseService } from '../services/DatabaseService';
|
||||
import type { GitOpsApplicationRow } from '../services/gitops/types';
|
||||
import type { ReconcileResult } from '../services/gitops/outcomes';
|
||||
|
||||
const TICK_MS = 60_000;
|
||||
const okResult: ReconcileResult = { outcome: 'no_source_change', reason: 'ok', nextAction: 'none' };
|
||||
const fetchedResult: ReconcileResult = { outcome: 'candidate_already_fetched', reason: 'ok', nextAction: 'none' };
|
||||
|
||||
type EvaluateCandidatePolicy = typeof import('../services/PolicyEnforcement')['evaluateCandidatePolicy'];
|
||||
|
||||
@@ -75,18 +78,32 @@ function getApp(id: string): GitOpsApplicationRow {
|
||||
}
|
||||
|
||||
const STAGED_COMPOSE = 'services:\n web:\n image: nginx:1.27\n';
|
||||
const STAGED_CONTENT_SHA = candidateContentFingerprint([{ path: 'compose.yaml', content: STAGED_COMPOSE }]);
|
||||
|
||||
/** Write the compose file the controller reads as candidate evidence. */
|
||||
function stageCandidateComposeFile(stackName: string, generationId: string): void {
|
||||
function writeCandidateCompose(stackName: string, generationId: string, compose: string): void {
|
||||
const dir = path.join(stackManagedRoot(stackName), 'generations', `candidate-${generationId}`);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, 'compose.yaml'), STAGED_COMPOSE);
|
||||
fs.writeFileSync(path.join(dir, 'compose.yaml'), compose);
|
||||
}
|
||||
|
||||
/** Write the applied copy of a generation, the content the guard diffs against. */
|
||||
function writeAppliedCompose(stackName: string, generationId: string, compose: string): void {
|
||||
const dir = path.join(stackManagedRoot(stackName), 'generations', `applied-${generationId}-0`);
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
fs.writeFileSync(path.join(dir, 'compose.yaml'), compose);
|
||||
}
|
||||
|
||||
type StageOpts = {
|
||||
blocked?: boolean;
|
||||
sourceEvidence?: string | null;
|
||||
securityEvidence?: string | null;
|
||||
composeInputs?: string | null;
|
||||
compose?: string;
|
||||
};
|
||||
|
||||
/**
|
||||
* Stage a live candidate through the real transitions, mirroring the fetch
|
||||
* success sequence GitSourceService drives: fetch opens, generation is
|
||||
* Insert a generation and drive it through the real fetch transitions,
|
||||
* mirroring the sequence GitSourceService drives: fetch opens, generation is
|
||||
* inserted, the commit lands, the candidate passes validation. reviewRequired
|
||||
* follows the production rule (anything not automatic needs a human sign-off).
|
||||
*
|
||||
@@ -95,24 +112,16 @@ function stageCandidateComposeFile(stackName: string, generationId: string): voi
|
||||
* conflict-blocked candidate, which the acceptance boundary must refuse no
|
||||
* matter what the policy evaluator says.
|
||||
*/
|
||||
function stageCandidate(
|
||||
appId: string,
|
||||
function insertAndStageGeneration(
|
||||
app: GitOpsApplicationRow,
|
||||
stackName: string,
|
||||
generationId: string,
|
||||
policy: 'manual' | 'review' | 'automatic' = 'automatic',
|
||||
opts: {
|
||||
blocked?: boolean;
|
||||
sourceEvidence?: string | null;
|
||||
securityEvidence?: string | null;
|
||||
composeInputs?: string | null;
|
||||
} = {},
|
||||
opts: StageOpts = {},
|
||||
): void {
|
||||
const app = { ...directApplicationFixture(appId, stackName), source_policy: policy };
|
||||
GitOpsTransitions.getInstance().activateDirect({
|
||||
application: app,
|
||||
nodeId: 1,
|
||||
envelope: { operationId: `seed-${appId}`, actor: 'test', trigger: 'config_change', at: Date.now() },
|
||||
});
|
||||
const appId = app.id;
|
||||
const policy = app.source_policy;
|
||||
const compose = opts.compose ?? STAGED_COMPOSE;
|
||||
const contentSha = candidateContentFingerprint([{ path: 'compose.yaml', content: compose }]);
|
||||
GitOpsStore.getInstance().insertGeneration({
|
||||
id: generationId,
|
||||
application_id: appId,
|
||||
@@ -136,7 +145,7 @@ function stageCandidate(
|
||||
redacted_limitations_json: '[]',
|
||||
portable_manifest_json: null,
|
||||
compose_inputs_json: opts.composeInputs === undefined
|
||||
? JSON.stringify({ candidateContentSha256: STAGED_CONTENT_SHA })
|
||||
? JSON.stringify({ candidateContentSha256: contentSha })
|
||||
: opts.composeInputs,
|
||||
source_policy_evidence_json: opts.sourceEvidence === undefined
|
||||
? JSON.stringify({ sourcePolicy: policy })
|
||||
@@ -157,7 +166,24 @@ function stageCandidate(
|
||||
} else {
|
||||
GitOpsTransitions.getInstance().candidateReady(appId, generationId, policy !== 'automatic', env);
|
||||
}
|
||||
stageCandidateComposeFile(stackName, generationId);
|
||||
writeCandidateCompose(stackName, generationId, compose);
|
||||
}
|
||||
|
||||
/** Activate a fresh Direct application and stage its first candidate. */
|
||||
function stageCandidate(
|
||||
appId: string,
|
||||
stackName: string,
|
||||
generationId: string,
|
||||
policy: 'manual' | 'review' | 'automatic' = 'automatic',
|
||||
opts: StageOpts = {},
|
||||
): void {
|
||||
const app = { ...directApplicationFixture(appId, stackName), source_policy: policy };
|
||||
GitOpsTransitions.getInstance().activateDirect({
|
||||
application: app,
|
||||
nodeId: 1,
|
||||
envelope: { operationId: `seed-${appId}`, actor: 'test', trigger: 'config_change', at: Date.now() },
|
||||
});
|
||||
insertAndStageGeneration(app, stackName, generationId, opts);
|
||||
}
|
||||
|
||||
/** Arm the poll cursor in the past through the real transition, making the row poll-due. */
|
||||
@@ -733,3 +759,366 @@ describe('SourceController automatic acceptance', () => {
|
||||
expect(GitOpsStore.getInstance().getGeneration('gen-ev-held')?.security_policy_evidence_json).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* The automatic path is the only acceptance route with no human in the loop,
|
||||
* so it must never withdraw a stateful workload on its own. Each test stages a
|
||||
* real accepted generation with a stateful service on disk, then drives the
|
||||
* controller against a follow-up candidate that either preserves or withdraws
|
||||
* it. The block falls back to the review behavior and must be visible as a
|
||||
* canonical attention state, never a silent skip.
|
||||
*/
|
||||
describe('SourceController stateful withdrawal guard', () => {
|
||||
const STATEFUL_COMPOSE = [
|
||||
'services:',
|
||||
' db:',
|
||||
' image: postgres:16',
|
||||
' volumes:',
|
||||
' - pgdata:/var/lib/postgresql/data',
|
||||
'volumes:',
|
||||
' pgdata:',
|
||||
'',
|
||||
].join('\n');
|
||||
const KEEPS_STATEFUL_COMPOSE = [
|
||||
'services:',
|
||||
' db:',
|
||||
' image: postgres:17',
|
||||
' volumes:',
|
||||
' - pgdata:/var/lib/postgresql/data',
|
||||
' cache:',
|
||||
' image: redis:7',
|
||||
'volumes:',
|
||||
' pgdata:',
|
||||
'',
|
||||
].join('\n');
|
||||
const WITHDRAWN_COMPOSE = 'services:\n web:\n image: nginx:1.27\n';
|
||||
|
||||
/** Accept the staged candidate through the real transition and lay down the applied copy. */
|
||||
function acceptAndApply(appId: string, stackName: string, generationId: string, compose: string): void {
|
||||
GitOpsTransitions.getInstance().sourceAccepted({
|
||||
applicationId: appId,
|
||||
generationId,
|
||||
artifactSetId: `as-${generationId}`,
|
||||
sourceAcceptanceId: `sa-${generationId}`,
|
||||
authority: 'operator',
|
||||
envelope: { operationId: `accept-${generationId}`, actor: 'operator', trigger: 'manual', at: Date.now() },
|
||||
});
|
||||
writeAppliedCompose(stackName, generationId, compose);
|
||||
}
|
||||
|
||||
function stageFollowUp(
|
||||
appId: string,
|
||||
stackName: string,
|
||||
generationId: string,
|
||||
compose: string,
|
||||
): void {
|
||||
insertAndStageGeneration(getApp(appId), stackName, generationId, { compose });
|
||||
}
|
||||
|
||||
it('holds a candidate that withdraws a stateful service instead of auto-accepting', async () => {
|
||||
stageCandidate('app-withdraw', 'withdraw-web', 'wd-prev', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
acceptAndApply('app-withdraw', 'withdraw-web', 'wd-prev', STATEFUL_COMPOSE);
|
||||
stageFollowUp('app-withdraw', 'withdraw-web', 'wd-next', WITHDRAWN_COMPOSE);
|
||||
mockDue([armDuePoll('app-withdraw')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
const row = getApp('app-withdraw');
|
||||
// The refusal falls back to review exactly as if the policy were
|
||||
// 'review': the candidate stays staged, nothing is accepted or
|
||||
// dispatched, and the reason names the stateful block.
|
||||
expect(row.accepted_generation_id).toBe('wd-prev');
|
||||
expect(row.candidate_generation_id).toBe('wd-next');
|
||||
expect(row.review_required).toBe(1);
|
||||
expect(row.review_block_reason).toBe('stateful_withdrawal');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
expect(warnSpy.mock.calls.some((args) =>
|
||||
String(args[0]).includes('[SourceController] automatic candidate held for app-withdraw')
|
||||
&& String(args[0]).includes('withdraws or renames stateful services')
|
||||
&& String(args[0]).includes('db'))).toBe(true);
|
||||
|
||||
// The block reaches the hub-side classifier as its own named reason.
|
||||
const projection = projectApplication('app-withdraw', false);
|
||||
expect(attentionReasons(projection)).toContain('stateful_withdrawal_blocked');
|
||||
});
|
||||
|
||||
it('blocks a renamed stateful service the same as a removed one', async () => {
|
||||
stageCandidate('app-rename', 'rename-web', 'rn-prev', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
acceptAndApply('app-rename', 'rename-web', 'rn-prev', STATEFUL_COMPOSE);
|
||||
// The same data-bearing service under a new name: the old name leaves
|
||||
// the stateful set, so the data it owned is orphaned by the rename.
|
||||
stageFollowUp('app-rename', 'rename-web', 'rn-next', [
|
||||
'services:',
|
||||
' database:',
|
||||
' image: postgres:17',
|
||||
' volumes:',
|
||||
' - pgdata:/var/lib/postgresql/data',
|
||||
'volumes:',
|
||||
' pgdata:',
|
||||
'',
|
||||
].join('\n'));
|
||||
mockDue([armDuePoll('app-rename')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(getApp('app-rename').review_block_reason).toBe('stateful_withdrawal');
|
||||
expect(getApp('app-rename').accepted_generation_id).toBe('rn-prev');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('still auto-accepts a change that keeps every stateful service', async () => {
|
||||
stageCandidate('app-additive', 'additive-web', 'ad-prev', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
acceptAndApply('app-additive', 'additive-web', 'ad-prev', STATEFUL_COMPOSE);
|
||||
stageFollowUp('app-additive', 'additive-web', 'ad-next', KEEPS_STATEFUL_COMPOSE);
|
||||
mockDue([armDuePoll('app-additive')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
const row = getApp('app-additive');
|
||||
expect(row.accepted_generation_id).toBe('ad-next');
|
||||
expect(row.review_required).toBe(0);
|
||||
expect(row.review_block_reason).toBeNull();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('does not re-evaluate a source that already fell back to review', async () => {
|
||||
stageCandidate('app-once', 'once-web', 'on-prev', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
acceptAndApply('app-once', 'once-web', 'on-prev', STATEFUL_COMPOSE);
|
||||
stageFollowUp('app-once', 'once-web', 'on-next', WITHDRAWN_COMPOSE);
|
||||
mockDue([armDuePoll('app-once')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
spyOnDispatch();
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
await advanceOneTick();
|
||||
|
||||
// The refusal is durable: the next tick re-runs the fetch mock but
|
||||
// never re-evaluates the policy, so the block cannot turn into a
|
||||
// retry loop that eventually slips an acceptance through.
|
||||
expect(evaluateCandidatePolicy).toHaveBeenCalledTimes(1);
|
||||
expect(getApp('app-once').review_block_reason).toBe('stateful_withdrawal');
|
||||
});
|
||||
|
||||
it('holds when the generation in force cannot be read', async () => {
|
||||
stageCandidate('app-unreadable', 'unreadable-web', 'un-prev', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
acceptAndApply('app-unreadable', 'unreadable-web', 'un-prev', STATEFUL_COMPOSE);
|
||||
// Both copies of the generation in force disappear (an operator pruned
|
||||
// the managed area by hand): the guard cannot prove the stateful
|
||||
// services survive, so it holds rather than guess.
|
||||
fs.rmSync(path.join(stackManagedRoot('unreadable-web'), 'generations', 'applied-un-prev-0'), { recursive: true, force: true });
|
||||
fs.rmSync(path.join(stackManagedRoot('unreadable-web'), 'generations', 'candidate-un-prev'), { recursive: true, force: true });
|
||||
stageFollowUp('app-unreadable', 'unreadable-web', 'un-next', KEEPS_STATEFUL_COMPOSE);
|
||||
mockDue([armDuePoll('app-unreadable')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
vi.spyOn(console, 'warn').mockImplementation(() => undefined);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(getApp('app-unreadable').accepted_generation_id).toBe('un-prev');
|
||||
expect(getApp('app-unreadable').review_block_reason).toBe('stateful_withdrawal');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('holds a candidate that keeps a stateful service name but strips its volumes', async () => {
|
||||
stageCandidate('app-strip', 'strip-web', 'st-prev', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
acceptAndApply('app-strip', 'strip-web', 'st-prev', STATEFUL_COMPOSE);
|
||||
// Same service name, no mount: the data the service owned is left
|
||||
// behind, which is a withdrawal even though the name survives.
|
||||
stageFollowUp('app-strip', 'strip-web', 'st-next', 'services:\n db:\n image: postgres:17\n');
|
||||
mockDue([armDuePoll('app-strip')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
// The withdrawal branch, not an unreadable-evidence hold.
|
||||
expect(warnSpy.mock.calls.some((args) =>
|
||||
String(args[0]).includes('withdraws or renames stateful services (db)'))).toBe(true);
|
||||
expect(getApp('app-strip').accepted_generation_id).toBe('st-prev');
|
||||
expect(getApp('app-strip').review_block_reason).toBe('stateful_withdrawal');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('holds when the candidate compose does not parse', async () => {
|
||||
// A stateless baseline, so no withdrawal is possible: only the parse
|
||||
// failure itself can hold. It must read as missing evidence, never as
|
||||
// "nothing stateful here".
|
||||
stageCandidate('app-badyaml', 'badyaml-web', 'by-prev', 'automatic', { compose: WITHDRAWN_COMPOSE });
|
||||
acceptAndApply('app-badyaml', 'badyaml-web', 'by-prev', WITHDRAWN_COMPOSE);
|
||||
stageFollowUp('app-badyaml', 'badyaml-web', 'by-next', 'services:\n db: [unclosed\n');
|
||||
mockDue([armDuePoll('app-badyaml')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => undefined);
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(warnSpy.mock.calls.some((args) => String(args[0]).includes('compose does not parse'))).toBe(true);
|
||||
expect(getApp('app-badyaml').accepted_generation_id).toBe('by-prev');
|
||||
expect(getApp('app-badyaml').review_block_reason).toBe('stateful_withdrawal');
|
||||
expect(dispatch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it('accepts the first candidate when no generation is in force yet', async () => {
|
||||
// No managed baseline exists, so the first acceptance defines it,
|
||||
// even for a stateful compose.
|
||||
stageCandidate('app-first', 'first-web', 'fi-first', 'automatic', { compose: STATEFUL_COMPOSE });
|
||||
mockDue([armDuePoll('app-first')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(getApp('app-first').accepted_generation_id).toBe('fi-first');
|
||||
expect(getApp('app-first').review_block_reason).toBeNull();
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* The durable block makes the controller skip the source on every tick, so
|
||||
* every exit from the held state must clear the reason. A reason left behind
|
||||
* would silently stop automatic acceptance for that application forever.
|
||||
*/
|
||||
describe('stateful review block lifecycle', () => {
|
||||
function envelope(id: string) {
|
||||
return { operationId: id, actor: 'operator', trigger: 'manual' as const, at: Date.now() };
|
||||
}
|
||||
|
||||
/** Stage an accepted baseline plus a follow-up candidate, then record the block. */
|
||||
function blocked(appId: string, stackName: string): void {
|
||||
stageCandidate(appId, stackName, `${appId}-prev`, 'automatic');
|
||||
GitOpsTransitions.getInstance().sourceAccepted({
|
||||
applicationId: appId,
|
||||
generationId: `${appId}-prev`,
|
||||
artifactSetId: `as-${appId}-prev`,
|
||||
sourceAcceptanceId: `sa-${appId}-prev`,
|
||||
authority: 'operator',
|
||||
envelope: envelope(`accept-${appId}-prev`),
|
||||
});
|
||||
insertAndStageGeneration(getApp(appId), stackName, `${appId}-next`);
|
||||
GitOpsTransitions.getInstance().sourceReviewBlocked({
|
||||
applicationId: appId,
|
||||
generationId: `${appId}-next`,
|
||||
reason: 'stateful_withdrawal',
|
||||
envelope: envelope(`block-${appId}`),
|
||||
});
|
||||
expect(getApp(appId).review_block_reason).toBe('stateful_withdrawal');
|
||||
expect(getApp(appId).review_required).toBe(1);
|
||||
}
|
||||
|
||||
it('clears on explicit operator acceptance', () => {
|
||||
blocked('blk-accept', 'blk-accept-web');
|
||||
GitOpsTransitions.getInstance().sourceAccepted({
|
||||
applicationId: 'blk-accept',
|
||||
generationId: 'blk-accept-next',
|
||||
artifactSetId: 'as-blk-accept-next',
|
||||
sourceAcceptanceId: 'sa-blk-accept-next',
|
||||
authority: 'operator',
|
||||
envelope: envelope('accept-blk-accept-next'),
|
||||
});
|
||||
const row = getApp('blk-accept');
|
||||
expect(row.accepted_generation_id).toBe('blk-accept-next');
|
||||
expect(row.review_block_reason).toBeNull();
|
||||
expect(row.review_required).toBe(0);
|
||||
});
|
||||
|
||||
it('clears on dismissal', () => {
|
||||
blocked('blk-dismiss', 'blk-dismiss-web');
|
||||
GitOpsTransitions.getInstance().dismissed('blk-dismiss', envelope('dismiss-blk'));
|
||||
const row = getApp('blk-dismiss');
|
||||
expect(row.candidate_generation_id).toBeNull();
|
||||
expect(row.review_block_reason).toBeNull();
|
||||
});
|
||||
|
||||
it('clears when a newer candidate is staged', () => {
|
||||
blocked('blk-newer', 'blk-newer-web');
|
||||
insertAndStageGeneration(getApp('blk-newer'), 'blk-newer-web', 'blk-newer-third');
|
||||
const row = getApp('blk-newer');
|
||||
expect(row.candidate_generation_id).toBe('blk-newer-third');
|
||||
expect(row.review_block_reason).toBeNull();
|
||||
expect(row.review_required).toBe(0);
|
||||
});
|
||||
|
||||
it.each(['review', 'manual'] as const)('clears when the policy moves to %s', (policy) => {
|
||||
const id = `blk-policy-${policy}`;
|
||||
blocked(id, `${id}-web`);
|
||||
GitOpsTransitions.getInstance().sourcePolicyChanged(id, policy, envelope(`policy-${id}`));
|
||||
const row = getApp(id);
|
||||
expect(row.review_block_reason).toBeNull();
|
||||
// The candidate still needs a decision; only the automatic refusal
|
||||
// label goes away, leaving the generic review state.
|
||||
expect(row.review_required).toBe(1);
|
||||
expect(attentionReasons(projectApplication(id, false))).not.toContain('stateful_withdrawal_blocked');
|
||||
});
|
||||
|
||||
it('resumes automatic acceptance once a newer candidate clears the block', async () => {
|
||||
blocked('blk-resume', 'blk-resume-web');
|
||||
insertAndStageGeneration(getApp('blk-resume'), 'blk-resume-web', 'blk-resume-third');
|
||||
mockDue([armDuePoll('blk-resume')]);
|
||||
evaluateCandidatePolicy.mockResolvedValue({ status: 'allowed' });
|
||||
spyOnReconcile().mockResolvedValue(fetchedResult);
|
||||
const dispatch = spyOnDispatch();
|
||||
|
||||
controller.start();
|
||||
await advanceOneTick();
|
||||
|
||||
expect(getApp('blk-resume').accepted_generation_id).toBe('blk-resume-third');
|
||||
expect(dispatch).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('keeps the block when the policy is re-saved as automatic', () => {
|
||||
blocked('blk-auto', 'blk-auto-web');
|
||||
GitOpsTransitions.getInstance().sourcePolicyChanged('blk-auto', 'automatic', envelope('policy-blk-auto'));
|
||||
expect(getApp('blk-auto').review_block_reason).toBe('stateful_withdrawal');
|
||||
});
|
||||
|
||||
it('refuses to record a block while an apply is in flight', () => {
|
||||
stageCandidate('blk-inflight', 'blk-inflight-web', 'blk-inflight-gen', 'automatic');
|
||||
GitOpsTransitions.getInstance().applyStarted('blk-inflight', 'blk-inflight-gen', envelope('apply-blk-inflight'));
|
||||
expect(() => GitOpsTransitions.getInstance().sourceReviewBlocked({
|
||||
applicationId: 'blk-inflight',
|
||||
generationId: 'blk-inflight-gen',
|
||||
reason: 'stateful_withdrawal',
|
||||
envelope: envelope('block-blk-inflight'),
|
||||
})).toThrow('cannot hold acceptance while an apply is in flight');
|
||||
expect(getApp('blk-inflight').review_block_reason).toBeNull();
|
||||
});
|
||||
|
||||
it('projects an unknown stored reason as the plain review state', () => {
|
||||
blocked('blk-unknown', 'blk-unknown-web');
|
||||
// A value written by a newer build this one does not know.
|
||||
DatabaseService.getInstance().getDb().prepare(
|
||||
'UPDATE gitops_applications SET review_block_reason = ? WHERE id = ?',
|
||||
).run('future_reason', 'blk-unknown');
|
||||
const projection = projectApplication('blk-unknown', false);
|
||||
const reasons = attentionReasons(projection);
|
||||
expect(reasons).toContain('source_review_pending');
|
||||
expect(reasons).not.toContain('stateful_withdrawal_blocked');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -209,6 +209,47 @@ export class BlueprintAnalyzer {
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Names of services that declare a named volume or bind mount.
|
||||
*
|
||||
* The same mount classification `analyze` applies document-wide, kept
|
||||
* per-service so a caller can diff one compose against another: a service
|
||||
* whose name leaves this set between revisions is a withdrawal of whatever
|
||||
* data it owned. External volumes count as stateful here even though
|
||||
* `analyze` calls the document unknown: unproven portability is a reason
|
||||
* to guard them, not to ignore them. Null when the content does not parse,
|
||||
* so callers can hold rather than read a parse failure as "nothing
|
||||
* stateful here".
|
||||
*/
|
||||
static statefulServiceNames(composeContent: string): Set<string> | null {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = parseYaml(composeContent);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (parsed == null || typeof parsed !== 'object') return null;
|
||||
const doc = parsed as ComposeShape;
|
||||
const services = doc.services ?? {};
|
||||
const out = new Set<string>();
|
||||
for (const [serviceName, serviceDef] of Object.entries(services)) {
|
||||
if (!serviceDef || typeof serviceDef !== 'object') continue;
|
||||
for (const v of serviceDef.volumes ?? []) {
|
||||
if (typeof v === 'string') {
|
||||
// A source-less entry such as "/data" is an anonymous
|
||||
// volume, but the mount target alone still marks a
|
||||
// data-bearing service, and `analyze` counts it too.
|
||||
if (v.split(':')[0]) out.add(serviceName);
|
||||
} else if (v && typeof v === 'object') {
|
||||
const type = (v.type ?? '').toLowerCase();
|
||||
if (type === 'tmpfs') continue;
|
||||
if (type === 'bind' || type === 'volume' || v.source) out.add(serviceName);
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
static extractImageRefs(composeContent: string): string[] {
|
||||
const doc = (parseYaml(composeContent) ?? {}) as ComposeShape;
|
||||
const services = doc.services ?? {};
|
||||
|
||||
@@ -1994,6 +1994,10 @@ export class DatabaseService {
|
||||
// existing installs already have. New installs get it from the
|
||||
// CREATE TABLE; older DBs need the additive column here.
|
||||
maybeAddCol('gitops_applications', 'source_suspended_reason', 'TEXT NULL');
|
||||
// Why an automatic acceptance fell back to review (for example a
|
||||
// candidate that withdraws a stateful service). New installs get it
|
||||
// from the CREATE TABLE; older DBs need the additive column here.
|
||||
maybeAddCol('gitops_applications', 'review_block_reason', 'TEXT NULL');
|
||||
// Portable accepted-generation contract fields. Additive and
|
||||
// nullable: existing generation rows decode these as an explicit
|
||||
// limitation rather than invented evidence.
|
||||
|
||||
@@ -47,6 +47,7 @@ import {
|
||||
type ReconcileTrigger,
|
||||
} from './gitops/triggers';
|
||||
import { classifyFailure, effectivePollIntervalSecs } from './gitops/backoff';
|
||||
import { checkStatefulWithdrawal, holdForStatefulReview } from './gitops/statefulGuard';
|
||||
import { BlueprintTargetAdapter, buildAcceptedGeneration, type AcceptedGeneration, type DispatchContext, type DispatchResult } from './gitops/handoff';
|
||||
import {
|
||||
GitOpsTransitions,
|
||||
@@ -2608,16 +2609,29 @@ export class GitSourceService {
|
||||
// for a different commit, or no candidate at all, mints anew:
|
||||
// staging after an apply is a new dispatch cycle and needs its
|
||||
// own generation to accept.
|
||||
//
|
||||
// A safety-held candidate counts as the expected state for an
|
||||
// automatic source even though its review flag reads 1: the
|
||||
// hold is durable and must survive repeat pulls of the same
|
||||
// commit. Minting a lookalike would clear the hold and re-run
|
||||
// the guard on every poll, churning generations and history.
|
||||
// A plan-blocked candidate carries no review meaning at all,
|
||||
// so its flag never takes part in the no-op identity.
|
||||
const reviewRequired = gitopsApp.source_policy !== 'automatic';
|
||||
const staged = gitopsApp.candidate_generation_id
|
||||
? GitOpsStore.getInstance().getGeneration(gitopsApp.candidate_generation_id)
|
||||
: undefined;
|
||||
const reviewStateMatches = staged?.plan_blocked === 1
|
||||
? true
|
||||
: reviewRequired
|
||||
? gitopsApp.review_required === 1
|
||||
: gitopsApp.review_required === 0 || gitopsApp.review_block_reason !== null;
|
||||
if (
|
||||
staged &&
|
||||
staged.commit_sha === fetched.commitSha &&
|
||||
staged.materialization_fingerprint === identity.fingerprint &&
|
||||
staged.plan_blocked === (plan?.blocked === true ? 1 : 0) &&
|
||||
gitopsApp.review_required === (reviewRequired ? 1 : 0) &&
|
||||
reviewStateMatches &&
|
||||
storedCandidateContentSha256(staged.compose_inputs_json) !== null
|
||||
) {
|
||||
return;
|
||||
@@ -7363,6 +7377,15 @@ export class GitSourceService {
|
||||
if (diag) console.log(`[GitSource:diag] webhook pending-only stack=${stackName} sha=${commitSha.slice(0, 7)}`);
|
||||
return { status: 'success', message: `Pending update ready at ${commitSha.slice(0, 7)}.` };
|
||||
}
|
||||
// Automatic modes have no human in the loop, so the same stateful
|
||||
// guard the poll path applies must hold a withdrawing delivery instead
|
||||
// of applying it. The delivery still reads as received; the operator
|
||||
// sees the block in the attention queue and accepts explicitly.
|
||||
const statefulHold = this.statefulWebhookHold(stackName, gitopsApp.id, trigger, actor);
|
||||
if (statefulHold !== null) {
|
||||
if (diag) console.log(`[GitSource:diag] webhook auto-apply held stack=${stackName} sha=${commitSha.slice(0, 7)}`);
|
||||
return { status: 'success', message: statefulHold };
|
||||
}
|
||||
|
||||
try {
|
||||
const request: ReconcileRequest = {
|
||||
@@ -7406,6 +7429,51 @@ export class GitSourceService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Hold an automatic webhook delivery when the staged candidate withdraws
|
||||
* or renames a stateful service, or when that cannot be proven. Returns
|
||||
* the operator-facing message when held, null when the delivery may apply.
|
||||
*/
|
||||
private statefulWebhookHold(
|
||||
stackName: string,
|
||||
applicationId: string,
|
||||
trigger: ReconcileTrigger,
|
||||
actor: string,
|
||||
): string | null {
|
||||
const app = GitOpsStore.getInstance().getApplication(applicationId);
|
||||
// The call site only reaches here for an automated delivery intent, so
|
||||
// the current policy value is not the gate: a stored redelivery keeps
|
||||
// its original automated authority even if the policy has moved since.
|
||||
if (!app) return null;
|
||||
if (app.review_block_reason !== null) {
|
||||
return 'Held for review: the pending update removes or renames a stateful service.';
|
||||
}
|
||||
if (!app.candidate_generation_id) return null;
|
||||
const generation = GitOpsStore.getInstance().getGeneration(app.candidate_generation_id);
|
||||
if (!generation) {
|
||||
console.warn(
|
||||
`[GitSource] webhook auto-apply held for ${sanitizeForLog(app.id)}: the candidate generation row is missing, so a stateful withdrawal cannot be ruled out`,
|
||||
);
|
||||
return 'Held for review: the pending update could not be checked for stateful services.';
|
||||
}
|
||||
const withdrawal = checkStatefulWithdrawal(stackName, app, generation);
|
||||
if (withdrawal.status === 'clear') return null;
|
||||
const recorded = holdForStatefulReview(app, generation, {
|
||||
operationId: crypto.randomUUID(),
|
||||
actor,
|
||||
trigger,
|
||||
at: Date.now(),
|
||||
});
|
||||
const detail = withdrawal.status === 'withdrawn'
|
||||
? `withdraws or renames stateful services (${withdrawal.services.map((name) => sanitizeForLog(name)).join(', ')})`
|
||||
: `cannot be proven safe for stateful services (${withdrawal.reason})`;
|
||||
console.warn(
|
||||
`[GitSource] webhook auto-apply held for ${sanitizeForLog(app.id)}: the candidate ${detail}; an operator must accept explicitly`,
|
||||
);
|
||||
const unrecorded = recorded ? '' : ' The block could not be recorded; check the server logs.';
|
||||
return `Held for review: the pending update ${detail}.${unrecorded}`;
|
||||
}
|
||||
|
||||
private webhookFailure(stackName: string, deliverySuffix: string, phase: 'pull' | 'apply', error: unknown): WebhookPullResult {
|
||||
const msg = error instanceof GitSourceError ? `${error.code}: ${error.message}` : error instanceof Error ? error.message : String(error);
|
||||
const scrubbed = scrubCredentials(msg);
|
||||
|
||||
@@ -1,19 +1,17 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { GitOpsStore } from './store';
|
||||
import { GitOpsTransitions } from './transitions';
|
||||
import { GitSourceService } from '../GitSourceService';
|
||||
import { buildAcceptedGeneration } from './handoff';
|
||||
import { checkStatefulWithdrawal, holdForStatefulReview, readStagedGeneration } from './statefulGuard';
|
||||
import { DatabaseService } from '../DatabaseService';
|
||||
import type { GitOpsApplicationRow, GitOpsGenerationRow } from './types';
|
||||
import { classifyFailure, nextRetryAt, isGitSourceErrorCode, effectivePollIntervalSecs } from './backoff';
|
||||
import { evaluateCandidatePolicy } from '../PolicyEnforcement';
|
||||
import { buildSystemPolicyGateOptions } from '../../helpers/policyGate';
|
||||
import { stackManagedRoot, newGitOpsId } from './directApplication';
|
||||
import { newGitOpsId } from './directApplication';
|
||||
import { NodeRegistry } from '../NodeRegistry';
|
||||
import { gitSourceLocalComposeFiles } from '../../utils/gitComposeFiles';
|
||||
import { extractImagesFromCompose, loadDotEnv } from '../ImageUpdateService';
|
||||
import { extractImagesFromCompose } from '../ImageUpdateService';
|
||||
import type { ReconcileOutcome } from './outcomes';
|
||||
import type { ReconcileTrigger } from './triggers';
|
||||
import { redactSensitiveText, sanitizeForLog } from '../../utils/safeLog';
|
||||
@@ -440,6 +438,12 @@ export class SourceController {
|
||||
trigger: ReconcileTrigger,
|
||||
): Promise<void> {
|
||||
if (app.source_policy !== 'automatic') return;
|
||||
// A source that already fell back to review under a safety refusal
|
||||
// waits for the operator: re-evaluating every tick would repeat the
|
||||
// refusal without changing anything. The reason field, not the plain
|
||||
// review flag, is the marker: a review-policy candidate later switched
|
||||
// to automatic has no refusal to honor and must evaluate normally.
|
||||
if (app.review_block_reason !== null) return;
|
||||
if (!app.candidate_generation_id || app.accepted_generation_id === app.candidate_generation_id) return;
|
||||
const generation = GitOpsStore.getInstance().getGeneration(app.candidate_generation_id);
|
||||
if (!generation) return;
|
||||
@@ -491,6 +495,27 @@ export class SourceController {
|
||||
this.warnSkipped(app.id, 'source revalidation failed', e);
|
||||
return;
|
||||
}
|
||||
// Stateful safety: this acceptance route has no human in the loop, so
|
||||
// it must prove the candidate does not withdraw or rename a stateful
|
||||
// workload relative to the generation in force. Unprovable evidence is
|
||||
// a hold, never an acceptance.
|
||||
const withdrawal = checkStatefulWithdrawal(stackName, app, acceptGeneration);
|
||||
if (withdrawal.status !== 'clear') {
|
||||
const recorded = holdForStatefulReview(app, acceptGeneration, {
|
||||
operationId: randomUUID(),
|
||||
actor: 'system:source-controller',
|
||||
trigger,
|
||||
at: Date.now(),
|
||||
});
|
||||
const detail = withdrawal.status === 'withdrawn'
|
||||
? `the candidate withdraws or renames stateful services (${withdrawal.services.map((name) => sanitizeForLog(name)).join(', ')})`
|
||||
: `a stateful withdrawal cannot be ruled out (${withdrawal.reason})`;
|
||||
const unrecorded = recorded ? '' : ' (the block could not be recorded; see server logs)';
|
||||
console.warn(
|
||||
`[SourceController] automatic candidate held for ${sanitizeForLog(app.id)}: ${detail}; an operator must accept explicitly${unrecorded}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
try {
|
||||
GitOpsTransitions.getInstance().sourceAccepted({
|
||||
applicationId: app.id,
|
||||
@@ -539,62 +564,22 @@ export class SourceController {
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the candidate's image refs off disk for policy evaluation: the
|
||||
* candidate's compose files (paths come from the application row, mapped
|
||||
* the same way staging laid them out) with its staged .env merged under
|
||||
* process.env, exactly as the update scanner resolves them. Returns null
|
||||
* when the evidence cannot be read (missing staging directory, unreadable
|
||||
* compose file, a .env that exists but cannot be read; an absent .env is
|
||||
* normal and interpolates compose-only): the caller then holds the
|
||||
* candidate rather than evaluating against an empty ref set.
|
||||
* Read the candidate's image refs off disk for policy evaluation. Null
|
||||
* means the evidence could not be read, so the caller holds the candidate
|
||||
* rather than evaluating against an empty ref set.
|
||||
*/
|
||||
private candidateImageRefs(
|
||||
stackName: string,
|
||||
app: GitOpsApplicationRow,
|
||||
generation: GitOpsGenerationRow,
|
||||
): string[] | null {
|
||||
try {
|
||||
const candidateDir = path.join(stackManagedRoot(stackName), generation.candidate_dir);
|
||||
const composePaths: string[] = app.compose_paths_json ? JSON.parse(app.compose_paths_json) : [];
|
||||
if (composePaths.length === 0) return null;
|
||||
const contents: string[] = [];
|
||||
for (const local of gitSourceLocalComposeFiles(composePaths)) {
|
||||
contents.push(fs.readFileSync(path.join(candidateDir, local), 'utf8'));
|
||||
}
|
||||
let envVars: Record<string, string> = {};
|
||||
try {
|
||||
envVars = loadDotEnv(fs.readFileSync(path.join(candidateDir, '.env'), 'utf8'));
|
||||
} catch (e) {
|
||||
// A missing .env is normal (the staging step only writes one
|
||||
// when sync_env produced it), but a present-yet-unreadable one
|
||||
// would silently narrow the interpolation inputs the policy
|
||||
// sees, so surface it and hold rather than evaluate on
|
||||
// incomplete evidence.
|
||||
const missing = (e as NodeJS.ErrnoException).code === 'ENOENT';
|
||||
if (!missing) {
|
||||
console.warn(
|
||||
`[SourceController] staged .env unreadable for ${sanitizeForLog(stackName)} (generation ${sanitizeForLog(generation.id)}); holding candidate:`,
|
||||
e instanceof Error ? e.message : String(e),
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
const merged: Record<string, string> = { ...envVars };
|
||||
for (const [k, v] of Object.entries(process.env)) {
|
||||
if (v !== undefined) merged[k] = v;
|
||||
}
|
||||
const refs = new Set<string>();
|
||||
for (const content of contents) {
|
||||
for (const img of extractImagesFromCompose(content, merged)) refs.add(img);
|
||||
}
|
||||
return [...refs];
|
||||
} catch (e) {
|
||||
console.warn(
|
||||
`[SourceController] candidate evidence unreadable for ${sanitizeForLog(stackName)} (generation ${sanitizeForLog(generation.id)}):`,
|
||||
e instanceof Error ? e.message : String(e),
|
||||
);
|
||||
return null;
|
||||
const staged = readStagedGeneration(stackName, app, generation);
|
||||
if (!staged) return null;
|
||||
const refs = new Set<string>();
|
||||
for (const content of staged.contents) {
|
||||
for (const img of extractImagesFromCompose(content, staged.mergedEnv)) refs.add(img);
|
||||
}
|
||||
return [...refs];
|
||||
}
|
||||
|
||||
private warnSkipped(appId: string, what: string, e: unknown): void {
|
||||
|
||||
@@ -36,6 +36,7 @@ export type GitOpsAttentionReason =
|
||||
| 'deploy_failed'
|
||||
| 'placement_review_pending'
|
||||
| 'stateful_confirmation_required'
|
||||
| 'stateful_withdrawal_blocked'
|
||||
| 'rollout_authorization_pending'
|
||||
| 'rollout_authorization_stale'
|
||||
| 'preflight_blocked'
|
||||
@@ -72,6 +73,7 @@ export const ATTENTION_TONE: Readonly<Record<GitOpsAttentionReason, 'failure' |
|
||||
deploy_failed: 'failure',
|
||||
placement_review_pending: 'pending',
|
||||
stateful_confirmation_required: 'pending',
|
||||
stateful_withdrawal_blocked: 'pending',
|
||||
rollout_authorization_pending: 'pending',
|
||||
rollout_authorization_stale: 'pending',
|
||||
preflight_blocked: 'pending',
|
||||
@@ -111,7 +113,14 @@ export function attentionReasons(projection: GitOpsRevisionProjection): GitOpsAt
|
||||
reasons.add('source_unknown_outcome');
|
||||
break;
|
||||
case 'source_review_pending':
|
||||
reasons.add('source_review_pending');
|
||||
// One reason, not both: when an automatic acceptance refused for safety
|
||||
// the specific block is the actionable fact, and the queue must not spend
|
||||
// two codes on one waiting decision.
|
||||
reasons.add(
|
||||
source.reviewBlockReason === 'stateful_withdrawal'
|
||||
? 'stateful_withdrawal_blocked'
|
||||
: 'source_review_pending',
|
||||
);
|
||||
break;
|
||||
case 'source_conflict_blocker':
|
||||
reasons.add('source_conflict_blocker');
|
||||
|
||||
@@ -376,6 +376,7 @@ export function blankInlineApplication(id: string, blueprintId: number, at: numb
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -39,6 +39,7 @@ import type {
|
||||
RuntimeFacet,
|
||||
SourceFacet,
|
||||
SourceIdentityFields,
|
||||
SourceReviewBlockReason,
|
||||
} from './types';
|
||||
|
||||
/**
|
||||
@@ -249,6 +250,18 @@ function targetObservationMatchesExpected(target: GitOpsTargetProjection): boole
|
||||
return expectedSetAgreesWithObservation(expected.artifactSetId, target.observedArtifactIdentity, expected.identity);
|
||||
}
|
||||
|
||||
/**
|
||||
* The persisted block reason, narrowed to the codes this build knows.
|
||||
*
|
||||
* A value from a newer writer is dropped rather than passed through: the
|
||||
* projection feeds a closed classifier, and an unknown code would have to be
|
||||
* treated as a code by every consumer anyway. Dropping degrades to the plain
|
||||
* review state, which is still truthful about what is happening.
|
||||
*/
|
||||
function normalizeSourceReviewBlockReason(value: string | null): SourceReviewBlockReason | null {
|
||||
return value === 'stateful_withdrawal' ? value : null;
|
||||
}
|
||||
|
||||
function deriveSource(app: GitOpsApplicationRow, limitations: GitOpsLimitation[]): SourceFacet {
|
||||
if (app.target_mode === 'inline_blueprint') return { status: 'not_applicable' };
|
||||
const identity = sourceIdentity(app, limitations);
|
||||
@@ -329,7 +342,13 @@ function deriveSource(app: GitOpsApplicationRow, limitations: GitOpsLimitation[]
|
||||
return { ...identity, status: 'source_reconcile_required' };
|
||||
}
|
||||
if (app.candidate_plan_blocked === 1) return { ...identity, status: 'source_conflict_blocker' };
|
||||
if (app.review_required === 1) return { ...identity, status: 'source_review_pending' };
|
||||
if (app.review_required === 1) {
|
||||
return {
|
||||
...identity,
|
||||
status: 'source_review_pending',
|
||||
reviewBlockReason: normalizeSourceReviewBlockReason(app.review_block_reason),
|
||||
};
|
||||
}
|
||||
return { ...identity, status: 'candidate_ready' };
|
||||
}
|
||||
if (app.accepted_generation_id) {
|
||||
|
||||
@@ -141,6 +141,7 @@ export function buildDirectApplicationRow(args: {
|
||||
accepted_generation_id: null,
|
||||
candidate_plan_blocked: 0,
|
||||
review_required: 0,
|
||||
review_block_reason: null,
|
||||
artifact_set_id: null,
|
||||
latest_artifact_set_id: null,
|
||||
intent_revision_id: null,
|
||||
|
||||
@@ -90,6 +90,7 @@ export type GitOpsHistoryStage =
|
||||
| 'source_reconcile_started'
|
||||
| 'source_reconcile_settled'
|
||||
| 'source_retry_scheduled'
|
||||
| 'source_review_blocked'
|
||||
| 'source_suspended'
|
||||
| 'source_unsuspended'
|
||||
| 'target_applied'
|
||||
|
||||
@@ -75,6 +75,7 @@ CREATE TABLE IF NOT EXISTS gitops_applications (
|
||||
accepted_generation_id TEXT NULL,
|
||||
candidate_plan_blocked INTEGER NOT NULL DEFAULT 0,
|
||||
review_required INTEGER NOT NULL DEFAULT 0,
|
||||
review_block_reason TEXT NULL,
|
||||
artifact_set_id TEXT NULL,
|
||||
latest_artifact_set_id TEXT NULL,
|
||||
intent_revision_id TEXT NULL,
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
/**
|
||||
* Stateful-withdrawal guard for automated GitOps acceptance.
|
||||
*
|
||||
* The automated acceptance paths (the source controller's poll acceptance and
|
||||
* the webhook auto-apply for automatic policies) have no human in the loop, so
|
||||
* before a staged candidate is accepted they must prove it does not withdraw
|
||||
* or rename a stateful service relative to the generation in force. A service
|
||||
* whose name leaves the stateful set (removed, renamed, or stripped of its
|
||||
* volumes) is a withdrawal: applying it can destroy or orphan the data that
|
||||
* service owned.
|
||||
*
|
||||
* The check reads managed evidence only: the candidate's staged compose files
|
||||
* and the applied copy (falling back to the staged copy) of the generation in
|
||||
* force. Unreadable evidence is a hold, never an acceptance, and callers
|
||||
* record the hold as a review block so the projection names the block instead
|
||||
* of reading as an unexplained review.
|
||||
*
|
||||
* With no accepted generation there is no managed baseline to compare
|
||||
* against, so the check passes: the first acceptance defines the baseline, and
|
||||
* protecting it is the operator's choice of source policy or of accepting
|
||||
* under review.
|
||||
*/
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { BlueprintAnalyzer } from '../BlueprintAnalyzer';
|
||||
import { gitSourceLocalComposeFiles } from '../../utils/gitComposeFiles';
|
||||
import { loadDotEnv } from '../ImageUpdateService';
|
||||
import { sanitizeForLog } from '../../utils/safeLog';
|
||||
import { stackManagedRoot } from './directApplication';
|
||||
import { GitOpsStore } from './store';
|
||||
import { GitOpsTransitions, type EventEnvelope } from './transitions';
|
||||
import type { GitOpsApplicationRow, GitOpsGenerationRow } from './types';
|
||||
|
||||
export type StatefulWithdrawalCheck =
|
||||
| { status: 'clear' }
|
||||
| { status: 'withdrawn'; services: string[] }
|
||||
| { status: 'unreadable'; reason: string };
|
||||
|
||||
/**
|
||||
* Read a generation's staged compose files off disk: the paths come from the
|
||||
* application row, mapped the same way staging laid them out, with the
|
||||
* generation's staged .env merged under process.env exactly as the update
|
||||
* scanner resolves them. Returns null when the evidence cannot be read
|
||||
* (missing staging directory, unreadable compose file, a .env that exists but
|
||||
* cannot be read; an absent .env is normal and interpolates compose-only):
|
||||
* callers then hold rather than proceed on missing evidence.
|
||||
*/
|
||||
export function readStagedGeneration(
|
||||
stackName: string,
|
||||
app: GitOpsApplicationRow,
|
||||
generation: GitOpsGenerationRow,
|
||||
): { contents: string[]; mergedEnv: Record<string, string> } | null {
|
||||
try {
|
||||
// The directory name arrives from a generation row, so it is contained
|
||||
// under the managed root before any file inside is read.
|
||||
const managedRoot = path.resolve(stackManagedRoot(stackName));
|
||||
const candidateDir = path.resolve(managedRoot, generation.candidate_dir);
|
||||
if (!candidateDir.startsWith(managedRoot + path.sep)) return null;
|
||||
const composePaths = parseComposePaths(app);
|
||||
if (composePaths === null || composePaths.length === 0) return null;
|
||||
const contents: string[] = [];
|
||||
for (const local of gitSourceLocalComposeFiles(composePaths)) {
|
||||
// Inline containment at the read sink: the analyzer credits the
|
||||
// guard only where it can see it at the call, so each read is
|
||||
// resolved and checked right here rather than through a helper.
|
||||
const abs = path.resolve(candidateDir, local.replace(/\\/g, '/'));
|
||||
if (!abs.startsWith(candidateDir + path.sep)) return null;
|
||||
contents.push(fs.readFileSync(abs, 'utf8'));
|
||||
}
|
||||
const envVars = readStagedEnv(candidateDir, stackName, generation.id);
|
||||
if (envVars === null) return null;
|
||||
const merged: Record<string, string> = { ...envVars };
|
||||
for (const [k, v] of Object.entries(process.env)) {
|
||||
if (v !== undefined) merged[k] = v;
|
||||
}
|
||||
return { contents, mergedEnv: merged };
|
||||
} catch (e) {
|
||||
console.warn(
|
||||
`[GitOps] staged candidate unreadable for ${sanitizeForLog(stackName)} (generation ${sanitizeForLog(generation.id)}):`,
|
||||
e instanceof Error ? e.message : String(e),
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The staged .env, or null when a present-but-unreadable one must hold the
|
||||
* caller. A missing .env is normal (the staging step only writes one when
|
||||
* sync_env produced it), but a present-yet-unreadable one would silently
|
||||
* narrow the interpolation inputs, so it is surfaced rather than skipped.
|
||||
*/
|
||||
function readStagedEnv(
|
||||
candidateDir: string,
|
||||
stackName: string,
|
||||
generationId: string,
|
||||
): Record<string, string> | null {
|
||||
try {
|
||||
// Same inline containment the compose reads carry, so the analyzer
|
||||
// sees the sink guarded in this function too.
|
||||
const envPath = path.resolve(candidateDir, '.env');
|
||||
if (!envPath.startsWith(path.resolve(candidateDir) + path.sep)) return null;
|
||||
return loadDotEnv(fs.readFileSync(envPath, 'utf8'));
|
||||
} catch (e) {
|
||||
const missing = (e as NodeJS.ErrnoException).code === 'ENOENT';
|
||||
if (missing) return {};
|
||||
console.warn(
|
||||
`[GitOps] staged .env unreadable for ${sanitizeForLog(stackName)} (generation ${sanitizeForLog(generationId)}); holding candidate:`,
|
||||
e instanceof Error ? e.message : String(e),
|
||||
);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stateful services the candidate would withdraw or rename relative to the
|
||||
* generation in force. `clear` also covers "nothing stateful ran before";
|
||||
* `unreadable` means the comparison could not be made from evidence, and the
|
||||
* caller must hold rather than accept.
|
||||
*/
|
||||
export function checkStatefulWithdrawal(
|
||||
stackName: string,
|
||||
app: GitOpsApplicationRow,
|
||||
generation: GitOpsGenerationRow,
|
||||
): StatefulWithdrawalCheck {
|
||||
const staged = readStagedGeneration(stackName, app, generation);
|
||||
if (!staged) {
|
||||
return { status: 'unreadable', reason: 'the staged candidate could not be read' };
|
||||
}
|
||||
const next = statefulNamesAcrossFiles(staged.contents);
|
||||
if (next === null) {
|
||||
return { status: 'unreadable', reason: 'the staged candidate compose does not parse' };
|
||||
}
|
||||
if (!app.accepted_generation_id) return { status: 'clear' };
|
||||
const previous = GitOpsStore.getInstance().getGeneration(app.accepted_generation_id);
|
||||
if (!previous || previous.application_id !== app.id) {
|
||||
return { status: 'unreadable', reason: 'the generation in force is missing or belongs to another application' };
|
||||
}
|
||||
const previousContents = readAppliedGenerationContents(stackName, app, previous);
|
||||
if (previousContents === null) {
|
||||
return { status: 'unreadable', reason: 'the generation in force could not be read' };
|
||||
}
|
||||
const previousNames = statefulNamesAcrossFiles(previousContents.contents);
|
||||
if (previousNames === null) {
|
||||
return { status: 'unreadable', reason: 'the generation in force compose does not parse' };
|
||||
}
|
||||
const withdrawn = [...previousNames].filter((name) => !next.has(name));
|
||||
return withdrawn.length > 0 ? { status: 'withdrawn', services: withdrawn } : { status: 'clear' };
|
||||
}
|
||||
|
||||
/**
|
||||
* Record that the automatic path refused a candidate for safety. The candidate
|
||||
* stays staged for review; the write is what lets the projection name the
|
||||
* block. A failed write is logged, not thrown (the hold itself already
|
||||
* stands), and reported back so the caller can say the reason was not
|
||||
* recorded instead of implying the attention queue will show it.
|
||||
*/
|
||||
export function holdForStatefulReview(
|
||||
app: GitOpsApplicationRow,
|
||||
generation: GitOpsGenerationRow,
|
||||
envelope: EventEnvelope,
|
||||
): boolean {
|
||||
try {
|
||||
GitOpsTransitions.getInstance().sourceReviewBlocked({
|
||||
applicationId: app.id,
|
||||
generationId: generation.id,
|
||||
reason: 'stateful_withdrawal',
|
||||
envelope,
|
||||
});
|
||||
return true;
|
||||
} catch (e) {
|
||||
console.warn(
|
||||
`[GitOps] stateful review block not recorded for ${sanitizeForLog(app.id)}:`,
|
||||
e instanceof Error ? e.message : String(e),
|
||||
);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the compose content of the generation in force: the applied copy (what
|
||||
* was promoted) first, falling back to the staged candidate copy (accepted but
|
||||
* not yet promoted). Reports the first readable copy, or a reason when neither
|
||||
* is readable.
|
||||
*/
|
||||
function readAppliedGenerationContents(
|
||||
stackName: string,
|
||||
app: GitOpsApplicationRow,
|
||||
generation: GitOpsGenerationRow,
|
||||
): { contents: string[] } | null {
|
||||
const managedRoot = path.resolve(stackManagedRoot(stackName));
|
||||
const dirs = [generation.applied_dir, generation.candidate_dir].filter((dir) => dir.trim() !== '');
|
||||
const failures: string[] = [];
|
||||
for (const dir of dirs) {
|
||||
// The directory names arrive from generation rows, so they are
|
||||
// contained under the managed root before anything inside is read.
|
||||
const resolvedDir = path.resolve(managedRoot, dir);
|
||||
if (!resolvedDir.startsWith(managedRoot + path.sep)) {
|
||||
failures.push(`${dir}: the generation directory escapes the managed root`);
|
||||
continue;
|
||||
}
|
||||
const read = readComposeContents(resolvedDir, app);
|
||||
if (read.ok) return { contents: read.contents };
|
||||
failures.push(`${dir}: ${read.error}`);
|
||||
}
|
||||
console.warn(
|
||||
`[GitOps] generation-in-force content unreadable for ${sanitizeForLog(stackName)} (generation ${sanitizeForLog(generation.id)}): ${failures.map((failure) => sanitizeForLog(failure)).join('; ')}`,
|
||||
);
|
||||
return null;
|
||||
}
|
||||
|
||||
type ComposeContentsRead =
|
||||
| { ok: true; contents: string[] }
|
||||
| { ok: false; error: string };
|
||||
|
||||
function readComposeContents(dir: string, app: GitOpsApplicationRow): ComposeContentsRead {
|
||||
try {
|
||||
const base = path.resolve(dir);
|
||||
const composePaths = parseComposePaths(app);
|
||||
if (composePaths === null) return { ok: false, error: 'the compose path list is not valid json' };
|
||||
if (composePaths.length === 0) return { ok: false, error: 'the application has no compose paths' };
|
||||
const contents: string[] = [];
|
||||
for (const local of gitSourceLocalComposeFiles(composePaths)) {
|
||||
// Inline containment at the read sink, as in readStagedGeneration.
|
||||
const abs = path.resolve(base, local.replace(/\\/g, '/'));
|
||||
if (!abs.startsWith(base + path.sep)) return { ok: false, error: 'a compose path escapes the generation directory' };
|
||||
contents.push(fs.readFileSync(abs, 'utf8'));
|
||||
}
|
||||
return { ok: true, contents };
|
||||
} catch (e) {
|
||||
return { ok: false, error: e instanceof Error ? e.message : String(e) };
|
||||
}
|
||||
}
|
||||
|
||||
function parseComposePaths(app: GitOpsApplicationRow): string[] | null {
|
||||
if (!app.compose_paths_json) return [];
|
||||
try {
|
||||
return JSON.parse(app.compose_paths_json) as string[];
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Names of services with a named volume or bind mount, across all files. Null when a file does not parse. */
|
||||
function statefulNamesAcrossFiles(contents: string[]): Set<string> | null {
|
||||
const names = new Set<string>();
|
||||
for (const content of contents) {
|
||||
const parsed = BlueprintAnalyzer.statefulServiceNames(content);
|
||||
if (parsed === null) return null;
|
||||
for (const name of parsed) names.add(name);
|
||||
}
|
||||
return names;
|
||||
}
|
||||
@@ -719,7 +719,7 @@ export class GitOpsStore {
|
||||
configured_repo_url, repo_identity_json, configured_ref, compose_paths_json,
|
||||
context_dir, sync_env, env_path, materialization_fingerprint, desired_commit_sha,
|
||||
fetched_commit_sha, fetched_resolved_ref_kind, candidate_generation_id, accepted_generation_id,
|
||||
candidate_plan_blocked, review_required, artifact_set_id, latest_artifact_set_id,
|
||||
candidate_plan_blocked, review_required, review_block_reason, artifact_set_id, latest_artifact_set_id,
|
||||
intent_revision_id, rollout_candidate_id, rollout_generation_id, source_acceptance_ref,
|
||||
placement_approval_ref, rollout_authorization_ref, legacy_combined_approval_ref,
|
||||
preflight_fingerprint, latest_preflight_evidence_json, latest_operation_id, active_operation_id, active_operation_stage,
|
||||
@@ -729,14 +729,14 @@ export class GitOpsStore {
|
||||
recovery_ref, recovery_phase, interruption_stage, interruption_at,
|
||||
interruption_operation_id, interruption_generation_id, evidence_fresh_at,
|
||||
evidence_limitations_json, created_at, updated_at
|
||||
) VALUES (${Array(62).fill('?').join(', ')})`,
|
||||
) VALUES (${Array(63).fill('?').join(', ')})`,
|
||||
).run(
|
||||
row.id, row.lifecycle_key, row.lifecycle_status, row.target_mode, row.stack_name,
|
||||
row.configured_source_stack_name, row.blueprint_id,
|
||||
row.configured_repo_url, row.repo_identity_json, row.configured_ref, row.compose_paths_json,
|
||||
row.context_dir, row.sync_env, row.env_path, row.materialization_fingerprint, row.desired_commit_sha,
|
||||
row.fetched_commit_sha, row.fetched_resolved_ref_kind, row.candidate_generation_id, row.accepted_generation_id,
|
||||
row.candidate_plan_blocked, row.review_required, row.artifact_set_id, row.latest_artifact_set_id,
|
||||
row.candidate_plan_blocked, row.review_required, row.review_block_reason, row.artifact_set_id, row.latest_artifact_set_id,
|
||||
row.intent_revision_id, row.rollout_candidate_id, row.rollout_generation_id, row.source_acceptance_ref,
|
||||
row.placement_approval_ref, row.rollout_authorization_ref, row.legacy_combined_approval_ref,
|
||||
row.preflight_fingerprint, row.latest_preflight_evidence_json, row.latest_operation_id, row.active_operation_id, row.active_operation_stage,
|
||||
|
||||
@@ -29,6 +29,7 @@ import type {
|
||||
GitOpsRolloutGenerationRow,
|
||||
GitOpsTargetCurrentRow,
|
||||
SourcePolicy,
|
||||
SourceReviewBlockReason,
|
||||
} from './types';
|
||||
|
||||
export type EventEnvelope = {
|
||||
@@ -407,6 +408,7 @@ export class GitOpsTransitions {
|
||||
this.supersedeCandidate(app, generationId, envelope, extras);
|
||||
app.candidate_generation_id = generationId;
|
||||
app.candidate_plan_blocked = 1;
|
||||
app.review_block_reason = null;
|
||||
this.forEachLiveDirectTarget(app, (target) => {
|
||||
target.candidate_generation_id = generationId;
|
||||
this.store().upsertTarget(target);
|
||||
@@ -430,6 +432,7 @@ export class GitOpsTransitions {
|
||||
app.candidate_generation_id = null;
|
||||
app.candidate_plan_blocked = 0;
|
||||
app.review_required = 0;
|
||||
app.review_block_reason = null;
|
||||
this.forEachLiveDirectTarget(app, (target) => {
|
||||
target.candidate_generation_id = null;
|
||||
this.store().upsertTarget(target);
|
||||
@@ -482,6 +485,7 @@ export class GitOpsTransitions {
|
||||
app.candidate_generation_id = null;
|
||||
app.candidate_plan_blocked = 0;
|
||||
app.review_required = 0;
|
||||
app.review_block_reason = null;
|
||||
this.clearAppFailure(app, ['fetch', 'validation']);
|
||||
this.forEachLiveDirectTarget(app, (target) => {
|
||||
target.candidate_generation_id = null;
|
||||
@@ -506,6 +510,7 @@ export class GitOpsTransitions {
|
||||
app.candidate_generation_id = generationId;
|
||||
app.candidate_plan_blocked = 0;
|
||||
app.review_required = reviewRequired ? 1 : 0;
|
||||
app.review_block_reason = null;
|
||||
this.clearAppFailure(app, ['validation']);
|
||||
this.forEachLiveDirectTarget(app, (target) => {
|
||||
target.candidate_generation_id = generationId;
|
||||
@@ -514,6 +519,36 @@ export class GitOpsTransitions {
|
||||
}, { generationId });
|
||||
}
|
||||
|
||||
/**
|
||||
* An automatic acceptance refused for safety and fell back to review.
|
||||
*
|
||||
* The candidate stays staged exactly as `candidateReady` left it; this
|
||||
* records that the source policy's automatic path declined it, and why, so
|
||||
* the projection names the block instead of reporting an unexplained review
|
||||
* (or, worse, no state at all). Operator acceptance, dismissal, or a newer
|
||||
* candidate clears it.
|
||||
*/
|
||||
sourceReviewBlocked(args: {
|
||||
applicationId: string;
|
||||
generationId: string;
|
||||
reason: SourceReviewBlockReason;
|
||||
envelope: EventEnvelope;
|
||||
}): TransitionResult {
|
||||
return this.mutateApp(args.applicationId, args.envelope, 'source_review_blocked', 'committed', (app) => {
|
||||
if (app.candidate_generation_id !== args.generationId) {
|
||||
throw new GitOpsTransitionError('blocked generation is not the current candidate');
|
||||
}
|
||||
if (app.candidate_plan_blocked === 1) {
|
||||
throw new GitOpsTransitionError('candidate is blocked by its plan, not by acceptance');
|
||||
}
|
||||
if (app.active_operation_stage === 'apply_started') {
|
||||
throw new GitOpsTransitionError('cannot hold acceptance while an apply is in flight');
|
||||
}
|
||||
app.review_required = 1;
|
||||
app.review_block_reason = args.reason;
|
||||
}, { generationId: args.generationId });
|
||||
}
|
||||
|
||||
applyStarted(applicationId: string, generationId: string, envelope: EventEnvelope): TransitionResult {
|
||||
return this.mutateApp(applicationId, envelope, 'apply_started', 'committed', (app) => {
|
||||
if (app.suspended_at) throw new GitOpsTransitionError('source is suspended');
|
||||
@@ -702,6 +737,7 @@ export class GitOpsTransitions {
|
||||
app.candidate_generation_id = args.generation.id;
|
||||
app.candidate_plan_blocked = 0;
|
||||
app.review_required = 0;
|
||||
app.review_block_reason = null;
|
||||
app.latest_operation_id = args.envelope.operationId;
|
||||
app.updated_at = args.envelope.at;
|
||||
this.writeApplication(app);
|
||||
@@ -1113,6 +1149,12 @@ export class GitOpsTransitions {
|
||||
throw new GitOpsTransitionError('cannot change the source policy while an operation is in flight');
|
||||
}
|
||||
app.source_policy = sourcePolicy;
|
||||
// The block reason explains why the automatic path refused; once the
|
||||
// policy is no longer automatic, the generic review state is the
|
||||
// truthful one. Review_required stays for the derivation to interpret.
|
||||
if (sourcePolicy !== 'automatic') {
|
||||
app.review_block_reason = null;
|
||||
}
|
||||
if (sourcePolicy === 'manual') {
|
||||
app.next_poll_at = null;
|
||||
app.retry_at = null;
|
||||
@@ -2976,6 +3018,7 @@ export class GitOpsTransitions {
|
||||
app.candidate_generation_id = null;
|
||||
app.candidate_plan_blocked = 0;
|
||||
app.review_required = 0;
|
||||
app.review_block_reason = null;
|
||||
if (args.activateCreating && app.lifecycle_status === 'creating') {
|
||||
app.lifecycle_status = 'active';
|
||||
}
|
||||
@@ -3514,7 +3557,7 @@ export class GitOpsTransitions {
|
||||
compose_paths_json=?, context_dir=?, sync_env=?, env_path=?,
|
||||
materialization_fingerprint=?, desired_commit_sha=?, fetched_commit_sha=?,
|
||||
fetched_resolved_ref_kind=?, candidate_generation_id=?, accepted_generation_id=?, candidate_plan_blocked=?,
|
||||
review_required=?, artifact_set_id=?, latest_artifact_set_id=?,
|
||||
review_required=?, review_block_reason=?, artifact_set_id=?, latest_artifact_set_id=?,
|
||||
intent_revision_id=?, rollout_candidate_id=?, rollout_generation_id=?,
|
||||
source_acceptance_ref=?, placement_approval_ref=?, rollout_authorization_ref=?,
|
||||
legacy_combined_approval_ref=?, preflight_fingerprint=?, latest_preflight_evidence_json=?,
|
||||
@@ -3532,7 +3575,7 @@ export class GitOpsTransitions {
|
||||
app.compose_paths_json, app.context_dir, app.sync_env, app.env_path,
|
||||
app.materialization_fingerprint, app.desired_commit_sha, app.fetched_commit_sha,
|
||||
app.fetched_resolved_ref_kind, app.candidate_generation_id, app.accepted_generation_id, app.candidate_plan_blocked,
|
||||
app.review_required, app.artifact_set_id, app.latest_artifact_set_id,
|
||||
app.review_required, app.review_block_reason, app.artifact_set_id, app.latest_artifact_set_id,
|
||||
app.intent_revision_id, app.rollout_candidate_id, app.rollout_generation_id,
|
||||
app.source_acceptance_ref, app.placement_approval_ref, app.rollout_authorization_ref,
|
||||
app.legacy_combined_approval_ref, app.preflight_fingerprint, app.latest_preflight_evidence_json,
|
||||
|
||||
@@ -65,6 +65,11 @@ export type GitOpsApplicationRow = {
|
||||
accepted_generation_id: string | null;
|
||||
candidate_plan_blocked: number;
|
||||
review_required: number;
|
||||
/**
|
||||
* Why an automatic acceptance refused and fell back to review. Null under
|
||||
* the ordinary review policy, which was never going to accept on its own.
|
||||
*/
|
||||
review_block_reason: SourceReviewBlockReason | null;
|
||||
artifact_set_id: string | null;
|
||||
latest_artifact_set_id: string | null;
|
||||
intent_revision_id: string | null;
|
||||
@@ -445,6 +450,12 @@ export type SourceIdentityFields = {
|
||||
acceptedGenerationId: string | null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Why the source policy's automatic path refused a candidate and left it for
|
||||
* review. A closed set: the projection carries the code, the surface names it.
|
||||
*/
|
||||
export type SourceReviewBlockReason = 'stateful_withdrawal';
|
||||
|
||||
export type SourceFacet =
|
||||
| { status: 'not_applicable' }
|
||||
| (SourceIdentityFields & {
|
||||
@@ -453,10 +464,14 @@ export type SourceFacet =
|
||||
| 'checking_fetching'
|
||||
| 'application_generation_accepted'
|
||||
| 'candidate_ready'
|
||||
| 'source_review_pending'
|
||||
| 'source_conflict_blocker'
|
||||
| 'source_reconcile_required';
|
||||
})
|
||||
| (SourceIdentityFields & {
|
||||
status: 'source_review_pending';
|
||||
/** Set when automatic acceptance refused for safety, null for a review policy. */
|
||||
reviewBlockReason: SourceReviewBlockReason | null;
|
||||
})
|
||||
| (SourceIdentityFields & { status: 'source_superseded'; supersededGenerationId: string })
|
||||
| (SourceIdentityFields & { status: 'applying'; activeOperationId: string; activeGenerationId: string })
|
||||
| (SourceIdentityFields & { status: 'source_retry_scheduled'; retryAt: number; retryCount: number })
|
||||
|
||||
@@ -143,6 +143,8 @@ You can always override on the spot: when you click **Apply** in the change plan
|
||||
|
||||
**Review only** maps to review policy. **Auto-write files** and **Auto-deploy** map to automatic policy. Suspend is the operator stop; there is no fourth Manual radio on the sheet.
|
||||
|
||||
Automatic modes carry one built-in safety check: when the incoming commit withdraws or renames a service that persists data (named volumes or bind mounts), or when Sencho cannot prove it does not, the change is held for review instead of being accepted or applied, and the stack shows a **stateful change blocked** attention state. The hold clears when you accept the change explicitly, dismiss it, or a newer commit is staged. Nothing is written or redeployed while it is held, so withdrawing a stateful service is always a deliberate choice.
|
||||
|
||||
Git Sources apply to stacks you link from the editor. Blueprint deployments do not evaluate a Git source.
|
||||
|
||||
## Polling
|
||||
|
||||
@@ -37,6 +37,7 @@ export const ATTENTION_LABEL: Partial<Record<GitOpsAttentionReason, PortfolioLab
|
||||
source_suspended: { label: 'source suspended', tone: 'neutral', line: 'Reconciliation is suspended for this source.' },
|
||||
placement_review_pending: { label: 'placement review', tone: 'warning', line: 'A target placement decision is waiting for review.' },
|
||||
stateful_confirmation_required: { label: 'stateful confirmation', tone: 'warning', line: 'Stateful changes need explicit confirmation before proceeding.' },
|
||||
stateful_withdrawal_blocked: { label: 'stateful change blocked', tone: 'warning', line: 'An automatic acceptance was held for review because the candidate removes or renames a stateful service, or that cannot be ruled out.' },
|
||||
rollout_authorization_pending: { label: 'rollout authorization', tone: 'warning', line: 'The rollout is waiting for authorization.' },
|
||||
rollout_authorization_stale: { label: 'authorization stale', tone: 'warning', line: 'The rollout authorization no longer matches the current intent; re-authorize to proceed.' },
|
||||
preflight_blocked: { label: 'preflight blocked', tone: 'warning', line: 'Preflight evidence blocks this rollout (for example an unready private-registry target).' },
|
||||
|
||||
Reference in New Issue
Block a user