mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-25 13:11:05 +00:00
fix(blueprints): retry a refused drift repair instead of stranding the row (#2028)
* fix(blueprints): retry a refused drift repair instead of stranding the row A drift repair that could not finish left the deployment row reading `correcting` forever, so nothing ever retried it. `drift_enforce_start` writes `correcting` before the repair runs, and a repair that then answered without ever reaching a deploy wrote no status of its own. That state is in-flight as far as the decision is concerned, so `computeDecision` skipped it and the preview projection rendered it as the informational `in_flight_correct`. No later tick re-checked the node: the workload kept running the moved image, one correction-failed alert fired, and then nothing, for as long as the drift lasted. Four ticks on a refused repair recorded one drift check, one repair attempt and one alert, and did nothing on the rest. A refusal, or a repair that throws, now settles the row back to `drifted` with the reason in `last_error`, so the next tick checks and retries it. A throw counts because resolving a remote node's platform label is a network call made before any deploy, so it can reject with nothing written and used to escape to the per-blueprint catch with the row still `correcting`. The row deliberately does not become `failed` or `pending`. Both are retry buckets for an ordinary deploy, so the next tick would repair by tag and adopt whatever the tag points at now, which is the one thing a digest-pinned repair exists to prevent. The failure alert fires when the row changed across the attempt, compared against a snapshot read before the repair ran. It cannot be decided from the row afterwards: a repair that reaches a deploy writes its own `last_error` and returns the very same string, so asking the row afterwards finds the failure already sitting there and mistakes a new incident for a repeat, dropping the first failure of every kind silently. Losing the deploy lock is a deferral rather than a failure, so the row still leaves `correcting` with a reason and the operator is not paged about a race they cannot act on. A converged row now forgets its last failure, as the deploy acknowledgement already did, so a failure that returns after a repair counts as a new incident. computeDecision also treats a `drifted` row at the current revision as a drift-check target. The projection already reached it through its own status fallback with the same action, severity and confirmable membership, so nothing an operator sees changes; it is here so the decision a tick acts on is not two files away in the preview projection. Verified: 10333 backend tests pass with no failures, typecheck and lint clean, and an e2e case drives drift, repair and reconvergence on a real Inline Blueprint, asserting the row returns to Active with no drift summary and the workload runs the approved digest again. * fix(blueprints): correct the drift repair settle found in review Four corrections to the refused-repair settle, all found by reviewing the change rather than by a failing test. The deferral note was the serious one. Losing the deploy lock is silent, and it wrote a note about the race into `last_error`, which is the replay key. One lost-lock tick between two identical failures therefore displaced the failure being replayed, and the next tick reported that same known failure again as though it were new. A deferral now leaves `last_error` alone: the row keeps saying what it last actually failed with, which is worth more to an operator than a note about a race that is already over. An alert now quotes the row rather than the outcome where the outcome carries only a shorthand. The name-conflict paths return the bare word `name_conflict` and then park the row for good, so that one alert is the only signal the operator will ever get, and it has to name the directory in the way. The row's own text is only preferred when the repair actually wrote a status: when the row is still `correcting`, nothing was written and the row is still carrying the previous failure, which must not be reported as the current one. A target that converges while its repair is in flight is no longer reported. The drift the failure belonged to no longer exists, so paging the operator about it is stale news. A repair that throws is now logged with its stack as well as alerted. This boundary cannot tell an operational failure from a bug in the repair and both arrive identically; an alert alone would repeat a bug every tick and never reach a developer, and re-throwing would strand the row again. Separately, the drift observation restamped `last_drift_at` every time it re-observed the same drift, so a drift outliving one tick reported itself as just found, forever. It is stamped only when the row was not already drifted. Both observation sites did this, the Inline one and the Git-managed one. Coverage added for the Git-managed repair path, which is the third dispatch and had no case, for the timestamp, and for the three alert rules above. 19 cases now, 5 of which fail with the whole change reverted.
This commit is contained in:
Anso
@@ -0,0 +1,589 @@
|
||||
/**
|
||||
* A drifted Inline deployment must be re-examined until it converges.
|
||||
*
|
||||
* The reconciler writes `correcting` before it attempts a drift repair, so every
|
||||
* repair path that refuses without writing a terminal status used to strand the
|
||||
* row there: `computeDecision` skips in-flight states and the projection renders
|
||||
* them informational, so no later tick re-checked, no repair was retried, and a
|
||||
* workload running the wrong image sat unrepaired behind a single alert.
|
||||
*
|
||||
* These drive the real tick. What is under test is what the reconciler does with
|
||||
* a refused repair, so the repair preconditions are left real where they can be
|
||||
* (the digest pin builder genuinely refuses) and stubbed only where the refusal
|
||||
* is the thing under test (lock contention).
|
||||
*/
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { intentFingerprint, serializeApprovedBlast } from '../services/blueprintApproval';
|
||||
import type { Blueprint, Node } from '../services/DatabaseService';
|
||||
import type { ReconcileDecision } from '../services/BlueprintReconciler';
|
||||
|
||||
type ReconcilerWithCompute = {
|
||||
computeDecision: (blueprint: Blueprint, allNodes: Node[]) => ReconcileDecision;
|
||||
};
|
||||
|
||||
let tmpDir: string;
|
||||
let DatabaseService: typeof import('../services/DatabaseService').DatabaseService;
|
||||
let BlueprintService: typeof import('../services/BlueprintService').BlueprintService;
|
||||
let BlueprintReconciler: typeof import('../services/BlueprintReconciler').BlueprintReconciler;
|
||||
let NotificationService: typeof import('../services/NotificationService').NotificationService;
|
||||
let buildBlueprintPreview: typeof import('../services/blueprintPreviewProjection').buildBlueprintPreview;
|
||||
let counter = 0;
|
||||
|
||||
const DRIFT_REASON = 'runtime artifact identity differs from the expected artifact set';
|
||||
const PIN_REFUSAL = 'approved digest unavailable for digest repair';
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
({ DatabaseService } = await import('../services/DatabaseService'));
|
||||
({ BlueprintService } = await import('../services/BlueprintService'));
|
||||
({ BlueprintReconciler } = await import('../services/BlueprintReconciler'));
|
||||
({ NotificationService } = await import('../services/NotificationService'));
|
||||
({ buildBlueprintPreview } = await import('../services/blueprintPreviewProjection'));
|
||||
});
|
||||
|
||||
afterAll(() => cleanupTestDb(tmpDir));
|
||||
|
||||
beforeEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
db.prepare('DELETE FROM blueprint_deployments').run();
|
||||
db.prepare('DELETE FROM blueprints').run();
|
||||
db.prepare('DELETE FROM node_labels').run();
|
||||
db.prepare("DELETE FROM nodes WHERE is_default = 0").run();
|
||||
counter += 1;
|
||||
});
|
||||
|
||||
function seedNode(): Node {
|
||||
counter += 1;
|
||||
const nodeId = DatabaseService.getInstance().getDb().prepare(
|
||||
`INSERT INTO nodes (name, type, mode, compose_dir, is_default, status, created_at)
|
||||
VALUES (?, 'local', 'proxy', '/tmp/compose', 0, 'online', ?)`,
|
||||
).run(`retry-node-${counter}`, Date.now()).lastInsertRowid as number;
|
||||
return DatabaseService.getInstance().getNode(nodeId)!;
|
||||
}
|
||||
|
||||
function seedEnforceBlueprint(node: Node): Blueprint {
|
||||
counter += 1;
|
||||
return DatabaseService.getInstance().createBlueprint({
|
||||
name: `retry-bp-${counter}`,
|
||||
description: null,
|
||||
compose_content: 'services:\n web:\n image: nginx:alpine\n',
|
||||
selector: { type: 'nodes', ids: [node.id] },
|
||||
drift_mode: 'enforce',
|
||||
classification: 'stateless',
|
||||
classification_reasons: [],
|
||||
enabled: true,
|
||||
created_by: 'tester',
|
||||
});
|
||||
}
|
||||
|
||||
/** Approve the place outcome the operator confirmed, as Confirm persists it. */
|
||||
function approvePlace(blueprint: Blueprint, nodeId: number): void {
|
||||
DatabaseService.getInstance().setBlueprintApproval(blueprint.id, {
|
||||
intentFingerprint: intentFingerprint(blueprint),
|
||||
blastJson: serializeApprovedBlast([{ nodeId, outcome: 'place' }]),
|
||||
approvedBy: 'tester',
|
||||
});
|
||||
}
|
||||
|
||||
function writeRow(blueprint: Blueprint, nodeId: number, status: string, extra: Record<string, unknown> = {}): void {
|
||||
DatabaseService.getInstance().upsertDeployment({
|
||||
blueprint_id: blueprint.id,
|
||||
node_id: nodeId,
|
||||
status,
|
||||
applied_revision: blueprint.revision,
|
||||
last_deployed_at: Date.now(),
|
||||
drift_summary: DRIFT_REASON,
|
||||
last_drift_at: Date.now(),
|
||||
...extra,
|
||||
} as Parameters<typeof DatabaseService.prototype.upsertDeployment>[0]);
|
||||
}
|
||||
|
||||
/** The drift a digest mismatch produces, counted so a retry is observable. */
|
||||
function countDriftChecks(): { calls: () => number } {
|
||||
let calls = 0;
|
||||
vi.spyOn(BlueprintService.getInstance(), 'checkForDrift').mockImplementation(async () => {
|
||||
calls += 1;
|
||||
return { kind: 'drifted', reason: DRIFT_REASON, cause: 'digest' };
|
||||
});
|
||||
return { calls: () => calls };
|
||||
}
|
||||
|
||||
function countAlerts(): { categories: () => string[] } {
|
||||
const categories: string[] = [];
|
||||
vi.spyOn(NotificationService.getInstance(), 'dispatchAlert').mockImplementation(async (_level, category) => {
|
||||
categories.push(category);
|
||||
return { persisted: true };
|
||||
});
|
||||
return { categories: () => categories };
|
||||
}
|
||||
|
||||
describe('a refused drift repair does not strand the deployment', () => {
|
||||
it('keeps the row drifted and retries on the next tick when the approved digest is unavailable', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
const checks = countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// The real digest repair, with no comparable approved artifact set behind
|
||||
// it: it refuses before it can touch the node, which is the shape of every
|
||||
// refusal that writes no status of its own.
|
||||
const repair = vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair');
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const afterFirst = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
|
||||
expect(afterFirst?.status).toBe('drifted');
|
||||
expect(afterFirst?.drift_summary).toBe(DRIFT_REASON);
|
||||
expect(afterFirst?.last_error).toBeTruthy();
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
expect(repair).toHaveBeenCalledTimes(1);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
// The next tick re-checks and re-attempts rather than skipping a row it
|
||||
// believes is still in flight.
|
||||
expect(checks.calls()).toBe(2);
|
||||
expect(repair).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it('returns the row to drifted when another operation holds the deploy lock', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
const checks = countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// Lock contention answers pending without writing anything: the attempt was
|
||||
// never made, so the drift is still owed.
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockResolvedValue({ status: 'pending' } as never);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const afterFirst = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
expect(afterFirst?.status).toBe('drifted');
|
||||
// Losing the lock for a moment is a deferral, not a failure, and it carries
|
||||
// no message of its own. The row is left saying whatever it last actually
|
||||
// failed with, because that text is the replay key, and the operator is not
|
||||
// paged about a race they cannot act on.
|
||||
expect(afterFirst?.last_error, 'a deferral does not overwrite what the row last failed with').toBeFalsy();
|
||||
expect(alerts.categories()).toEqual([]);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
expect(checks.calls()).toBe(2);
|
||||
expect(alerts.categories(), 'still silent on the retry').toEqual([]);
|
||||
});
|
||||
|
||||
it('settles a container-level refusal, which takes the other repair path', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
const alerts = countAlerts();
|
||||
// A stopped container is the other drift cause, and it repairs through
|
||||
// deployToNode rather than the digest path, so a refusal has to settle the
|
||||
// same way whichever method ran.
|
||||
const message = 'service "web" exited with code 1';
|
||||
const deploy = vi.spyOn(BlueprintService.getInstance(), 'deployToNode')
|
||||
.mockResolvedValue({ status: 'failed', error: message } as never);
|
||||
let checks = 0;
|
||||
vi.spyOn(BlueprintService.getInstance(), 'checkForDrift').mockImplementation(async () => {
|
||||
checks += 1;
|
||||
return { kind: 'drifted', reason: 'no containers running for this blueprint', cause: 'container' };
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const afterFirst = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
|
||||
expect(deploy, 'a container-level drift repairs through the deploy path').toHaveBeenCalledTimes(1);
|
||||
expect(afterFirst?.status).toBe('drifted');
|
||||
expect(afterFirst?.last_error).toBe(message);
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
expect(checks, 'the retry re-checks the node').toBe(2);
|
||||
expect(deploy, 'and re-attempts the repair').toHaveBeenCalledTimes(2);
|
||||
expect(alerts.categories(), 'and does not page about the same refusal twice').toEqual([
|
||||
'blueprint_drift_correction_failed',
|
||||
]);
|
||||
});
|
||||
|
||||
it('alerts on the first failure a deploy recorded itself, because that failure is new', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
const checks = countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// What a repair that reaches a deploy actually does when compose fails: it
|
||||
// writes its own terminal status and its own last_error, and it returns the
|
||||
// very same string. Reading the row afterwards would therefore find the
|
||||
// failure already sitting there and mistake a new incident for a repeat.
|
||||
const message = 'service "web" failed to start: port 80 is already allocated';
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockImplementation(async () => {
|
||||
writeRow(blueprint, node.id, 'failed', { drift_summary: null, last_error: message });
|
||||
return { status: 'failed', error: message } as never;
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status).toBe('failed');
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
|
||||
// The row is the deploy bucket's now, so the drift path is not re-entered.
|
||||
// One alert for the incident is the whole promise, not one per tick.
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
expect(checks.calls(), 'a failed row is retried as a deploy, not as a drift check').toBe(1);
|
||||
});
|
||||
|
||||
it('alerts once for a name conflict and does not re-repair the blocked row', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
const repair = vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockImplementation(async () => {
|
||||
writeRow(blueprint, node.id, 'name_conflict', {
|
||||
drift_summary: null,
|
||||
last_error: 'A stack named this already exists on this node and is not managed by Sencho.',
|
||||
});
|
||||
return { status: 'name_conflict', error: 'name_conflict' } as never;
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status).toBe('name_conflict');
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
// A name conflict is in-flight as far as the decision is concerned, so the
|
||||
// second tick never reaches the repair again and cannot alert again.
|
||||
expect(repair).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('leaves a name conflict the repair already recorded alone', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// A deploy that reaches the name check writes name_conflict itself, so the
|
||||
// row is already the blocker and must not be rewritten back to drifted.
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockImplementation(async () => {
|
||||
writeRow(blueprint, node.id, 'name_conflict', {
|
||||
drift_summary: null,
|
||||
last_error: 'A stack named this already exists on this node and is not managed by Sencho.',
|
||||
});
|
||||
return { status: 'name_conflict', error: 'name_conflict' } as never;
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status).toBe('name_conflict');
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
});
|
||||
|
||||
it('settles a repair that throws, which is a refusal that reached no deploy', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
const checks = countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// Resolving a remote node's platform label is a network call made before any
|
||||
// deploy, so a repair can throw with nothing written at all. Left unhandled
|
||||
// the throw escapes to the per-blueprint catch with the row still correcting.
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair')
|
||||
.mockRejectedValue(new Error('getaddrinfo ENOTFOUND leaf.example.test'));
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const afterFirst = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
|
||||
expect(afterFirst?.status).toBe('drifted');
|
||||
expect(afterFirst?.last_error).toMatch(/ENOTFOUND/);
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
expect(checks.calls(), 'the tick after a thrown repair still re-checks').toBe(2);
|
||||
expect(alerts.categories(), 'and the same throw is not reported twice').toEqual([
|
||||
'blueprint_drift_correction_failed',
|
||||
]);
|
||||
});
|
||||
|
||||
it('alerts again when the same failure returns after the drift converged', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
const alerts = countAlerts();
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair')
|
||||
.mockResolvedValue({ status: 'failed', error: PIN_REFUSAL } as never);
|
||||
const drift = vi.spyOn(BlueprintService.getInstance(), 'checkForDrift')
|
||||
.mockResolvedValue({ kind: 'drifted', reason: DRIFT_REASON, cause: 'digest' });
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
// The cause clears on its own, so the next check finds the target matching
|
||||
// and the row goes back to active. That has to forget the failure: it is
|
||||
// repaired, and a later failure of the same kind is a new incident.
|
||||
drift.mockResolvedValue({ kind: 'matched' });
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const converged = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
expect(converged?.status).toBe('active');
|
||||
expect(converged?.last_error, 'a converged row carries no failure').toBeNull();
|
||||
|
||||
// The same refusal happens again later on a fresh drift.
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
drift.mockResolvedValue({ kind: 'drifted', reason: DRIFT_REASON, cause: 'digest' });
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(alerts.categories()).toEqual([
|
||||
'blueprint_drift_correction_failed',
|
||||
'blueprint_drift_correction_failed',
|
||||
]);
|
||||
});
|
||||
|
||||
it('does not report an already-reported failure again just because a lock was lost in between', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
const repair = vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair')
|
||||
.mockResolvedValue({ status: 'failed', error: PIN_REFUSAL } as never);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
|
||||
// A concurrent operation takes the deploy lock for one tick.
|
||||
repair.mockResolvedValue({ status: 'pending' } as never);
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const deferred = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
expect(deferred?.status).toBe('drifted');
|
||||
expect(alerts.categories(), 'a lost lock race is not news').toEqual([
|
||||
'blueprint_drift_correction_failed',
|
||||
]);
|
||||
|
||||
// The same refusal comes back. It is the failure already reported, so the
|
||||
// note about the race must not have displaced it from the row.
|
||||
repair.mockResolvedValue({ status: 'failed', error: PIN_REFUSAL } as never);
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
});
|
||||
|
||||
it('names the directory in the alert when a name conflict blocks the repair for good', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const messages: string[] = [];
|
||||
vi.spyOn(NotificationService.getInstance(), 'dispatchAlert')
|
||||
.mockImplementation(async (_level, _category, message) => {
|
||||
messages.push(message);
|
||||
return { persisted: true };
|
||||
});
|
||||
// The real name-conflict path returns the bare shorthand but writes the
|
||||
// sentence onto the row, and the row then stays blocked for good.
|
||||
const sentence = 'A stack named "retry-bp" already exists on this node and is not managed by Sencho.';
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockImplementation(async () => {
|
||||
writeRow(blueprint, node.id, 'name_conflict', { drift_summary: null, last_error: sentence });
|
||||
return { status: 'name_conflict', error: 'name_conflict' } as never;
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(messages).toHaveLength(1);
|
||||
expect(messages[0], 'the one alert this operator gets has to be actionable').toContain(sentence);
|
||||
});
|
||||
|
||||
it('stays silent when the target converged while the repair was in flight', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// Someone else converged the target while this tick's repair was running, so
|
||||
// there is no drift left for this failure to be about.
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockImplementation(async () => {
|
||||
writeRow(blueprint, node.id, 'active', { drift_summary: null, last_drift_at: null });
|
||||
return { status: 'failed', error: 'service "web" failed to start' } as never;
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status).toBe('active');
|
||||
expect(alerts.categories(), 'a healthy target is not paged about a stale failure').toEqual([]);
|
||||
});
|
||||
|
||||
it('settles a refusal on a Git-managed blueprint, which repairs by reapplying the generation', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
DatabaseService.getInstance().updateBlueprintContentOrigin(blueprint.id, 'git', 'app-git-managed');
|
||||
const gitBlueprint = DatabaseService.getInstance().getBlueprint(blueprint.id)!;
|
||||
approvePlace(gitBlueprint, node.id);
|
||||
writeRow(gitBlueprint, node.id, 'drifted');
|
||||
const alerts = countAlerts();
|
||||
// Container drift, not a frozen-image disagreement, so the dispatch reaches
|
||||
// the Git-managed branch. Git-managed content has no authored compose to
|
||||
// deploy, so its repair is the authorized reapply, and it has to settle the
|
||||
// same way or a Git-managed target strands exactly like an Inline one.
|
||||
vi.spyOn(BlueprintService.getInstance(), 'checkForDrift').mockResolvedValue({
|
||||
kind: 'drifted', reason: 'no containers running for this blueprint', cause: 'container',
|
||||
});
|
||||
const reapply = vi.spyOn(BlueprintService.getInstance(), 'reapplyAuthorizedMaterialization')
|
||||
.mockResolvedValue({ status: 'failed', error: 'acknowledged generation missing for authorized reapply' } as never);
|
||||
const deploy = vi.spyOn(BlueprintService.getInstance(), 'deployToNode');
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(gitBlueprint.id);
|
||||
|
||||
expect(reapply, 'a Git-managed drift repairs by reapplying, not by deploying authored compose').toHaveBeenCalledTimes(1);
|
||||
expect(deploy).not.toHaveBeenCalled();
|
||||
const after = DatabaseService.getInstance().getDeployment(gitBlueprint.id, node.id);
|
||||
expect(after?.status).toBe('drifted');
|
||||
expect(after?.last_error).toMatch(/acknowledged generation missing/);
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
});
|
||||
|
||||
it('keeps the original drift timestamp when a later tick re-observes the same drift', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
countAlerts();
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair')
|
||||
.mockResolvedValue({ status: 'failed', error: PIN_REFUSAL } as never);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
const firstSeen = DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.last_drift_at;
|
||||
expect(firstSeen, 'the first observation stamps when the drift began').toBeTruthy();
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
// The drift episode began at the first observation. Restamping on every tick
|
||||
// made a drift that outlived one tick report itself as just found, forever,
|
||||
// which is the one thing the drift age on the row exists to say.
|
||||
const dep = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
expect(dep?.last_drift_at).toBe(firstSeen);
|
||||
});
|
||||
|
||||
it('alerts once for a failure that persists instead of once per tick', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockResolvedValue({
|
||||
status: 'failed',
|
||||
error: PIN_REFUSAL,
|
||||
} as never);
|
||||
|
||||
for (let tick = 0; tick < 3; tick++) {
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
}
|
||||
|
||||
// Retrying every tick is the point; paging the operator every tick is not.
|
||||
expect(alerts.categories()).toEqual(['blueprint_drift_correction_failed']);
|
||||
expect(DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status).toBe('drifted');
|
||||
});
|
||||
|
||||
it('alerts again when the failure changes, because a different refusal needs a different fix', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
const repair = vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair')
|
||||
.mockResolvedValue({ status: 'failed', error: PIN_REFUSAL } as never);
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
repair.mockResolvedValue({ status: 'failed', error: 'compose up failed' } as never);
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
expect(alerts.categories()).toEqual([
|
||||
'blueprint_drift_correction_failed',
|
||||
'blueprint_drift_correction_failed',
|
||||
]);
|
||||
});
|
||||
|
||||
it('leaves a successful repair to the deploy path that already wrote the row', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
approvePlace(blueprint, node.id);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
countDriftChecks();
|
||||
const alerts = countAlerts();
|
||||
// What a successful repair does: the deploy writes active and clears drift.
|
||||
vi.spyOn(BlueprintService.getInstance(), 'enforceDigestRepair').mockImplementation(async () => {
|
||||
writeRow(blueprint, node.id, 'active', {
|
||||
drift_summary: null,
|
||||
last_drift_at: null,
|
||||
last_error: null,
|
||||
});
|
||||
return { status: 'active' } as never;
|
||||
});
|
||||
|
||||
await BlueprintReconciler.getInstance().reconcileOne(blueprint.id);
|
||||
|
||||
const dep = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
expect(dep?.status).toBe('active');
|
||||
expect(dep?.drift_summary).toBeNull();
|
||||
expect(alerts.categories()).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('a drifted row is a drift-check target in its own right', () => {
|
||||
it('enters the drift-check bucket when its applied revision still matches', () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
|
||||
const decision = (BlueprintReconciler.getInstance() as unknown as ReconcilerWithCompute)
|
||||
.computeDecision(blueprint, DatabaseService.getInstance().getNodes());
|
||||
|
||||
expect(decision.check.map((n) => n.id)).toContain(node.id);
|
||||
expect(decision.deploy).toEqual([]);
|
||||
expect(decision.withdraw).toEqual([]);
|
||||
});
|
||||
|
||||
it('keeps an in-flight row out of every bucket', () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
writeRow(blueprint, node.id, 'correcting');
|
||||
|
||||
const decision = (BlueprintReconciler.getInstance() as unknown as ReconcilerWithCompute)
|
||||
.computeDecision(blueprint, DatabaseService.getInstance().getNodes());
|
||||
|
||||
expect(decision.check).toEqual([]);
|
||||
expect(decision.deploy).toEqual([]);
|
||||
expect(decision.withdraw).toEqual([]);
|
||||
});
|
||||
|
||||
it('projects one drift check for a drifted row, not one per path that mentions it', async () => {
|
||||
const node = seedNode();
|
||||
const blueprint = seedEnforceBlueprint(node);
|
||||
writeRow(blueprint, node.id, 'drifted');
|
||||
|
||||
const preview = await buildBlueprintPreview(blueprint.id);
|
||||
const forNode = (preview?.changes ?? []).filter((c) => c.nodeId === node.id);
|
||||
|
||||
expect(forNode).toHaveLength(1);
|
||||
expect(forNode[0]?.action).toBe('check_enforce');
|
||||
// The whole projected row is pinned, not just the action name. A drifted row
|
||||
// already reached the projection through the status fallback before the
|
||||
// decision bucket learned about it, so what must not change is the shape an
|
||||
// operator confirms against: one row, this severity, this kind, and the same
|
||||
// executor action the apply executes.
|
||||
expect(forNode[0]).toMatchObject({ severity: 'warning', kind: 'executor', detail: expect.any(String) });
|
||||
expect(preview?.executorActions).toEqual([{ nodeId: node.id, action: 'check_enforce' }]);
|
||||
expect(preview?.confirmableActions).toEqual([{ nodeId: node.id, action: 'check_enforce' }]);
|
||||
});
|
||||
});
|
||||
@@ -428,10 +428,17 @@ export class BlueprintReconciler {
|
||||
}
|
||||
}
|
||||
const reason = driftResult.reason;
|
||||
// The drift episode began when the row first read drifted, so a
|
||||
// tick that re-observes the same drift must not restamp it.
|
||||
// Restamping made a drift that outlived one tick report itself
|
||||
// as just found, forever, which is the one thing the drift age
|
||||
// on the row exists to say.
|
||||
const alreadyDrifted =
|
||||
DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status === 'drifted';
|
||||
commitBlueprintDeploymentCause('drift_observed', blueprint.id, node.id, {
|
||||
status: 'drifted',
|
||||
last_checked_at: Date.now(),
|
||||
last_drift_at: Date.now(),
|
||||
...(alreadyDrifted ? {} : { last_drift_at: Date.now() }),
|
||||
drift_summary: reason,
|
||||
}, null);
|
||||
// A block about auto-repair means nothing to Observe or Suggest,
|
||||
@@ -747,6 +754,17 @@ export class BlueprintReconciler {
|
||||
decision.check.push(node);
|
||||
continue;
|
||||
}
|
||||
// A drifted row at the current revision belongs to the drift check,
|
||||
// for the same reason an active one does: the drift is still there
|
||||
// until a check says otherwise, and under Enforce the check is what
|
||||
// repairs it. The projection also reaches such a row through its own
|
||||
// status fallback, so this changes nothing an operator sees; it is
|
||||
// here so the decision a tick acts on is not two files away in the
|
||||
// preview projection, which is no place to keep a recovery path.
|
||||
if (dep.status === 'drifted' && dep.applied_revision === blueprint.revision) {
|
||||
decision.check.push(node);
|
||||
continue;
|
||||
}
|
||||
// A held target keeps its check. The hold is a state, not a latch:
|
||||
// the cause can clear (a node returns, a rollout lands, a rollback
|
||||
// finishes), and this is the only thing that notices. Without it a
|
||||
@@ -832,15 +850,39 @@ export class BlueprintReconciler {
|
||||
});
|
||||
return;
|
||||
}
|
||||
// Read before the attempt, because the answer to "is this failure
|
||||
// new" cannot come from the row afterwards: a repair that reaches
|
||||
// a deploy writes its own status and its own `last_error`, and it
|
||||
// writes the very string it returns, so the row it leaves behind
|
||||
// already looks like a failure someone has reported.
|
||||
const before = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
commitBlueprintDeploymentCause('drift_enforce_start', blueprint.id, node.id, {
|
||||
status: 'correcting',
|
||||
last_checked_at: Date.now(),
|
||||
}, null);
|
||||
const result = cause === 'digest'
|
||||
? await BlueprintService.getInstance().enforceDigestRepair(blueprint, node)
|
||||
: isGitManagedBlueprint(blueprint)
|
||||
? await BlueprintService.getInstance().reapplyAuthorizedMaterialization(blueprint, node)
|
||||
: await BlueprintService.getInstance().deployToNode(blueprint, node);
|
||||
let result: DeployOutcome;
|
||||
try {
|
||||
result = await this.runDriftRepair(blueprint, node, cause);
|
||||
} catch (err) {
|
||||
// A repair that throws has not written anything either, and
|
||||
// the row is already reading `correcting`, so it is settled
|
||||
// exactly like a refusal. Without this the throw escapes to
|
||||
// the per-blueprint catch and strands the row, which is the
|
||||
// same ambiguity a refusal used to leave.
|
||||
//
|
||||
// The throw is logged rather than only alerted, because this
|
||||
// boundary cannot tell an operational failure from a bug in
|
||||
// the repair itself, and both look identical here. Letting a
|
||||
// bug through as an alert alone would repeat it every tick
|
||||
// with a cryptic message and never reach a developer, while
|
||||
// re-throwing would strand the row again. Logging covers the
|
||||
// developer and the alert covers the operator.
|
||||
console.error(
|
||||
`[BlueprintReconciler] drift repair threw for blueprint ${blueprint.id} on node ${node.id}:`,
|
||||
err instanceof Error ? err.stack ?? err.message : String(err),
|
||||
);
|
||||
result = { status: 'failed', error: BlueprintService.formatError(err) };
|
||||
}
|
||||
// A hold is a decision, not a failed attempt. The repair
|
||||
// re-resolved its authority and found it unprovable, so this
|
||||
// tick records the hold and leaves no `correcting` row behind.
|
||||
@@ -858,18 +900,148 @@ export class BlueprintReconciler {
|
||||
return;
|
||||
}
|
||||
if (result.status !== 'active') {
|
||||
notifications.dispatchAlert(
|
||||
'error',
|
||||
'blueprint_drift_correction_failed',
|
||||
`Auto-fix for "${blueprint.name}" ${nodeLocationClause(node)} failed: ${result.error ?? 'unknown error'}`,
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
// Losing a race for the deploy lock is not a failure and
|
||||
// carries no message of its own. The row still has to leave
|
||||
// `correcting` so the next tick picks it up, but there is
|
||||
// nothing to page about.
|
||||
const deferred = result.status === 'pending';
|
||||
// A refusal that wrote no status of its own is settled here,
|
||||
// so the row never keeps reading `correcting` and the next
|
||||
// tick gets to try again. The settle decides whether this is
|
||||
// news, because that question can only be answered against
|
||||
// the row and the snapshot together.
|
||||
const report = this.settleRefusedDriftRepair(
|
||||
blueprint,
|
||||
node,
|
||||
result,
|
||||
before,
|
||||
deferred,
|
||||
);
|
||||
if (report) {
|
||||
notifications.dispatchAlert(
|
||||
'error',
|
||||
'blueprint_drift_correction_failed',
|
||||
`Auto-fix for "${blueprint.name}" ${nodeLocationClause(node)} failed: ${report}`,
|
||||
{ stackName: blueprint.name, actor: 'system:blueprint' },
|
||||
);
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the repair a drift calls for.
|
||||
*
|
||||
* Three methods, one per cause, because a frozen image identity must be
|
||||
* restored through the approved digest rather than by tag, and a Git-managed
|
||||
* target has no authored compose to deploy at all. A throw is left to the
|
||||
* caller, which settles the row: every one of these can reject before it
|
||||
* writes anything.
|
||||
*/
|
||||
private runDriftRepair(blueprint: Blueprint, node: Node, cause: DriftCause): Promise<DeployOutcome> {
|
||||
if (cause === 'digest') {
|
||||
return BlueprintService.getInstance().enforceDigestRepair(blueprint, node);
|
||||
}
|
||||
if (isGitManagedBlueprint(blueprint)) {
|
||||
return BlueprintService.getInstance().reapplyAuthorizedMaterialization(blueprint, node);
|
||||
}
|
||||
return BlueprintService.getInstance().deployToNode(blueprint, node);
|
||||
}
|
||||
|
||||
/**
|
||||
* Settle a drift repair that refused, and report whether the operator needs
|
||||
* telling about it.
|
||||
*
|
||||
* `drift_enforce_start` writes `correcting` before the repair runs, so any
|
||||
* repair path that answers without reaching a deploy leaves the row there.
|
||||
* A `correcting` row is skipped by the decision buckets and projected as
|
||||
* informational, so nothing would re-examine it: the drift would sit
|
||||
* unrepaired, with the wrong image still running, behind a single alert.
|
||||
* A repair that reaches a deploy writes its own terminal status, and a hold
|
||||
* writes its own row, so the row is only rewritten when the refusal wrote
|
||||
* nothing at all.
|
||||
*
|
||||
* Such a row returns to `drifted` because the drift is still real and still
|
||||
* owed a repair, and `drifted` is re-checked on every following tick.
|
||||
* Writing `failed` or `pending` instead would be a worse answer: both are
|
||||
* retry buckets for an ordinary deploy, so the next tick would repair by tag
|
||||
* and adopt whatever the tag points at now, which is exactly what a
|
||||
* digest-pinned repair exists to prevent.
|
||||
*
|
||||
* The alert is for a failure that is **new**, judged against the row as it
|
||||
* stood before this tick's attempt. That snapshot is the only honest basis
|
||||
* for it: a repair that reaches a deploy writes its own status and its own
|
||||
* `last_error`, and it writes the very string it returns, so the row it
|
||||
* leaves behind already looks like a failure someone reported. Comparing
|
||||
* against the row afterwards would silence the first real failure of every
|
||||
* kind, and comparing `last_error` alone would go on silencing a failure
|
||||
* that recurs after the drift converged in between.
|
||||
*
|
||||
* The key is a message rather than a code because every refusal that reaches
|
||||
* here answers with a constant string: the preconditions in the digest repair
|
||||
* and the authorized reapply, a lock the deploy could not take, or the thrown
|
||||
* error of a repair that failed outright. A code column is the thing to add
|
||||
* if a refusal ever starts composing its own text.
|
||||
*
|
||||
* Every tick of a persistent refusal records a `blueprint_correcting` and a
|
||||
* `blueprint_drifted` observation, so the history grows by two rows a minute
|
||||
* while it persists. That is left in place deliberately: the row really does
|
||||
* move between the two states on every attempt, and the projection reads the
|
||||
* latest stage, so suppressing either write would leave the history claiming
|
||||
* a state the row is not in. A hold, which does not move, suppresses instead.
|
||||
*
|
||||
* Returns the message to report, or null when the operator has already been
|
||||
* told this one.
|
||||
*/
|
||||
private settleRefusedDriftRepair(
|
||||
blueprint: Blueprint,
|
||||
node: Node,
|
||||
outcome: DeployOutcome,
|
||||
before: BlueprintDeployment | undefined,
|
||||
deferred: boolean,
|
||||
): string | null {
|
||||
const row = DatabaseService.getInstance().getDeployment(blueprint.id, node.id);
|
||||
// A repair that reached a deploy wrote its own terminal status, and a
|
||||
// hold wrote its own row. Only the refusals that wrote nothing are
|
||||
// settled here.
|
||||
const stranded = row?.status === 'correcting';
|
||||
// When the repair reached a deploy, the row it left behind spells this
|
||||
// failure out and the outcome may only carry a shorthand: the
|
||||
// name-conflict paths return the bare word `name_conflict` and then park
|
||||
// the row for good, so that one alert is the only signal the operator
|
||||
// gets and it has to name the directory in the way. When the row is
|
||||
// still `correcting` nothing was written, so the row's `last_error` is
|
||||
// the *previous* failure and the outcome is the only honest source.
|
||||
const message = stranded
|
||||
? outcome.error ?? 'unknown error'
|
||||
: row?.last_error ?? outcome.error ?? 'unknown error';
|
||||
if (stranded) {
|
||||
// Neither `last_drift_at` nor `last_error` is restamped by a
|
||||
// deferral. The timestamp answers when this drift episode began, and
|
||||
// the error is the replay key: writing a note about a lost lock race
|
||||
// over the failure being replayed would make the very next tick
|
||||
// report that same failure again as though it were new. What the row
|
||||
// already says is worth more to an operator than a note about a race
|
||||
// that is over.
|
||||
commitBlueprintDeploymentCause('drift_observed', blueprint.id, node.id, {
|
||||
status: 'drifted',
|
||||
last_checked_at: Date.now(),
|
||||
...(deferred ? {} : { last_error: message }),
|
||||
}, null);
|
||||
}
|
||||
const nowStatus = stranded ? 'drifted' : row?.status;
|
||||
const nowError = stranded && !deferred ? message : row?.last_error;
|
||||
if (deferred) return null;
|
||||
// A row that converged while the repair was in flight has no drift left
|
||||
// for this failure to be about, so reporting it would page the operator
|
||||
// over a target that is already healthy.
|
||||
if (nowStatus === 'active') return null;
|
||||
const isNew = nowStatus !== before?.status || nowError !== before?.last_error;
|
||||
return isNew ? message : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Git-managed Blueprints skip place/withdraw/content apply on the tick, but
|
||||
* still re-observe runtime identity against the already-authorized generation
|
||||
@@ -926,10 +1098,12 @@ export class BlueprintReconciler {
|
||||
}
|
||||
}
|
||||
const reason = driftResult.reason;
|
||||
const alreadyDrifted =
|
||||
DatabaseService.getInstance().getDeployment(blueprint.id, node.id)?.status === 'drifted';
|
||||
commitBlueprintDeploymentCause('drift_observed', blueprint.id, node.id, {
|
||||
status: 'drifted',
|
||||
last_checked_at: Date.now(),
|
||||
last_drift_at: Date.now(),
|
||||
...(alreadyDrifted ? {} : { last_drift_at: Date.now() }),
|
||||
drift_summary: reason,
|
||||
}, null);
|
||||
await this.handleDrift(blueprint, node, reason, driftResult.cause);
|
||||
@@ -1006,6 +1180,11 @@ export class BlueprintReconciler {
|
||||
last_checked_at: Date.now(),
|
||||
last_drift_at: null,
|
||||
drift_summary: null,
|
||||
// Cleared with the drift, like the deploy acknowledgement clears it.
|
||||
// Left behind, it would keep naming a failure that has since been
|
||||
// repaired, and a failure that recurs after this would be read as one
|
||||
// already reported.
|
||||
last_error: null,
|
||||
}, null);
|
||||
}
|
||||
|
||||
|
||||
@@ -446,7 +446,7 @@ Local-target failure reasons stay node-neutral (`Target node is offline`, `Conta
|
||||
### Blueprints
|
||||
|
||||
- `warning`/`blueprint_drift_detected`: `Blueprint "<n>" drifted on this node: <reason>` for a local target, or `Blueprint "<n>" drifted on node "<remote>": <reason>` for a remote. Stateful marker-loss uses the same local/remote clause.
|
||||
- `error`/`blueprint_drift_correction_failed`: `Auto-fix for "<n>" on this node failed: <err>` (local) or `Auto-fix for "<n>" on node "<remote>" failed: <err>` (remote).
|
||||
- `error`/`blueprint_drift_correction_failed`: `Auto-fix for "<n>" on this node failed: <err>` (local) or `Auto-fix for "<n>" on node "<remote>" failed: <err>` (remote). The repair is retried on the next reconciler tick, so this fires once for a given reason rather than on every retry, and fires again when the reason changes.
|
||||
- `warning`/`blueprint_drift_repair_held`: `Auto-fix for "<n>" on this node was declined: <reason>` (local) or the same on the remote node name (remote). This fires once per hold, when the reason changes, not on every tick.
|
||||
|
||||
See [Blueprints](/features/blueprint-model).
|
||||
|
||||
@@ -136,7 +136,7 @@ The three policy cards in the editor map exactly to the three modes the reconcil
|
||||
|---|---|
|
||||
| **Observe** · Detect & display, no notifications | Drift surfaces in the deployment table; no notification, no auto-fix. |
|
||||
| **Suggest** · Detect & notify, operator decides | Sencho dispatches a `blueprint_drift_detected` notification through your notification routes, if any. |
|
||||
| **Enforce** · Detect & auto-fix, silent on success | Sencho re-deploys the blueprint silently when drift is detected. A `blueprint_drift_correction_failed` notification fires only when the auto-fix attempt fails, and a `blueprint_drift_repair_held` notification fires when auto-fix is declined. |
|
||||
| **Enforce** · Detect & auto-fix, silent on success | Sencho re-deploys the blueprint silently when drift is detected, and retries on each following tick for as long as the row stays **Drifted**. A `blueprint_drift_correction_failed` notification fires when an auto-fix attempt fails, once for that reason rather than on every retry, and a `blueprint_drift_repair_held` notification fires when auto-fix is declined. |
|
||||
|
||||
Even **Observe** keeps Sencho honest about what it found: the deployment row shows "drifted 3h ago: service caddy exited code 1". Silence would forfeit Sencho's authority over your fleet.
|
||||
|
||||
@@ -190,8 +190,8 @@ Possible status values:
|
||||
| **Awaiting confirmation** | A stateful blueprint reached a new node. The reconciler refuses to deploy until you click **Confirm deploy** in the action column. |
|
||||
| **Deploying** | The compose action is in flight on this node. |
|
||||
| **Active** | The deployment is placed and its containers are running for this blueprint revision. Exact image identity is a separate GitOps / Drift proof; a row can read Active while Drift shows runtime artifact disagreement. |
|
||||
| **Drifted** | The reconciler detected a difference between the live state and the desired one (marker, revision, container health, or frozen image digest). |
|
||||
| **Correcting** | An Enforce-mode auto-fix is in flight. |
|
||||
| **Drifted** | The reconciler detected a difference between the live state and the desired one (marker, revision, container health, or frozen image digest). Under **Enforce** the next tick tries to repair it, and tries again for as long as the row stays in this state. Drift that is held or blocked instead does not come back here, because **Repair held** and **Name conflict** are waiting on you, not on the reconciler. |
|
||||
| **Correcting** | An Enforce-mode auto-fix is in flight. An attempt that reports back without finishing leaves the row on **Drifted** with the reason in the notes column, for the next tick to try again. A row that stays here means the attempt never reported back at all, which a restart during a repair can cause, and it will not pick itself up. |
|
||||
| **Repair held** | Drift was found and auto-fix was declined, with the reason on the row. Nothing was changed. |
|
||||
| **Failed** | The deployment or withdrawal errored. The notes column carries the message. |
|
||||
| **Withdrawing** | A withdraw is in flight. |
|
||||
@@ -207,7 +207,7 @@ Possible status values:
|
||||
|
||||
The common paths through the status enum.
|
||||
|
||||
- **Stateless first deploy.** No row → **Pending** → **Deploying** → **Active**. Drift detected later on Enforce flips to **Drifted** → **Correcting** → **Active**. Drift that cannot be repaired safely goes to **Repair held** instead and stays there until the cause clears.
|
||||
- **Stateless first deploy.** No row → **Pending** → **Deploying** → **Active**. Drift detected later on Enforce flips to **Drifted** → **Correcting** → **Active**. Drift that cannot be repaired safely goes to **Repair held** instead and stays there until the cause clears. An auto-fix that fails goes back to **Drifted** with the reason in the notes column, and the next tick tries again, so a repair that fails once is not the end of it.
|
||||
- **Stateful first deploy.** No row → **Awaiting confirmation**. Operator clicks **Confirm deploy** and picks **Deploy fresh** → **Deploying** → **Active**.
|
||||
- **Compose change on a stateless deployment.** **Active** → **Deploying** on the next tick → **Active**.
|
||||
- **Compose change on a stateful or unknown deployment.** **Active** → **Awaiting confirmation** on every targeted node so the operator can decide whether the new revision is safe for that node's local data. Operator confirms → **Deploying** → **Active**.
|
||||
@@ -457,6 +457,8 @@ A future Volume Migration feature will automate this with app-aware backup tooli
|
||||
</Accordion>
|
||||
<Accordion title="Drift is reported but never gets corrected">
|
||||
Confirm the drift policy is **Enforce** and that **Reconciler enabled** is on. Open the detail sheet to check the row's status and the most recent drift summary. If the row is sitting in **Awaiting confirmation** instead of **Drifted**, that is expected: a compose edit on a stateful or state-unknown blueprint never auto-corrects under any drift mode, including Enforce. It always waits for an explicit **Confirm deploy** so a human signs off before the new revision touches that node's data. Enforce's auto-fix applies to container-level drift (a stopped or crashed container) and to frozen image-digest disagreement on a deployment that is already **Active**. Unreachable nodes and missing digest evidence stay unverified and never auto-correct.
|
||||
|
||||
When an auto-fix cannot finish, the row returns to **Drifted** with the reason in the notes column and the next tick tries again, so the reason there is what to fix: an unreachable node, a node running a Sencho release too old to honor digest pins, or a registry that cannot resolve the approved image. A row left reading **Correcting** is different: it means the attempt never reported back at all, which a restart in the middle of a repair can cause. Such a row does not retry on its own and Confirm does not act on it either, so treat it as needing your decision rather than as work in progress.
|
||||
</Accordion>
|
||||
<Accordion title="'Apply now' is greyed out">
|
||||
**Apply now** requires the blueprint to be enabled and Inline. Open the detail sheet, click **Enable** to flip the reconciler back on, then click **Apply now**. The button is also disabled while the detail sheet is in edit mode; save or cancel the edit first. Git-managed Blueprints do not offer Apply now: content comes from Git, and place runs from an authorized accepted generation rather than from the editor or the detach snapshot. Use **Detach Git** if you need Inline editing and Confirm Apply again.
|
||||
@@ -488,7 +490,7 @@ A future Volume Migration feature will automate this with app-aware backup tooli
|
||||
A Blueprint is a fleet-wide declaration: one compose YAML targeting a set of nodes. Each node where the blueprint resolves materializes as a Stack on that node, in the same compose directory layout the rest of the per-stack lane uses, with a `.blueprint.json` marker added. You can browse the materialized stack in the per-node Stacks view; the Blueprint is the source of truth that drives it.
|
||||
</Accordion>
|
||||
<Accordion title="How fast does drift get noticed?">
|
||||
After you confirm the updated rollout, the next reconciler tick (or the Confirm Apply execution itself) places or removes as approved. The reconciler ticks every 60 seconds with a 5-second initial delay after startup. Within Enforce mode, drift correction for an already-approved place outcome begins on the same tick that detects the drift.
|
||||
After you confirm the updated rollout, the next reconciler tick (or the Confirm Apply execution itself) places or removes as approved. The reconciler ticks every 60 seconds with a 5-second initial delay after startup. Within Enforce mode, drift correction for an already-approved place outcome begins on the same tick that detects the drift, and a row that is still **Drifted** is checked and repaired again on each tick after that, so a repair that could not complete is retried rather than left in place.
|
||||
</Accordion>
|
||||
<Accordion title="Can I roll back to a previous revision?">
|
||||
Not through a one-click history. Blueprints intentionally do not keep a versioned revision history. To revert an Inline Blueprint, paste the prior compose into the editor and save; the reconciler treats the change as a new revision and redeploys. Git-managed Blueprints place from an authorized accepted generation, not from the detach snapshot. Detach Git first if you need Inline paste-and-apply.
|
||||
|
||||
+211
-24
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Digest drift on a real Blueprint deployment.
|
||||
* Digest drift on a real Blueprint deployment, and what each policy does about it.
|
||||
*
|
||||
* A tag is a promise, not an identity, so the drift this exercises is created
|
||||
* the way it happens in the field: the compose pins an approved image digest,
|
||||
@@ -8,6 +8,16 @@
|
||||
* tab, and leave the approved identity where it was instead of adopting whatever
|
||||
* is actually running.
|
||||
*
|
||||
* Two policies are covered against that same drift, because the interesting part
|
||||
* is what happens next. Observe reports it and leaves the row drifted, which is
|
||||
* what the first test asserts. Enforce has to take that same drifted row and put
|
||||
* it back: the second test watches the row return to Active with no drift
|
||||
* summary and the workload running the approved digest again, which is the whole
|
||||
* cycle rather than only its detection. It switches the policy on *after* the
|
||||
* drift is recorded, so the repair lands on a pass later than the one that
|
||||
* recorded it, and the pass that picks a drifted row back up is genuinely
|
||||
* exercised instead of being raced.
|
||||
*
|
||||
* The approved digest is discovered from `nginx:alpine` and the digest the
|
||||
* workload is moved to is discovered from `nginx:1.27`, both of them real
|
||||
* upstream tags of one repository. A locally invented repository cannot work
|
||||
@@ -172,6 +182,7 @@ interface ServiceEvidence {
|
||||
|
||||
interface DeploymentRow {
|
||||
status: string;
|
||||
drift_summary: string | null;
|
||||
}
|
||||
|
||||
interface PreviewPayload {
|
||||
@@ -201,17 +212,56 @@ interface DriftPayload {
|
||||
} | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Preview the plan and confirm it, the way an operator does.
|
||||
*
|
||||
* Confirming against a live Blueprint is a two-step dance: the tick can re-save
|
||||
* the row between the preview and the confirm, which the server refuses with
|
||||
* PREVIEW_STALE. That one case is worth retrying with a fresh preview, because
|
||||
* the UI asks the operator to confirm again. Any other failure is reported as it
|
||||
* is, since retrying it would only repeat it.
|
||||
*/
|
||||
async function confirmPlan(page: Page, blueprintId: number): Promise<void> {
|
||||
let applied: { status: number; body: ApplyPayload } | null = null;
|
||||
for (let attempt = 0; attempt < 4; attempt += 1) {
|
||||
const preview = expectOk(
|
||||
await jsonRequest<PreviewPayload>(page, `/api/blueprints/${blueprintId}/preview`),
|
||||
'previewing the plan',
|
||||
);
|
||||
const result = await jsonRequest<ApplyPayload>(page, `/api/blueprints/${blueprintId}/apply`, {
|
||||
method: 'POST',
|
||||
body: {
|
||||
planFingerprint: preview.planFingerprint,
|
||||
// The apply confirms the authority evidence the preview displayed, so
|
||||
// what it showed has to be echoed back with the plan.
|
||||
...(preview.gitopsFingerprint ? { gitopsFingerprint: preview.gitopsFingerprint } : {}),
|
||||
actions: preview.confirmableActions,
|
||||
},
|
||||
});
|
||||
applied = result;
|
||||
if (result.status === 200) break;
|
||||
if (result.status !== 409 || result.body.code !== 'PREVIEW_STALE') break;
|
||||
}
|
||||
expect(applied?.status, `applying the plan: ${JSON.stringify(applied?.body)}`).toBe(200);
|
||||
}
|
||||
|
||||
test.describe('GitOps digest drift', () => {
|
||||
test.skip(!dockerAvailable() && !process.env.CI, 'Docker with the Compose plugin is not available');
|
||||
|
||||
// The reconciler ticks once a minute, so the drift poll covers a tick and the
|
||||
// observation that follows it, and the baseline poll covers the compose run.
|
||||
// These are ceilings rather than expected durations: the happy path takes about
|
||||
// half a minute, and the CI job has a whole-suite budget to protect.
|
||||
// half a minute, and the CI job has a whole-suite budget to protect. Both tests
|
||||
// here are reconciler-driven, so each carries its own ceiling and the repair's
|
||||
// ceiling covers a second tick on top of the detection it waits for.
|
||||
test.setTimeout(360_000);
|
||||
|
||||
const stamp = Date.now();
|
||||
const blueprintName = `e2e-digest-${stamp}`;
|
||||
// One stack name per case, not one per file. Both cases deploy a real compose
|
||||
// project under this name, so a shared one would make the second case collide
|
||||
// with whatever the first case's teardown left behind, and the failure would
|
||||
// surface as a confusing container count rather than as the leak it is.
|
||||
const blueprintNameFor = (caseName: string) => `e2e-digest-${caseName}-${stamp}`;
|
||||
// Two real tags of one repository, pinned by digest and then swapped, so the
|
||||
// approved reference resolves locally and the observed digest matches the
|
||||
// authored repository. The digests themselves are read from the daemon, never
|
||||
@@ -220,6 +270,9 @@ test.describe('GitOps digest drift', () => {
|
||||
const movedImage = 'nginx:1.27';
|
||||
const overrideFile = `/tmp/e2e-digest-override-${stamp}.yaml`;
|
||||
let blueprintId: number | null = null;
|
||||
// Set by each case as it starts, so teardown can look for the containers that
|
||||
// case created even when the case failed before it created the blueprint.
|
||||
let blueprintName = '';
|
||||
|
||||
test.beforeAll(() => {
|
||||
execFileSync('docker', ['pull', '--quiet', approvedImage], { timeout: DOCKER_TIMEOUT_MS, stdio: 'pipe' });
|
||||
@@ -232,6 +285,7 @@ test.describe('GitOps digest drift', () => {
|
||||
|
||||
test('a replaced image digest is reported per service with both digests, and the approved identity holds', async ({ page }) => {
|
||||
await loginAs(page);
|
||||
blueprintName = blueprintNameFor('observe');
|
||||
|
||||
const approvedDigests = repoDigests(approvedImage);
|
||||
const movedDigests = repoDigests(movedImage);
|
||||
@@ -273,27 +327,7 @@ test.describe('GitOps digest drift', () => {
|
||||
// refuses with PREVIEW_STALE. That one case is worth retrying with a fresh
|
||||
// preview, because the UI asks the operator to confirm again. Any other
|
||||
// failure is reported as it is, since retrying it would only repeat it.
|
||||
let applied: { status: number; body: ApplyPayload } | null = null;
|
||||
for (let attempt = 0; attempt < 4; attempt += 1) {
|
||||
const preview = expectOk(
|
||||
await jsonRequest<PreviewPayload>(page, `/api/blueprints/${blueprintId}/preview`),
|
||||
'previewing the plan',
|
||||
);
|
||||
const result = await jsonRequest<ApplyPayload>(page, `/api/blueprints/${blueprintId}/apply`, {
|
||||
method: 'POST',
|
||||
body: {
|
||||
planFingerprint: preview.planFingerprint,
|
||||
// The apply confirms the authority evidence the preview displayed, so
|
||||
// what it showed has to be echoed back with the plan.
|
||||
...(preview.gitopsFingerprint ? { gitopsFingerprint: preview.gitopsFingerprint } : {}),
|
||||
actions: preview.confirmableActions,
|
||||
},
|
||||
});
|
||||
applied = result;
|
||||
if (result.status === 200) break;
|
||||
if (result.status !== 409 || result.body.code !== 'PREVIEW_STALE') break;
|
||||
}
|
||||
expect(applied?.status, `applying the plan: ${JSON.stringify(applied?.body)}`).toBe(200);
|
||||
await confirmPlan(page, blueprintId);
|
||||
|
||||
// Only the projection is read after its poll, so it is the only field that
|
||||
// needs to outlive the callbacks that fill it.
|
||||
@@ -407,6 +441,159 @@ test.describe('GitOps digest drift', () => {
|
||||
expect(stillApproved, 'reading the expectation must not adopt the running image').toBe(pinned);
|
||||
});
|
||||
|
||||
test('an Enforce Blueprint repairs a moved image back to the approved digest and reconverges', async ({ page }) => {
|
||||
await loginAs(page);
|
||||
blueprintName = blueprintNameFor('enforce');
|
||||
|
||||
const approvedDigests = repoDigests(approvedImage);
|
||||
const movedDigests = repoDigests(movedImage);
|
||||
const [pinned] = approvedDigests;
|
||||
if (pinned === undefined) throw new Error(`${approvedImage} reported no repo digest`);
|
||||
expect(
|
||||
movedDigests.filter((digest) => approvedDigests.includes(digest)),
|
||||
`${approvedImage} and ${movedImage} resolve to the same digest, so nothing would drift`,
|
||||
).toEqual([]);
|
||||
const approvedRef = `nginx@${pinned}`;
|
||||
const compose = `services:\n web:\n image: ${approvedRef}\n`;
|
||||
|
||||
const nodes = expectOk(await jsonRequest<NodeRow[]>(page, '/api/nodes'), 'listing nodes');
|
||||
const local = nodes.find((node) => node.type === 'local');
|
||||
expect(local, 'expected a local node to target').toBeTruthy();
|
||||
const stackDir = `${local!.compose_dir}/${blueprintName}`;
|
||||
|
||||
// Observe first, then switch to Enforce once the drift is on the row. Going
|
||||
// straight to Enforce would let the tick that detects the drift repair it in
|
||||
// the same pass, so the row would never be observably drifted and the pass
|
||||
// that picks a drifted row back up would go unproven. This way the row is
|
||||
// provably drifted under Enforce before anything may repair it.
|
||||
const created = await jsonRequest<{ id?: number; error?: string }>(page, '/api/blueprints', {
|
||||
method: 'POST',
|
||||
body: {
|
||||
name: blueprintName,
|
||||
compose_content: compose,
|
||||
selector: { type: 'nodes', ids: [local!.id] },
|
||||
drift_mode: 'observe',
|
||||
},
|
||||
});
|
||||
expect(created.status, `creating the blueprint: ${JSON.stringify(created.body)}`).toBe(201);
|
||||
if (typeof created.body.id !== 'number') {
|
||||
throw new Error(`the create response carried no blueprint id: ${JSON.stringify(created.body)}`);
|
||||
}
|
||||
blueprintId = created.body.id;
|
||||
|
||||
await confirmPlan(page, blueprintId);
|
||||
|
||||
// The baseline has to be proven, not assumed: a mismatch that was already
|
||||
// there at deploy time would otherwise be credited to the override below.
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const detail = expectOk(
|
||||
await jsonRequest<BlueprintDetail>(page, `/api/blueprints/${blueprintId}`),
|
||||
'reading the blueprint',
|
||||
);
|
||||
const approved =
|
||||
detail.gitopsRevision?.targets?.[0]?.artifact?.expected?.services?.[0]?.platformDigest ?? null;
|
||||
return approved === pinned && detail.deployments.some((row) => row.status === 'active');
|
||||
},
|
||||
{ timeout: 150_000, intervals: [1_000] },
|
||||
)
|
||||
.toBe(true);
|
||||
expect(
|
||||
containerImageReference(blueprintName, 'web'),
|
||||
'the deploy must run the approved reference, or the override proves nothing',
|
||||
).toContain(`nginx@${pinned}`);
|
||||
|
||||
// Replace the running workload under the same project. The compose on disk
|
||||
// still pins the approved digest, so the only thing that changed is what is
|
||||
// actually running.
|
||||
writeFileSync(overrideFile, `services:\n web:\n image: ${movedImage}\n`);
|
||||
execFileSync(
|
||||
'docker',
|
||||
[
|
||||
'compose', '-p', blueprintName,
|
||||
'-f', `${stackDir}/compose.yaml`,
|
||||
'-f', overrideFile,
|
||||
'up', '-d', '--force-recreate', '--pull', 'never',
|
||||
],
|
||||
{ timeout: DOCKER_TIMEOUT_MS, stdio: 'pipe' },
|
||||
);
|
||||
|
||||
// Observe records the drift and leaves it, which is what makes the row
|
||||
// drifted rather than repaired on the pass that finds it.
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const detail = expectOk(
|
||||
await jsonRequest<BlueprintDetail>(page, `/api/blueprints/${blueprintId}`),
|
||||
'reading the blueprint',
|
||||
);
|
||||
return detail.deployments.some((row) => row.status === 'drifted');
|
||||
},
|
||||
{ timeout: 150_000, intervals: [2_000] },
|
||||
)
|
||||
.toBe(true);
|
||||
const drifted = expectOk(
|
||||
await jsonRequest<BlueprintDetail>(page, `/api/blueprints/${blueprintId}`),
|
||||
'reading the drifted row',
|
||||
).deployments.find((row) => row.status === 'drifted');
|
||||
expect(drifted?.drift_summary, 'a drifted row carries why it drifted').toBeTruthy();
|
||||
expect(
|
||||
containerImageReference(blueprintName, 'web'),
|
||||
'the moved image is what is running until the repair lands',
|
||||
).not.toContain(`nginx@${pinned}`);
|
||||
|
||||
// Switch the policy on. The edit clears the approval, because the plan the
|
||||
// operator confirmed described a different policy, so the plan is confirmed
|
||||
// again against the same intent.
|
||||
const switched = await jsonRequest<{ error?: string }>(page, `/api/blueprints/${blueprintId}`, {
|
||||
method: 'PUT',
|
||||
body: { drift_mode: 'enforce' },
|
||||
});
|
||||
expect(switched.status, `switching to enforce: ${JSON.stringify(switched.body)}`).toBe(200);
|
||||
await confirmPlan(page, blueprintId);
|
||||
|
||||
// The row was already drifted when Enforce started, so the repair runs on a
|
||||
// later pass than the one that recorded the drift. Either the confirm
|
||||
// executes it or the next reconciler tick does; the poll covers both.
|
||||
//
|
||||
// What this covers is the cycle and the reconvergence, not the settle of a
|
||||
// repair that could not complete: here the repair succeeds, so the row is
|
||||
// never stranded in `correcting` across ticks. That path needs a refusal
|
||||
// arranged on purpose, and it is pinned deterministically in
|
||||
// `backend/src/__tests__/blueprints-inline-drift-repair-retry.test.ts`.
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
const detail = expectOk(
|
||||
await jsonRequest<BlueprintDetail>(page, `/api/blueprints/${blueprintId}`),
|
||||
'reading the blueprint',
|
||||
);
|
||||
const row = detail.deployments[0];
|
||||
return row?.status === 'active' && row.drift_summary === null;
|
||||
},
|
||||
{ timeout: 180_000, intervals: [2_000] },
|
||||
)
|
||||
.toBe(true);
|
||||
|
||||
// The repair restored the approved identity rather than adopting whatever
|
||||
// the tag points at now, which is the whole point of a digest-pinned repair.
|
||||
const repaired = containerImageReference(blueprintName, 'web');
|
||||
expect(
|
||||
repaired,
|
||||
'the repaired workload must run the approved image identity',
|
||||
).toContain(`nginx@${pinned}`);
|
||||
|
||||
const after = expectOk(
|
||||
await jsonRequest<DriftPayload>(page, `/api/stacks/${blueprintName}/drift`),
|
||||
'reading drift after the repair',
|
||||
);
|
||||
expect(
|
||||
after.gitopsRevision?.targets?.[0]?.artifact?.expected?.services?.[0]?.platformDigest,
|
||||
'the expectation never moved to the running image',
|
||||
).toBe(pinned);
|
||||
});
|
||||
|
||||
test.afterEach(async ({ page }) => {
|
||||
if (blueprintId === null) return;
|
||||
const id = blueprintId;
|
||||
|
||||
Reference in New Issue
Block a user