mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-10-25 13:11:05 +00:00
fix(gitops): bound GitOps history with a retention window (#2105)
* fix(gitops): bound GitOps history with a retention window `gitops_history` is insert-only and the reconcile tick appends to it continuously, so it grew for the life of an installation. A new `gitops_history_retention_days` setting (default 30, range 1 to 365) seeds with the other GitOps settings and prunes history from the monitor's cleanup pass. Drained outbox rows are pruned with the history they announced; an undrained row gets a grace period past the window because the boot repair can still drain it from its payload, and a row still undrained after the grace is counted, logged, and dropped so it cannot accumulate for ever. The window is instance-local, like the artifact retry interval beside it: the history being pruned is the one this instance holds, and the monitor that prunes it runs here. * fix(gitops): keep each live application's newest history row through the prune `cleanupOldGitOpsHistory` deleted `gitops_history` purely by age, but `latestTransitionByApplication` reads `MAX(created_at)` per application and feeds the "last activity" fact the portfolio and application sheet render. An application with no history inside the window, because polling is off, a source is suspended, a node is detached, or a Blueprint is idle, lost every row and its activity column silently went empty. The prune now exempts each live application's newest row, so the claim the JSDoc, the PR body and the settings helper text already make is true. Scoped to live applications rather than to every application_id: a tombstone in `gitops_applications` is permanent, so an unscoped exemption would retain one history row per stack ever retired and leave exactly the growth this setting exists to stop. A retired application's rows are now pruned in full. Also folds in three follow-ups from the audit: - The prune is chunked to GITOPS_PRUNE_CHUNK_ROWS rows per table per pass, so a first prune on a long-lived install clears its backlog over several janitor ticks instead of holding SQLite's single writer for the whole delete. The method reports historyRemaining and outboxRemaining so the debug log says whether a pass is still catching up. - The outbox OR-delete is two statements, so the undrained branch uses idx_gitops_settled_outbox_undrained instead of scanning for the drained one. - The retention tests assert exact counts and exact surviving row ids rather than toBeGreaterThanOrEqual over a shared database. New tests pin each guard: a quiet application's newest row survives every window, its older rows are pruned, a tombstoned application clears entirely, an unsettled attempt is still recoverable after a prune both inside the window and outside it, and a backlog clears in bounded passes with the remainder reported. Removing the newest-row exemption turns 10 of the 11 tests red; removing only the liveness clause turns the tombstone test red. Docs: the GitOps portfolio page describes the window and its setting, and now states that the newest entry per application is kept, which is what the code does. * test(gitops): drop the unused db handles from the recovery assertions * fix(gitops): keep a settled history row ahead of its started row A reconcile attempt stamps both rows with the same time, and the prune broke that tie on a random id, so it could delete the settlement and leave startup recovery to reopen the attempt. Recency now follows insertion order, in the exemption and in each chunk. Remaining counts run only while developer mode is on, drained outbox deletes use an age index, and the settings copy says the kept row is the newest one for each live application. * fix(gitops): log the undrained rows a history prune actually deletes A pass is capped, so the warning was reporting the whole backlog. The history-window tests now fail if the permission, the page save, the monitor window, or the getter fallback stops matching the code. * fix(gitops): prune a superseded unsettled attempt past the window An unsettled reservation survives past the retention window only while it is that live application's newest row. Creating applications keep that row; detached applications do not. * fix(gitops): skip the history prune when the retention setting cannot be read A failed settings read used to fall back to 30 days and delete rows the operator had chosen to keep. The getter now throws, and that pass deletes nothing. A missing or unusable stored value still uses 30. * fix(gitops): restore the history window when the retention spec fails The retention spec restored the seeded 30 days as its last statement, not in a finally, so a failed assertion left the shared instance on 14. The CI retry then read 14 and failed the fresh-install assertion, turning one failure into a persistent red. Everything after the first save now runs inside a try whose finally puts the window back, so the spec leaves the instance as it found it whether it passes or fails. * docs(settings): list the GitOps history window in the retention reference The Data Retention table listed every other retention window and not the GitOps one, so an operator checking retention windows did not find it. The portfolio page described the setting in a paragraph, which is not where retention windows are looked up. Adds the row with its default and range, and a line under the table saying the window is set per Sencho instance and where the control lives. * fix(gitops): report an unmeasured prune remainder as unknown The prune reported a remainder of 0 whenever developer mode was off, so the diagnostic log read "none left" for a table it had never counted. It now reports null, and the log line says the remainder was not measured. Three smaller corrections from the same review: - The diagnostics allowlist omitted the history window, so a support bundle could not explain a timeline that had already been pruned. - The GitOps registry entry described every control as belonging to this node, while the retry interval and the history window are instance local. The sentence now names no scope it does not have. * fix(gitops): correct what the history window is documented to govern The copy said the window governs notifications. It does not: the prune deletes history rows and their outbox rows, and the bell entries those outbox rows announce age out under the Notification log window instead. An operator setting 7 days would have expected their bell to follow. It also promised to keep the database small, which nothing delivers. The prune stops old rows from being written; it never reclaims the file. - The helper now names the record behind the GitOps timeline, points bell entries at the Notification log, and says the window stops history from growing. - The settings reference gains the GitOps section it never had, with the window under it rather than under a per-node table that ends in a Save button the control does not use. - The portfolio page stops claiming a shorter window never changes what an application is doing: a webhook redelivery reads this same record, which the same paragraph already said. - A fraction below one day now clamps to a day instead of flooring to zero, which as a cutoff is now and would delete everything not exempt. The only caller passes an integer, so this is a guard, not a reachable fix. * fix(settings): snap a typed number chip to its own step on commit The chip clamped to min and max but never rounded, so typing 1.5 wrote "1.5" and every integer setting rejected it at the route with a bare "Validation failed" toast. Nothing was saved, so no data was at risk, but the control accepted input it could not store. Rounding to the control's step fixes that and leaves a genuinely fractional chip alone: the Docker janitor asks for 0.5 GiB and keeps it. Clamping still runs after rounding, so a fraction below the floor cannot round down under it. The settings reference also listed GitOps last under Infrastructure while the sidebar renders it third. * docs(gitops): replace the unsourced window figure with what the code does The helper and both docs said a window under about 3 days can let a Git host retry a webhook. Nothing in the code picks three days, and the dedupe the sentence describes only applies to the direct webhook path. The text now states the condition instead of a number: a redelivery can run the work again once the record of its first delivery is pruned. A provider endpoint keeps its own delivery record, pruned to a fixed depth, so the window does not affect it, and the sentence says so. * docs(settings): say where a remote instance's window is set The reference explained that the window belongs to an instance rather than a node, but not how to change the window of a remote one, which is on that instance.
This commit is contained in:
Anso
@@ -51,6 +51,7 @@ describe('collectDiagnostics', () => {
|
||||
db.updateGlobalSetting('auth_jwt_secret', 'top-secret-signing-key');
|
||||
db.updateGlobalSetting('cloud_backup_secret_key', 'enc:deadbeef');
|
||||
db.updateGlobalSetting('host_cpu_limit', '80');
|
||||
db.updateGlobalSetting('gitops_history_retention_days', '45');
|
||||
|
||||
const report = await collectDiagnostics();
|
||||
|
||||
@@ -58,6 +59,9 @@ describe('collectDiagnostics', () => {
|
||||
expect(report.config.cloud_backup_secret_key).toBeUndefined();
|
||||
// The allowlisted, non-secret value is present.
|
||||
expect(report.config.host_cpu_limit).toBe('80');
|
||||
// The retention window decides what history an instance keeps, so a
|
||||
// support bundle that omits it cannot explain a missing timeline.
|
||||
expect(report.config.gitops_history_retention_days).toBe('45');
|
||||
// No emitted value is one of the seeded secrets, regardless of key name.
|
||||
expect(Object.values(report.config)).not.toContain('top-secret-signing-key');
|
||||
expect(Object.values(report.config)).not.toContain('enc:deadbeef');
|
||||
|
||||
@@ -0,0 +1,591 @@
|
||||
/**
|
||||
* GitOps history retention: the monitor's cleanup pass is the only bound on the
|
||||
* insert-only history and outbox tables.
|
||||
*
|
||||
* The reconcile tick appends to `gitops_history` continuously, and an
|
||||
* unresolved artifact expectation retries on its own interval, so without the
|
||||
* window both tables grow for the life of the installation. These tests pin the
|
||||
* window's behavior on real rows rather than on a mocked delete.
|
||||
*
|
||||
* One invariant is load-bearing and easy to break: the newest row of each
|
||||
* application survives the prune, because the portfolio's "last activity"
|
||||
* fact reads `MAX(created_at)` over history. Every test here that prunes a row
|
||||
* therefore writes a newer row for the same application first, so it is testing
|
||||
* the prune rather than accidentally testing that exemption.
|
||||
*/
|
||||
import { randomUUID } from 'crypto';
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest';
|
||||
import { setupTestDb, cleanupTestDb } from './helpers/setupTestDb';
|
||||
import { directApplicationFixture } from './helpers/gitopsFixtures';
|
||||
import { DatabaseService, GITOPS_PRUNE_CHUNK_ROWS } from '../services/DatabaseService';
|
||||
import { insertHistory, latestTransitionByApplication } from '../services/gitops/history';
|
||||
import { GitOpsStore } from '../services/gitops/store';
|
||||
import { GitOpsTransitions } from '../services/gitops/transitions';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
const DAY_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
beforeAll(async () => {
|
||||
tmpDir = await setupTestDb();
|
||||
GitOpsStore.resetForTests();
|
||||
GitOpsTransitions.resetForTests();
|
||||
});
|
||||
|
||||
afterAll(() => cleanupTestDb(tmpDir));
|
||||
|
||||
/**
|
||||
* Write one history row. `applicationId` is what several rows share when a test
|
||||
* needs one application's timeline rather than several applications'; the
|
||||
* unique index is (application_id, operation_id, stage, dedupe_target), so
|
||||
* distinct rows need a distinct operation id or dedupe target.
|
||||
*
|
||||
* The application row itself is inserted because the prune exempts the newest
|
||||
* row of a *live* application, and that liveness is read from
|
||||
* `gitops_applications`: a history row with no application behind it is orphaned
|
||||
* history the prune is free to take in full. Production always has the row, so
|
||||
* the test fixtures do too.
|
||||
*/
|
||||
function writeRow(
|
||||
operationId: string,
|
||||
at: number,
|
||||
opts: { applicationId?: string; stage?: 'source_reconcile_settled' | 'source_reconcile_started' } = {},
|
||||
): string {
|
||||
const applicationId = opts.applicationId ?? `app-${operationId}`;
|
||||
const application = directApplicationFixture(applicationId, `stack-${applicationId}`);
|
||||
const store = GitOpsStore.getInstance();
|
||||
// Several rows on one application insert the application once.
|
||||
const existing = DatabaseService.getInstance().getDb()
|
||||
.prepare('SELECT 1 FROM gitops_applications WHERE id = ?')
|
||||
.get(applicationId);
|
||||
if (!existing) store.insertApplication(application);
|
||||
const id = insertHistory(DatabaseService.getInstance().getDb(), {
|
||||
application,
|
||||
nodeId: 3,
|
||||
dedupeTarget: 'app',
|
||||
operationId,
|
||||
stage: opts.stage ?? 'source_reconcile_settled',
|
||||
outcome: 'committed',
|
||||
trigger: 'poll',
|
||||
actor: 'system:source-controller',
|
||||
before: {},
|
||||
after: { outcome: 'pending_review', reason: 'ok', nextAction: 'none' },
|
||||
commitSha: null,
|
||||
at,
|
||||
});
|
||||
if (!id) throw new Error('expected history insert');
|
||||
return id;
|
||||
}
|
||||
|
||||
/** Mark the outbox row drained, which is what the fanout does on the normal path. */
|
||||
function markOutboxDrained(historyId: string): void {
|
||||
DatabaseService.getInstance().getDb()
|
||||
.prepare('UPDATE gitops_settled_outbox SET drained_at = ? WHERE settled_history_id = ?')
|
||||
.run(Date.now(), historyId);
|
||||
}
|
||||
|
||||
/** History rows still standing for one application, newest first. */
|
||||
function remainingIdsFor(db: import('better-sqlite3').Database, applicationId: string): string[] {
|
||||
return (
|
||||
db.prepare('SELECT id FROM gitops_history WHERE application_id = ? ORDER BY created_at DESC, id DESC')
|
||||
.all(applicationId) as { id: string }[]
|
||||
).map((row) => row.id);
|
||||
}
|
||||
|
||||
/**
|
||||
* One started or settled row with a caller-chosen id, so a test can force the
|
||||
* UUID order independently of insertion order. The application row is created
|
||||
* the same way `writeRow` does. Call started first when the settlement must
|
||||
* have the higher rowid.
|
||||
*/
|
||||
function insertStageRow(
|
||||
id: string,
|
||||
operationId: string,
|
||||
at: number,
|
||||
applicationId: string,
|
||||
stage: 'source_reconcile_started' | 'source_reconcile_settled',
|
||||
): void {
|
||||
const application = directApplicationFixture(applicationId, `stack-${applicationId}`);
|
||||
const database = DatabaseService.getInstance().getDb();
|
||||
const existing = database.prepare('SELECT 1 FROM gitops_applications WHERE id = ?').get(applicationId);
|
||||
if (!existing) GitOpsStore.getInstance().insertApplication(application);
|
||||
database.prepare(
|
||||
`INSERT INTO gitops_history (id, created_at, application_id, target_mode, lifecycle_key,
|
||||
node_id, dedupe_target, operation_id, stage, outcome, trigger, before_json, after_json)
|
||||
VALUES (?, ?, ?, 'direct', ?, 3, 'app', ?, ?, 'committed', 'poll', '{}', '{}')`,
|
||||
).run(id, at, applicationId, application.lifecycle_key, operationId, stage);
|
||||
}
|
||||
|
||||
/** Run the prune until a pass deletes nothing, so a later test's counts are its own rows. */
|
||||
function sweepUntilQuiet(): void {
|
||||
const db = DatabaseService.getInstance();
|
||||
for (let pass = 0; pass < 20; pass++) {
|
||||
const result = db.cleanupOldGitOpsHistory(30);
|
||||
if (result.history === 0 && result.outbox === 0) return;
|
||||
}
|
||||
throw new Error('retention prune did not finish');
|
||||
}
|
||||
|
||||
/** Remaining counts are computed only while developer mode is on. */
|
||||
function pruneWithCounts(days: number): ReturnType<DatabaseService['cleanupOldGitOpsHistory']> {
|
||||
const db = DatabaseService.getInstance();
|
||||
db.updateGlobalSetting('developer_mode', '1');
|
||||
try {
|
||||
return db.cleanupOldGitOpsHistory(days);
|
||||
} finally {
|
||||
db.updateGlobalSetting('developer_mode', '0');
|
||||
}
|
||||
}
|
||||
|
||||
describe('GitOps history retention', () => {
|
||||
// The delete is global and capped, so a row left by an earlier test can
|
||||
// change the next test's count or steal a chunk. Clear those first.
|
||||
beforeEach(() => {
|
||||
sweepUntilQuiet();
|
||||
});
|
||||
|
||||
it('seeds a 30-day default and clamps an out-of-range stored value', () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
expect(db.getGlobalSettings().gitops_history_retention_days).toBe('30');
|
||||
expect(db.getGitOpsHistoryRetentionDays()).toBe(30);
|
||||
|
||||
db.updateGlobalSetting('gitops_history_retention_days', '9999');
|
||||
expect(db.getGitOpsHistoryRetentionDays()).toBe(365);
|
||||
db.updateGlobalSetting('gitops_history_retention_days', '30');
|
||||
});
|
||||
|
||||
it('reads a stored window and falls back to 30 for garbage, zero, negative, and a missing row', () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
db.updateGlobalSetting('gitops_history_retention_days', '45');
|
||||
expect(db.getGitOpsHistoryRetentionDays()).toBe(45);
|
||||
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
try {
|
||||
for (const stored of ['banana', '0', '-3']) {
|
||||
db.updateGlobalSetting('gitops_history_retention_days', stored);
|
||||
expect(db.getGitOpsHistoryRetentionDays(), stored).toBe(30);
|
||||
}
|
||||
|
||||
const developerMode = db.getGlobalSettings().developer_mode;
|
||||
db.getDb().prepare('DELETE FROM global_settings WHERE key = ?').run('gitops_history_retention_days');
|
||||
db.updateGlobalSetting('developer_mode', developerMode);
|
||||
expect(db.getGitOpsHistoryRetentionDays()).toBe(30);
|
||||
expect(warn.mock.calls.some((call) => String(call[0]).includes('invalid gitops_history_retention_days'))).toBe(true);
|
||||
} finally {
|
||||
warn.mockRestore();
|
||||
db.updateGlobalSetting('gitops_history_retention_days', '30');
|
||||
}
|
||||
});
|
||||
|
||||
it('throws when the retention setting cannot be read, so the caller can skip the delete', () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
const read = vi.spyOn(db, 'getGlobalSettings').mockImplementation(() => {
|
||||
throw new Error('settings unavailable');
|
||||
});
|
||||
try {
|
||||
expect(() => db.getGitOpsHistoryRetentionDays()).toThrow(/settings unavailable/);
|
||||
} finally {
|
||||
read.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('prunes history and drained outbox rows older than the window and keeps the newest', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const now = Date.now();
|
||||
const app = 'app-retention-classic';
|
||||
const oldId = writeRow('op-retention-classic-old', now - 40 * DAY_MS, { applicationId: app });
|
||||
const freshId = writeRow('op-retention-classic-fresh', now - 1 * DAY_MS, { applicationId: app });
|
||||
markOutboxDrained(oldId);
|
||||
markOutboxDrained(freshId);
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(1);
|
||||
expect(result.outbox).toBe(1);
|
||||
expect(remainingIdsFor(db, app)).toEqual([freshId]);
|
||||
});
|
||||
|
||||
it('keeps an application\'s newest history row even when every row is older than the window', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
// A quiet application whose only history is older than the window. Its row
|
||||
// must survive, and the "last activity" the portfolio and the application
|
||||
// sheet read must still resolve.
|
||||
const app = 'app-retention-quiet';
|
||||
const onlyRow = writeRow('op-retention-quiet-only', Date.now() - 400 * DAY_MS, { applicationId: app });
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(0);
|
||||
expect(remainingIdsFor(db, app)).toEqual([onlyRow]);
|
||||
expect(latestTransitionByApplication(db, [app]).get(app)).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('prunes the older rows of a quiet application but not its newest one', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const now = Date.now();
|
||||
// A quiet application with several rows, all older than the window. Only the
|
||||
// newest survives, so history is still bounded and the activity fact still
|
||||
// resolves.
|
||||
const app = 'app-retention-quiet-many';
|
||||
const ids = [45, 44, 43, 42].map((days) =>
|
||||
writeRow(`op-retention-quiet-many-${days}`, now - days * DAY_MS, { applicationId: app }));
|
||||
const newest = ids[ids.length - 1];
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(3);
|
||||
expect(remainingIdsFor(db, app)).toEqual([newest]);
|
||||
expect(latestTransitionByApplication(db, [app]).get(app)).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('still returns an unsettled attempt from inside the window after a prune', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
// The recovery claim: a reservation with no settlement inside the window is
|
||||
// still recoverable after a prune. A newer row is written first so the
|
||||
// reservation is not also the application's newest row: the newest-row
|
||||
// exemption would keep it even if the window filter were wrong.
|
||||
const inside = 'app-retention-recovery';
|
||||
writeRow('op-retention-recovery-inside-newer', Date.now() - 1 * DAY_MS, { applicationId: inside });
|
||||
writeRow('op-retention-recovery-inside', Date.now() - 20 * DAY_MS, {
|
||||
applicationId: inside,
|
||||
stage: 'source_reconcile_started',
|
||||
});
|
||||
|
||||
DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
const returned = store.listUnsettledReconcileAttempts();
|
||||
expect(returned.some((row) => row.application_id === inside)).toBe(true);
|
||||
});
|
||||
|
||||
it('keeps the newest row of a quiet application recoverable even outside the window', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
// A process that crashed and was not restarted for longer than the window:
|
||||
// its reservation is the application's newest row, so the exemption keeps it
|
||||
// and startup recovery can still settle it rather than leaving it open.
|
||||
const app = 'app-retention-recovery-old';
|
||||
writeRow('op-retention-recovery-old-reserved', Date.now() - 60 * DAY_MS, {
|
||||
applicationId: app,
|
||||
stage: 'source_reconcile_started',
|
||||
});
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(0);
|
||||
expect(store.listUnsettledReconcileAttempts().some((row) => row.application_id === app)).toBe(true);
|
||||
});
|
||||
|
||||
it('drops an unsettled attempt older than the window once a newer row exists', () => {
|
||||
const store = GitOpsStore.getInstance();
|
||||
// An unsettled reservation older than the window, then later history for the
|
||||
// same live application. The reservation is no longer the newest row, so the
|
||||
// window takes it and the next startup has nothing to recover.
|
||||
const app = 'app-retention-unsettled-superseded';
|
||||
writeRow('op-retention-unsettled-superseded-newer', Date.now() - 1 * DAY_MS, { applicationId: app });
|
||||
writeRow('op-retention-unsettled-superseded-started', Date.now() - 60 * DAY_MS, {
|
||||
applicationId: app,
|
||||
stage: 'source_reconcile_started',
|
||||
});
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(1);
|
||||
expect(store.listUnsettledReconcileAttempts().some((row) => row.application_id === app)).toBe(false);
|
||||
});
|
||||
|
||||
it('keeps an undrained outbox row through the grace period, and prunes it once drained', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const app = 'app-retention-undrained';
|
||||
// A newer row is what makes the 32-day-old row a prune candidate at all.
|
||||
writeRow('op-retention-undrained-newer', Date.now() - 1 * DAY_MS, { applicationId: app });
|
||||
const historyId = writeRow('op-retention-undrained-old', Date.now() - 32 * DAY_MS, { applicationId: app });
|
||||
|
||||
// The history row is pruned by age. The notification the fanout has not
|
||||
// written yet is kept through the grace. The repair that could still drain
|
||||
// it runs at startup, so the grace only helps an instance that restarts.
|
||||
const first = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
expect(first.history).toBe(1);
|
||||
expect(
|
||||
db.prepare('SELECT drained_at FROM gitops_settled_outbox WHERE settled_history_id = ?').get(historyId),
|
||||
).toEqual({ drained_at: null });
|
||||
|
||||
markOutboxDrained(historyId);
|
||||
const second = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
expect(second.outbox).toBe(1);
|
||||
expect(
|
||||
db.prepare('SELECT COUNT(*) AS n FROM gitops_settled_outbox WHERE settled_history_id = ?').get(historyId),
|
||||
).toEqual({ n: 0 });
|
||||
});
|
||||
|
||||
it('drops an undrained row past the grace period and says so', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const app = 'app-retention-poison';
|
||||
writeRow('op-retention-poison-newer', Date.now() - 1 * DAY_MS, { applicationId: app });
|
||||
const historyId = writeRow('op-retention-poison-old', Date.now() - 40 * DAY_MS, { applicationId: app });
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
try {
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
expect(result.outbox).toBe(1);
|
||||
expect(
|
||||
db.prepare('SELECT COUNT(*) AS n FROM gitops_settled_outbox WHERE settled_history_id = ?').get(historyId),
|
||||
).toEqual({ n: 0 });
|
||||
// A silent backlog is the failure mode the log exists to prevent.
|
||||
expect(warn.mock.calls.some((call) => String(call[0]).includes('undrained GitOps outbox'))).toBe(true);
|
||||
} finally {
|
||||
warn.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('falls back to the default window for a non-positive value', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const now = Date.now();
|
||||
const app = 'app-retention-fallback';
|
||||
writeRow('op-retention-fallback-old', now - 40 * DAY_MS, { applicationId: app });
|
||||
const freshId = writeRow('op-retention-fallback-fresh', now - 1 * DAY_MS, { applicationId: app });
|
||||
|
||||
// 0 is not an off switch: retention off would leave the tables unbounded,
|
||||
// which is the accumulation the setting exists to stop.
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(0);
|
||||
|
||||
expect(result.history).toBe(1);
|
||||
expect(remainingIdsFor(db, app)).toEqual([freshId]);
|
||||
});
|
||||
|
||||
it('clamps a fraction below one day to a day rather than to nothing', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const now = Date.now();
|
||||
const app = 'app-retention-fraction';
|
||||
const olderId = writeRow('op-retention-fraction-old', now - 12 * 60 * 60 * 1000, { applicationId: app });
|
||||
const freshId = writeRow('op-retention-fraction-fresh', now - 1 * 60 * 1000, { applicationId: app });
|
||||
|
||||
// 0.5 passes a `> 0` guard and floors to 0, which as a cutoff is now and
|
||||
// deletes everything not exempt. One day is the smallest real window, so
|
||||
// the 12-hour-old row survives a clamped pass and would not survive a
|
||||
// floored one. Asserted on this application's rows rather than on the
|
||||
// returned count, which also covers whatever an earlier test left behind.
|
||||
DatabaseService.getInstance().cleanupOldGitOpsHistory(0.5);
|
||||
|
||||
expect(remainingIdsFor(db, app)).toEqual([freshId, olderId]);
|
||||
});
|
||||
|
||||
it('bounds each pass to a chunk of rows, so a backlog cannot hold the writer', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
// The unchunked alternative deletes the whole backlog in one transaction,
|
||||
// holding SQLite's single writer and blocking every other write on the
|
||||
// instance for its duration. Rows are written directly rather than through
|
||||
// insertHistory: the outbox and publication side effects are not what this
|
||||
// asserts, and thousands of them would slow the suite without testing more.
|
||||
const app = 'app-retention-chunk';
|
||||
const backlogAt = Date.now() - 400 * DAY_MS;
|
||||
const backlogRows = GITOPS_PRUNE_CHUNK_ROWS * 2 + 1000;
|
||||
const windowCutoff = Date.now() - 30 * DAY_MS;
|
||||
const insert = db.prepare(
|
||||
`INSERT INTO gitops_history (id, created_at, application_id, target_mode, lifecycle_key,
|
||||
node_id, dedupe_target, operation_id, stage, outcome, trigger, before_json, after_json)
|
||||
VALUES (?, ?, ?, 'direct', ?, 3, ?, ?, 'runtime_artifact_observed', 'committed', 'poll', '{}', '{}')`,
|
||||
);
|
||||
db.transaction(() => {
|
||||
for (let i = 0; i < backlogRows; i++) {
|
||||
insert.run(randomUUID(), backlogAt, app, `direct:${app}`, `target-${i}`, `op-chunk-${i}`);
|
||||
}
|
||||
})();
|
||||
const backlogStanding = () =>
|
||||
(
|
||||
db.prepare('SELECT COUNT(*) AS n FROM gitops_history WHERE application_id = ? AND created_at < ?')
|
||||
.get(app, windowCutoff) as { n: number }
|
||||
).n;
|
||||
// A newer row keeps every backlog row a prune candidate: without it the
|
||||
// newest of them would be exempt and the counts would be off by one.
|
||||
writeRow('op-retention-chunk-newest', Date.now() - 1 * DAY_MS, { applicationId: app });
|
||||
expect(backlogStanding()).toBe(backlogRows);
|
||||
|
||||
// Developer mode is off, so this pass still deletes one chunk and reports
|
||||
// no remainder at all, even though rows are left. Unknown is not the same
|
||||
// as none left, so the counts are null. The exact remainder is what the
|
||||
// next pass reports once the diagnostic counts are on.
|
||||
const blind = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
expect(blind.history).toBe(GITOPS_PRUNE_CHUNK_ROWS);
|
||||
expect(blind.historyRemaining).toBeNull();
|
||||
expect(blind.outboxRemaining).toBeNull();
|
||||
expect(backlogStanding()).toBe(backlogRows - GITOPS_PRUNE_CHUNK_ROWS);
|
||||
|
||||
const first = pruneWithCounts(30);
|
||||
// One pass deletes exactly one chunk. An unbounded delete, or a cap raised
|
||||
// past the backlog, deletes a different number.
|
||||
expect(first.history).toBe(GITOPS_PRUNE_CHUNK_ROWS);
|
||||
expect(first.historyRemaining).toBe(backlogRows - 2 * GITOPS_PRUNE_CHUNK_ROWS);
|
||||
expect(backlogStanding()).toBe(first.historyRemaining);
|
||||
expect(first.outboxRemaining).toBe(0);
|
||||
|
||||
// Later passes finish the backlog, and a pass with nothing to do is a no-op.
|
||||
while (DatabaseService.getInstance().cleanupOldGitOpsHistory(30).history > 0) {
|
||||
// Clear the backlog one bounded pass at a time, as the monitor pass does.
|
||||
}
|
||||
expect(backlogStanding()).toBe(0);
|
||||
const empty = pruneWithCounts(30);
|
||||
expect(empty.history).toBe(0);
|
||||
expect(empty.historyRemaining).toBe(0);
|
||||
expect(remainingIdsFor(db, app)).toHaveLength(1);
|
||||
});
|
||||
|
||||
it('prunes a tombstoned application\'s whole history, including its newest row', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
// A retired stack must not leave a row behind for ever: the tombstone in
|
||||
// gitops_applications already keeps the application's frozen record, so the
|
||||
// exemption has nothing to preserve here.
|
||||
const app = 'app-retention-tombstoned';
|
||||
writeRow('op-retention-tombstoned-only', Date.now() - 60 * DAY_MS, { applicationId: app });
|
||||
db.prepare("UPDATE gitops_applications SET lifecycle_status = 'deleted' WHERE id = ?").run(app);
|
||||
|
||||
const result = pruneWithCounts(30);
|
||||
|
||||
expect(result.history).toBe(1);
|
||||
expect(result.historyRemaining).toBe(0);
|
||||
expect(remainingIdsFor(db, app)).toEqual([]);
|
||||
expect(latestTransitionByApplication(db, [app]).has(app)).toBe(false);
|
||||
});
|
||||
|
||||
it('keeps the newest row of a creating application when every row is older than the window', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
// Rows are written before activation. Dropping them would leave an
|
||||
// application that goes live with no recorded activity.
|
||||
const app = 'app-retention-creating';
|
||||
writeRow('op-retention-creating-older', Date.now() - 60 * DAY_MS, { applicationId: app });
|
||||
const newest = writeRow('op-retention-creating-newest', Date.now() - 40 * DAY_MS, { applicationId: app });
|
||||
db.prepare("UPDATE gitops_applications SET lifecycle_status = 'creating' WHERE id = ?").run(app);
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(1);
|
||||
expect(remainingIdsFor(db, app)).toEqual([newest]);
|
||||
});
|
||||
|
||||
it('prunes a detached application\'s whole history, including its newest row', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const app = 'app-retention-detached';
|
||||
writeRow('op-retention-detached-only', Date.now() - 60 * DAY_MS, { applicationId: app });
|
||||
db.prepare("UPDATE gitops_applications SET lifecycle_status = 'detached' WHERE id = ?").run(app);
|
||||
|
||||
const result = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(result.history).toBe(1);
|
||||
expect(remainingIdsFor(db, app)).toEqual([]);
|
||||
});
|
||||
|
||||
it('keeps the settled row when it ties the started row and the started id sorts later', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const store = GitOpsStore.getInstance();
|
||||
// Both rows of one attempt share envelope.at. The started id sorts after
|
||||
// the settled id, which is the UUID order that used to treat started as
|
||||
// newer. Started is inserted first, so its rowid is lower.
|
||||
const app = 'app-retention-tie';
|
||||
const operationId = 'op-retention-tie';
|
||||
const at = Date.now() - 60 * DAY_MS;
|
||||
insertStageRow('z-tie-started', operationId, at, app, 'source_reconcile_started');
|
||||
insertStageRow('a-tie-settled', operationId, at, app, 'source_reconcile_settled');
|
||||
|
||||
DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(
|
||||
db.prepare('SELECT id FROM gitops_history WHERE id = ?').get('a-tie-settled'),
|
||||
).toEqual({ id: 'a-tie-settled' });
|
||||
expect(store.listUnsettledReconcileAttempts().some((row) => row.operation_id === operationId)).toBe(false);
|
||||
});
|
||||
|
||||
it('does not drop a settled row when a chunk boundary falls inside its attempt', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
const store = GitOpsStore.getInstance();
|
||||
sweepUntilQuiet();
|
||||
// 4999 older rows, then a tied pair. Ordered by created_at then rowid, the
|
||||
// chunk of 5000 takes the older rows and the started row (inserted first)
|
||||
// and leaves the settlement. Ordered by id, the settled id sorts first and
|
||||
// would be the row the chunk deletes.
|
||||
const app = 'app-retention-chunk-tie';
|
||||
const operationId = 'op-retention-chunk-tie';
|
||||
const now = Date.now();
|
||||
writeRow('op-retention-chunk-tie-newest', now - 1 * DAY_MS, { applicationId: app });
|
||||
const fillerAt = now - 400 * DAY_MS;
|
||||
const pairAt = now - 60 * DAY_MS;
|
||||
const insert = db.prepare(
|
||||
`INSERT INTO gitops_history (id, created_at, application_id, target_mode, lifecycle_key,
|
||||
node_id, dedupe_target, operation_id, stage, outcome, trigger, before_json, after_json)
|
||||
VALUES (?, ?, ?, 'direct', ?, 3, ?, ?, 'runtime_artifact_observed', 'committed', 'poll', '{}', '{}')`,
|
||||
);
|
||||
const fillers = GITOPS_PRUNE_CHUNK_ROWS - 1;
|
||||
db.transaction(() => {
|
||||
for (let i = 0; i < fillers; i++) {
|
||||
insert.run(randomUUID(), fillerAt, app, `direct:stack-${app}`, `filler-${i}`, `op-chunk-tie-filler-${i}`);
|
||||
}
|
||||
})();
|
||||
insertStageRow('z-chunk-tie-started', operationId, pairAt, app, 'source_reconcile_started');
|
||||
insertStageRow('a-chunk-tie-settled', operationId, pairAt, app, 'source_reconcile_settled');
|
||||
|
||||
const first = DatabaseService.getInstance().cleanupOldGitOpsHistory(30);
|
||||
|
||||
expect(first.history).toBe(GITOPS_PRUNE_CHUNK_ROWS);
|
||||
expect(
|
||||
db.prepare('SELECT id FROM gitops_history WHERE id = ?').get('a-chunk-tie-settled'),
|
||||
).toEqual({ id: 'a-chunk-tie-settled' });
|
||||
expect(store.listUnsettledReconcileAttempts().some((row) => row.operation_id === operationId)).toBe(false);
|
||||
sweepUntilQuiet();
|
||||
});
|
||||
|
||||
it('bounds an outbox pass to one chunk', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
sweepUntilQuiet();
|
||||
const backlogAt = Date.now() - 400 * DAY_MS;
|
||||
const backlogRows = GITOPS_PRUNE_CHUNK_ROWS + 1000;
|
||||
const insert = db.prepare(
|
||||
`INSERT INTO gitops_settled_outbox
|
||||
(settled_history_id, payload_json, payload_version, created_at, updated_at, drained_at)
|
||||
VALUES (?, '{}', 1, ?, ?, ?)`,
|
||||
);
|
||||
db.transaction(() => {
|
||||
for (let i = 0; i < backlogRows; i++) {
|
||||
insert.run(`outbox-chunk-${i}`, backlogAt, backlogAt, backlogAt);
|
||||
}
|
||||
})();
|
||||
|
||||
const first = pruneWithCounts(30);
|
||||
|
||||
expect(first.outbox).toBe(GITOPS_PRUNE_CHUNK_ROWS);
|
||||
expect(first.outboxRemaining).toBe(backlogRows - GITOPS_PRUNE_CHUNK_ROWS);
|
||||
sweepUntilQuiet();
|
||||
expect(
|
||||
db.prepare("SELECT COUNT(*) AS n FROM gitops_settled_outbox WHERE settled_history_id LIKE 'outbox-chunk-%'").get(),
|
||||
).toEqual({ n: 0 });
|
||||
});
|
||||
|
||||
it('counts an undrained row past the grace that one pass did not delete', () => {
|
||||
const db = DatabaseService.getInstance().getDb();
|
||||
sweepUntilQuiet();
|
||||
// Past the window plus the grace, so the undrained branch is the only one
|
||||
// that can see these rows. One more than a chunk is inserted, so one row
|
||||
// is left after the pass. A remaining count that only sees drained rows
|
||||
// reports 0.
|
||||
const backlogAt = Date.now() - 40 * DAY_MS;
|
||||
const backlogRows = GITOPS_PRUNE_CHUNK_ROWS + 1;
|
||||
const insert = db.prepare(
|
||||
`INSERT INTO gitops_settled_outbox
|
||||
(settled_history_id, payload_json, payload_version, created_at, updated_at, drained_at)
|
||||
VALUES (?, '{}', 1, ?, ?, NULL)`,
|
||||
);
|
||||
db.transaction(() => {
|
||||
for (let i = 0; i < backlogRows; i++) {
|
||||
insert.run(`outbox-undrained-${i}`, backlogAt, backlogAt);
|
||||
}
|
||||
})();
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
|
||||
try {
|
||||
const first = pruneWithCounts(30);
|
||||
expect(first.outbox).toBe(GITOPS_PRUNE_CHUNK_ROWS);
|
||||
expect(first.outboxRemaining).toBe(1);
|
||||
expect(warn.mock.calls.some((call) => String(call[0]).includes(`dropping ${GITOPS_PRUNE_CHUNK_ROWS} undrained`))).toBe(true);
|
||||
} finally {
|
||||
warn.mockRestore();
|
||||
}
|
||||
sweepUntilQuiet();
|
||||
expect(
|
||||
db.prepare("SELECT COUNT(*) AS n FROM gitops_settled_outbox WHERE settled_history_id LIKE 'outbox-undrained-%'").get(),
|
||||
).toEqual({ n: 0 });
|
||||
});
|
||||
});
|
||||
@@ -8,7 +8,8 @@ import { installArcstatsFsMock, arcstatsBody, DEFAULT_ARC_PATH, type ArcstatsFsM
|
||||
// ── Hoisted mocks ──────────────────────────────────────────────────────
|
||||
|
||||
const { mockGetGlobalSettings, mockGetNodes, mockGetStackAlerts, mockAddContainerMetric,
|
||||
mockCleanupOldMetrics, mockCleanupOldNotifications, mockCleanupOldAuditLogs,
|
||||
mockCleanupOldMetrics, mockCleanupOldNotifications, mockCleanupOldAuditLogs, mockCleanupOldDeliveryEvents,
|
||||
mockGetGitOpsHistoryRetentionDays, mockCleanupOldGitOpsHistory,
|
||||
mockUpdateStackAlertLastFired, mockGetStackAlertServiceCooldown,
|
||||
mockHasAnyStackAlertServiceCooldown, mockUpsertStackAlertServiceCooldown,
|
||||
mockGetSystemState, mockSetSystemState,
|
||||
@@ -34,6 +35,9 @@ const { mockGetGlobalSettings, mockGetNodes, mockGetStackAlerts, mockAddContaine
|
||||
mockCleanupOldMetrics: vi.fn(),
|
||||
mockCleanupOldNotifications: vi.fn(),
|
||||
mockCleanupOldAuditLogs: vi.fn(),
|
||||
mockCleanupOldDeliveryEvents: vi.fn(),
|
||||
mockGetGitOpsHistoryRetentionDays: vi.fn().mockReturnValue(45),
|
||||
mockCleanupOldGitOpsHistory: vi.fn().mockReturnValue({ history: 0, outbox: 0, historyRemaining: null, outboxRemaining: null }),
|
||||
mockUpdateStackAlertLastFired: vi.fn(),
|
||||
mockGetStackAlertServiceCooldown: vi.fn().mockReturnValue(null),
|
||||
mockHasAnyStackAlertServiceCooldown: vi.fn().mockReturnValue(false),
|
||||
@@ -83,6 +87,9 @@ vi.mock('../services/DatabaseService', () => ({
|
||||
cleanupOldMetrics: mockCleanupOldMetrics,
|
||||
cleanupOldNotifications: mockCleanupOldNotifications,
|
||||
cleanupOldAuditLogs: mockCleanupOldAuditLogs,
|
||||
cleanupOldDeliveryEvents: mockCleanupOldDeliveryEvents,
|
||||
getGitOpsHistoryRetentionDays: mockGetGitOpsHistoryRetentionDays,
|
||||
cleanupOldGitOpsHistory: mockCleanupOldGitOpsHistory,
|
||||
updateStackAlertLastFired: mockUpdateStackAlertLastFired,
|
||||
getStackAlertServiceCooldown: mockGetStackAlertServiceCooldown,
|
||||
hasAnyStackAlertServiceCooldown: mockHasAnyStackAlertServiceCooldown,
|
||||
@@ -1164,6 +1171,10 @@ describe('MonitorService - cleanup triggers', () => {
|
||||
expect(mockCleanupOldMetrics).toHaveBeenCalledWith(48);
|
||||
expect(mockCleanupOldNotifications).toHaveBeenCalledWith(7);
|
||||
expect(mockCleanupOldAuditLogs).toHaveBeenCalledWith(30);
|
||||
// The history window is read through its own getter, which applies the
|
||||
// default and the ceiling; the cleanup pass receives that value.
|
||||
expect(mockGetGitOpsHistoryRetentionDays).toHaveBeenCalled();
|
||||
expect(mockCleanupOldGitOpsHistory).toHaveBeenCalledWith(45);
|
||||
});
|
||||
|
||||
it('uses defaults when settings are NaN', async () => {
|
||||
@@ -1182,6 +1193,66 @@ describe('MonitorService - cleanup triggers', () => {
|
||||
expect(mockCleanupOldNotifications).toHaveBeenCalledWith(30);
|
||||
expect(mockCleanupOldAuditLogs).toHaveBeenCalledWith(90);
|
||||
});
|
||||
|
||||
it('still prunes GitOps history when an unrelated cleanup step fails', async () => {
|
||||
mockGetNodes.mockReturnValue([]);
|
||||
mockGetStackAlerts.mockReturnValue([]);
|
||||
mockGetGlobalSettings.mockReturnValue({});
|
||||
// The metrics/notifications/audit/scan block is one try/catch. A failure
|
||||
// there must not disable the only bound on the insert-only GitOps tables,
|
||||
// so the history prune has its own.
|
||||
mockCleanupOldMetrics.mockImplementationOnce(() => {
|
||||
throw new Error('metrics cleanup failed');
|
||||
});
|
||||
|
||||
const svc = MonitorService.getInstance();
|
||||
await (svc as any).evaluate();
|
||||
|
||||
expect(mockCleanupOldGitOpsHistory).toHaveBeenCalledWith(45);
|
||||
});
|
||||
|
||||
it('finishes the evaluation when the GitOps history prune throws', async () => {
|
||||
mockGetNodes.mockReturnValue([]);
|
||||
mockGetStackAlerts.mockReturnValue([]);
|
||||
mockGetGlobalSettings.mockReturnValue({});
|
||||
mockCleanupOldGitOpsHistory.mockImplementationOnce(() => {
|
||||
throw new Error('gitops history prune failed');
|
||||
});
|
||||
const svc = MonitorService.getInstance();
|
||||
const sweep = vi.spyOn(svc as unknown as { sweepStaleUpdateTrackers: () => void }, 'sweepStaleUpdateTrackers')
|
||||
.mockImplementation(() => {});
|
||||
try {
|
||||
await (svc as unknown as { evaluate: () => Promise<void> }).evaluate();
|
||||
expect(mockCleanupOldGitOpsHistory).toHaveBeenCalledWith(45);
|
||||
expect(sweep).toHaveBeenCalled();
|
||||
expect((svc as unknown as { isProcessing: boolean }).isProcessing).toBe(false);
|
||||
} finally {
|
||||
sweep.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it('skips the GitOps history prune when the retention read throws', async () => {
|
||||
mockGetNodes.mockReturnValue([]);
|
||||
mockGetStackAlerts.mockReturnValue([]);
|
||||
mockGetGlobalSettings.mockReturnValue({});
|
||||
mockGetGitOpsHistoryRetentionDays.mockImplementationOnce(() => {
|
||||
throw new Error('settings unavailable');
|
||||
});
|
||||
const error = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
const svc = MonitorService.getInstance();
|
||||
const sweep = vi.spyOn(svc as unknown as { sweepStaleUpdateTrackers: () => void }, 'sweepStaleUpdateTrackers')
|
||||
.mockImplementation(() => {});
|
||||
try {
|
||||
await (svc as unknown as { evaluate: () => Promise<void> }).evaluate();
|
||||
expect(mockCleanupOldGitOpsHistory).not.toHaveBeenCalled();
|
||||
expect(error.mock.calls.some((call) => String(call[0]).includes('failed to cleanup GitOps history'))).toBe(true);
|
||||
expect(sweep).toHaveBeenCalled();
|
||||
expect((svc as unknown as { isProcessing: boolean }).isProcessing).toBe(false);
|
||||
} finally {
|
||||
sweep.mockRestore();
|
||||
error.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// ── isProcessing guard ─────────────────────────────────────────────────
|
||||
|
||||
@@ -141,6 +141,36 @@ describe('PATCH /api/settings permission buckets', () => {
|
||||
expect(res.body.code).toBe('PERMISSION_DENIED');
|
||||
});
|
||||
|
||||
it('rejects a node admin setting the GitOps history window', async () => {
|
||||
// The window deletes rows. A node admin holds node:manage and does not hold
|
||||
// system:settings, so this write must be refused.
|
||||
const before = DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days;
|
||||
const patch = await request(app)
|
||||
.patch('/api/settings')
|
||||
.set('Cookie', roleCookie['node-admin']!)
|
||||
.send({ gitops_history_retention_days: 14 });
|
||||
expect(patch.status).toBe(403);
|
||||
expect(patch.body.code).toBe('PERMISSION_DENIED');
|
||||
|
||||
const post = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', roleCookie['node-admin']!)
|
||||
.send({ key: 'gitops_history_retention_days', value: '14' });
|
||||
expect(post.status).toBe(403);
|
||||
expect(post.body.code).toBe('PERMISSION_DENIED');
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).toBe(before);
|
||||
});
|
||||
|
||||
it('lets an admin set the GitOps history window', async () => {
|
||||
const res = await request(app)
|
||||
.patch('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ gitops_history_retention_days: 14 });
|
||||
expect(res.status).toBe(200);
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).toBe('14');
|
||||
DatabaseService.getInstance().updateGlobalSetting('gitops_history_retention_days', '30');
|
||||
});
|
||||
|
||||
it('honors node-scoped node-admin grants for node:manage writes', async () => {
|
||||
const db = DatabaseService.getInstance();
|
||||
const defaultNodeId = db.getDefaultNode()!.id!;
|
||||
|
||||
@@ -241,6 +241,87 @@ describe('prune_on_update (auto-prune after updates)', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('gitops_history_retention_days (GitOps history and outbox retention)', () => {
|
||||
it('defaults to 30 days in a freshly seeded database', () => {
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).toBe('30');
|
||||
});
|
||||
|
||||
it('is exposed through the settings GET projection', async () => {
|
||||
const res = await request(app).get('/api/settings').set('Cookie', adminCookie);
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.gitops_history_retention_days).toBe('30');
|
||||
});
|
||||
|
||||
it('accepts a well-formed write and persists it', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'gitops_history_retention_days', value: '90' });
|
||||
try {
|
||||
expect(res.status).toBe(200);
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).toBe('90');
|
||||
} finally {
|
||||
DatabaseService.getInstance().updateGlobalSetting('gitops_history_retention_days', '30');
|
||||
}
|
||||
});
|
||||
|
||||
it('rejects an out-of-range value (400) and does not write it', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'gitops_history_retention_days', value: '366' });
|
||||
expect(res.status).toBe(400);
|
||||
expect(res.body.error).toBe('Validation failed');
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).not.toBe('366');
|
||||
});
|
||||
|
||||
it('rejects a non-numeric value (400) and does not write it', async () => {
|
||||
const res = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'gitops_history_retention_days', value: 'banana' });
|
||||
expect(res.status).toBe(400);
|
||||
expect(res.body.error).toBe('Validation failed');
|
||||
});
|
||||
|
||||
it('accepts the bounds the settings page saves', async () => {
|
||||
// The page PATCHes the stored string. 1 and 365 are the schema edges.
|
||||
// Restore in a finally: a failed assertion would otherwise leave the
|
||||
// shared test database on the last value this test wrote.
|
||||
try {
|
||||
for (const value of ['1', '365']) {
|
||||
const res = await request(app)
|
||||
.patch('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ gitops_history_retention_days: value });
|
||||
expect(res.status, value).toBe(200);
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).toBe(value);
|
||||
}
|
||||
} finally {
|
||||
DatabaseService.getInstance().updateGlobalSetting('gitops_history_retention_days', '30');
|
||||
}
|
||||
});
|
||||
|
||||
it('rejects an off value, a value past the ceiling, and a non-numeric value on PATCH', async () => {
|
||||
// 0 is not an off switch. The page save is a PATCH, so a POST-only check
|
||||
// would miss a schema that accepted it there.
|
||||
for (const value of [0, 366, 'banana']) {
|
||||
const res = await request(app)
|
||||
.patch('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ gitops_history_retention_days: value });
|
||||
expect(res.status, String(value)).toBe(400);
|
||||
expect(res.body.error).toBe('Validation failed');
|
||||
}
|
||||
const postZero = await request(app)
|
||||
.post('/api/settings')
|
||||
.set('Cookie', adminCookie)
|
||||
.send({ key: 'gitops_history_retention_days', value: 0 });
|
||||
expect(postZero.status).toBe(400);
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().gitops_history_retention_days).toBe('30');
|
||||
});
|
||||
});
|
||||
|
||||
describe('recovery_retention_days (superseded rollback generation retention)', () => {
|
||||
it('defaults to 7 days in a freshly seeded database', () => {
|
||||
expect(DatabaseService.getInstance().getGlobalSettings().recovery_retention_days).toBe('7');
|
||||
|
||||
@@ -30,6 +30,11 @@ export const SETTING_WRITE_PERMISSIONS: Record<string, PermissionAction> = {
|
||||
// manages, so one value governs the fleet. The sibling node-scoped keys keep
|
||||
// `node:manage` because each governs the node it is stored against.
|
||||
gitops_artifact_retry_interval_mins: 'system:settings',
|
||||
// `system:settings` for the same reason as the retry interval above: the
|
||||
// history being pruned is the one this instance holds, and the monitor that
|
||||
// prunes it runs here. A value written to a remote node's own database would
|
||||
// govern that node's history, not this instance's.
|
||||
gitops_history_retention_days: 'system:settings',
|
||||
env_block_deploy_on_missing_required: 'node:manage',
|
||||
auto_create_missing_external_networks: 'node:manage',
|
||||
notification_dispatch_retries: 'node:manage',
|
||||
@@ -120,6 +125,7 @@ const SettingsPatchSchema = z.object({
|
||||
health_gate_enabled: z.enum(['0', '1']),
|
||||
health_gate_window_seconds: z.coerce.number().int().min(15).max(600).transform(String),
|
||||
gitops_artifact_retry_interval_mins: z.coerce.number().int().min(1).max(1440).transform(String),
|
||||
gitops_history_retention_days: z.coerce.number().int().min(1).max(365).transform(String),
|
||||
env_block_deploy_on_missing_required: z.enum(['0', '1']),
|
||||
auto_create_missing_external_networks: z.enum(['0', '1']),
|
||||
image_update_sidebar_indicators: z.enum(['0', '1']),
|
||||
|
||||
@@ -89,6 +89,79 @@ const DEFAULT_RECOVERY_MAX_GENERATIONS = 0;
|
||||
// value from parking a target unverified for longer than a day.
|
||||
const DEFAULT_ARTIFACT_RETRY_INTERVAL_MINS = 5;
|
||||
const MAX_ARTIFACT_RETRY_INTERVAL_MINS = 1440;
|
||||
// Bounds of gitops_history_retention_days. The window is what keeps the
|
||||
// insert-only history and outbox tables from growing for the life of an
|
||||
// installation; the ceiling keeps a stored value from retaining rows longer
|
||||
// than a year.
|
||||
const DEFAULT_GITOPS_HISTORY_RETENTION_DAYS = 30;
|
||||
const MAX_GITOPS_HISTORY_RETENTION_DAYS = 365;
|
||||
// How long past the history window an undrained outbox row is kept. The repair
|
||||
// that drains these rows from their payload runs at startup, so a restart
|
||||
// inside this grace can still announce them. A process that stays up does not
|
||||
// retry them. A row still undrained after the grace is dropped so it cannot
|
||||
// accumulate for ever.
|
||||
const UNDRAINED_OUTBOX_GRACE_DAYS = 7;
|
||||
// History and outbox rows deleted per prune pass. A first prune on a long-lived
|
||||
// install has months of backlog to clear and SQLite has one writer, so an
|
||||
// unbounded transaction would block every other write on the instance for its
|
||||
// duration. The pass runs at the end of each monitor evaluation (every 30
|
||||
// seconds), not on the 15-minute janitor, and clears the rest on later passes.
|
||||
export const GITOPS_PRUNE_CHUNK_ROWS = 5000;
|
||||
// What a prune may delete, in one place so the delete statement and the
|
||||
// remaining-count, when developer mode runs it, can never disagree.
|
||||
//
|
||||
// A row older than the window is pruned unless it is the newest row of a live
|
||||
// application: only a live application has a "last activity" fact to preserve.
|
||||
//
|
||||
// The portfolio's "last activity" fact reads `MAX(created_at)` over history
|
||||
// (`latestTransitionByApplication`), and its readers select only active
|
||||
// applications. Deleting an idle application's last row would silently empty its
|
||||
// activity column with no change on the application itself, since polling off, a
|
||||
// suspended source, a detached node and an idle Blueprint are indistinguishable
|
||||
// from "no activity" once the rows are gone.
|
||||
//
|
||||
// Ties on created_at are broken by rowid, SQLite's insertion-order key. The id
|
||||
// column is a random UUID and does not sort by recency (`latestSettledAttempt`
|
||||
// already orders by rowid for that reason). One attempt's started and settled
|
||||
// rows share the same envelope timestamp, so a UUID tie-break can keep the
|
||||
// started row and delete the settled one. Startup recovery then treats that
|
||||
// attempt as open and re-settles it. rowid keeps the later insert, which is
|
||||
// the settlement. The chunk delete orders the same way, so a pass that stops
|
||||
// between the two rows leaves the settlement in place.
|
||||
//
|
||||
// Scoped to live applications rather than to every application_id, because the
|
||||
// tombstone is the only thing that outlives a stack: `gitops_applications` keeps a
|
||||
// deleted or detached row as a frozen record forever, so an unscoped exemption
|
||||
// would retain one history row per stack ever retired and leave exactly the
|
||||
// growth this setting exists to stop. 'creating' counts as live because its rows
|
||||
// are written before the application is activated, and dropping them would leave
|
||||
// an application that goes live with no recorded activity at all. The application
|
||||
// lookup is a primary-key seek. The newer-row probe uses
|
||||
// idx_gitops_history_app_created: one seek for a later timestamp and one for
|
||||
// the same timestamp, with the rowid tie applied on that second seek. A
|
||||
// retired application stops after the primary-key seek.
|
||||
//
|
||||
// An unsettled reservation older than the window is pruned once a newer row
|
||||
// exists for that application. Startup recovery then does not see it. The
|
||||
// reservation survives past the window only when it is still that application's
|
||||
// newest row, which is the quiet-application case. Keeping every unsettled row
|
||||
// would retain attempts the store documents as staying unsettled forever.
|
||||
const GITOPS_HISTORY_PRUNEABLE = `created_at < ?
|
||||
AND (
|
||||
-- The application is retired, so no projection reads its history.
|
||||
NOT EXISTS (
|
||||
SELECT 1 FROM gitops_applications application
|
||||
WHERE application.id = gitops_history.application_id
|
||||
AND application.lifecycle_status IN ('active','creating')
|
||||
)
|
||||
-- Or this row is not the newest one, for any application.
|
||||
OR EXISTS (
|
||||
SELECT 1 FROM gitops_history newer
|
||||
WHERE newer.application_id = gitops_history.application_id
|
||||
AND (newer.created_at > gitops_history.created_at
|
||||
OR (newer.created_at = gitops_history.created_at AND newer.rowid > gitops_history.rowid))
|
||||
)
|
||||
)`;
|
||||
|
||||
function isStackServiceStatus(value: unknown): value is StackServiceStatus {
|
||||
if (!value || typeof value !== 'object') return false;
|
||||
@@ -2348,6 +2421,11 @@ stmt.run('gitops_schema_version', '1');
|
||||
// reachable floor. Kept off the generic read path on purpose: it is only
|
||||
// consulted for a target whose approved identity is still unresolved.
|
||||
stmt.run('gitops_artifact_retry_interval_mins', '5');
|
||||
// Days of GitOps history and settled-outbox rows kept before the
|
||||
// monitor's cleanup pass prunes them. Both tables are insert-only and
|
||||
// the reconcile tick appends to them continuously, so the window is the
|
||||
// only bound on their size; it defaults on rather than off.
|
||||
stmt.run('gitops_history_retention_days', '30');
|
||||
// SSO role sync defaults off: admin-set roles persist across SSO sign-ins;
|
||||
// operators who want IdP group membership to drive roles opt in via Settings > SSO.
|
||||
stmt.run('sso_role_sync', '0');
|
||||
@@ -5049,6 +5127,27 @@ stmt.run('gitops_schema_version', '1');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Days of GitOps history to keep.
|
||||
*
|
||||
* History is insert-only and the reconcile tick appends to it continuously,
|
||||
* so this is the only bound on the table's size. There is no off value:
|
||||
* turning retention off would leave the history and its outbox growing for
|
||||
* the life of the installation, which is the accumulation this setting
|
||||
* exists to stop. A missing or unusable stored value uses the default. A
|
||||
* failed read throws, so the caller can skip the delete instead of pruning
|
||||
* with a window the operator did not choose.
|
||||
*/
|
||||
public getGitOpsHistoryRetentionDays(): number {
|
||||
const stored = this.getGlobalSettings()['gitops_history_retention_days'];
|
||||
const raw = parseInt(stored ?? '', 10);
|
||||
if (!Number.isFinite(raw) || raw < 1) {
|
||||
console.warn(`[DatabaseService] invalid gitops_history_retention_days "${String(stored)}"; treating as ${DEFAULT_GITOPS_HISTORY_RETENTION_DAYS}`);
|
||||
return DEFAULT_GITOPS_HISTORY_RETENTION_DAYS;
|
||||
}
|
||||
return Math.min(raw, MAX_GITOPS_HISTORY_RETENTION_DAYS);
|
||||
}
|
||||
|
||||
/** Total generations retained per stack, current included (0 = unlimited). */
|
||||
public getRecoveryMaxGenerations(): number {
|
||||
try {
|
||||
@@ -5561,6 +5660,110 @@ stmt.run('gitops_schema_version', '1');
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop one bounded chunk of GitOps history and its consumed outbox rows
|
||||
* older than the window.
|
||||
*
|
||||
* What a prune may delete, and why each live application's newest row is exempt
|
||||
* from it, is documented on `GITOPS_HISTORY_PRUNEABLE`.
|
||||
* `historyRemaining` and `outboxRemaining` are exact only while developer mode
|
||||
* is on. The diagnostic log is the only reader, and the counts walk the
|
||||
* backlog, so they are skipped otherwise and reported as `null`: an
|
||||
* unmeasured remainder is unknown, not zero.
|
||||
*
|
||||
* A drained outbox row is pruned with the history it announced. An undrained
|
||||
* row is a notification the fanout has not written yet, or a payload a later
|
||||
* decoder is meant to repair, so it gets a grace period past the window. The
|
||||
* repair runs at startup, so the grace only helps an instance that restarts
|
||||
* in time. A row still undrained after the grace is dropped, and the warning
|
||||
* reports how many this pass deleted, because one pass is capped and keeping
|
||||
* the row for ever is the unbounded growth the window exists to stop.
|
||||
*/
|
||||
public cleanupOldGitOpsHistory(
|
||||
daysToKeep = DEFAULT_GITOPS_HISTORY_RETENTION_DAYS,
|
||||
): { history: number; outbox: number; historyRemaining: number | null; outboxRemaining: number | null } {
|
||||
const days = Number.isFinite(daysToKeep) && daysToKeep > 0
|
||||
// Floor before the guard, not after it: a fraction between 0 and 1
|
||||
// floors to 0, which is not an off value and would delete everything
|
||||
// non-exempt.
|
||||
? Math.min(Math.max(Math.floor(daysToKeep), 1), MAX_GITOPS_HISTORY_RETENTION_DAYS)
|
||||
: DEFAULT_GITOPS_HISTORY_RETENTION_DAYS;
|
||||
const cutoff = Date.now() - (days * 24 * 60 * 60 * 1000);
|
||||
const undrainedCutoff = cutoff - (UNDRAINED_OUTBOX_GRACE_DAYS * 24 * 60 * 60 * 1000);
|
||||
|
||||
const deleteHistoryChunk = this.db.prepare(
|
||||
`DELETE FROM gitops_history
|
||||
WHERE id IN (
|
||||
SELECT id FROM gitops_history WHERE ${GITOPS_HISTORY_PRUNEABLE}
|
||||
ORDER BY created_at ASC, rowid ASC
|
||||
LIMIT ?
|
||||
)`,
|
||||
);
|
||||
// Two statements rather than one OR, so each branch keeps its own row
|
||||
// cap. One statement with both predicates and one LIMIT would still use
|
||||
// the two partial indexes, and it would cap the combined delete. Both
|
||||
// branches are bounded by the same row cap, so a first prune with
|
||||
// months of outbox backlog clears it over later passes rather than
|
||||
// holding the writer for all of it.
|
||||
const deleteDrained = this.db.prepare(
|
||||
`DELETE FROM gitops_settled_outbox
|
||||
WHERE settled_history_id IN (
|
||||
SELECT settled_history_id FROM gitops_settled_outbox
|
||||
WHERE drained_at IS NOT NULL AND created_at < ?
|
||||
LIMIT ?
|
||||
)`,
|
||||
);
|
||||
const deleteUndrained = this.db.prepare(
|
||||
`DELETE FROM gitops_settled_outbox
|
||||
WHERE settled_history_id IN (
|
||||
SELECT settled_history_id FROM gitops_settled_outbox
|
||||
WHERE drained_at IS NULL AND created_at < ?
|
||||
LIMIT ?
|
||||
)`,
|
||||
);
|
||||
// One transaction per pass, so a long backlog is cleared over several
|
||||
// passes rather than holding SQLite's single writer for all of it.
|
||||
const pruned = this.db.transaction(() => {
|
||||
const deletedHistory = deleteHistoryChunk.run(cutoff, GITOPS_PRUNE_CHUNK_ROWS);
|
||||
const deletedDrained = deleteDrained.run(cutoff, GITOPS_PRUNE_CHUNK_ROWS);
|
||||
const deletedUndrained = deleteUndrained.run(undrainedCutoff, GITOPS_PRUNE_CHUNK_ROWS);
|
||||
return {
|
||||
history: Number(deletedHistory.changes ?? 0),
|
||||
outbox: Number(deletedDrained.changes ?? 0) + Number(deletedUndrained.changes ?? 0),
|
||||
undrained: Number(deletedUndrained.changes ?? 0),
|
||||
};
|
||||
})();
|
||||
// Counted after the delete, and only for the diagnostic log. The history
|
||||
// count walks whatever backlog is still left, and the monitor tick runs
|
||||
// this every 30 seconds. Null rather than 0 when they are skipped, so the
|
||||
// log cannot read "none left" for a table that was never measured.
|
||||
let historyRemaining: number | null = null;
|
||||
let outboxRemaining: number | null = null;
|
||||
if (this.getGlobalSettings().developer_mode === '1') {
|
||||
historyRemaining = (this.db.prepare(
|
||||
`SELECT COUNT(*) AS n FROM gitops_history WHERE ${GITOPS_HISTORY_PRUNEABLE}`,
|
||||
).get(cutoff) as { n: number }).n;
|
||||
outboxRemaining = (this.db.prepare(
|
||||
`SELECT COUNT(*) AS n FROM gitops_settled_outbox
|
||||
WHERE (drained_at IS NOT NULL AND created_at < ?)
|
||||
OR (drained_at IS NULL AND created_at < ?)`,
|
||||
).get(cutoff, undrainedCutoff) as { n: number }).n;
|
||||
}
|
||||
|
||||
if (pruned.undrained > 0) {
|
||||
console.warn(
|
||||
`[DatabaseService] dropping ${pruned.undrained} undrained GitOps outbox row(s) older than ${days + UNDRAINED_OUTBOX_GRACE_DAYS} days`,
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
history: pruned.history,
|
||||
outbox: pruned.outbox,
|
||||
historyRemaining,
|
||||
outboxRemaining,
|
||||
};
|
||||
}
|
||||
|
||||
// --- Nodes ---
|
||||
|
||||
private decryptNodeRow(row: any): Node {
|
||||
|
||||
@@ -35,6 +35,7 @@ const SAFE_SETTING_KEYS = [
|
||||
'metrics_retention_hours',
|
||||
'log_retention_days',
|
||||
'audit_retention_days',
|
||||
'gitops_history_retention_days',
|
||||
'mesh_auto_recreate',
|
||||
'scan_history_per_image_limit',
|
||||
'cloud_backup_provider',
|
||||
|
||||
@@ -855,6 +855,28 @@ export class MonitorService {
|
||||
} catch (e) {
|
||||
console.error('MonitorService: failed to cleanup old data', e);
|
||||
}
|
||||
|
||||
// Separate from the metrics and notification cleanup above, so a failure
|
||||
// there does not skip this prune. A throw from the global-settings
|
||||
// evaluation earlier in this cycle does skip it until the next
|
||||
// evaluation, because that cycle never reaches this pass. A hang in the
|
||||
// Docker enumeration above holds this evaluation, so later ticks do not
|
||||
// run the prune until that call returns. A failed read of the window
|
||||
// throws out of the getter, and this catch logs it without deleting.
|
||||
try {
|
||||
const gitopsHistoryRetentionDays = db.getGitOpsHistoryRetentionDays();
|
||||
const gitopsHistoryPruned = db.cleanupOldGitOpsHistory(gitopsHistoryRetentionDays);
|
||||
if (isDebugEnabled()) {
|
||||
// Null remainder means the counts were skipped (developer mode
|
||||
// off), which is unknown rather than empty.
|
||||
const remaining = gitopsHistoryPruned.historyRemaining === null
|
||||
? 'not measured'
|
||||
: `${gitopsHistoryPruned.historyRemaining}+${gitopsHistoryPruned.outboxRemaining}`;
|
||||
console.log(`[Monitor:diag] Cleanup: gitops history ${gitopsHistoryRetentionDays}d (history=${gitopsHistoryPruned.history} outbox=${gitopsHistoryPruned.outbox} left=${remaining})`);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('MonitorService: failed to cleanup GitOps history', e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -645,6 +645,11 @@ CREATE TABLE IF NOT EXISTS gitops_settled_outbox (
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_settled_outbox_undrained
|
||||
ON gitops_settled_outbox(created_at)
|
||||
WHERE drained_at IS NULL;
|
||||
-- Drained rows are the steady-state majority. The prune deletes them by age on
|
||||
-- the monitor tick, and the undrained partial index above cannot serve that scan.
|
||||
CREATE INDEX IF NOT EXISTS idx_gitops_settled_outbox_drained
|
||||
ON gitops_settled_outbox(created_at)
|
||||
WHERE drained_at IS NOT NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS git_provider_endpoints (
|
||||
id TEXT PRIMARY KEY,
|
||||
|
||||
@@ -31,6 +31,8 @@ The application list shows, per application:
|
||||
|
||||
Rows with pending or failed items are tinted so the list reads as a signal, not a spreadsheet; the reasons are in the **Attention required** list above it (and, for a row beyond the list's cap, on hovering the state dot and in the application's own sheet). Statuses that arrived from a node running a newer Sencho build are shown with their raw state and flagged as unknown evidence rather than being reinterpreted as something else.
|
||||
|
||||
Each row's activity timestamp comes from Sencho's GitOps history, the record of every source attempt, authority decision, and rollout change. Sencho keeps that record for 30 days by default and prunes older entries on a cleanup pass; **Keep GitOps history for** under **Settings → GitOps** changes the window from 1 to 365 days. The newest entry for each live application is kept, so a row's activity stays readable however short the window is. History views show the entries inside the window, and that kept entry when it is older. An attempt that never finished is kept past the window only when it is that newest entry. Once a newer entry exists, the older attempt is pruned with the rest, and a later restart does not reopen it. That record is also what tells a redelivered webhook apart from a new one, so a redelivery can run the work again once the record of its first delivery is pruned; a delivery to a provider endpoint keeps its own record and is unaffected. Everything else the application shows, including the current state, the authority in force, and any running rollout, is held outside this record, so a shorter window does not change what an application is running.
|
||||
|
||||
## The attention queue
|
||||
|
||||
Above the list, the **Attention required** section lists everything currently waiting: failed reconciliations, pending reviews or authorizations, preflight blocks like an unready private registry, partial or paused rollouts, drift, stale or unreachable targets, and recovery states. Each application appears once, with its most urgent reason in a line of its own (so it is readable without hovering anything) and a count of any others. Each entry carries the step that resolves it, and the step happens where you are: **Retry** for a source that failed to fetch and **Resume** for a suspended one run right there, **Review update** opens that stack's Git Source sheet over the portfolio, pulled and showing the diff when an update is waiting, so you can apply it without leaving GitOps, **Open stack** takes you to a failed deploy or health check on a stack, and **Review** opens a hub-local Git-managed Blueprint's own sheet, where the accept, approve, and authorize actions live. A node-scoped remote Blueprint offers **Inspect** instead. When your account cannot retry or resume a source, the entry opens the Git Source sheet instead of offering a button that would be refused.
|
||||
|
||||
@@ -16,7 +16,7 @@ Open the Settings Hub by clicking the **Profile** icon in the top bar and select
|
||||
|-------|----------------|
|
||||
| **Personal** | Account, Appearance |
|
||||
| **Access** | Admiral Account, Users, SSO, API Tokens |
|
||||
| **Infrastructure** | Nodes, Stacks, Fleet, Registries, Recovery Vault, App Store |
|
||||
| **Infrastructure** | Nodes, Stacks, GitOps, Fleet, Registries, Recovery Vault, App Store |
|
||||
| **Monitoring** | Host Alerts, Container Alerts, Docker & Storage |
|
||||
| **Notifications** | Channels, Notification Routing, Mute Rules |
|
||||
| **Automation** | Image update checks, Webhooks |
|
||||
@@ -391,6 +391,49 @@ See [Fleet Snapshots & Backups](/features/fleet-backups) for restore workflows.
|
||||
|
||||
---
|
||||
|
||||
## GitOps
|
||||
|
||||
How Git sources are fetched, how the images a Blueprint target runs are verified, and how long the GitOps history is kept. The section mixes scopes: **Git sources** follows the active node, while the other two groups belong to the Sencho instance serving it.
|
||||
|
||||
### Git sources
|
||||
|
||||
**Scope:** Per-node
|
||||
|
||||
When on, this node fetches linked Git sources on a schedule. Off means polling is off (interval 0); sources still pull on demand, on webhook, and on retry. Linked sources with no override inherit this interval. Off by default.
|
||||
|
||||
| Setting | Default | Max | Description |
|
||||
|---------|---------|-----|-------------|
|
||||
| **Poll Git sources** | Off | - | Master switch for scheduled fetches on this node. |
|
||||
| **Poll interval** | - | 10,080 min (1 week) | Minutes between unattended fetches, shown while polling is on. |
|
||||
|
||||
### Drift verification
|
||||
|
||||
**Scope:** Per instance
|
||||
|
||||
How Sencho proves which images a Blueprint target is actually running.
|
||||
|
||||
| Setting | Default | Max | Description |
|
||||
|---------|---------|-----|-------------|
|
||||
| **Retry an unresolved image identity** | 5 min | 1,440 min | How often a drift check retries resolving the exact image identity a Blueprint target was approved at, when the first attempt could not reach the registry. Lower it for faster recovery once a registry comes back, raise it to keep Sencho off a rate-limited or private registry. Covers the Blueprint targets this Sencho manages, whichever node each one runs on. It has no effect while every target's identity is already resolved. |
|
||||
|
||||
### History
|
||||
|
||||
**Scope:** Per instance
|
||||
|
||||
How long the record behind the GitOps timeline is kept.
|
||||
|
||||
| Setting | Default | Max | Description |
|
||||
|---------|---------|-----|-------------|
|
||||
| **Keep GitOps history for** | 30 days | 365 | How long to keep the record behind the GitOps timeline before older entries are pruned. Lower it to stop a busy Sencho's history from growing, raise it to keep a longer record. The newest entry for each live application is always kept, so its activity stays readable. A webhook redelivered after the record of its first delivery is pruned can run again; a delivery to a provider endpoint keeps its own record and is unaffected. Bell entries for those changes follow the **Notification log** window in [Data Retention](#data-retention) instead. |
|
||||
|
||||
The window belongs to the Sencho instance, not to a node: each instance prunes the history it holds, so the window set here governs this instance and a remote instance's window is set on that instance. Pruning deletes entries older than the window; it does not shrink the database file.
|
||||
|
||||
The [GitOps portfolio](/features/gitops-portfolio) page explains what a cleanup pass keeps.
|
||||
|
||||
Click **Save settings** to apply.
|
||||
|
||||
---
|
||||
|
||||
## Nodes
|
||||
|
||||
**Scope:** Global
|
||||
@@ -579,6 +622,8 @@ How long Sencho keeps historical data on this node before pruning it.
|
||||
|
||||
Removing scans for deleted artifacts runs in the background a few minutes after an image or stack disappears, and immediately when you delete a stack. An image that is still present on the node keeps its scan results, even when no stack uses it.
|
||||
|
||||
The GitOps history window is a retention window too, but it belongs to this Sencho instance rather than to a node, so it is set under **Settings → GitOps** and is listed on that section's page.
|
||||
|
||||
Click **Save settings** to apply.
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* The GitOps history window, end to end against a running Sencho.
|
||||
*
|
||||
* The unit tests drive the prune against seeded rows. This one proves the
|
||||
* setting an operator would hit: a fresh install keeps 30 days, a value in
|
||||
* range persists, and a value outside 1 to 365 is refused.
|
||||
*
|
||||
* No Docker needed: settings API traffic only, with no deploy involved.
|
||||
*/
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { loginAs } from './helpers';
|
||||
|
||||
async function jsonRequest<T>(
|
||||
page: import('@playwright/test').Page,
|
||||
path: string,
|
||||
init?: { method?: string; body?: unknown },
|
||||
): Promise<{ status: number; body: T }> {
|
||||
const result = await page.evaluate(
|
||||
async ([url, method, payload]) => {
|
||||
const response = await fetch(url, {
|
||||
method: method ?? 'GET',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
...(payload === null ? {} : { body: JSON.stringify(payload) }),
|
||||
});
|
||||
const text = await response.text();
|
||||
return { status: response.status, text };
|
||||
},
|
||||
[path, init?.method ?? 'GET', init?.body ?? null] as const,
|
||||
);
|
||||
return { status: result.status, body: (result.text ? JSON.parse(result.text) : null) as T };
|
||||
}
|
||||
|
||||
interface SettingsRow {
|
||||
gitops_history_retention_days?: string;
|
||||
}
|
||||
|
||||
test.describe('GitOps history retention', () => {
|
||||
test('the history window round-trips and is refused outside its bounds', async ({ page }) => {
|
||||
await loginAs(page);
|
||||
|
||||
const seeded = await jsonRequest<SettingsRow>(page, '/api/settings');
|
||||
expect(seeded.status).toBe(200);
|
||||
expect(
|
||||
seeded.body.gitops_history_retention_days,
|
||||
'a fresh install keeps 30 days of GitOps history',
|
||||
).toBe('30');
|
||||
|
||||
const saved = await jsonRequest<{ error?: string }>(page, '/api/settings', {
|
||||
method: 'PATCH',
|
||||
body: { gitops_history_retention_days: 14 },
|
||||
});
|
||||
expect(saved.status, `saving the window: ${JSON.stringify(saved.body)}`).toBe(200);
|
||||
|
||||
// Everything below reads the window back after mutating it, so the restore
|
||||
// belongs in a finally. A failed assertion has to leave the instance on the
|
||||
// seeded 30: the retry of a failed run (CI runs with one) and every later
|
||||
// spec would otherwise start from whatever this run left behind.
|
||||
try {
|
||||
await expect
|
||||
.poll(async () => (await jsonRequest<SettingsRow>(page, '/api/settings')).body.gitops_history_retention_days)
|
||||
.toBe('14');
|
||||
|
||||
for (const value of [0, 366]) {
|
||||
const rejected = await jsonRequest<{ error?: string }>(page, '/api/settings', {
|
||||
method: 'PATCH',
|
||||
body: { gitops_history_retention_days: value },
|
||||
});
|
||||
expect(rejected.status, `value ${value} must be refused`).toBe(400);
|
||||
}
|
||||
} finally {
|
||||
await jsonRequest(page, '/api/settings', {
|
||||
method: 'PATCH',
|
||||
body: { gitops_history_retention_days: 30 },
|
||||
});
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -20,10 +20,11 @@ interface GitOpsSectionProps {
|
||||
onDirtyChange?: (dirty: boolean) => void;
|
||||
}
|
||||
|
||||
type GitOpsFields = Pick<PatchableSettings, 'gitops_artifact_retry_interval_mins'>;
|
||||
type GitOpsFields = Pick<PatchableSettings, 'gitops_artifact_retry_interval_mins' | 'gitops_history_retention_days'>;
|
||||
|
||||
const DEFAULT_FIELDS: GitOpsFields = {
|
||||
gitops_artifact_retry_interval_mins: DEFAULT_SETTINGS.gitops_artifact_retry_interval_mins,
|
||||
gitops_history_retention_days: DEFAULT_SETTINGS.gitops_history_retention_days,
|
||||
};
|
||||
|
||||
function RetrySkeleton() {
|
||||
@@ -37,31 +38,34 @@ function RetrySkeleton() {
|
||||
/**
|
||||
* How this instance behaves on its own schedule.
|
||||
*
|
||||
* The two controls here answer different questions and neither implies the
|
||||
* other. Git polling is about fetching new intent on a cadence the operator
|
||||
* opts into, and it is off by default because an unattended fetch writes to a
|
||||
* repository. The artifact retry interval is about how fast Sencho proves what
|
||||
* is already running, and it is on by default because leaving it unset would
|
||||
* hold a target's approved image identity unresolved until something unrelated
|
||||
* happened to redeploy it.
|
||||
* The controls here answer different questions and none implies another. Git
|
||||
* polling is about fetching new intent on a cadence the operator opts into, and
|
||||
* it is off by default because an unattended fetch writes to a repository. The
|
||||
* artifact retry interval is about how fast Sencho proves what is already
|
||||
* running, and it is on by default because leaving it unset would hold a
|
||||
* target's approved image identity unresolved until something unrelated
|
||||
* happened to redeploy it. The history window is about how long the record of
|
||||
* all of it is kept, and it is on by default because the tables it prunes are
|
||||
* insert-only.
|
||||
*
|
||||
* They share a section because both configure background work and an operator
|
||||
* adjusting one usually wants to see the other. They are saved differently on
|
||||
* purpose: the poll interval has to reschedule live fetchers, so it owns an
|
||||
* endpoint and is node-scoped, while the retry interval is a value read at the
|
||||
* moment a check needs it, so it rides the shared settings and is read from this
|
||||
* instance rather than from the selected node.
|
||||
* They share a section because all three configure background work and an
|
||||
* operator adjusting one usually wants to see the others. They are saved
|
||||
* differently on purpose: the poll interval has to reschedule live fetchers, so
|
||||
* it owns an endpoint and is node-scoped, while the retry interval and the
|
||||
* history window are values read at the moment they are needed, so they ride
|
||||
* the shared settings and are read from this instance rather than from the
|
||||
* selected node.
|
||||
*/
|
||||
export function GitOpsSection({ onDirtyChange }: GitOpsSectionProps) {
|
||||
const { can } = useAuth();
|
||||
// The two controls gate differently, because their permissions differ, and
|
||||
// only the retry interval is wrapped in a disabled fieldset. The poll
|
||||
// interval is node-scoped and the backend requires node:manage against the
|
||||
// active node, which `GitPollingControl` checks for itself; the retry
|
||||
// interval is instance-scoped and requires system:settings, since one value
|
||||
// governs every node this instance manages.
|
||||
const canEditRetry = can('system:settings');
|
||||
const readOnly = !canEditRetry;
|
||||
// The three controls gate differently, because their permissions differ,
|
||||
// and only the retry interval and history window are wrapped in a disabled
|
||||
// fieldset. The poll interval is node-scoped and the backend requires
|
||||
// node:manage against the active node, which `GitPollingControl` checks for
|
||||
// itself; the other two are instance-scoped and require system:settings,
|
||||
// since one value governs every node this instance manages.
|
||||
const canEditInstanceSettings = can('system:settings');
|
||||
const readOnly = !canEditInstanceSettings;
|
||||
const { settings, setSettings, dirtyCount, hasChanges, reset, markSaved } = useSettingsDirty<GitOpsFields>({ ...DEFAULT_FIELDS });
|
||||
const [phase, setPhase] = useState<LoadPhase>('loading');
|
||||
const [isSaving, setIsSaving] = useState(false);
|
||||
@@ -108,6 +112,10 @@ export function GitOpsSection({ onDirtyChange }: GitOpsSectionProps) {
|
||||
typeof body.gitops_artifact_retry_interval_mins === 'string'
|
||||
? body.gitops_artifact_retry_interval_mins
|
||||
: DEFAULT_SETTINGS.gitops_artifact_retry_interval_mins,
|
||||
gitops_history_retention_days:
|
||||
typeof body.gitops_history_retention_days === 'string'
|
||||
? body.gitops_history_retention_days
|
||||
: DEFAULT_SETTINGS.gitops_history_retention_days,
|
||||
});
|
||||
setPhase('ready');
|
||||
} catch {
|
||||
@@ -170,6 +178,24 @@ export function GitOpsSection({ onDirtyChange }: GitOpsSectionProps) {
|
||||
</SettingsField>
|
||||
</SettingsSection>
|
||||
|
||||
<SettingsSection title="History" kicker="this Sencho">
|
||||
<p className="pb-2 text-sm leading-relaxed text-stat-subtitle">
|
||||
How long the record behind the GitOps timeline is kept.
|
||||
</p>
|
||||
<SettingsField
|
||||
label="Keep GitOps history for"
|
||||
helper="How long to keep the record behind the GitOps timeline before older entries are pruned. Lower it to stop a busy Sencho's history from growing, raise it to keep a longer record. The newest entry for each live application is always kept, so its activity stays readable. A webhook redelivered after the record of its first delivery is pruned can run again; a delivery to a provider endpoint keeps its own record and is unaffected. Bell entries for those changes follow the Notification log window instead. Default 30 days."
|
||||
>
|
||||
<NumberChip
|
||||
value={settings.gitops_history_retention_days || '30'}
|
||||
onChange={(v) => setSettings(prev => ({ ...prev, gitops_history_retention_days: v }))}
|
||||
suffix="days"
|
||||
min={1}
|
||||
max={365}
|
||||
/>
|
||||
</SettingsField>
|
||||
</SettingsSection>
|
||||
|
||||
<SettingsActions hint={readOnly ? 'Read-only · permission required to edit' : (hasChanges ? `${dirtyCount} unsaved` : undefined)}>
|
||||
{!readOnly && (
|
||||
<SettingsPrimaryButton onClick={save} disabled={isSaving || !hasChanges || phase !== 'ready'}>
|
||||
|
||||
@@ -38,6 +38,13 @@ export function NumberChip({ value, onChange, suffix, min, max, step = 1, warnOv
|
||||
const parsed = Number(trimmed);
|
||||
if (trimmed !== '' && Number.isFinite(parsed)) {
|
||||
let next = parsed;
|
||||
// Snap to the control's own step before clamping. A chip with the
|
||||
// default step of 1 writes an integer, which is what the settings
|
||||
// route validates for every integer key; without this a typed "1.5"
|
||||
// is clamped but not rounded, saved as a fraction, and rejected by
|
||||
// the route with a bare validation error. A chip that asks for a
|
||||
// fractional step (docker janitor, 0.5 GiB) keeps it.
|
||||
next = Math.round(next / step) * step;
|
||||
if (typeof min === 'number') next = Math.max(min, next);
|
||||
if (typeof max === 'number') next = Math.min(max, next);
|
||||
onChange(String(next));
|
||||
@@ -47,6 +54,7 @@ export function NumberChip({ value, onChange, suffix, min, max, step = 1, warnOv
|
||||
|
||||
const numeric = Number(value);
|
||||
const warn = typeof warnOver === 'number' && Number.isFinite(numeric) && numeric > warnOver;
|
||||
const shownSuffix = numeric === 1 && suffix.length > 1 && suffix.endsWith('s') ? suffix.slice(0, -1) : suffix;
|
||||
|
||||
const chipClass = cn(
|
||||
'inline-flex items-baseline gap-1 rounded-md border px-2.5 py-1 font-mono text-sm tabular-nums tracking-tight transition-colors min-w-[78px] justify-end focus-within:ring-2 focus-within:ring-brand/50 focus-within:outline-none',
|
||||
@@ -74,7 +82,7 @@ export function NumberChip({ value, onChange, suffix, min, max, step = 1, warnOv
|
||||
className="w-12 bg-transparent text-right outline-none [&::-webkit-inner-spin-button]:appearance-none [&::-webkit-outer-spin-button]:appearance-none"
|
||||
disabled={disabled}
|
||||
/>
|
||||
<span className="text-stat-subtitle">{suffix}</span>
|
||||
<span className="text-stat-subtitle">{shownSuffix}</span>
|
||||
</span>
|
||||
);
|
||||
}
|
||||
@@ -87,7 +95,7 @@ export function NumberChip({ value, onChange, suffix, min, max, step = 1, warnOv
|
||||
disabled={disabled}
|
||||
>
|
||||
<span>{value || '0'}</span>
|
||||
<span className="text-stat-subtitle">{suffix}</span>
|
||||
<span className="text-stat-subtitle">{shownSuffix}</span>
|
||||
</button>
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
/**
|
||||
* The GitOps section: how often this node fetches Git sources, and how fast it
|
||||
* retries proving the images a Blueprint target is running.
|
||||
* The GitOps section: how often this node fetches Git sources, how fast it
|
||||
* retries proving the images a Blueprint target is running, and how long it
|
||||
* keeps the history of all of it.
|
||||
*
|
||||
* The two controls save through different routes on purpose, so the tests assert
|
||||
* The controls save through different routes on purpose, so the tests assert
|
||||
* each one reaches its own: the poll interval owns an endpoint because writing it
|
||||
* reschedules live fetchers, and the retry interval rides the shared node
|
||||
* settings because it is only read when a check needs it.
|
||||
* reschedules live fetchers, and the retry interval and history window ride the
|
||||
* shared node settings because they are only read when needed.
|
||||
*/
|
||||
import { describe, it, expect, vi, beforeEach } from 'vitest';
|
||||
import { render, screen, fireEvent, waitFor } from '@testing-library/react';
|
||||
@@ -49,7 +50,10 @@ import { apiFetch } from '@/lib/api';
|
||||
|
||||
const mockedFetch = apiFetch as unknown as ReturnType<typeof vi.fn>;
|
||||
|
||||
const SETTINGS: Record<string, string> = { gitops_artifact_retry_interval_mins: '5' };
|
||||
const SETTINGS: Record<string, string> = {
|
||||
gitops_artifact_retry_interval_mins: '5',
|
||||
gitops_history_retention_days: '30',
|
||||
};
|
||||
|
||||
beforeEach(() => {
|
||||
mockedFetch.mockReset();
|
||||
@@ -94,15 +98,40 @@ describe('GitOpsSection', () => {
|
||||
mockedFetch.mockImplementation(async (path: string) => (
|
||||
path === '/git-sources/polling'
|
||||
? { ok: true, json: async () => ({ poll_interval_mins: 0 }) }
|
||||
: { ok: true, json: async () => ({ gitops_artifact_retry_interval_mins: '17' }) }
|
||||
: { ok: true, json: async () => ({ gitops_artifact_retry_interval_mins: '17', gitops_history_retention_days: '45' }) }
|
||||
));
|
||||
render(<GitOpsSection />);
|
||||
await waitFor(() => expect(screen.getByText('Drift verification')).toBeInTheDocument());
|
||||
expect(screen.getByText('Retry an unresolved image identity')).toBeInTheDocument();
|
||||
await waitFor(() => expect(chipShowing('17')).toBeInTheDocument());
|
||||
// The history window reads the same payload and renders its own stored value.
|
||||
expect(screen.getByText('Keep GitOps history for')).toBeInTheDocument();
|
||||
// The helper states what the window governs and where bell entries
|
||||
// actually come from. Asserted on substance, not on whole sentences,
|
||||
// so a copy edit does not need a test change. The History subtitle
|
||||
// and the chip suffix share phrases with the helper, so each match
|
||||
// below is on text the helper alone carries.
|
||||
expect(screen.getByText(/before older entries are pruned/)).toBeInTheDocument();
|
||||
expect(screen.getByText(/follow the Notification log window/)).toBeInTheDocument();
|
||||
// The 3-day figure had no basis in the code; the helper now names the
|
||||
// real condition and scopes it to the direct webhook path.
|
||||
expect(screen.getByText(/provider endpoint keeps its own record/)).toBeInTheDocument();
|
||||
expect(screen.getByText('days')).toBeInTheDocument();
|
||||
await waitFor(() => expect(chipShowing('45')).toBeInTheDocument());
|
||||
});
|
||||
|
||||
it('saves a changed interval through the shared node settings, carrying only that key', async () => {
|
||||
it('reads day when the history window is 1', async () => {
|
||||
mockedFetch.mockImplementation(async (path: string) => (
|
||||
path === '/git-sources/polling'
|
||||
? { ok: true, json: async () => ({ poll_interval_mins: 0 }) }
|
||||
: { ok: true, json: async () => ({ gitops_artifact_retry_interval_mins: '5', gitops_history_retention_days: '1' }) }
|
||||
));
|
||||
render(<GitOpsSection />);
|
||||
await waitFor(() => expect(screen.getByText('Keep GitOps history for')).toBeInTheDocument());
|
||||
expect(screen.getByText('day')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('saves a changed interval through the shared node settings, carrying the section values', async () => {
|
||||
render(<GitOpsSection />);
|
||||
await waitFor(() => expect(chipShowing('5')).toBeInTheDocument());
|
||||
|
||||
@@ -115,7 +144,24 @@ describe('GitOpsSection', () => {
|
||||
);
|
||||
expect(call).toBeDefined();
|
||||
expect(JSON.parse((call![1] as { body: string }).body))
|
||||
.toEqual({ gitops_artifact_retry_interval_mins: '30' });
|
||||
.toEqual({ gitops_artifact_retry_interval_mins: '30', gitops_history_retention_days: '30' });
|
||||
});
|
||||
});
|
||||
|
||||
it('saves a changed history window through the shared node settings', async () => {
|
||||
render(<GitOpsSection />);
|
||||
await waitFor(() => expect(chipShowing('30')).toBeInTheDocument());
|
||||
|
||||
await editChip('30', '90');
|
||||
fireEvent.click(screen.getByText('Save settings'));
|
||||
|
||||
await waitFor(() => {
|
||||
const call = mockedFetch.mock.calls.find(
|
||||
(c) => c[0] === '/settings' && (c[1] as { method?: string } | undefined)?.method === 'PATCH',
|
||||
);
|
||||
expect(call).toBeDefined();
|
||||
expect(JSON.parse((call![1] as { body: string }).body))
|
||||
.toEqual({ gitops_artifact_retry_interval_mins: '5', gitops_history_retention_days: '90' });
|
||||
});
|
||||
});
|
||||
|
||||
@@ -184,10 +230,31 @@ describe('GitOpsSection', () => {
|
||||
});
|
||||
render(<GitOpsSection />);
|
||||
await waitFor(() => expect(screen.getByText('Drift verification')).toBeInTheDocument());
|
||||
expect(document.querySelectorAll('fieldset[disabled]').length).toBeGreaterThan(0);
|
||||
const fieldset = document.querySelector('fieldset[disabled]');
|
||||
expect(fieldset).not.toBeNull();
|
||||
expect(fieldset?.textContent).toContain('Keep GitOps history for');
|
||||
expect(fieldset?.textContent).toContain('Retry an unresolved image identity');
|
||||
expect(fieldset?.textContent).not.toContain('Poll Git sources');
|
||||
expect(screen.queryByText('Save settings')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('shows the load error when settings cannot be read', async () => {
|
||||
mockedFetch.mockImplementation(async () => ({ ok: false, json: async () => ({}) }));
|
||||
render(<GitOpsSection />);
|
||||
expect(await screen.findByText('Could not load settings')).toBeInTheDocument();
|
||||
expect(screen.queryByText('Keep GitOps history for')).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('falls back to 30 days when the history window is missing from the payload', async () => {
|
||||
mockedFetch.mockImplementation(async (path: string) => {
|
||||
if (path === '/git-sources/polling') return { ok: true, json: async () => ({ poll_interval_mins: 0 }) };
|
||||
return { ok: true, json: async () => ({ gitops_artifact_retry_interval_mins: '5' }) };
|
||||
});
|
||||
render(<GitOpsSection />);
|
||||
await waitFor(() => expect(screen.getByText('Keep GitOps history for')).toBeInTheDocument());
|
||||
expect(chipShowing('30')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('leaves the poll control usable for a node admin who cannot edit the retry interval', async () => {
|
||||
useAuthMock.mockReturnValue({
|
||||
isAdmin: false,
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
/**
|
||||
* The shared number chip singularizes a plural suffix at 1, and snaps a typed
|
||||
* value to its own step so a fraction never reaches an integer setting's
|
||||
* route validation. The history window covers "day"; the generations chip in
|
||||
* the Stacks section uses the same control; the Docker janitor chip covers a
|
||||
* genuinely fractional step.
|
||||
*/
|
||||
import { describe, it, expect, vi, afterEach } from 'vitest';
|
||||
import { render, screen, fireEvent, cleanup } from '@testing-library/react';
|
||||
import { NumberChip } from '../SystemControls';
|
||||
|
||||
// One render per test, so the role queries below only ever see the chip the
|
||||
// test itself mounted.
|
||||
afterEach(() => cleanup());
|
||||
|
||||
describe('NumberChip suffix', () => {
|
||||
it('reads generation when one rollback generation is retained', () => {
|
||||
render(<NumberChip value="1" onChange={() => {}} suffix="generations" min={0} max={50} />);
|
||||
expect(screen.getByText('generation')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
it('keeps the plural suffix above 1', () => {
|
||||
render(<NumberChip value="2" onChange={() => {}} suffix="generations" min={0} max={50} />);
|
||||
expect(screen.getByText('generations')).toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
|
||||
describe('NumberChip step', () => {
|
||||
function commitValue(draft: string, props: Partial<React.ComponentProps<typeof NumberChip>> = {}) {
|
||||
const onChange = vi.fn();
|
||||
render(<NumberChip value="30" onChange={onChange} suffix="days" min={1} max={365} {...props} />);
|
||||
fireEvent.click(screen.getByRole('button'));
|
||||
const input = screen.getByRole('spinbutton');
|
||||
fireEvent.change(input, { target: { value: draft } });
|
||||
fireEvent.blur(input);
|
||||
return onChange;
|
||||
}
|
||||
|
||||
it('rounds a typed fraction up to the whole step an integer setting validates', () => {
|
||||
// 1.5 used to be clamped but written verbatim, so the save sent "1.5"
|
||||
// and the route rejected it with a bare validation error.
|
||||
expect(commitValue('1.5')).toHaveBeenCalledWith('2');
|
||||
});
|
||||
|
||||
it('rounds a typed fraction down to the whole step', () => {
|
||||
expect(commitValue('1.4')).toHaveBeenCalledWith('1');
|
||||
});
|
||||
|
||||
it('keeps a genuinely fractional step on a chip that asks for one', () => {
|
||||
// The Docker janitor chip is a float setting, so 0.5 must survive.
|
||||
expect(commitValue('10.3', { min: 0, step: 0.5 })).toHaveBeenCalledWith('10.5');
|
||||
});
|
||||
|
||||
it('clamps after rounding, so a fraction below the floor cannot round down under it', () => {
|
||||
expect(commitValue('0.4')).toHaveBeenCalledWith('1');
|
||||
});
|
||||
});
|
||||
@@ -67,6 +67,7 @@ const FULL_SETTINGS: Record<string, string> = {
|
||||
env_block_deploy_on_missing_required: '0',
|
||||
auto_create_missing_external_networks: '0',
|
||||
gitops_artifact_retry_interval_mins: '5',
|
||||
gitops_history_retention_days: '30',
|
||||
};
|
||||
|
||||
function patchedKeys(): string[] {
|
||||
@@ -129,7 +130,7 @@ describe('split section save payloads', () => {
|
||||
]);
|
||||
});
|
||||
|
||||
it('GitOpsSection patches only the artifact retry interval', async () => {
|
||||
it('GitOpsSection patches only the GitOps section keys', async () => {
|
||||
// The polling control saves to its own endpoint, and this file's
|
||||
// patchedKeys() reads the last PATCH, so the capability is off here to
|
||||
// keep that control out of the way and leave one PATCH to inspect.
|
||||
@@ -140,7 +141,7 @@ describe('split section save payloads', () => {
|
||||
fireEvent.blur(screen.getByRole('spinbutton'));
|
||||
fireEvent.click(save);
|
||||
await waitFor(() => expect(mockedFetch.mock.calls.some(c => c[1]?.method === 'PATCH')).toBe(true));
|
||||
expect(patchedKeys()).toEqual(['gitops_artifact_retry_interval_mins']);
|
||||
expect(patchedKeys()).toEqual(['gitops_artifact_retry_interval_mins', 'gitops_history_retention_days']);
|
||||
});
|
||||
|
||||
it('DockerStorageSection patches only docker and storage keys', async () => {
|
||||
|
||||
@@ -164,6 +164,13 @@ describe('requiredPermission registry mapping', () => {
|
||||
expect(byId.get('labels')?.requiredPermission).toBe('stack:read');
|
||||
});
|
||||
|
||||
it('registers GitOps history retention in the GitOps search keywords', () => {
|
||||
const gitops = SETTINGS_ITEMS.find(i => i.id === 'gitops');
|
||||
for (const term of ['history', 'retention', 'prune', 'cleanup']) {
|
||||
expect(gitops?.keywords, `keyword ${term}`).toContain(term);
|
||||
}
|
||||
});
|
||||
|
||||
it('keeps adminOnly on identity, credentials, and emergency surfaces', () => {
|
||||
for (const id of [
|
||||
'sso',
|
||||
|
||||
@@ -140,8 +140,8 @@ export const SETTINGS_ITEMS: readonly SettingsItemMeta[] = [
|
||||
id: 'gitops',
|
||||
group: 'infrastructure',
|
||||
label: 'GitOps',
|
||||
description: 'How this node fetches Git sources and verifies the images a Blueprint target is running.',
|
||||
keywords: ['git', 'gitops', 'source', 'poll', 'polling', 'interval', 'cadence', 'fetch', 'schedule', 'drift', 'verify', 'verification', 'artifact', 'digest', 'image', 'identity', 'registry', 'freeze', 'expected', 'retry', 'blueprint'],
|
||||
description: 'How Git sources are fetched, how the images a Blueprint target runs are verified, and how long the GitOps history is kept.',
|
||||
keywords: ['git', 'gitops', 'source', 'poll', 'polling', 'interval', 'cadence', 'fetch', 'schedule', 'drift', 'verify', 'verification', 'artifact', 'digest', 'image', 'identity', 'registry', 'freeze', 'expected', 'retry', 'blueprint', 'history', 'retention', 'prune', 'cleanup'],
|
||||
tier: null,
|
||||
scope: 'node',
|
||||
},
|
||||
|
||||
@@ -24,6 +24,7 @@ export interface PatchableSettings {
|
||||
env_block_deploy_on_missing_required?: '0' | '1';
|
||||
auto_create_missing_external_networks?: '0' | '1';
|
||||
gitops_artifact_retry_interval_mins?: string;
|
||||
gitops_history_retention_days?: string;
|
||||
image_update_sidebar_indicators?: '0' | '1';
|
||||
notification_dispatch_retries?: string;
|
||||
session_sliding_refresh?: '0' | '1';
|
||||
@@ -55,6 +56,7 @@ export const DEFAULT_SETTINGS: PatchableSettings = {
|
||||
env_block_deploy_on_missing_required: '0',
|
||||
auto_create_missing_external_networks: '0',
|
||||
gitops_artifact_retry_interval_mins: '5',
|
||||
gitops_history_retention_days: '30',
|
||||
image_update_sidebar_indicators: '1',
|
||||
notification_dispatch_retries: '0',
|
||||
session_sliding_refresh: '1',
|
||||
|
||||
Reference in New Issue
Block a user