gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-20 07:13:05 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)

Browse Source 
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.

Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
  `sticky_error_got` columns to `fleet_sync_status` via an idempotent
  migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
  clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
  flag on a clean push. getFailedSyncTargets SQL adds
  `AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
  sticky is set (covers event-driven pushResourceAsync calls). On a 409
  with code CONTROL_IDENTITY_MISMATCH, records the failure once and
  pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
  Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
  the peer with `{override:true}` using the stored Bearer node_proxy
  token. On peer 200, clears every sticky row for the node so the next
  push re-anchors and resumes replication. Distinct 502 / 504 responses
  for peer-rejected / peer-unreachable so the UI can show a useful toast.

Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
  (30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
  fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
  Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
  SummaryRow per remote node card. In sync / degraded / paused with
  a tooltip; no action buttons (the action lives in NodeManager).

Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
  mismatch, short-circuit on subsequent pushes, null fingerprints,
  and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
  contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
  happy path, peer 401 -> 502, peer unreachable -> 504, local-node
  rejection, unknown node id, and community-tier 403.

Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.

Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.

Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
This commit is contained in:
Anso
2026-05-19 19:49:16 -04:00
committed by GitHub
parent 69bc955c3b
commit e05099f2a1
10 changed files with 876 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/services/DatabaseService.ts
+77 -4
View File
@@ -555,6 +555,9 @@ export interface FleetSyncStatus {
last_success_at: number | null;
last_failure_at: number | null;
last_error: string | null;
sticky_error_code: string | null;
sticky_error_expected: string | null;
sticky_error_got: string | null;
}
export interface CveSuppression {
@@ -659,6 +662,7 @@ export class DatabaseService {
this.migrateAddNodeCordonFields();
this.migrateAddBlueprintPinnedNode();
this.migrateAutoHealNodeId();
this.migrateFleetSyncStickyError();
// Reset the cache once at end of constructor in case any migration
// populated it via getGlobalSettings() and a subsequent migration
@@ -1581,6 +1585,12 @@ export class DatabaseService {
this.tryAddColumn('blueprints', 'pinned_node_id', 'INTEGER');
}
private migrateFleetSyncStickyError(): void {
this.tryAddColumn('fleet_sync_status', 'sticky_error_code', 'TEXT');
this.tryAddColumn('fleet_sync_status', 'sticky_error_expected', 'TEXT');
this.tryAddColumn('fleet_sync_status', 'sticky_error_got', 'TEXT');
}
private migrateAutoHealNodeId(): void {
const markerKey = 'migration_auto_heal_node_scope_v1';
const markerDone = this.getGlobalSettings()[markerKey] === '1';
@@ -3931,11 +3941,15 @@ export class DatabaseService {
const now = Date.now();
this.db
.prepare(
`INSERT INTO fleet_sync_status (node_id, resource, last_success_at, last_failure_at, last_error)
VALUES (?, ?, ?, NULL, NULL)
`INSERT INTO fleet_sync_status (node_id, resource, last_success_at, last_failure_at, last_error,
sticky_error_code, sticky_error_expected, sticky_error_got)
VALUES (?, ?, ?, NULL, NULL, NULL, NULL, NULL)
ON CONFLICT(node_id, resource) DO UPDATE SET
last_success_at = excluded.last_success_at,
last_error = NULL`,
last_error = NULL,
sticky_error_code = NULL,
sticky_error_expected = NULL,
sticky_error_got = NULL`,
)
.run(nodeId, resource, now);
}
@@ -3953,6 +3967,64 @@ export class DatabaseService {
.run(nodeId, resource, now, error);
}
/**
* Mark a (node, resource) pair as having hit a non-retriable failure. The
* retry service skips sticky rows and the push paths short-circuit before
* any HTTP call. The first such failure still records `last_failure_at` +
* `last_error` via `recordFleetSyncFailure`; the sticky write is additive.
*
* `expected` and `got` carry the fingerprints from a 409
* CONTROL_IDENTITY_MISMATCH response so the UI can render
* "anchored to <expected>, this central is <got>" without parsing the
* error string.
*/
public setFleetSyncSticky(
nodeId: number,
resource: string,
code: string,
expected: string | null,
got: string | null,
): void {
this.db
.prepare(
`INSERT INTO fleet_sync_status (node_id, resource, sticky_error_code,
sticky_error_expected, sticky_error_got)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(node_id, resource) DO UPDATE SET
sticky_error_code = excluded.sticky_error_code,
sticky_error_expected = excluded.sticky_error_expected,
sticky_error_got = excluded.sticky_error_got`,
)
.run(nodeId, resource, code, expected, got);
}
public getFleetSyncStickyCode(nodeId: number, resource: string): string | null {
const row = this.db
.prepare(
`SELECT sticky_error_code FROM fleet_sync_status
WHERE node_id = ? AND resource = ?`,
)
.get(nodeId, resource) as { sticky_error_code: string | null } | undefined;
return row?.sticky_error_code ?? null;
}
/**
* Clear every sticky-error row for one node id. Used by the
* reset-anchor endpoint after the peer has acknowledged a reanchor; the
* next push attempt re-tries normally.
*/
public clearFleetSyncStickyForNode(nodeId: number): void {
this.db
.prepare(
`UPDATE fleet_sync_status
SET sticky_error_code = NULL,
sticky_error_expected = NULL,
sticky_error_got = NULL
WHERE node_id = ?`,
)
.run(nodeId);
}
public getFailedSyncTargets(resource: string, maxAgeMs: number): FleetSyncStatus[] {
const cutoff = Date.now() - maxAgeMs;
return this.db
@@ -3960,7 +4032,8 @@ export class DatabaseService {
`SELECT * FROM fleet_sync_status
WHERE resource = ?
AND (last_failure_at IS NOT NULL AND last_failure_at > ?)
AND (last_success_at IS NULL OR last_success_at < last_failure_at)`,
AND (last_success_at IS NULL OR last_success_at < last_failure_at)
AND sticky_error_code IS NULL`,
)
.all(resource, cutoff) as FleetSyncStatus[];
}
backend/src/services/FleetSyncService.ts
+47 -1
View File
@@ -449,12 +449,38 @@ export class FleetSyncService {
return next;
}
/**
* Concurrency note: the sticky-set in the CONTROL_IDENTITY_MISMATCH catch
* branch is best-effort against an operator-initiated reset that lands
* during a push's HTTP round-trip. The reset endpoint clears
* `sticky_error_code` to NULL; if this push's 409 arrives after that
* clear, it will re-pin the row. The operator clicks Reset again. The
* window is bounded by one HTTP round-trip per resource; no lock or
* generation counter is justified.
*/
private async executePushToNode(
node: Node & { id: number },
resource: FleetResource,
partial: Omit<FleetSyncPayload, 'targetIdentity'>,
): Promise<void> {
const db = DatabaseService.getInstance();
// Sticky-error short-circuit. When a previous push hit a non-retriable
// failure (today: CONTROL_IDENTITY_MISMATCH), every subsequent push
// would re-issue the same 409 and re-spam the log every 5 minutes. The
// sticky flag is cleared by either (a) the operator resetting the
// anchor via POST /api/nodes/:id/fleet-sync/reset-anchor, or (b) a
// successful push to the same node (recordFleetSyncSuccess clears
// it). Until then, skip the HTTP call entirely.
if (db.getFleetSyncStickyCode(node.id, resource)) {
if (isDebugEnabled()) {
console.debug(
`[FleetSync:debug] Skipping ${resource} push to "${node.name}": sticky error blocks retries.`,
);
}
return;
}
const apiUrl = node.api_url ?? '';
const baseUrl = apiUrl.replace(/\/$/, '');
const payload: FleetSyncPayload = { ...partial, targetIdentity: apiUrl };
@@ -479,7 +505,7 @@ export class FleetSyncService {
// healthy; suppress the failure record so it does not surface as
// an alert in the sync-status panel.
if (err instanceof AxiosError && err.response?.status === 409) {
const data = err.response.data as { code?: string } | undefined;
const data = err.response.data as { code?: string; expected?: string; got?: string } | undefined;
if (data?.code === SYNC_ERROR_CODES.staleSyncPush) {
if (isDebugEnabled()) {
console.debug(
@@ -488,6 +514,26 @@ export class FleetSyncService {
}
return;
}
// CONTROL_IDENTITY_MISMATCH is permanent until the operator
// explicitly resets the peer anchor. Record the failure once
// (last_error + last_failure_at) plus a sticky flag so the
// retry service and event-driven push path skip subsequent
// attempts.
if (data?.code === SYNC_ERROR_CODES.controlIdentityMismatch) {
const message = this.formatError(err);
console.warn(
`[FleetSync] Failed to push ${resource} to "${node.name}" (${baseUrl}): ${message}`,
);
db.recordFleetSyncFailure(node.id, resource, message);
db.setFleetSyncSticky(
node.id,
resource,
SYNC_ERROR_CODES.controlIdentityMismatch,
typeof data.expected === 'string' ? data.expected : null,
typeof data.got === 'string' ? data.got : null,
);
return;
}
}
const message = this.formatError(err);
console.warn(