mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-31 04:38:11 +00:00
fix(fleet-sync): make control-identity-mismatch sticky and surface in UI (#1117)
Treat 409 CONTROL_IDENTITY_MISMATCH from a replica as a non-retriable
failure instead of looping the same 409 through the 5-minute retry
service forever and silently writing identical failure rows.
Backend
- DatabaseService: add `sticky_error_code`, `sticky_error_expected`,
`sticky_error_got` columns to `fleet_sync_status` via an idempotent
migration. New methods setFleetSyncSticky, getFleetSyncStickyCode,
clearFleetSyncStickyForNode. recordFleetSyncSuccess clears the sticky
flag on a clean push. getFailedSyncTargets SQL adds
`AND sticky_error_code IS NULL` so the retry loop skips sticky rows.
- FleetSyncService.executePushToNode: short-circuits at the top when
sticky is set (covers event-driven pushResourceAsync calls). On a 409
with code CONTROL_IDENTITY_MISMATCH, records the failure once and
pins sticky with the expected/got fingerprints carried in the 409 body.
- routes/nodes.ts: new POST /api/nodes/:id/fleet-sync/reset-anchor.
Admin + paid + node:manage. Proxies POST /api/fleet/role/reanchor to
the peer with `{override:true}` using the stored Bearer node_proxy
token. On peer 200, clears every sticky row for the node so the next
push re-anchors and resumes replication. Distinct 502 / 504 responses
for peer-rejected / peer-unreachable so the UI can show a useful toast.
Frontend
- New lib/fleetSyncApi.ts + hooks/useFleetSyncStatus.ts. Polling hook
(30s visibilityInterval) skips fetch when !isPaid.
- NodeManager.tsx: destructive banner per affected node listing both
fingerprints, with `Reset anchor on peer` and `Remove node` buttons.
Hidden for community-tier users via empty hook data.
- FleetConfiguration.tsx (Fleet -> Status): read-only `Policy sync`
SummaryRow per remote node card. In sync / degraded / paused with
a tooltip; no action buttons (the action lives in NodeManager).
Tests
- fleet-sync-service.test.ts: 4 new cases for sticky-set on first
mismatch, short-circuit on subsequent pushes, null fingerprints,
and non-mismatch failures not setting sticky.
- database-fleet-sync-sticky.test.ts (new): 6 cases pinning the DB
contract incl. retry-loop SQL filter and migration idempotency.
- nodes-fleet-sync-reset-anchor.test.ts (new): 6 cases covering
happy path, peer 401 -> 502, peer unreachable -> 504, local-node
rejection, unknown node id, and community-tier 403.
Gate parity (Directive 30): the new POST .../reset-anchor enforces
requireAdmin + requirePaid + node:manage (matches the existing read at
GET /api/fleet/sync-status). UI banner + SummaryRow only render when the
hook returns data, which it only does for paid-tier authed users. No
existing tier-gate file moved; this is greenfield parity.
Auth audit: the peer's POST /api/fleet/role/reanchor route already uses
requireAdmin, which accepts the central's stored node_proxy Bearer
token because authMiddleware maps `scope === 'node_proxy'` to
`req.user = { username: 'node-proxy', role: 'admin', userId: 0 }`.
No widening required.
Backend tsc clean. Frontend tsc -b clean. 59 fleet-sync tests pass; full
backend suite green minus the pre-existing Windows-only file-lock flake
on filesystem-backup.test.ts that reproduces unchanged on main.
This commit is contained in:
@@ -449,12 +449,38 @@ export class FleetSyncService {
|
||||
return next;
|
||||
}
|
||||
|
||||
/**
|
||||
* Concurrency note: the sticky-set in the CONTROL_IDENTITY_MISMATCH catch
|
||||
* branch is best-effort against an operator-initiated reset that lands
|
||||
* during a push's HTTP round-trip. The reset endpoint clears
|
||||
* `sticky_error_code` to NULL; if this push's 409 arrives after that
|
||||
* clear, it will re-pin the row. The operator clicks Reset again. The
|
||||
* window is bounded by one HTTP round-trip per resource; no lock or
|
||||
* generation counter is justified.
|
||||
*/
|
||||
private async executePushToNode(
|
||||
node: Node & { id: number },
|
||||
resource: FleetResource,
|
||||
partial: Omit<FleetSyncPayload, 'targetIdentity'>,
|
||||
): Promise<void> {
|
||||
const db = DatabaseService.getInstance();
|
||||
|
||||
// Sticky-error short-circuit. When a previous push hit a non-retriable
|
||||
// failure (today: CONTROL_IDENTITY_MISMATCH), every subsequent push
|
||||
// would re-issue the same 409 and re-spam the log every 5 minutes. The
|
||||
// sticky flag is cleared by either (a) the operator resetting the
|
||||
// anchor via POST /api/nodes/:id/fleet-sync/reset-anchor, or (b) a
|
||||
// successful push to the same node (recordFleetSyncSuccess clears
|
||||
// it). Until then, skip the HTTP call entirely.
|
||||
if (db.getFleetSyncStickyCode(node.id, resource)) {
|
||||
if (isDebugEnabled()) {
|
||||
console.debug(
|
||||
`[FleetSync:debug] Skipping ${resource} push to "${node.name}": sticky error blocks retries.`,
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
const apiUrl = node.api_url ?? '';
|
||||
const baseUrl = apiUrl.replace(/\/$/, '');
|
||||
const payload: FleetSyncPayload = { ...partial, targetIdentity: apiUrl };
|
||||
@@ -479,7 +505,7 @@ export class FleetSyncService {
|
||||
// healthy; suppress the failure record so it does not surface as
|
||||
// an alert in the sync-status panel.
|
||||
if (err instanceof AxiosError && err.response?.status === 409) {
|
||||
const data = err.response.data as { code?: string } | undefined;
|
||||
const data = err.response.data as { code?: string; expected?: string; got?: string } | undefined;
|
||||
if (data?.code === SYNC_ERROR_CODES.staleSyncPush) {
|
||||
if (isDebugEnabled()) {
|
||||
console.debug(
|
||||
@@ -488,6 +514,26 @@ export class FleetSyncService {
|
||||
}
|
||||
return;
|
||||
}
|
||||
// CONTROL_IDENTITY_MISMATCH is permanent until the operator
|
||||
// explicitly resets the peer anchor. Record the failure once
|
||||
// (last_error + last_failure_at) plus a sticky flag so the
|
||||
// retry service and event-driven push path skip subsequent
|
||||
// attempts.
|
||||
if (data?.code === SYNC_ERROR_CODES.controlIdentityMismatch) {
|
||||
const message = this.formatError(err);
|
||||
console.warn(
|
||||
`[FleetSync] Failed to push ${resource} to "${node.name}" (${baseUrl}): ${message}`,
|
||||
);
|
||||
db.recordFleetSyncFailure(node.id, resource, message);
|
||||
db.setFleetSyncSticky(
|
||||
node.id,
|
||||
resource,
|
||||
SYNC_ERROR_CODES.controlIdentityMismatch,
|
||||
typeof data.expected === 'string' ? data.expected : null,
|
||||
typeof data.got === 'string' ? data.got : null,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
const message = this.formatError(err);
|
||||
console.warn(
|
||||
|
||||
Reference in New Issue
Block a user