feat(fleet-sync): retry failed pushes and backfill on add-node (#970)

A control instance now retries fleet-sync pushes that hit a transient
failure and backfills the security state on a freshly registered remote
without waiting for the next policy edit.

New service:
- FleetSyncRetryService (singleton, start/stop) wakes 30s after boot
  and ticks every 5min. For each fleet resource, queries
  getFailedSyncTargets within a 24h window and re-pushes via
  FleetSyncService.pushResourceToNode through the same per-node mutex,
  so a normal fanout in flight serializes naturally with a retry.
- After STALE_THRESHOLD_MS (1h) of continuous failure for a
  previously-working node, dispatches a single warning notification
  per cooldown window. Brand-new nodes that have never succeeded do
  not alert via this path; misconfigured remotes are caught by the
  test-connection affordance at registration time.
- Wired into bootstrap startup/shutdown next to AutoHealService.

Public surface:
- FleetSyncService.pushResourceToNode(node, resource): targeted push
  to one node that re-uses the per-node mutex. Used by the retry
  service and any future targeted-resync flow.
- routes/nodes.ts POST /api/nodes fires pushResourceAsync for both
  resources after a remote-proxy node row commits.

Tuning constants centralized in fleetSyncConstants.ts:
- RETRY_MAX_AGE_MS = 24h
- STALE_THRESHOLD_MS = 1h

Tests:
- 8 vitest cases covering replica skip, retry dispatch, missing-node
  skip, alert-once-per-cooldown across the threshold window, no-alert
  for recent failures, no-alert for brand-new never-succeeded nodes,
  no-alert when the retry itself succeeds, start/stop idempotency.
- Full backend suite: 1781 pass / 5 skipped.
This commit is contained in:
Anso
2026-05-07 13:16:24 -04:00
committed by GitHub
parent 7dde257e1f
commit 33b15d6cba
7 changed files with 389 additions and 0 deletions
+24
View File
@@ -224,6 +224,30 @@ export class FleetSyncService {
});
}
/**
* Push a resource to a single remote node. Used by the retry service to
* re-send to a node that previously failed without disturbing other nodes
* in the fleet. Goes through the same per-node mutex as `pushResource`,
* so a normal fanout and a retry never overlap on one node.
*
* No-op when this instance is a replica or when the node is not a
* proxy-mode remote with credentials configured.
*/
public async pushResourceToNode(
node: Node & { id: number },
resource: FleetResource,
): Promise<void> {
if (FleetSyncService.getRole() === 'replica') return;
if (node.type !== 'remote' || !node.api_url || !node.api_token) return;
const rows = this.loadResource(resource);
const payload: Omit<FleetSyncPayload, 'targetIdentity'> = {
rows,
pushedAt: this.nextPushedAt(),
controlIdentity: FleetSyncService.getControlIdentity(),
};
await this.enqueuePushToNode(node, resource, payload);
}
/**
* Apply a received sync payload on a replica. Runs control-anchor check,
* staleness comparison, role flip, identity cache, row replacement, and