mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-30 03:59:41 +00:00
feat(fleet-sync): retry failed pushes and backfill on add-node (#970)
A control instance now retries fleet-sync pushes that hit a transient failure and backfills the security state on a freshly registered remote without waiting for the next policy edit. New service: - FleetSyncRetryService (singleton, start/stop) wakes 30s after boot and ticks every 5min. For each fleet resource, queries getFailedSyncTargets within a 24h window and re-pushes via FleetSyncService.pushResourceToNode through the same per-node mutex, so a normal fanout in flight serializes naturally with a retry. - After STALE_THRESHOLD_MS (1h) of continuous failure for a previously-working node, dispatches a single warning notification per cooldown window. Brand-new nodes that have never succeeded do not alert via this path; misconfigured remotes are caught by the test-connection affordance at registration time. - Wired into bootstrap startup/shutdown next to AutoHealService. Public surface: - FleetSyncService.pushResourceToNode(node, resource): targeted push to one node that re-uses the per-node mutex. Used by the retry service and any future targeted-resync flow. - routes/nodes.ts POST /api/nodes fires pushResourceAsync for both resources after a remote-proxy node row commits. Tuning constants centralized in fleetSyncConstants.ts: - RETRY_MAX_AGE_MS = 24h - STALE_THRESHOLD_MS = 1h Tests: - 8 vitest cases covering replica skip, retry dispatch, missing-node skip, alert-once-per-cooldown across the threshold window, no-alert for recent failures, no-alert for brand-new never-succeeded nodes, no-alert when the retry itself succeeds, start/stop idempotency. - Full backend suite: 1781 pass / 5 skipped.
This commit is contained in:
@@ -224,6 +224,30 @@ export class FleetSyncService {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Push a resource to a single remote node. Used by the retry service to
|
||||
* re-send to a node that previously failed without disturbing other nodes
|
||||
* in the fleet. Goes through the same per-node mutex as `pushResource`,
|
||||
* so a normal fanout and a retry never overlap on one node.
|
||||
*
|
||||
* No-op when this instance is a replica or when the node is not a
|
||||
* proxy-mode remote with credentials configured.
|
||||
*/
|
||||
public async pushResourceToNode(
|
||||
node: Node & { id: number },
|
||||
resource: FleetResource,
|
||||
): Promise<void> {
|
||||
if (FleetSyncService.getRole() === 'replica') return;
|
||||
if (node.type !== 'remote' || !node.api_url || !node.api_token) return;
|
||||
const rows = this.loadResource(resource);
|
||||
const payload: Omit<FleetSyncPayload, 'targetIdentity'> = {
|
||||
rows,
|
||||
pushedAt: this.nextPushedAt(),
|
||||
controlIdentity: FleetSyncService.getControlIdentity(),
|
||||
};
|
||||
await this.enqueuePushToNode(node, resource, payload);
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply a received sync payload on a replica. Runs control-anchor check,
|
||||
* staleness comparison, role flip, identity cache, row replacement, and
|
||||
|
||||
Reference in New Issue
Block a user