feat(fleet-sync): retry failed pushes and backfill on add-node (#970)

A control instance now retries fleet-sync pushes that hit a transient
failure and backfills the security state on a freshly registered remote
without waiting for the next policy edit.

New service:
- FleetSyncRetryService (singleton, start/stop) wakes 30s after boot
  and ticks every 5min. For each fleet resource, queries
  getFailedSyncTargets within a 24h window and re-pushes via
  FleetSyncService.pushResourceToNode through the same per-node mutex,
  so a normal fanout in flight serializes naturally with a retry.
- After STALE_THRESHOLD_MS (1h) of continuous failure for a
  previously-working node, dispatches a single warning notification
  per cooldown window. Brand-new nodes that have never succeeded do
  not alert via this path; misconfigured remotes are caught by the
  test-connection affordance at registration time.
- Wired into bootstrap startup/shutdown next to AutoHealService.

Public surface:
- FleetSyncService.pushResourceToNode(node, resource): targeted push
  to one node that re-uses the per-node mutex. Used by the retry
  service and any future targeted-resync flow.
- routes/nodes.ts POST /api/nodes fires pushResourceAsync for both
  resources after a remote-proxy node row commits.

Tuning constants centralized in fleetSyncConstants.ts:
- RETRY_MAX_AGE_MS = 24h
- STALE_THRESHOLD_MS = 1h

Tests:
- 8 vitest cases covering replica skip, retry dispatch, missing-node
  skip, alert-once-per-cooldown across the threshold window, no-alert
  for recent failures, no-alert for brand-new never-succeeded nodes,
  no-alert when the retry itself succeeds, start/stop idempotency.
- Full backend suite: 1781 pass / 5 skipped.
This commit is contained in:
Anso
2026-05-07 13:16:24 -04:00
committed by GitHub
parent 7dde257e1f
commit 33b15d6cba
7 changed files with 389 additions and 0 deletions
+2
View File
@@ -3,6 +3,7 @@ import { DatabaseService } from '../services/DatabaseService';
import { LicenseService } from '../services/LicenseService';
import { MonitorService } from '../services/MonitorService';
import { AutoHealService } from '../services/AutoHealService';
import { FleetSyncRetryService } from '../services/FleetSyncRetryService';
import { DockerEventManager } from '../services/DockerEventManager';
import { ImageUpdateService } from '../services/ImageUpdateService';
import { SchedulerService } from '../services/SchedulerService';
@@ -29,6 +30,7 @@ export function installShutdownHandlers(server: Server): void {
console.warn('[Shutdown] MonitorService cleanup failed:', (e as Error).message);
}
try { AutoHealService.getInstance().stop(); } catch (e) { console.warn('[Shutdown] AutoHealService cleanup failed:', (e as Error).message); }
try { FleetSyncRetryService.getInstance().stop(); } catch (e) { console.warn('[Shutdown] FleetSyncRetryService cleanup failed:', (e as Error).message); }
try { DockerEventManager.getInstance().stop(); } catch (e) {
console.warn('[Shutdown] DockerEventManager cleanup failed:', (e as Error).message);
}
+2
View File
@@ -5,6 +5,7 @@ import { LicenseService } from '../services/LicenseService';
import SelfUpdateService from '../services/SelfUpdateService';
import { MonitorService } from '../services/MonitorService';
import { AutoHealService } from '../services/AutoHealService';
import { FleetSyncRetryService } from '../services/FleetSyncRetryService';
import { DockerEventManager } from '../services/DockerEventManager';
import TrivyService from '../services/TrivyService';
import { ImageUpdateService } from '../services/ImageUpdateService';
@@ -39,6 +40,7 @@ export async function startServer(server: Server): Promise<void> {
// safely run alongside the async initializers below.
MonitorService.getInstance().start();
AutoHealService.getInstance().start();
FleetSyncRetryService.getInstance().start();
ImageUpdateService.getInstance().start();
SchedulerService.getInstance().start();
MfaService.getInstance().start();