mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-10 18:56:53 +00:00
feat(fleet-sync): retry failed pushes and backfill on add-node (#970)
A control instance now retries fleet-sync pushes that hit a transient failure and backfills the security state on a freshly registered remote without waiting for the next policy edit. New service: - FleetSyncRetryService (singleton, start/stop) wakes 30s after boot and ticks every 5min. For each fleet resource, queries getFailedSyncTargets within a 24h window and re-pushes via FleetSyncService.pushResourceToNode through the same per-node mutex, so a normal fanout in flight serializes naturally with a retry. - After STALE_THRESHOLD_MS (1h) of continuous failure for a previously-working node, dispatches a single warning notification per cooldown window. Brand-new nodes that have never succeeded do not alert via this path; misconfigured remotes are caught by the test-connection affordance at registration time. - Wired into bootstrap startup/shutdown next to AutoHealService. Public surface: - FleetSyncService.pushResourceToNode(node, resource): targeted push to one node that re-uses the per-node mutex. Used by the retry service and any future targeted-resync flow. - routes/nodes.ts POST /api/nodes fires pushResourceAsync for both resources after a remote-proxy node row commits. Tuning constants centralized in fleetSyncConstants.ts: - RETRY_MAX_AGE_MS = 24h - STALE_THRESHOLD_MS = 1h Tests: - 8 vitest cases covering replica skip, retry dispatch, missing-node skip, alert-once-per-cooldown across the threshold window, no-alert for recent failures, no-alert for brand-new never-succeeded nodes, no-alert when the retry itself succeeds, start/stop idempotency. - Full backend suite: 1781 pass / 5 skipped.
This commit is contained in:
@@ -3,6 +3,7 @@ import { DatabaseService } from '../services/DatabaseService';
|
||||
import { LicenseService } from '../services/LicenseService';
|
||||
import { MonitorService } from '../services/MonitorService';
|
||||
import { AutoHealService } from '../services/AutoHealService';
|
||||
import { FleetSyncRetryService } from '../services/FleetSyncRetryService';
|
||||
import { DockerEventManager } from '../services/DockerEventManager';
|
||||
import { ImageUpdateService } from '../services/ImageUpdateService';
|
||||
import { SchedulerService } from '../services/SchedulerService';
|
||||
@@ -29,6 +30,7 @@ export function installShutdownHandlers(server: Server): void {
|
||||
console.warn('[Shutdown] MonitorService cleanup failed:', (e as Error).message);
|
||||
}
|
||||
try { AutoHealService.getInstance().stop(); } catch (e) { console.warn('[Shutdown] AutoHealService cleanup failed:', (e as Error).message); }
|
||||
try { FleetSyncRetryService.getInstance().stop(); } catch (e) { console.warn('[Shutdown] FleetSyncRetryService cleanup failed:', (e as Error).message); }
|
||||
try { DockerEventManager.getInstance().stop(); } catch (e) {
|
||||
console.warn('[Shutdown] DockerEventManager cleanup failed:', (e as Error).message);
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import { LicenseService } from '../services/LicenseService';
|
||||
import SelfUpdateService from '../services/SelfUpdateService';
|
||||
import { MonitorService } from '../services/MonitorService';
|
||||
import { AutoHealService } from '../services/AutoHealService';
|
||||
import { FleetSyncRetryService } from '../services/FleetSyncRetryService';
|
||||
import { DockerEventManager } from '../services/DockerEventManager';
|
||||
import TrivyService from '../services/TrivyService';
|
||||
import { ImageUpdateService } from '../services/ImageUpdateService';
|
||||
@@ -39,6 +40,7 @@ export async function startServer(server: Server): Promise<void> {
|
||||
// safely run alongside the async initializers below.
|
||||
MonitorService.getInstance().start();
|
||||
AutoHealService.getInstance().start();
|
||||
FleetSyncRetryService.getInstance().start();
|
||||
ImageUpdateService.getInstance().start();
|
||||
SchedulerService.getInstance().start();
|
||||
MfaService.getInstance().start();
|
||||
|
||||
Reference in New Issue
Block a user