mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-25 09:46:47 +00:00
fix(fleet-sync): version the wire protocol and serialize per-node pushes (#967)
Hardens the scan-policy and CVE-suppression replication channel as the foundation of a multi-PR fleet sync hardening track. No new endpoints, no new tables, no schema changes; receivers still tolerate legacy payloads (absent pushedAt and controlIdentity) for rollout safety. Wire protocol: - Sender stamps every push with a strictly-increasing pushedAt and a placeholder controlIdentity. Receiver rejects strictly-older pushedAt with 409 STALE_SYNC_PUSH so the next write retries. - pushedAt comparison plus row replacement plus watermark write run in a single SQLite transaction; a partial-write window cannot leave the watermark behind the row state. Concurrency and limits: - Per-node mutex on the sender so concurrent control writes serialize per remote and never apply older state on top of newer. - Sender-side row cap at MAX_SYNC_ROWS=5000 with a 6-hour throttled truncation alert so flapping configs cannot flood the operator. - Route-level body limit raised to 5MB on POST /api/fleet/sync/:resource only; the global 100KB cap is unchanged. Oversize bodies return a structured 413 SYNC_PAYLOAD_TOO_LARGE. Determinism and hygiene: - getMatchingPolicy gains an id-ASC tiebreaker so two replicas resolve the same winner when policies tie on scope class. - Inline comment documents why getMatchingPolicy filters node_id at SQL yet still relies on JS identity matching for replicated rows. - Comment on the receive endpoint documents why no requirePaid is enforced (control's tier authorizes; replica trusts the bearer). - STALE_SYNC_PUSH 409s no longer record a node failure; they are expected protocol outcomes, not health issues. Public surface additions: - DatabaseService.transaction(fn): generic SAVEPOINT-friendly wrapper. - DatabaseService.getLocalScanPolicies / getLocalCveSuppressions: SQL filter on replicated_from_control = 0. - StaleSyncPushError: typed sentinel the route translates to 409. - fleetSyncConstants: shared MAX_SYNC_ROWS, body limit, state-key and error-code maps so the wire protocol has one source of truth. Tests: - 24 new vitest cases across fleet-sync-service, fleet-sync-routes, and database-matching-policy covering monotonic pushedAt, per-node serialization, row truncation and throttle, stale-push suppression, receiver back-compat, oversize-body 413, deterministic matching. - Full backend suite: 1757 pass / 5 skipped.
This commit is contained in:
@@ -4,7 +4,8 @@ import semver from 'semver';
|
||||
import si from 'systeminformation';
|
||||
import type Dockerode from 'dockerode';
|
||||
import { DatabaseService, type Node } from '../services/DatabaseService';
|
||||
import { FleetSyncService } from '../services/FleetSyncService';
|
||||
import { FleetSyncService, StaleSyncPushError } from '../services/FleetSyncService';
|
||||
import { MAX_SYNC_ROWS, SYNC_ERROR_CODES } from '../services/fleetSyncConstants';
|
||||
import { FleetUpdateTrackerService } from '../services/FleetUpdateTrackerService';
|
||||
import { NodeRegistry } from '../services/NodeRegistry';
|
||||
import DockerController from '../services/DockerController';
|
||||
@@ -37,7 +38,6 @@ const UPDATE_TIMEOUT_MS = 5 * 60 * 1000; // 5 minutes
|
||||
const UPDATE_TIMEOUT_MSG = 'Node did not come back online within 5 minutes.';
|
||||
const EARLY_FAIL_MS = 180 * 1000; // 3 minutes before declaring a probable pull failure
|
||||
|
||||
const MAX_SYNC_ROWS = 5000;
|
||||
const CVE_ID_RE = /^(CVE-\d{4}-\d{4,}|GHSA-[\w-]{14,})$/;
|
||||
|
||||
const isIntFlag = (v: unknown): v is 0 | 1 => v === 0 || v === 1;
|
||||
@@ -298,6 +298,11 @@ fleetRouter.get('/role', authMiddleware, (req: Request, res: Response): void =>
|
||||
|
||||
// Receive a full replacement of a replicated resource from the control.
|
||||
// Restricted to node_proxy Bearer tokens so only a sibling Sencho can push.
|
||||
//
|
||||
// No requirePaid here: the control instance has already enforced its tier
|
||||
// before issuing the push. The replica trusts a valid node_proxy bearer
|
||||
// signed against THIS instance's secret and applies the payload regardless
|
||||
// of the replica's own tier.
|
||||
fleetRouter.post('/sync/:resource', authMiddleware, (req: Request, res: Response): void => {
|
||||
if (!requireNodeProxy(req, res)) return;
|
||||
const resource = req.params.resource;
|
||||
@@ -308,6 +313,14 @@ fleetRouter.post('/sync/:resource', authMiddleware, (req: Request, res: Response
|
||||
const body = req.body ?? {};
|
||||
const rows = Array.isArray(body.rows) ? body.rows : null;
|
||||
const targetIdentity = typeof body.targetIdentity === 'string' ? body.targetIdentity : '';
|
||||
// pushedAt is optional for back-compat with older controls that predate the
|
||||
// versioning protocol. When present and strictly older than the most recent
|
||||
// applied push for this resource, reject with 409 STALE_SYNC_PUSH so the
|
||||
// control's retry logic can fall back to the next write. Negative or zero
|
||||
// values are treated as absent: the sender always uses Date.now().
|
||||
const pushedAt = typeof body.pushedAt === 'number' && Number.isFinite(body.pushedAt) && body.pushedAt > 0
|
||||
? body.pushedAt
|
||||
: null;
|
||||
if (!rows) {
|
||||
res.status(400).json({ error: 'rows array is required' });
|
||||
return;
|
||||
@@ -325,9 +338,21 @@ fleetRouter.post('/sync/:resource', authMiddleware, (req: Request, res: Response
|
||||
}
|
||||
}
|
||||
try {
|
||||
FleetSyncService.getInstance().applyIncomingSync(resource, rows, targetIdentity);
|
||||
FleetSyncService.getInstance().applyIncomingSync(
|
||||
resource,
|
||||
rows,
|
||||
targetIdentity,
|
||||
pushedAt ?? undefined,
|
||||
);
|
||||
res.json({ success: true, applied: rows.length });
|
||||
} catch (error) {
|
||||
if (error instanceof StaleSyncPushError) {
|
||||
res.status(409).json({
|
||||
error: error.message,
|
||||
code: SYNC_ERROR_CODES.staleSyncPush,
|
||||
});
|
||||
return;
|
||||
}
|
||||
console.error('[FleetSync] Failed to apply incoming sync:', error);
|
||||
res.status(500).json({ error: 'Failed to apply sync' });
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user