gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-08-25 17:57:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(fleet): route remaining fleet dispatches through getProxyTarget for pilot-agent nodes (#1152)

Browse Source 
* fix(fleet): route remaining fleet dispatches through getProxyTarget for pilot-agent nodes

PR #1123 migrated POST /api/fleet/nodes/:id/update to use
NodeRegistry.getProxyTarget so pilot-agent rows (no api_url / api_token)
participate in remote update via the tunnel loopback. The same bug shape
lived on at ten sibling fleet-dispatch sites: each read node.api_url and
node.api_token directly, returning "Remote node not configured." against
pilots, or silently filtered pilot rows out of a fan-out loop.

Migrate every remaining fleet-wide remote dispatch to the same pattern:

- routes/fleet.ts: fleet-stop, fleet-prune, prune/estimate, snapshot
  restore (4 sites) -> getProxyTarget + mode-aware error copy +
  conditional Authorization header
- routes/imageUpdates.ts: fleet status + fleet refresh (2 sites) -> swap
  n.api_url filter for getProxyTarget != null and use target.apiUrl, so
  pilot rows appear in the aggregated image-updates view instead of
  being silently excluded
- utils/snapshot-capture.ts: captureRemoteNodeFiles (1 site) -> same
  pattern; CaptureNode interface gains required mode field so the
  thrown error message picks the pilot-tunnel copy automatically
- services/SecretsService.ts: resolveEnvFileRemote, readEnvRemote,
  writeEnvRemote (3 sites) -> same pattern; thrown errors now use the
  shared formatNoTargetError helper instead of leaking api_url/api_token
  field names

Extract the previously-private noTargetMessage helper from fleet.ts
into utils/remoteTarget.ts as formatNoTargetError so SecretsService,
snapshot-capture, and the fleet routes share one copy of the
mode-aware error string.

Add 10 regression tests in fleet-pilot-dispatch-parity.test.ts covering
each migrated route + the snapshot-capture utility: dispatch through
the loopback target with no Authorization header for pilots, and a
mode-aware error when the tunnel is disconnected.

FleetSyncService (4 additional sites) carries an api_url-anchored
targetIdentity in the wire protocol; pilot support there needs a
protocol-level identity decision and stays as a separate follow-up.

* fix(fleet): throw tunnel-disconnected error from resolveEnvFileRemote

Codex audit flagged that resolveEnvFileRemote returned null when
getProxyTarget was null. That predates the parity migration but the
migration was the right place to fix it: the null flowed through
readExistingEnv into previewPushDiff / executePush as "env file not
found", which is wrong (the env exists, the node is unreachable).

Throwing formatNoTargetError(node) here lets the existing catch arms
in previewPushDiff (lines 450-453) surface reachable=false with the
tunnel-disconnected message on the right axis, and executePush picks
up the same shape via its outer catch.

Also drop overstated coverage claims from the parity test header
(snapshot restore + SecretsService were never actually exercised in
this file, only structurally identical via tsc), and fix two describe
labels that read /api/labels/* instead of the mounted /api/fleet/labels/*.
This commit is contained in:
Anso
2026-05-22 00:30:51 -04:00
committed by GitHub
parent 60f893a81f
commit 2f2401df68
6 changed files with 468 additions and 56 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/fleet.ts
+36 -29
View File
@@ -25,6 +25,7 @@ import { getErrorMessage } from '../utils/errors';
import { parseIntParam } from '../utils/parseIntParam';
import { POLICY_SEVERITIES } from '../utils/severity';
import { sanitizeForLog } from '../utils/safeLog';
import { formatNoTargetError } from '../utils/remoteTarget';
import { CloudBackupService } from '../services/CloudBackupService';
import { NotificationService } from '../services/NotificationService';
import { invalidateNodeCaches } from '../helpers/cacheInvalidation';
@@ -228,12 +229,6 @@ function pilotLastSeenSeconds(node: Node): number | null {
: null;
}
function noTargetMessage(node: Node): string {
return node.mode === 'pilot_agent'
? `Pilot tunnel to "${node.name}" is disconnected. Operations resume when the agent reconnects.`
: 'Remote node not configured';
}
function offlineRemoteOverview(node: Node, status: 'online' | 'offline'): FleetNodeOverview {
const pilotSeen = pilotLastSeenSeconds(node);
// For pilot-agent rows the tunnel heartbeat is the contact signal. Mirror
@@ -619,7 +614,7 @@ fleetRouter.get('/node/:nodeId/stacks', authMiddleware, async (req: Request, res
if (node.type === 'remote') {
const target = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!target) {
res.status(503).json({ error: noTargetMessage(node) });
res.status(503).json({ error: formatNoTargetError(node) });
return;
}
const response = await fetch(`${target.apiUrl.replace(/\/$/, '')}/api/stacks`, {
@@ -663,7 +658,7 @@ fleetRouter.get('/node/:nodeId/stacks/:stackName/containers', authMiddleware, as
if (node.type === 'remote') {
const target = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!target) {
res.status(503).json({ error: noTargetMessage(node) });
res.status(503).json({ error: formatNoTargetError(node) });
return;
}
const response = await fetch(`${target.apiUrl.replace(/\/$/, '')}/api/stacks/${encodeURIComponent(stackName)}/containers`, {
@@ -893,10 +888,7 @@ fleetRouter.post('/nodes/:nodeId/update', authMiddleware, async (req: Request, r
const target = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!target) {
const msg = node.mode === 'pilot_agent'
? `Pilot tunnel to "${node.name}" is disconnected.`
: 'Remote node not configured.';
res.status(503).json({ error: msg });
res.status(503).json({ error: formatNoTargetError(node) });
return;
}
@@ -1100,16 +1092,20 @@ fleetRouter.post('/labels/fleet-stop', authMiddleware, async (req: Request, res:
}
}
if (!node.api_url || !node.api_token) {
const target = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!target) {
const error = formatNoTargetError(node);
return {
nodeId: node.id, nodeName: node.name, matched: true,
stackResults: stackNames.map(stackName => ({ stackName, success: false, error: 'Remote node not configured' })),
stackResults: stackNames.map(stackName => ({ stackName, success: false, error })),
};
}
try {
const response = await fetch(`${node.api_url.replace(/\/$/, '')}/api/labels/${label.id}/action`, {
const headers: Record<string, string> = { 'Content-Type': 'application/json' };
if (target.apiToken) headers.Authorization = `Bearer ${target.apiToken}`;
const response = await fetch(`${target.apiUrl.replace(/\/$/, '')}/api/labels/${label.id}/action`, {
method: 'POST',
headers: { Authorization: `Bearer ${node.api_token}`, 'Content-Type': 'application/json' },
headers,
body: JSON.stringify({ action: 'stop', dryRun: isDryRun }),
signal: AbortSignal.timeout(60000),
});
@@ -1227,13 +1223,17 @@ fleetRouter.post('/labels/fleet-prune', authMiddleware, async (req: Request, res
// Remote node: POST /api/system/prune/system per target, short-circuiting
// on the first transport-level failure so we don't hammer a dead node.
if (!node.api_url || !node.api_token) {
const proxyTarget = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!proxyTarget) {
const error = formatNoTargetError(node);
return {
nodeId: node.id, nodeName: node.name, reachable: false, error: 'Remote node not configured',
targets: targets.map(t => ({ target: t, success: false, reclaimedBytes: 0, error: 'Remote node not configured' })),
nodeId: node.id, nodeName: node.name, reachable: false, error,
targets: targets.map(t => ({ target: t, success: false, reclaimedBytes: 0, error })),
};
}
const baseUrl = node.api_url.replace(/\/$/, '');
const baseUrl = proxyTarget.apiUrl.replace(/\/$/, '');
const remoteHeaders: Record<string, string> = { 'Content-Type': 'application/json' };
if (proxyTarget.apiToken) remoteHeaders.Authorization = `Bearer ${proxyTarget.apiToken}`;
const targetResults: TargetResult[] = [];
let nodeUnreachable: string | null = null;
for (const target of targets) {
@@ -1244,7 +1244,7 @@ fleetRouter.post('/labels/fleet-prune', authMiddleware, async (req: Request, res
try {
const response = await fetch(`${baseUrl}/api/system/prune/system`, {
method: 'POST',
headers: { Authorization: `Bearer ${node.api_token}`, 'Content-Type': 'application/json' },
headers: remoteHeaders,
body: JSON.stringify({ target, scope, dryRun: isDryRun }),
signal: AbortSignal.timeout(120000),
});
@@ -1388,13 +1388,16 @@ fleetRouter.post('/prune/estimate', authMiddleware, async (req: Request, res: Re
}
}
if (!node.api_url || !node.api_token) {
const proxyTarget = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!proxyTarget) {
return {
nodeId: node.id, nodeName: node.name, reclaimableBytes: 0, reachable: false,
error: 'Remote node not configured',
error: formatNoTargetError(node),
};
}
const baseUrl = node.api_url.replace(/\/$/, '');
const baseUrl = proxyTarget.apiUrl.replace(/\/$/, '');
const estimateHeaders: Record<string, string> = { 'Content-Type': 'application/json' };
if (proxyTarget.apiToken) estimateHeaders.Authorization = `Bearer ${proxyTarget.apiToken}`;
// Estimate is a live readout; fan out the per-target fetches in parallel
// so wall time matches the slowest single call rather than the sum.
// (The destructive sibling stays serial because Docker prune is internally
@@ -1403,7 +1406,7 @@ fleetRouter.post('/prune/estimate', authMiddleware, async (req: Request, res: Re
try {
const response = await fetch(`${baseUrl}/api/system/prune/estimate`, {
method: 'POST',
headers: { Authorization: `Bearer ${node.api_token}`, 'Content-Type': 'application/json' },
headers: estimateHeaders,
body: JSON.stringify({ target, scope }),
signal: AbortSignal.timeout(15000),
});
@@ -1656,19 +1659,23 @@ fleetRouter.post('/snapshots/:id/restore', authMiddleware, async (req: Request,
await composeService.deployStack(stackName);
}
} else {
if (!node.api_url || !node.api_token) {
res.status(503).json({ error: 'Remote node not configured' });
const proxyTarget = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!proxyTarget) {
res.status(503).json({ error: formatNoTargetError(node) });
return;
}
const baseUrl = node.api_url.replace(/\/$/, '');
const baseUrl = proxyTarget.apiUrl.replace(/\/$/, '');
const proxyHeaders = LicenseService.getInstance().getProxyHeaders();
// Tier/variant headers describe the central instance and stay
// unconditional; the Bearer header is gated on a non-empty token
// because pilot-loopback dispatch carries auth via the tunnel.
const headers: Record<string, string> = {
Authorization: `Bearer ${node.api_token}`,
'Content-Type': 'application/json',
[PROXY_TIER_HEADER]: proxyHeaders.tier,
[PROXY_VARIANT_HEADER]: proxyHeaders.variant ?? '',
};
if (proxyTarget.apiToken) headers.Authorization = `Bearer ${proxyTarget.apiToken}`;
for (const file of files) {
if (file.filename === 'compose.yaml') {
backend/src/routes/imageUpdates.ts
+20 -10
View File
@@ -71,16 +71,21 @@ imageUpdatesRouter.get('/fleet', authMiddleware, async (_req: Request, res: Resp
}
// Remote nodes: parallel fetches with per-request timeouts.
const remoteNodes = nodes.filter(n => n.type === 'remote' && n.status === 'online' && n.api_url);
// Pilot-agent rows have no api_url; rely on getProxyTarget for the
// reachability predicate AND the base URL so pilots with an active
// tunnel participate in the fan-out.
const remoteCandidates = nodes
.filter(n => n.type === 'remote' && n.status === 'online')
.map(node => ({ node, proxyTarget: nr.getProxyTarget(node.id) }))
.filter((entry): entry is { node: typeof entry.node; proxyTarget: NonNullable<typeof entry.proxyTarget> } => entry.proxyTarget !== null);
const remoteResults = await Promise.allSettled(
remoteNodes.map(async (node) => {
const proxyTarget = nr.getProxyTarget(node.id);
const baseUrl = node.api_url!.replace(/\/$/, '');
remoteCandidates.map(async ({ node, proxyTarget }) => {
const baseUrl = proxyTarget.apiUrl.replace(/\/$/, '');
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), REMOTE_NODE_FETCH_TIMEOUT_MS);
try {
const resp = await fetch(`${baseUrl}/api/image-updates`, {
headers: proxyTarget?.apiToken
headers: proxyTarget.apiToken
? { Authorization: `Bearer ${proxyTarget.apiToken}` }
: {},
signal: controller.signal,
@@ -138,17 +143,22 @@ imageUpdatesRouter.post('/fleet/refresh', authMiddleware, async (_req: Request,
}
}
const remoteNodes = nodes.filter(n => n.type === 'remote' && n.status === 'online' && n.api_url);
// Pilot-agent rows have no api_url; rely on getProxyTarget for the
// reachability predicate AND the base URL so pilots with an active
// tunnel participate in the fan-out.
const remoteCandidates = nodes
.filter(n => n.type === 'remote' && n.status === 'online')
.map(node => ({ node, proxyTarget: nr.getProxyTarget(node.id) }))
.filter((entry): entry is { node: typeof entry.node; proxyTarget: NonNullable<typeof entry.proxyTarget> } => entry.proxyTarget !== null);
const remoteResults = await Promise.allSettled(
remoteNodes.map(async (node) => {
const proxyTarget = nr.getProxyTarget(node.id);
const baseUrl = node.api_url!.replace(/\/$/, '');
remoteCandidates.map(async ({ node, proxyTarget }) => {
const baseUrl = proxyTarget.apiUrl.replace(/\/$/, '');
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), REMOTE_NODE_FETCH_TIMEOUT_MS);
try {
const resp = await fetch(`${baseUrl}/api/image-updates/refresh`, {
method: 'POST',
headers: proxyTarget?.apiToken
headers: proxyTarget.apiToken
? { Authorization: `Bearer ${proxyTarget.apiToken}` }
: {},
signal: controller.signal,