gsadmin/sencho
1
0
Fork 0
mirror of https://github.com/Studio-Saelix/sencho.git synced 2026-09-03 22:25:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(fleet-actions): stop-by-label works on Community remote nodes (#1270)

Browse Source 
* fix(fleet-actions): stop-by-label works on Community remote nodes

Fleet-stop's remote leg fanned out to POST /api/labels/:id/action, which
is gated to Skipper/Admiral, so on a Community fleet the control node
stopped its own stacks but every remote node returned 403. Fleet-stop
itself is admin-only and available on every license, so the remote leg
contradicted the feature's own gate.

Extract the label-match plus bulk-stop logic into a shared
runLocalLabelStop helper and add an admin-only, every-license
POST /api/fleet-actions/labels/local-stop receiver. The control now fans
out to that receiver, so remote stacks stop on every tier. Each node runs
under its own per-node bulk lock, so a fleet-stop and a per-label action
still serialize cleanly instead of double-stopping containers.

Also degrade the control's own leg per-node instead of failing the whole
fan-out when its filesystem read throws, and gate fleet-stop and
fleet-prune diagnostics behind developer_mode.

Tests: local-stop auth, tier, validation, and behavior; a remote-leg
routing guard that asserts the fan-out targets local-stop and never the
paid route; local-leg graceful degradation; and the three Fleet Action
card UIs.

* fix(fleet-actions): honor the remote stop receiver's matched flag

The control reached the remote leg only because its own mirror had the
label, then hardcoded matched:true and trusted results without guarding
its shape. A mirror-skewed control (mirror has the label, remote does
not) then showed a remote mismatch as "matched, 0 stacks" instead of
"no matching label", and a malformed 200 body could flow a non-array
into the per-stack renderers.

Honor the remote's own matched flag and coerce results to an array when
the body is malformed. Add regression tests for the matched:false skew
case and the non-array results case.
This commit is contained in:
Anso
2026-06-01 14:18:44 -04:00
committed by GitHub
parent 085267b466
commit 7e0cffa376
9 changed files with 797 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/src/routes/fleet.ts
+64 -43
View File
@@ -34,8 +34,8 @@ import { formatNoTargetError } from '../utils/remoteTarget';
import { CloudBackupService } from '../services/CloudBackupService';
import { NotificationService } from '../services/NotificationService';
import { invalidateNodeCaches, invalidateRemoteMetaCache } from '../helpers/cacheInvalidation';
import { containerActionForStack } from './stacks';
import { activeBulkActions } from './labels';
import { runLocalLabelStop, type LabelLocalStopResponse, type StackStopResult } from '../helpers/fleetLabelStop';
import { buildLocalConfigurationStatus, type ConfigurationStatus } from './dashboard';
import { PROXY_TIER_HEADER, PROXY_VARIANT_HEADER } from '../services/license-headers';
import { LicenseService } from '../services/LicenseService';
@@ -1047,6 +1047,12 @@ fleetRouter.delete('/update-status', authMiddleware, async (req: Request, res: R
// in `routes/fleetActions.ts`. The endpoint below is gateway-orchestrated and
// lives here so it sits behind the `/api/fleet/` proxy-exempt prefix.
// Attribute one error to every stack a node was supposed to act on. Used for
// the fleet-stop failure paths (no proxy target, non-ok remote, transport
// error, local exception) so each stack carries the same node-level cause.
const failAllStacks = (stacks: string[], error: string): StackStopResult[] =>
stacks.map(stackName => ({ stackName, success: false, error }));
// Fleet-wide stop by label name. Matches each node's labels by name and runs
// container stops on each matching stack.
// Tier: requireAdmin (admin-only fleet plumbing; available on every license).
@@ -1067,7 +1073,36 @@ fleetRouter.post('/labels/fleet-stop', authMiddleware, async (req: Request, res:
try {
const db = DatabaseService.getInstance();
const nodes = db.getNodes();
if (isDebugEnabled()) console.debug('[Fleet:debug] fleet-stop:', { labelName: trimmed, dryRun: isDryRun, nodes: nodes.length });
const results = await Promise.all(nodes.map(async (node) => {
if (node.type === 'local') {
// Match + stop runs in-process against the control's own Docker. The
// helper shares the per-node `bulk:<id>` lock with the per-label action
// route so the two cannot double-stop the same containers. A control-side
// failure (e.g. the compose dir is unreadable) degrades to a per-stack
// error for this node only, the same way the remote leg does, so one bad
// node never discards the rest of the fleet's results.
try {
const outcome = await runLocalLabelStop(node.id, trimmed, isDryRun);
return { nodeId: node.id, nodeName: node.name, matched: outcome.matched, stackResults: outcome.stackResults };
} catch (err) {
const errorMsg = getErrorMessage(err, 'Failed to stop local stacks');
const localLabel = db.getLabels(node.id).find(l => l.name === trimmed);
const localStacks = localLabel ? db.getStacksForLabel(localLabel.id, node.id) : [];
return {
nodeId: node.id, nodeName: node.name, matched: !!localLabel,
stackResults: failAllStacks(localStacks, errorMsg),
};
}
}
// Remote node. The control's mirror tells us which stacks *should* match
// so transport errors can be attributed per stack; the authoritative stop
// runs on the remote via its admin-only local-stop receiver, which reuses
// the same name-matched helper under the remote's own bulk lock. This
// replaces the previous fan-out to `POST /api/labels/:id/action`, a
// paid-tier route that 403'd on Community remotes even though fleet-stop
// itself is available on every license.
const label = db.getLabels(node.id).find(l => l.name === trimmed);
if (!label) {
return { nodeId: node.id, nodeName: node.name, matched: false, stackResults: [] };
@@ -1077,56 +1112,21 @@ fleetRouter.post('/labels/fleet-stop', authMiddleware, async (req: Request, res:
return { nodeId: node.id, nodeName: node.name, matched: true, stackResults: [] };
}
if (node.type === 'local') {
// Share the per-node bulk lock with `POST /api/labels/:id/action` so
// a fleet-stop and a per-label action cannot double-stop the same
// containers concurrently on the same local node. Dry run acquires
// the same lock so the rehearsal exercises the same contention path.
const lockKey = `bulk:${node.id}`;
if (activeBulkActions.has(lockKey)) {
return {
nodeId: node.id, nodeName: node.name, matched: true,
stackResults: stackNames.map(stackName => ({ stackName, success: false, error: 'A bulk action is already running on this node' })),
};
}
activeBulkActions.add(lockKey);
try {
const fsStacks = await FileSystemService.getInstance(node.id).getStacks();
const fsStackSet = new Set(fsStacks);
const validStacks = stackNames.filter(name => fsStackSet.has(name));
const stackResults: { stackName: string; success: boolean; error?: string; dryRun?: boolean }[] = [];
for (const stackName of validStacks) {
if (isDryRun) {
stackResults.push({ stackName, success: true, dryRun: true });
continue;
}
const outcome = await containerActionForStack(node.id, stackName, 'stop');
if (outcome.kind === 'ok') stackResults.push({ stackName, success: true });
else if (outcome.kind === 'no-containers') stackResults.push({ stackName, success: false, error: 'No containers found for this stack' });
else stackResults.push({ stackName, success: false, error: outcome.message });
}
if (!isDryRun && stackResults.some(r => r.success)) invalidateNodeCaches(node.id);
return { nodeId: node.id, nodeName: node.name, matched: true, stackResults };
} finally {
activeBulkActions.delete(lockKey);
}
}
const target = NodeRegistry.getInstance().getProxyTarget(node.id);
if (!target) {
const error = formatNoTargetError(node);
return {
nodeId: node.id, nodeName: node.name, matched: true,
stackResults: stackNames.map(stackName => ({ stackName, success: false, error })),
stackResults: failAllStacks(stackNames, error),
};
}
try {
const headers: Record<string, string> = { 'Content-Type': 'application/json' };
if (target.apiToken) headers.Authorization = `Bearer ${target.apiToken}`;
const response = await fetch(`${target.apiUrl.replace(/\/$/, '')}/api/labels/${label.id}/action`, {
const response = await fetch(`${target.apiUrl.replace(/\/$/, '')}/api/fleet-actions/labels/local-stop`, {
method: 'POST',
headers,
body: JSON.stringify({ action: 'stop', dryRun: isDryRun }),
body: JSON.stringify({ labelName: trimmed, dryRun: isDryRun }),
signal: AbortSignal.timeout(60000),
});
if (!response.ok) {
@@ -1134,19 +1134,34 @@ fleetRouter.post('/labels/fleet-stop', authMiddleware, async (req: Request, res:
const message = err.error || `Remote returned ${response.status}`;
return {
nodeId: node.id, nodeName: node.name, matched: true,
stackResults: stackNames.map(stackName => ({ stackName, success: false, error: message })),
stackResults: failAllStacks(stackNames, message),
};
}
const remote = (await response.json()) as { results?: { stackName: string; success: boolean; error?: string; dryRun?: boolean }[] };
return { nodeId: node.id, nodeName: node.name, matched: true, stackResults: remote.results ?? [] };
// Trust the remote's own matched flag over the control's mirror: a
// mirror-skewed control could believe the label exists while the remote
// has no such label, which the remote reports as matched:false. Guard
// results as an array so a malformed 200 body degrades to empty rather
// than flowing a non-array into the per-stack renderers.
const remote = (await response.json()) as Partial<LabelLocalStopResponse>;
return {
nodeId: node.id, nodeName: node.name,
matched: remote.matched ?? true,
stackResults: Array.isArray(remote.results) ? remote.results : [],
};
} catch (err) {
const errorMsg = getErrorMessage(err, 'Failed to reach remote node');
return {
nodeId: node.id, nodeName: node.name, matched: true,
stackResults: stackNames.map(stackName => ({ stackName, success: false, error: errorMsg })),
stackResults: failAllStacks(stackNames, errorMsg),
};
}
}));
if (isDebugEnabled()) {
const matched = results.filter(r => r.matched).length;
const stopped = results.reduce((n, r) => n + r.stackResults.filter(s => s.success).length, 0);
const failed = results.reduce((n, r) => n + r.stackResults.filter(s => !s.success).length, 0);
console.debug('[Fleet:debug] fleet-stop complete:', { matched, stopped, failed });
}
res.json({ results });
} catch (error) {
console.error('[Fleet] fleet-stop error:', error);
@@ -1199,6 +1214,7 @@ fleetRouter.post('/labels/fleet-prune', authMiddleware, async (req: Request, res
try {
const db = DatabaseService.getInstance();
const nodes = db.getNodes();
if (isDebugEnabled()) console.debug('[Fleet:debug] fleet-prune:', { targets, scope, dryRun: isDryRun, nodes: nodes.length });
const results: NodeResult[] = await Promise.all(nodes.map(async (node): Promise<NodeResult> => {
if (node.type === 'local') {
@@ -1307,6 +1323,11 @@ fleetRouter.post('/labels/fleet-prune', authMiddleware, async (req: Request, res
};
}));
if (isDebugEnabled()) {
const reachable = results.filter(r => r.reachable).length;
const reclaimed = results.reduce((n, r) => n + r.targets.reduce((m, t) => m + t.reclaimedBytes, 0), 0);
console.debug('[Fleet:debug] fleet-prune complete:', { reachable, unreachable: results.length - reachable, reclaimedBytes: reclaimed });
}
res.json({ results });
} catch (error) {
console.error('[Fleet] fleet-prune error:', error);