fix: gate cross-node HTTP and stop-by-label on remote RBAC capability (#1509)

An older remote node ignores the forwarded actor-role header (running proxied
requests as admin) and ignores the stop-by-label stack allowlist (stopping
every label-matched stack). The control could neither detect nor prevent this
on a mixed-version fleet.

Instances now advertise a cross-node-rbac capability, and the control refuses
to act when a remote lacks it:

- HTTP proxy: a non-admin user's request is not forwarded to a remote that does
  not advertise the capability (fails closed when it cannot be determined).
  Admins are unaffected.
- Stop-by-label: a real stop bound to a confirmed stack set is not sent to a
  remote lacking the capability; the node is reported as needing an upgrade. As
  defense in depth, a node whose results name stacks outside the confirmed set
  is failed rather than rendered as a clean stop.

Separately, the stop's lock-contention path now reports every confirmed stack
as a contention failure (including one that lost its label), so a confirmed
stack is never silently dropped and the result is never empty.
This commit is contained in:
Anso
2026-06-28 18:28:47 -04:00
committed by GitHub
parent ef164f0e5b
commit 997a6bb79a
11 changed files with 365 additions and 1 deletions
+44
View File
@@ -0,0 +1,44 @@
import { CacheService } from '../services/CacheService';
import { NodeRegistry } from '../services/NodeRegistry';
import { CROSS_NODE_RBAC_CAPABILITY, type RemoteMeta } from '../services/CapabilityRegistry';
import { REMOTE_META_NAMESPACE } from './cacheInvalidation';
import { getErrorMessage } from '../utils/errors';
// Mirrors the node-meta endpoint's TTL and shares its `remote-meta:<id>` cache
// key, so a recent /api/nodes/:id/meta read warms this check and vice versa.
const REMOTE_META_CACHE_TTL = 3 * 60 * 1000;
/**
* Whether a remote node advertises that it enforces cross-node RBAC: the
* forwarded actor role on HTTP requests and the exact-stack allowlist on
* stop-by-label. Reads the shared remote-meta cache, fetching once on a cold
* miss.
*
* Fails closed: when the capability cannot be established (an un-upgraded
* remote, or a cold cache whose meta fetch fails) it returns false, so the
* caller denies rather than risk escalating a non-admin request or over-stopping
* on an un-upgraded node. A node previously cached as supported may be served
* that value while a later refresh is failing (getOrFetch serves stale on
* error); that is safe because capabilities are append-only and a request to an
* unreachable node fails at the transport regardless.
*/
export async function remoteSupportsCrossNodeRbac(nodeId: number): Promise<boolean> {
try {
const meta = await CacheService.getInstance().getOrFetch<RemoteMeta>(
`${REMOTE_META_NAMESPACE}:${nodeId}`,
REMOTE_META_CACHE_TTL,
async () => {
const fetched = await NodeRegistry.getInstance().fetchMetaForNode(nodeId);
if (fetched.version === null) throw new Error('Remote meta fetch returned null version');
return fetched;
},
);
return meta.capabilities.includes(CROSS_NODE_RBAC_CAPABILITY);
} catch (err) {
console.warn(
`[CrossNodeRBAC] Could not determine capability for node ${nodeId}; treating as unsupported:`,
getErrorMessage(err, 'unknown'),
);
return false;
}
}