The cross-node capability gate cached its verdict, so a remote replaced by older
code at the same URL stayed trusted until the cache expired, reopening the
non-admin HTTP escalation and the over-broad stop. The probe now hits the
remote's live /api/meta on every gated action (concurrent calls deduped, never
cached across requests, fail-closed), so a downgraded remote is detected
immediately.
Two stop-result gaps are also closed:
- A remote stop result must now cover exactly the confirmed stacks (one per
stack, no extras, no omissions), not merely exclude extras, so a dropped
confirmed stack is no longer accepted as clean. runLocalLabelStop reports one
result per confirmed stack even when the label has vanished, so a current
remote always satisfies the check.
- The local stop exception path now reports the full confirmed set, so a
confirmed stack that lost its label is not dropped when the local stop throws.
An older remote node ignores the forwarded actor-role header (running proxied
requests as admin) and ignores the stop-by-label stack allowlist (stopping
every label-matched stack). The control could neither detect nor prevent this
on a mixed-version fleet.
Instances now advertise a cross-node-rbac capability, and the control refuses
to act when a remote lacks it:
- HTTP proxy: a non-admin user's request is not forwarded to a remote that does
not advertise the capability (fails closed when it cannot be determined).
Admins are unaffected.
- Stop-by-label: a real stop bound to a confirmed stack set is not sent to a
remote lacking the capability; the node is reported as needing an upgrade. As
defense in depth, a node whose results name stacks outside the confirmed set
is failed rather than rendered as a clean stop.
Separately, the stop's lock-contention path now reports every confirmed stack
as a contention failure (including one that lost its label), so a confirmed
stack is never silently dropped and the result is never empty.
A fleet stop re-matched stacks by label name at execution, so a stack that
gained the label between the operator's preview and confirmation could be
stopped even though it never appeared in the confirmation. A confirmed node
that was deleted after the preview also vanished from the results, letting the
remaining successes read as a clean stop.
The confirm flow now sends the exact node and stack list resolved in the
preview. Each node's stop is bound to that set: only stacks that are still
label-matched and confirmed are stopped, and a confirmed node missing from the
registry is reported as an explicit failure rather than dropped.
* fix: harden cross-node fleet label actions and guard container reads
Release-stabilization fixes for the Fleet Actions surface:
- Stop-by-label binds execution to the nodes shown in the confirmed
preview. The real stop sends the confirmed node ids and the backend
restricts the fan-out to them, so a node that was unreachable during
preview and reconnects before the stop can no longer enter execution
and have unlisted stacks stopped.
- Bulk label assign validates each remote node's result against the
stacks it was asked to label: a body whose results are empty, partial,
duplicated, or shaped wrong is a per-node failure instead of reading as
a successful zero-stack assign. The card mirrors this, rejecting a
missing or non-array results body and only reporting success when at
least one stack was assigned.
- Bulk label assign re-reads authoritative per-node stacks and labels on
demand via a Refresh control, and the confirmation lists the affected
node and stack names rather than bare counts.
- The stack-specific and fleet container/stack read routes require the
stack:read permission, matching the generic container and stack routes.
Every shipped role already carries stack:read, so reachability is
unchanged; the guard closes the routes that were auth-only.
Adds unit coverage for the assign-result validator, route coverage for
the stop allowlist and assign membership checks, and authorization
coverage for the newly guarded reads.
* test: assert the confirmed node allowlist in the fleet stop-card test
The stop-card component test pinned the real-stop request body to
{ labelName, dryRun } and broke once the stop began carrying the
confirmed-preview node ids. Update it to expect the nodeIds allowlist
derived from the resolved preview, so the test asserts the binding
rather than the pre-fix shape.
* feat(fleet): cross-node bulk label assign with authoritative label discovery
Make Fleet Actions > Bulk label assign work across the fleet. Pick a stack
label that exists anywhere in the fleet, select stacks on one or more nodes,
and the control orchestrates: each target node resolves the label by name,
creating it with the same name and color if missing, then adds it to the
selected stacks while preserving their existing labels. The local node runs
in process; each remote runs its own admin-only local-assign receiver over
the node proxy. Per-node failures (unknown node, no proxy target, unreachable,
mixed-version remote) degrade that node only and are reported per node in the
result. Assignment writes use a transactional INSERT OR IGNORE so the
add-preserve path is idempotent and race-free.
Also make the shared fleet label discovery authoritative: suggestions,
match-preview, and the fleet-stop remote leg now read each node's labels live
over the proxy instead of the control database, which does not mirror remote
labels. A propagated label therefore appears in, and is stoppable by,
Stop-by-label across the fleet, and unreachable nodes are surfaced rather than
silently dropped.
Fleet Actions runs against the unfiltered node list, so overview filters no
longer narrow its scope. The previous node-scoped, replace-by-id bulk-assign
endpoint is removed.
* fix(fleet): treat malformed remote label responses as per-node failures
A 200 response from a remote node whose body is not the expected shape was
treated as a benign empty result, so a malformed remote could read as a clean
zero-stack assign or a "matched, nothing to stop" no-op and even surface a
success toast. Validate the wire shape in the bulk-assign and fleet-stop remote
legs and in the authoritative label discovery fan-out; on a malformed body,
report the node as a per-node failure with the error attributed to its stacks
instead of silently dropping it.
* chore: drop accidentally committed temp file
* fix(fleet-actions): stop-by-label works on Community remote nodes
Fleet-stop's remote leg fanned out to POST /api/labels/:id/action, which
is gated to Skipper/Admiral, so on a Community fleet the control node
stopped its own stacks but every remote node returned 403. Fleet-stop
itself is admin-only and available on every license, so the remote leg
contradicted the feature's own gate.
Extract the label-match plus bulk-stop logic into a shared
runLocalLabelStop helper and add an admin-only, every-license
POST /api/fleet-actions/labels/local-stop receiver. The control now fans
out to that receiver, so remote stacks stop on every tier. Each node runs
under its own per-node bulk lock, so a fleet-stop and a per-label action
still serialize cleanly instead of double-stopping containers.
Also degrade the control's own leg per-node instead of failing the whole
fan-out when its filesystem read throws, and gate fleet-stop and
fleet-prune diagnostics behind developer_mode.
Tests: local-stop auth, tier, validation, and behavior; a remote-leg
routing guard that asserts the fan-out targets local-stop and never the
paid route; local-leg graceful degradation; and the three Fleet Action
card UIs.
* fix(fleet-actions): honor the remote stop receiver's matched flag
The control reached the remote leg only because its own mirror had the
label, then hardcoded matched:true and trusted results without guarding
its shape. A mirror-skewed control (mirror has the label, remote does
not) then showed a remote mismatch as "matched, 0 stacks" instead of
"no matching label", and a malformed 200 body could flow a non-array
into the per-stack renderers.
Honor the remote's own matched flag and coerce results to an array when
the body is malformed. Add regression tests for the matched:false skew
case and the non-array results case.
* feat(fleet): open Fleet Actions tab to Community (admin-only)
Removes the requirePaid guard from the five Fleet Actions endpoints
(fleet-stop, fleet-prune, match-preview, prune/estimate, bulk-assign)
and drops the matching isPaid parent gate on FleetActionsTab so Community
admins can run fleet-wide bulk operations. requireAdmin stays on every
endpoint; operator and viewer roles still 403 on apply.
Tests flipped from "403 PAID_REQUIRED on community" to positive
"reachable on community + admin" assertions. Docs (fleet-actions,
fleet-view, licensing, overview, stack-labels) rewritten to state the
admin-role requirement once and drop the prior Skipper framing.
* fix(fleet): apply audit findings from PR #1153 review
- stack-labels.mdx: fix the page intro that still framed fleet label
actions as "Operators on a Skipper or Admiral license". The cards are
now Community + admin, so the intro reads "Admins also get a pair of
fleet-wide actions".
- Collapse redundant role-rule statements on the two affected pages.
fleet-actions.mdx now states the admin gate once in the lead-in Note
and again only in the troubleshooting accordion (the Prerequisites
row was duplicative). stack-labels.mdx trims the "Limits and rules"
bullet to the value-add half (label authoring is open to every role)
and drops the Fleet Actions repetition.
- Strip now-no-op mockTier('paid') calls from non-tier tests across the
three fleet test files, plus the test-wide default in the
fleet-action-card-endpoints beforeEach. Those mocks were misleading
after the routes stopped consulting tier; if a future change re-adds
requirePaid the tests will fail loudly instead of silently passing.
* feat(fleet): add Fleet Actions tab for cross-node bulk operations
Introduces a new "Actions" sub-tab in Fleet view with two Skipper+ cards
that fill gaps in the existing surface:
- Stop fleet by label: matches a label name across every node and stops
every stack assigned to it, reporting per-node and per-stack results.
- Bulk label assign: applies the same label set to many stacks on one
node in a single round trip.
Other bulk operations stay in their existing homes (sidebar bulk mode,
Schedules, NodeUpdatesSheet) to avoid duplicate surfaces.
Backend:
- POST /api/fleet/labels/fleet-stop (gateway-orchestrated, multi-node)
- POST /api/fleet-actions/labels/bulk-assign (per-node, capped at 1000)
- Tightens /api/fleet proxy-exempt prefix to /api/fleet/ so
/api/fleet-actions/* is routed through the proxy for per-node calls.
- Exports activeBulkActions from labels.ts so fleet-stop and label-action
share the per-node lock and cannot double-stop the same containers.
- Extracts containerActionForStack helper from stacks.ts for reuse.
* chore(fleet): rename Actions tab to Fleet Actions and reorder Fleet sub-tabs
- Tab label "Actions" -> "Fleet Actions" so the surface is unambiguous
alongside Schedules and the sidebar bulk bar.
- Reorder Fleet sub-tabs as Overview / Snapshots / Status | Deployments /
Traffic / Fleet Actions, with the separator after Status.
- Rename "Traffic · Routing" -> "Traffic" and update Sencho Mesh docs to
match the shorter label.
- Update Fleet Actions docs to the new tab name and placement.
Anso