feat(fleet): cross-node bulk label assign with authoritative label discovery (#1389)

* feat(fleet): cross-node bulk label assign with authoritative label discovery

Make Fleet Actions > Bulk label assign work across the fleet. Pick a stack
label that exists anywhere in the fleet, select stacks on one or more nodes,
and the control orchestrates: each target node resolves the label by name,
creating it with the same name and color if missing, then adds it to the
selected stacks while preserving their existing labels. The local node runs
in process; each remote runs its own admin-only local-assign receiver over
the node proxy. Per-node failures (unknown node, no proxy target, unreachable,
mixed-version remote) degrade that node only and are reported per node in the
result. Assignment writes use a transactional INSERT OR IGNORE so the
add-preserve path is idempotent and race-free.

Also make the shared fleet label discovery authoritative: suggestions,
match-preview, and the fleet-stop remote leg now read each node's labels live
over the proxy instead of the control database, which does not mirror remote
labels. A propagated label therefore appears in, and is stoppable by,
Stop-by-label across the fleet, and unreachable nodes are surfaced rather than
silently dropped.

Fleet Actions runs against the unfiltered node list, so overview filters no
longer narrow its scope. The previous node-scoped, replace-by-id bulk-assign
endpoint is removed.

* fix(fleet): treat malformed remote label responses as per-node failures

A 200 response from a remote node whose body is not the expected shape was
treated as a benign empty result, so a malformed remote could read as a clean
zero-stack assign or a "matched, nothing to stop" no-op and even surface a
success toast. Validate the wire shape in the bulk-assign and fleet-stop remote
legs and in the authoritative label discovery fan-out; on a malformed body,
report the node as a per-node failure with the error attributed to its stacks
instead of silently dropping it.

* chore: drop accidentally committed temp file

docs/features/fleet-actions.mdx

@@ -1,6 +1,6 @@
 ---
 title: "Fleet Actions"
-description: "Bulk operations across the fleet from one tab: stop stacks by label, replace labels on a batch of stacks, and reclaim Docker disk space on every node."
+description: "Bulk operations across the fleet from one tab: stop stacks by label, assign a label to stacks across nodes, and reclaim Docker disk space on every node."
 ---
 
 The **Actions** tab on the Fleet view groups bulk operations that touch more than a single stack on a single node. Each action lives in its own card, runs from the control instance, and reports per-node and per-stack results inline so you never have to click through a modal to learn what happened.
@@ -34,10 +34,10 @@ The cards share a tab and a role gate, but they don't share an execution path. K
 | Card | Endpoint | Where it runs | Scope |
 |---|---|---|---|
 | Stop fleet by label | `POST /api/fleet/labels/fleet-stop` | Control instance orchestrates; fans out to each node | Every configured node |
-| Bulk label assign | `POST /api/fleet-actions/labels/bulk-assign` | Target node (request is proxied) | The single node you pick |
+| Bulk label assign | `POST /api/fleet/labels/bulk-assign` | Control instance orchestrates; fans out to each target node | The stacks you select across nodes |
 | Prune Docker resources fleet-wide | `POST /api/fleet/labels/fleet-prune` | Control instance orchestrates; fans out to each node | Every reachable node |
 
-The two fan-out cards (Stop and Prune) iterate every node in **Settings → Nodes**, including offline remotes; unreachable nodes show up in the results with a transport error rather than blocking the rest of the batch. The single-node card (Bulk label assign) proxies the request through the standard `x-node-id` header to the node you select, so the work happens locally on that node.
+All three cards orchestrate from the control instance and report results grouped by node. Stop and Prune iterate every node in **Settings → Nodes**; Bulk label assign iterates only the nodes whose stacks you selected. Each card runs the authoritative work on the executing node (the local node in process, every remote over the node proxy), so unreachable nodes show up in the results with a transport error rather than blocking the rest of the batch.
 
 ## Stop fleet by label
 
@@ -77,30 +77,30 @@ A few quirks worth knowing:
 
 ## Bulk label assign
 
-Replace the label set on a batch of stacks on a single node, in one round trip. Use this when you've decided on a taxonomy change (e.g. splitting `prod` into `prod-edge` and `prod-core`) and need to relabel many stacks without clicking through each stack's editor.
+Add a stack label to stacks across the fleet in one round trip. Pick a label that exists anywhere in the fleet, select stacks on one or more nodes, and Sencho assigns that label on each target node, creating it there first if the node does not have it yet. Existing labels on the selected stacks are preserved. Use this to keep a label like `media` consistent across nodes without visiting each one to recreate and assign it by hand.
 
 <Frame>
-  <img src="/images/fleet-actions/fleet-actions-bulk-assign.png" alt="Bulk label assign card with the Local node selected, three stacks checked (bazarr, plex; counter reads 'Stacks (3/14)'), two labels toggled on (Media and Network; counter reads 'Labels (2/3)'), and the primary button reading 'Apply to 3 stacks'." />
+  <img src="/images/fleet-actions/fleet-actions-bulk-assign.png" alt="Bulk label assign card. A label source row of pills with 'Media' selected, a target-stacks list grouped by node (Local and a remote) with several stacks checked, a preview showing 'create' on the remote and 'reuse' on Local, and a primary button reading 'Apply'." />
 </Frame>
 
-### Single-node scope (and why)
+### Cross-node by label identity
 
-Unlike the other two cards, Bulk label assign targets exactly one node. The request is proxied via `x-node-id` to the node you pick from the **Select a node** dropdown, so the label rewrites happen on that node's local database. There is no fleet-wide variant in v1; if you need to retag the same stacks on multiple nodes, run the card once per node.
+Labels are node-local: each node owns its own copy of a label. Bulk label assign treats the label you pick as a name plus color, then resolves it per target node. If the node already has a label with that exact name, its own label is used; if not, Sencho creates one with the same name and color. The target node's own label is always used for the assignment, never the control node's.
 
 ### Step by step
 
-1. Pick a node from the **Select a node** dropdown. The card loads that node's stacks and labels in parallel.
-2. Check the stacks you want to update. The counter in the **STACKS** header reads `Stacks (selected/total)`; the **Select all** affordance flips to **Clear** once everything is selected.
-3. Toggle the label pills you want as the new label set. The counter in the **LABELS** header tracks the selection.
-4. Click **Apply to N stacks**. A confirmation appears titled `Apply N labels to M stacks?` (or the singular forms). Confirm to commit.
+1. Pick a label under **Label · source**. The pills list every stack label defined across the reachable fleet. If the same name carries different colors on different nodes, the local node's color is used where the label is created.
+2. Under **Target stacks**, check the stacks you want, grouped by node. Use the filter to narrow long lists and **Select all** to take a whole node. Unreachable nodes are shown and cannot be selected.
+3. The **Preview** shows, per node, whether the label will be **created** or **reused** and how many stacks it will touch.
+4. Click **Apply**. A confirmation summarizes the blast radius; confirm to commit. Results render grouped by node, each row noting whether the label was created or reused alongside the per-stack outcome.
 
-### Replace, not append
+### Add, preserving existing labels
 
-The selected label set **replaces** each chosen stack's existing label set on this node. Selecting zero labels clears assignments on those stacks; the confirmation modal calls this out so the destructive case is hard to miss.
+The selected label is **added** to each chosen stack; the stack keeps its other labels. The card only ever adds the one label you picked, so a fleet-wide propagation cannot accidentally wipe labels a remote node already carries.
 
 ### Batch ceiling
 
-The endpoint accepts up to **1,000 stack assignments per call** and returns `400` over the limit. In practice this is well above any sensible UI selection. Per-entry validation is lenient: an invalid stack name returns a per-entry failure row in the **Per-stack results** card, and the rest of the batch still applies.
+A single Apply accepts up to **1,000 stack assignments** summed across every target node and returns `400` over the limit. In practice this is well above any sensible UI selection. A stack that is missing on disk or has an invalid name returns a per-stack failure row, and the rest of the batch still applies.
 
 ## Prune Docker resources fleet-wide
 
@@ -145,8 +145,8 @@ Scope is a segmented control with two options:
 
 Fleet Actions is intentionally narrow in v1. The following are deliberately out of scope:
 
-- **No fleet-wide bulk label assign.** Bulk label assign targets one node at a time. Re-tagging the same stacks on multiple nodes is two clicks of the node selector and two confirmations.
-- **No fleet-wide bulk start, restart, or update.** Stop is the only fleet-wide stack action today. Per-node multi-stack start, restart, and update live in the sidebar's Bulk mode.
+- **No replace mode for bulk label assign.** The card only adds the label you pick; it never removes a stack's existing labels. To remove or swap labels on a stack, edit them from that stack's own view.
+- **No fleet-wide bulk start, restart, or update.** Stop is the only fleet-wide stack lifecycle action today. Per-node multi-stack start, restart, and update live in the sidebar's Bulk mode.
 - **No label-set selectors.** Stop fleet by label matches one label name. Combinations like "stacks labelled A AND B" are not supported.
 - **No partial-stop ceiling.** The Stop card stops every stack the label matches; there is no "stop the first N" knob.
 - **No undo.** A stopped stack stays stopped until you start it again; a pruned image is gone until it is pulled or rebuilt.
@@ -159,9 +159,9 @@ Fleet Actions is intentionally narrow in v1. The following are deliberately out
 
 Tag the stacks you want to bring down with a dedicated label (for example `evening-shutdown`). Run **Stop fleet by label** with that label name. The per-node breakdown confirms each stack's stop result; restart from the sidebar when power is back.
 
-### Migrate a label taxonomy on one node
+### Propagate a label across the fleet
 
-Split a coarse label like `prod` into `prod-edge` and `prod-core` by creating the new labels in **Settings → Labels** on the affected node, then opening **Bulk label assign**, picking that node, checking the stacks that should move, and toggling the new label set. The replace-not-append semantic guarantees the old label is removed in the same write.
+Define a label like `media` on one node (for example the local node) under **Settings → Labels**, then open **Bulk label assign**, pick `media` under the label source, and check the matching stacks on every node. Sencho creates `media` on each node that does not have it and assigns it, leaving each stack's other labels intact. The label then matches fleet-wide in **Stop fleet by label** and in per-node stack filtering.
 
 ### Free disk before a heavy deploy
 
@@ -176,11 +176,11 @@ Run **Prune Docker resources fleet-wide** with **Images** selected and **Managed
   <Accordion title="The autocomplete didn't suggest a stack label I know exists">
     The picker queries each reachable node for its own stack labels, so a name is missing when no reachable node has a **stack label** by that name, or when the node that owns it could not be reached when the picker loaded (the picker flags this case). Node labels never appear here, because this action targets stack labels only. You can always type a name by hand; the fleet-stop request still iterates every node and asks each one authoritatively.
   </Accordion>
-  <Accordion title="Bulk label assign reports 'Invalid stack name' for one row">
-    Stack names must match the standard validator: alphanumeric plus dash and underscore, no spaces, no path separators. The endpoint validates each assignment independently, so a single bad name does not block the rest of the batch. Fix the offending entry and re-run; the rows that already succeeded won't be re-applied.
+  <Accordion title="Bulk label assign reports 'Invalid stack name' or 'Stack not found' for one row">
+    Stack names must match the standard validator: alphanumeric plus dash and underscore, no spaces, no path separators. A name that does not exist on its node returns *Stack not found*. Each stack is validated independently, so one bad row does not block the rest of the batch. Fix the offending entry and re-run; the rows that already succeeded are simply re-added with no change.
   </Accordion>
-  <Accordion title="Apply to N stacks button is disabled">
-    The button requires at least one stack selected; the label selection can be empty (which clears assignments). Check that the node selector resolved (the **Loading…** spinner has cleared) and that the stack list is populated.
+  <Accordion title="The Apply button on Bulk label assign is disabled">
+    Apply needs both a label picked under the source and at least one stack checked. An empty label source means no stack labels are defined on any reachable node yet; create one under **Settings → Labels** first. Wait for the per-node lists to finish loading before selecting.
   </Accordion>
   <Accordion title="Fleet stop timed out on one remote node">
     Each remote node call carries a 60-second timeout. A remote with many stacks or a slow Docker daemon can outlast that budget. Check the affected remote's logs; the stop usually completed on the remote even though the control instance stopped waiting. Re-running the same fleet-stop is safe: stacks that are already stopped return the per-stack error *No containers found for this stack* and do not toggle anything else.

docs/features/licensing.mdx

@@ -28,7 +28,7 @@ See [the pricing page](https://sencho.io/pricing) for current pricing.
 - Multi-node management in both Proxy and Pilot Agent modes
 - Fleet View with search, sort, filter, and node-card drill-down
 - Manual and scheduled fleet snapshots (create, browse, restore, delete) and Remote OTA node updates (per-node and **Update all**)
-- Actions tab (stop stacks fleet-wide by label, bulk-assign labels to many stacks on a node, prune Docker resources fleet-wide; admin role required), plus fleet-wide bulk Sencho restart
+- Actions tab (stop stacks fleet-wide by label, assign a label to stacks across nodes, prune Docker resources fleet-wide; admin role required), plus fleet-wide bulk Sencho restart
 - Bulk actions on a label (deploy, stop, or restart every stack tagged with it)
 - Atomic deployments with automatic rollback, and one-click rollback to the previous deployment
 - Auto-update policies for stack images and auto-heal policies for failed containers

docs/features/overview.mdx

@@ -117,7 +117,7 @@ Operator-driven placement controls for fleets running Blueprints. Cordon nodes t
 
 ### Fleet Actions
 
-Run fleet-wide bulk operations from one place: stop stacks across nodes by label selector, bulk-assign labels to many stacks on a single node, or prune Docker resources fleet-wide. Admin-only on every tier. [Learn more →](/features/fleet-actions)
+Run fleet-wide bulk operations from one place: stop stacks across nodes by label selector, propagate a label to stacks across nodes (creating it where missing), or prune Docker resources fleet-wide. Admin-only on every tier. [Learn more →](/features/fleet-actions)
 
 ### Fleet Sync
 

docs/features/stack-labels.mdx

@@ -3,7 +3,7 @@ title: "Stack Labels"
 description: "Per-node tags that group your stacks by purpose, surface them under collapsible headers in the sidebar, and unlock cross-stack bulk actions across the fleet."
 ---
 
-A **Stack Label** is a per-node tag (name plus color) you can stick on any stack. Once a stack carries a label, the sidebar groups it under that label's header instead of dumping every stack into a flat list, and Fleet View can filter the overview by tag. Admins also get a pair of fleet-wide actions powered by labels: stop every stack labeled `prod` across every node, or replace the label set on a batch of stacks in one shot.
+A **Stack Label** is a per-node tag (name plus color) you can stick on any stack. Once a stack carries a label, the sidebar groups it under that label's header instead of dumping every stack into a flat list, and Fleet View can filter the overview by tag. Admins also get a pair of fleet-wide actions powered by labels: stop every stack labeled `prod` across every node, or add a label to stacks across nodes in one shot.
 
 <Frame>
   <img src="/images/stack-labels/sidebar-grouping.png" alt="Sidebar showing stacks grouped under three uppercase label headers (MEDIA 6, UTILITIES 5, NETWORK 3) and an UNLABELED 1 group at the bottom. Each stack row carries a small colored dot on the trailing edge that matches the group's color." />
@@ -88,7 +88,7 @@ A stack can carry multiple labels and will then appear under each label's group
 Two cards in the **Fleet · Actions** tab use labels to drive cross-node operations. See [Fleet Actions](/features/fleet-actions) for the full reference.
 
 <Frame>
-  <img src="/images/stack-labels/fleet-actions.png" alt="Fleet Actions tab with two cards side by side. Left card 'Stop fleet by label' (rose accent rail) has a Stack label combobox containing 'Media' and a Stop fleet button beneath. Right card 'Bulk label assign' (purple accent rail) has a node selector reading Local (local), a stacks checklist showing plex and radarr ticked, a Labels row with a highlighted Media pill plus inactive Network and Utilities, and an Apply to 3 stacks button." />
+  <img src="/images/stack-labels/fleet-actions.png" alt="Fleet Actions tab with two cards side by side. Left card 'Stop fleet by label' (rose accent rail) has a Stack label combobox containing 'Media' and a Stop fleet button beneath. Right card 'Bulk label assign' (purple accent rail) has a label source row with a highlighted Media pill, a target-stacks list grouped by node with several stacks ticked, a per-node create-or-reuse preview, and an Apply button." />
 </Frame>
 
 ### Stop fleet by label
@@ -99,16 +99,14 @@ A node that carries no matching stack label is shown as such, and a node Sencho
 
 ### Bulk label assign
 
-Pick a node from the dropdown; the card loads that node's stacks and labels in parallel. Tick the stacks you want to relabel and click the label pills you want to apply. The footer button reads `Apply to N stack(s)` and the confirmation modal title reads `Apply N label(s) to M stack(s)?` so the scope is unambiguous before you commit. A line under the controls makes the replacement semantics explicit:
+Pick a label that exists anywhere in the fleet, then tick the stacks you want across one or more nodes (grouped by node, with a filter and per-node select-all). The preview shows, per node, whether the label will be created or reused; **Apply** adds the label to each chosen stack on its node, creating it there first with the same name and color if the node does not have it yet. Existing labels on the selected stacks are preserved. The confirmation modal summarizes the blast radius before you commit, and the result list breaks down per node, noting whether the label was created or reused.
 
-> Selected labels replace each chosen stack's existing label set on this node. Selecting no labels clears assignments.
-
-The **Bulk label assign** card is per-node only by design. To re-tag stacks on a different node, switch the picker; the stack and label list refreshes and your previous selection clears.
+Because labels are node-local, each target node uses its own copy of the label rather than the control node's, so propagating a label keeps every node's label table self-consistent.
 
 ## Limits and rules
 
 - **50 labels per node.** Settings hides the **New label** button at the cap; the inline `+ New label` entry in the stack menu hides itself too.
-- **Names are unique per node**, case-sensitive. The same name on two nodes is two separate label rows. Cross-node fleet stop matches on name; bulk assign always operates on one node's labels at a time.
+- **Names are unique per node**, case-sensitive. The same name on two nodes is two separate label rows. Cross-node fleet stop and bulk assign both match by name across nodes; each node resolves the name to its own label (and bulk assign creates it there if missing).
 - **Allowed name characters**: letters, digits, spaces, and hyphens. Empty names and names beyond 30 characters are rejected at the API.
 - **Bulk-action concurrency**: only one label-driven bulk action can run on a single node at a time. A second concurrent attempt against the same node returns HTTP 429 and the operator sees an error toast; the in-flight action keeps running.
 - **Role visibility**: label authoring is open to every signed-in role. Sidebar grouping, trailing dots on stack rows, the **Settings · Organization · Labels** panel, the inline create form in the stack menu, and the Fleet View **Tags** filter all work for every user.
@@ -131,11 +129,11 @@ The **Bulk label assign** card is per-node only by design. To re-tag stacks on a
   <Accordion title="`Stop fleet by label` reports `No node carries a stack label by that name`">
     Stack labels are per-node, so the fleet-stop matches by name across nodes. If the stack label you typed only exists on one node and you typed the wrong case (`prod` versus `Prod`), no node will match. The picker queries each reachable node for its own stack labels; node labels never appear there. A label on a node Sencho cannot currently reach will not be suggested, and the picker flags that the list may be incomplete. Pick from the suggestion list rather than typing freehand to avoid case mistakes.
   </Accordion>
-  <Accordion title="`Bulk label assign` cleared every label on my stacks unexpectedly">
-    The card replaces, it does not merge. Selecting no labels and clicking **Apply to N stack(s)** is the documented way to clear assignments, and the confirmation copy on the **Bulk label assign** modal restates this: `No labels selected, this will clear existing assignments on the selected stacks.` Re-pick the labels you want and run the action again to restore them.
+  <Accordion title="`Bulk label assign` did not remove the old labels on my stacks">
+    By design it never does: the card only adds the label you picked, leaving each stack's other labels intact. There is no clear or replace mode. To remove or swap a stack's labels, edit them from that stack's own **Labels** menu.
   </Accordion>
-  <Accordion title="Two stacks with the same name on different nodes only got relabeled on one">
-    `Bulk label assign` is per-node by design. The node picker at the top is the source of truth and the stacks list only shows stacks on that node. Run the card a second time with the other node selected, or use **Stop fleet by label** instead if the goal is fleet-wide.
+  <Accordion title="A label did not appear on a remote node after bulk assign">
+    Confirm the node was reachable when you applied: an unreachable node is shown in the target list and reported in the per-node results rather than silently skipped. If the node was reachable, the label is created there by name with the chosen color and assigned; re-run to retry any node that failed.
   </Accordion>
   <Accordion title="The Fleet Actions cards return an error when I click Apply">
     The cards run admin-only. Confirm the active user has the admin role under **Settings · Users**; operator and viewer roles see the cards rendered but cannot apply them. All other Stack Labels surfaces (sidebar grouping, trailing dots, the Settings panel, the inline create form, the Fleet View Tags filter) work for every role.