mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-20 07:13:05 +00:00
chore: merge main for audit fix
This commit is contained in:
SaelixCode
@@ -1,10 +1,10 @@
|
||||
---
|
||||
title: Audit Log
|
||||
description: Track every mutating action on your Sencho instance with a searchable, exportable trail for team accountability.
|
||||
description: Track every mutating action on your Sencho instance with a searchable trail for team accountability; export and extended retention on Admiral.
|
||||
---
|
||||
|
||||
<Note>
|
||||
Community keeps a rolling 14-day recent-activity audit API window, but has no **Audit** tab in navigation to browse it. The Audit navigation view, CSV and JSON export, anomaly detection, and configurable retention beyond the recent window all require a Sencho **Admiral** license.
|
||||
Audit requires the `system:audit` permission (Admin, or Auditor on Admiral). Community shows the rolling 14-day recent-activity window. Admiral adds the 24h signal rail, CSV/JSON export, anomaly detection, and configurable retention beyond that window.
|
||||
</Note>
|
||||
|
||||
<Note>
|
||||
@@ -66,21 +66,25 @@ Expanding a row in the Table view reveals additional detail:
|
||||
|
||||
## Viewing the audit log
|
||||
|
||||
The **Audit** view in navigation is an Admiral governance surface: the tab itself only appears on an Admiral license, and Community accounts have no navigation entry point to Audit at all, though the underlying recent-activity API stays reachable directly. On Admiral, the tab is further limited to users whose role grants the `system:audit` permission, which by default means **Admin** or **Auditor**.
|
||||
The **Audit** tab appears for any signed-in user whose role grants the `system:audit` permission (by default **Admin**, and **Auditor** when that role is available). Community shows the last 14 days of activity. Admiral shows the full retained history; retention defaults to 90 days and is configurable up to 365.
|
||||
|
||||
Navigate to the **Audit** tab in the top navigation when it is available for your role and plan. The feed then shows your full retained history rather than any fixed lookback window; retention defaults to 90 days and is configurable up to 365.
|
||||
Navigate to the **Audit** tab in the top navigation when it is available for your role.
|
||||
|
||||
The page has two views, toggled from the segmented control in the card header: **Stream** (default) and **Table**. The card subtitle reports the total number of entries that match the current filters.
|
||||
|
||||
### Stream view
|
||||
|
||||
Stream gives you an at-a-glance read on activity. A signal rail at the top summarizes the last 24 hours across four tiles, and the feed below groups entries by day with severity dots, relative times, and inline anomaly callouts.
|
||||
Stream gives you an at-a-glance read on activity. The feed groups entries by day with severity dots and relative times.
|
||||
|
||||
<Note>
|
||||
The 24h signal rail and inline anomaly callouts require a Sencho **Admiral** license.
|
||||
</Note>
|
||||
|
||||
<Frame>
|
||||
<img src="/images/audit-log/audit-stream.png" alt="Audit Log Stream view with the four-tile signal rail (Events 53 +218% vs 7d avg, Actors 2, Failure rate 0%, Peak hour 21:00) above a day-banded chronological feed of admin POST and DELETE entries, including a first seen anomaly flag on one entry." />
|
||||
</Frame>
|
||||
|
||||
**Signal rail tiles:**
|
||||
**Signal rail tiles (Admiral):**
|
||||
|
||||
| Tile | What it shows |
|
||||
|------|---------------|
|
||||
@@ -105,7 +109,7 @@ Table keeps the full-featured detail grid for power users: exact timestamps, met
|
||||
<img src="/images/audit-log/audit-log-expanded.png" alt="Audit Log Table view with the second row expanded to reveal Request Path /api/webhooks/4, IP Address ::ffff:203.0.113.10, Node ID 1, and Entry ID #2472 in a four-cell detail strip." />
|
||||
</Frame>
|
||||
|
||||
Both views share the **Refresh** button and the **Export** dropdown in the card header, and both paginate at 50 entries per page with chevron controls at the bottom of the feed or table.
|
||||
Both views share the **Refresh** button in the card header and paginate at 50 entries per page with chevron controls at the bottom of the feed or table. On Admiral, both views also share the **Export** dropdown.
|
||||
|
||||
## Anomaly detection
|
||||
|
||||
@@ -190,7 +194,7 @@ Sensitive database values (such as remote node API tokens) are encrypted at rest
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="The Audit tab is missing from the navigation">
|
||||
The Audit tab is an Admiral governance view: it only appears on an Admiral license, and on Community it is not shown at all, regardless of role. On Admiral, the tab is further limited to users whose role grants the `system:audit` permission, by default **Admin** or **Auditor**. If you are on Admiral but signed in as a Deployer or Viewer, ask an admin to assign you the Auditor role from **Settings · Users**. The recent-activity API stays reachable on Community even without a navigation entry point to it.
|
||||
The Audit tab appears only for users with the `system:audit` permission (by default **Admin**, or **Auditor** when that role is available). If you are signed in as a Viewer, Deployer, or Node Admin, ask an admin to grant you a role that includes audit access. On Community, Admin is the role that can open Audit; the Auditor role requires Admiral to assign.
|
||||
</Accordion>
|
||||
<Accordion title="Stream view shows everything but I want to filter to a specific user, action, or date">
|
||||
Filters live in **Table view only**. Toggle the segmented control in the card header from **Stream** to **Table** and the search box, method dropdown, and From / To date pickers will appear above the grid. Switching back to Stream clears the filter strip but does not remember the last filter.
|
||||
@@ -202,6 +206,6 @@ Sensitive database values (such as remote node API tokens) are encrypted at rest
|
||||
Each export is capped at 10,000 entries. If your filter selects more than that, narrow the date range using the **From** and **To** pickers and download in chunks. The cap protects the API from generating very large CSVs in a single response; for full archives, schedule periodic exports from your own tooling.
|
||||
</Accordion>
|
||||
<Accordion title="Old entries vanished even though I never deleted anything">
|
||||
Cleanup runs automatically against the **Audit log** retention value in **Settings · Operations · Data Retention** (default 90 days). Entries older than the configured window are pruned on the next maintenance tick. Increase the value (up to 365 days) before the next cleanup runs to retain a longer history; the change applies forward only and cannot bring back already-pruned entries.
|
||||
On Community, the list shows only the last 14 days of activity, so older rows are not returned even when they still exist in the database. On Admiral, cleanup also runs against the **Audit log** retention value in **Settings · Operations · Data Retention** (default 90 days). Entries older than the configured window are pruned on the next maintenance tick. Increase the value (up to 365 days) before the next cleanup runs to retain a longer history; the change applies forward only and cannot bring back already-pruned entries.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
@@ -153,7 +153,7 @@ When the node is reachable, the tab compares the declared effective model agains
|
||||
| **Declared but unused** | A network is declared in the Compose file but no currently running service is connected to it. Often seen when a service is stopped or removed without `docker compose down`. |
|
||||
| **Missing from runtime** | A network is declared but does not exist in Docker. The stack may not have been deployed, or the network was deleted externally. |
|
||||
|
||||
System-managed networks (`bridge`, `host`, `none`) and Docker's implicit default bridge are excluded from all drift findings.
|
||||
System-managed networks (`bridge`, `host`, `none`) and Docker's implicit default bridge are excluded from all drift findings. Attachments to `sencho_mesh` are also excluded when Sencho verifies that the container is its own instance or that the stack is opted into Sencho Mesh. A manual attachment from an opted-out stack remains visible as drift.
|
||||
|
||||
When the runtime matches the Compose file, the section shows a green **runtime matches compose** card.
|
||||
|
||||
|
||||
@@ -48,7 +48,9 @@ The gauge bars (and the corresponding numeric values) pick up amber at 80% and r
|
||||
While the dashboard is loading the CPU tile reads `--` and the caption shows `collecting metrics…`; bars and sparklines render once the first sample arrives.
|
||||
|
||||
<Note>
|
||||
**ZFS hosts:** the memory tile and host RAM alerts are ZFS ARC-aware. Reclaimable ARC cache is treated as available memory rather than used, so a large ARC does not inflate the gauge or trigger false low-memory alerts. See [ZFS ARC-aware host memory](/getting-started/configuration#zfs-arc-aware-host-memory) for how to expose ARC stats to a Docker install.
|
||||
**ZFS hosts:** the memory tile and host RAM alerts are ZFS ARC-aware. Reclaimable ARC cache is treated as available memory rather than used, so a large ARC does not inflate the gauge or trigger false low-memory alerts. The memory tile shows the reclaimable amount as a context line when ARC stats are readable and the reclaimable amount is nonzero. See [ZFS ARC-aware host memory](/getting-started/configuration#zfs-arc-aware-host-memory) for how to expose ARC stats to a Docker install.
|
||||
|
||||
**Virtual machines:** the memory tile shows hypervisor-ballooned memory (TrueNAS/KVM, Proxmox) as informational context. Unlike ARC, ballooned pages are host-reclaimed and the guest cannot get them back on demand, so the gauge, health verdict, and alerts continue to use the standard working-set percentage. See [VM memory ballooning](/getting-started/configuration#vm-memory-ballooning) for details.
|
||||
</Note>
|
||||
|
||||
## Stack health
|
||||
|
||||
@@ -5,7 +5,7 @@ description: "Bulk operations across the fleet from one tab: stop stacks by labe
|
||||
|
||||
The **Actions** tab on the Fleet view groups bulk operations that touch more than a single stack on a single node. Each action lives in its own card, orchestrates from the control instance, and reports per-node and per-stack results inline so you never have to click through a modal to learn what happened.
|
||||
|
||||
Three cards ship today: **Prune Docker resources fleet-wide**, **Bulk label assign**, and **Stop by label**. Every card follows the same discipline before it touches anything: a live, debounced readout resolves the exact blast radius as you type or select, and the destructive or state-changing button stays disabled until that readout resolves to a concrete node and stack list. You confirm against real names, not a label string or a byte estimate.
|
||||
Three cards ship today: **Prune Docker resources fleet-wide**, **Bulk label assign**, and **Stop by label**. Every card resolves its blast radius before it touches anything. Fleet Prune also requires an itemized dry run, so its destructive button stays disabled until every reachable node has returned the exact resources and fingerprint that will authorize execution.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/fleet-actions/fleet-actions-overview.png" alt="Fleet view with the Actions tab selected. A two-column grid: Prune fleet-wide (top left, Maintenance chip), Bulk label assign (top right, Transformative chip), and Stop by label (bottom, Destructive chip). Each card shows a toolbar row with an action-class chip, a live blast-radius readout, a Dry run or Reset button, and the primary action." />
|
||||
@@ -26,17 +26,17 @@ Fleet Actions is the home for operations that span the fleet but don't fit anywh
|
||||
| Trigger a Sencho self-update across remote nodes | **Check Updates** button on the Fleet masthead |
|
||||
| Steer where new blueprint deployments land | [Fleet Federation](/features/fleet-federation) |
|
||||
| Replicate scan policies and CVE suppressions to remotes | [Fleet Sync](/features/fleet-sync) |
|
||||
| Reclaim disk space on a single node with an itemized, undo-safe preview | **Resources → Prune** on that node (a different endpoint and flow from the fleet-wide card; see [Prune Docker resources fleet-wide](#prune-docker-resources-fleet-wide)) |
|
||||
| Reclaim disk space on a single node with an itemized preview | **Resources → Prune** on that node (the single-node version of the fingerprint-bound flow described in [Prune Docker resources fleet-wide](#prune-docker-resources-fleet-wide)) |
|
||||
|
||||
## How every card works: preview, confirm, execute
|
||||
|
||||
All three cards share one execution model, and understanding it explains every result panel, timeout, and edge case below.
|
||||
|
||||
1. **You describe the target.** A stack label name (Stop by label), a label plus checked stacks (Bulk label assign), or a set of resource targets and a scope (Prune).
|
||||
2. **A live, debounced readout resolves the real blast radius.** Typing a label name or checking a target fires a non-destructive preview call (`POST /api/fleet/labels/match-preview` or `POST /api/fleet/prune/estimate`) roughly 350-500ms after you stop changing input. The readout in the card's toolbar shows `awaiting target` until something is selected, `resolving…` while the call is in flight, and then a concrete count (`7 stacks · 1 nodes`, `~ 6.09 GB reclaimable`). The primary button stays disabled until this resolves to a non-zero, non-loading result.
|
||||
3. **You confirm against the resolved list, not the input.** Clicking the primary button opens a confirmation dialog that lists the actual nodes and stacks (Stop, Bulk assign) or restates the scope (Prune). For Stop by label specifically, the confirmation carries the exact node/stack list the preview resolved, and the real stop only touches stacks that are still in that list *and* still carry the label at execution time: a stack that gains the label after you opened the confirmation is never touched, and a node that reconnects after the preview does not get pulled into the stop.
|
||||
4. **The control instance fans the confirmed action out to every node in parallel.** The local node runs in-process; each remote node is called over the standard Bearer-token proxy path. A node that cannot be reached, returns a non-2xx response, or returns a shape Sencho does not recognize is reported as a failure for that node only; the fan-out to every other node still completes.
|
||||
5. **Results render per node, grouped and expandable**, in a `Per-node breakdown` section below the form. Stop and Prune also expose a **Dry run** button that walks the identical code path and locks without performing the destructive step, so you can rehearse the exact fan-out before committing.
|
||||
2. **A live, debounced readout estimates the blast radius.** Typing a label name or checking a target fires a non-destructive preview call (`POST /api/fleet/labels/match-preview` or `POST /api/fleet/prune/estimate`) roughly 350-500ms after you stop changing input. The toolbar shows a stack count or approximate reclaimable bytes while you refine the action.
|
||||
3. **You review the resolved list, not only the input.** Stop and Bulk assign resolve concrete stacks. Prune requires **Dry run**, which lists every candidate image, volume, and network for each reachable node. Changing the targets, scope, node roster, or node reachability clears that authorization.
|
||||
4. **The control instance verifies before mutation.** Fleet Prune rebuilds every reviewed plan and checks the complete node roster before any node starts deleting. A stale plan or changed reachability rejects the whole preflight. A later race can still produce an explicit partial result because each node revalidates again immediately before deletion.
|
||||
5. **Results render per node, grouped and expandable.** Prune retains the reviewed item identity and, when the node returns item outcomes, marks each candidate Removed, Skipped, or Failed after execution. Unreachable nodes remain visible as excluded rather than appearing as successful empty plans.
|
||||
|
||||
## The three cards
|
||||
|
||||
@@ -46,7 +46,7 @@ All three cards share one execution model, and understanding it explains every r
|
||||
| Bulk label assign | Transformative | `POST /api/fleet/labels/bulk-assign` | (computed client-side from `/api/labels` and `/api/fleet/node/:id/stacks` per node) | Only the nodes whose stacks you select |
|
||||
| Prune Docker resources fleet-wide | Maintenance | `POST /api/fleet/labels/fleet-prune` | `POST /api/fleet/prune/estimate` | Every configured node |
|
||||
|
||||
Every card is admin-only and available on every license tier. Stop and Prune iterate every node in **Settings → Nodes**; Bulk label assign iterates only the nodes whose stacks you actually checked. Each card runs the authoritative work on the executing node (the local node in process, every remote over the node proxy), so an unreachable node shows up in the results with a transport error rather than blocking the rest of the batch.
|
||||
Every card is admin-only and available on every license tier. Stop and Prune iterate every node in **Settings → Nodes**; Bulk label assign iterates only the nodes whose stacks you actually checked. Each card runs the authoritative work on the executing node (the local node in process, every remote over the node proxy). Stop and Bulk label assign report unreachable nodes without blocking work elsewhere. Fleet Prune excludes unreachable nodes during review, then rejects execution if that reviewed reachability changes.
|
||||
|
||||
## Stop by label
|
||||
|
||||
@@ -120,7 +120,7 @@ A single Apply accepts up to **1,000 stack assignments** summed across every tar
|
||||
|
||||
## Prune Docker resources fleet-wide
|
||||
|
||||
Reclaim disk space on every reachable node by deleting unused images, volumes, and networks. The control instance fans out to each node and reports reclaimed bytes per node and per target.
|
||||
Reclaim disk space on every reachable node by deleting unused images, volumes, and networks. A dry run lists the exact candidates on each node, and the real prune is authorized by the fingerprint of each reviewed plan.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/fleet-actions/fleet-actions-prune.png" alt="Prune fleet-wide card with Images and Volumes targets checked, scope set to All unused, and a live per-node estimate: Local 153.85 MB, Opsix 3.06 GB, Pitt-Moba 1.37 GB, SLX-Mars 1.51 GB, totaling roughly 6.09 GB reclaimable in the toolbar readout." />
|
||||
@@ -135,33 +135,48 @@ The **Targets** checkboxes are independent and at least one must be ticked: **Im
|
||||
Scope is a segmented control with two options:
|
||||
|
||||
- **Managed only** (default). Sencho looks up the stacks it knows about on the node, then prunes only resources owned by those stacks. Active containers and resources placed by other tools are untouched.
|
||||
- **All unused**. Sencho runs the equivalent of `docker system prune` for each selected target. Any image, volume, or network not currently in use is deleted, including resources from workloads Sencho does not manage. The confirmation title flips to **Prune ALL unused resources across the fleet?**.
|
||||
- **All unused**. Sencho applies the target-specific Docker prune eligibility rules to each selected resource type. Any selected image, volume, or network not currently in use is deleted, including resources from workloads Sencho does not manage. The confirmation title flips to **Prune ALL unused resources across the fleet?**.
|
||||
|
||||
### Live estimate and behaviour
|
||||
### Review the itemized dry run
|
||||
|
||||
- Changing a target or the scope re-triggers a debounced call to `POST /api/fleet/prune/estimate`, which walks the same Docker enumeration the destructive path uses so the estimate matches what pruning would actually reclaim. A completed real prune that succeeds on at least one target re-triggers the same estimate so the toolbar total and per-node list reflect post-prune Docker state. **Prune fleet** stays disabled until the estimate resolves: you cannot confirm a destructive fleet-wide prune with no context on what it will reclaim.
|
||||
- Each remote node receives one `POST /api/system/prune/system` call per selected target, with a 120-second timeout. If a transport error fires for one target, the remaining targets on that node are short-circuited with the same error rather than retried, so a dead node doesn't absorb the full multi-target timeout budget.
|
||||
- Local nodes serialize against a per-node lock (`bulk-prune:<nodeId>`). A second fleet prune launched against the same local node while the first is still in flight returns *A prune is already running on this node* for each target.
|
||||
- Reclaimed bytes are reported by the Docker daemon and are approximate. Per-node rows in the results panel sum the per-target reclaim; the per-target children show how much each individual prune actually freed.
|
||||
Click **Dry run** after choosing targets and scope. Each reachable node returns one multi-target plan grouped into Images, Volumes, and Networks. Candidate rows show the stable ID, display name, reclaimable size when Docker provides one, why the resource is unused, managed or unmanaged ownership, and the associated stack when it can be resolved. Images also show available digest and creation details; volumes show their driver; networks show driver and scope. Only Compose ownership labels are shown, not arbitrary Docker labels.
|
||||
|
||||
<Note>
|
||||
Fleet Actions' prune card calls the same node-local prune route as the single-node **Resources → Prune** page, but without that page's itemized plan-and-fingerprint flow. It never returns the `PRUNE_PLAN_STALE` (409) error you can see on Resources; each fleet prune call targets exactly one resource type per node and executes immediately. If you want an itemized, reviewable plan before pruning a specific node, use that node's own Resources page instead.
|
||||
</Note>
|
||||
An untagged image is identified as `<none>:<none>` alongside its short ID. Under **All unused**, unmanaged candidates carry an **UNMANAGED** badge. Nodes that cannot be reached are shown as **excluded** and never as zero-candidate success. A reachable plan with zero items is still valid.
|
||||
|
||||
The node total is the sum of the sizes shown in that node's candidate rows. Image totals are estimates because Docker layers may be shared; the actual bytes reclaimed can differ after Docker accounts for layers still referenced by other images.
|
||||
|
||||
### Fingerprint-bound execution
|
||||
|
||||
**Prune fleet** remains disabled until the current targets, scope, and node roster have a valid reviewed plan for every reachable node. Execution sends one fingerprint per reviewed reachable node. Before deletion begins, the control instance rebuilds all plans, confirms that reviewed-unreachable nodes are still unreachable, and compares the complete configured-node roster.
|
||||
|
||||
If a node was added, removed, connected, disconnected, or changed candidates after the dry run, no node starts pruning. Run **Dry run** again to review the new state. Once fleet-wide preflight passes, each node revalidates immediately before deletion. A race at that point can produce a partial result, which is reported rather than hidden.
|
||||
|
||||
Local plan enumeration has an eight-second Docker-daemon timeout. Real local execution holds the per-node prune lock from preflight through mutation. Proxy remotes and Pilot nodes use one multi-target plan request and one fingerprint-bound execute request through their normal fleet transport. Mesh-managed stacks follow the transport of the node that hosts them.
|
||||
|
||||
### Read post-prune outcomes
|
||||
|
||||
The result keeps the reviewed name and metadata for every candidate and adds one outcome when the node returns itemized outcomes:
|
||||
|
||||
- **Removed** means the reviewed resource was deleted.
|
||||
- **Skipped** means it became active, was already absent, or became protected before deletion.
|
||||
- **Failed** includes the resource-level error returned by the node.
|
||||
|
||||
If a remote reports only its reclaimed total, the node shows that total without inventing per-item statuses. A completed mutation refreshes the live estimate.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
| Requirement | Why it matters |
|
||||
|---|---|
|
||||
| **Configured remote nodes in Settings → Nodes** | Stop and Prune iterate the configured node list; Bulk label assign iterates whichever nodes you select stacks on. A node missing its `api_url` or `api_token`, or one that cannot be reached, is reported once per node as unreachable and never blocks the reachable nodes. |
|
||||
| **Configured remote nodes in Settings → Nodes** | Stop and Prune iterate the configured node list; Bulk label assign iterates whichever nodes you select stacks on. Stop and Bulk label assign report an unreachable node without blocking other nodes. Prune excludes it from the reviewed plan and rejects execution if its reachability later changes. |
|
||||
| **Admin role** | Every card requires the admin role to apply. |
|
||||
| **Labels you intend to target** | Stop by label and its autocomplete depend on stack labels existing on at least one node; Bulk label assign depends on at least one stack label existing anywhere in the fleet. See [Stack Labels](/features/stack-labels) for the authoring flow. |
|
||||
|
||||
## Behaviour and lifecycle
|
||||
|
||||
- **Always returns 200.** Every destructive endpoint is structured so the HTTP status reflects the request shape, not the operational outcome. Partial failure is encoded in per-row fields, not in the status code.
|
||||
- **Operational outcomes are itemized.** Normal fan-out results use per-node and per-item fields. Fleet Prune uses `409` when the reviewed roster, reachability, or fingerprint changes before mutation, because that rejection guarantees no node has started deleting.
|
||||
- **No retry, no scheduling, no undo.** Fleet Actions runs synchronously and is operator-driven; there is no background scheduler and no roll-back. For recurrence, use [Scheduled Operations](/features/scheduled-operations).
|
||||
- **Offline remotes still receive the request.** A node that is down at the moment of the action returns a transport-error row but does not block the fan-out across the rest of the fleet.
|
||||
- **Concurrent runs serialize per node.** All three cards take per-node locks before touching Docker or the label tables, so kicking off a second prune, a second fleet stop, or a fleet stop overlapping a per-label stop on the same node yields a calm "already running on this node" row rather than silent double-execution.
|
||||
- **Offline remotes stay visible.** Dry run marks an unreachable node as excluded. If its reachability changes before Prune executes, the reviewed authorization is rejected and must be rebuilt.
|
||||
- **Concurrent mutations serialize per node.** A real local Fleet Prune holds its prune lock through preflight and execution. Dry-run enumeration stays outside the destructive lock.
|
||||
|
||||
## Limitations and non-goals
|
||||
|
||||
@@ -174,7 +189,7 @@ Fleet Actions is intentionally narrow. The following are deliberately out of sco
|
||||
- **No undo.** A stopped stack stays stopped until you start it again; a pruned image is gone until it is pulled or rebuilt.
|
||||
- **Approximate reclaim numbers.** The bytes the Prune card reports come from the Docker daemon and are best-effort, not authoritative.
|
||||
- **Confirmed-target stops need a current remote.** A real (non-dry-run) stop bound to specific stacks refuses to run against a remote that doesn't advertise support for confirmed-target binding; upgrade the remote to retry.
|
||||
- **Timeouts scale with the fan-out, not with any one node.** 60 seconds per remote on fleet-stop and bulk-assign, 120 seconds per remote per prune target. A remote with many stacks or a very slow filesystem may produce a timeout row before the underlying work fully completes; the action itself usually still finishes on the remote, the control instance just stopped waiting.
|
||||
- **Timeouts scale with the fan-out, not with any one node.** Remote fleet-stop and bulk-assign calls allow 60 seconds. Fleet Prune allows 120 seconds for each node's combined multi-target plan or execute request. A remote with many stacks or a very slow filesystem may produce a timeout row before the underlying work fully completes; during execution, check that remote's logs before retrying because the control instance may have stopped waiting after mutation began.
|
||||
|
||||
## Practical workflows
|
||||
|
||||
@@ -184,7 +199,7 @@ Tag the stacks you want to bring down with a dedicated label (for example `eveni
|
||||
|
||||
### Rehearse a destructive action before committing
|
||||
|
||||
For Stop and Prune, click **Dry run** first. It walks the identical lock, fan-out, and per-node logic as the real action but skips the destructive leaf call, so the results panel shows exactly what would happen (including which nodes are unreachable right now) before you commit to it.
|
||||
For Stop and Prune, click **Dry run** first. Fleet Prune shows the exact Docker candidates, including which nodes are excluded, and stores the fingerprints needed to unlock the destructive action.
|
||||
|
||||
### Propagate a label across the fleet
|
||||
|
||||
@@ -192,7 +207,7 @@ Define a label like `Media` on one node (for example the local node) under **Set
|
||||
|
||||
### Free disk before a heavy deploy
|
||||
|
||||
Run **Prune Docker resources fleet-wide** with **Images** selected and **Managed only** scope, and check the live per-node estimate before confirming. It gives a quick read on which hosts have accumulated the most stale layers. Switch to **All unused** if you want the prune to reach workloads that Sencho does not manage.
|
||||
Run **Prune Docker resources fleet-wide** with **Images** selected and **Managed only** scope. Check the live estimate, run the itemized dry run, and review each image before confirming. Switch to **All unused** if you want the plan to include workloads that Sencho does not manage.
|
||||
|
||||
## Common questions
|
||||
|
||||
@@ -201,13 +216,13 @@ Run **Prune Docker resources fleet-wide** with **Images** selected and **Managed
|
||||
Bulk mode operates on a hand-picked set of stacks **on one node** and supports start, stop, restart, and update. Fleet Actions operates **across every configured node** by selector (a label, or a checked cross-node set), and only Stop is a lifecycle action here (Bulk mode covers restart and update, Fleet Actions does not).
|
||||
</Accordion>
|
||||
<Accordion title="Does Dry run touch anything?">
|
||||
No. Dry run walks the same code path, including acquiring the per-node lock, but every card skips the destructive Docker or label-table call and returns what it would have done instead. It is safe to run repeatedly.
|
||||
No. Fleet Prune enumerates candidates without calling Docker remove methods or invalidating caches. It is safe to run repeatedly.
|
||||
</Accordion>
|
||||
<Accordion title="Why is the primary button disabled even though I typed a label or checked a target?">
|
||||
Every destructive or state-changing button stays disabled until the live preview or estimate resolves to a non-zero, non-loading result. This is deliberate: you always confirm against a concrete, current blast radius rather than an unresolved input.
|
||||
Fleet Prune requires a successful **Dry run** for the current targets, scope, and node roster. Run it again after any of those inputs or a node's reachability changes.
|
||||
</Accordion>
|
||||
<Accordion title="Why does Fleet's Prune never show the 'stale plan' error I've seen on Resources?">
|
||||
Resources → Prune builds an itemized plan with a fingerprint and re-validates it at execute time, which is where that error comes from. Fleet Actions' prune card calls the simpler legacy single-target path on each node instead, so there is no plan to go stale.
|
||||
<Accordion title="Why did Fleet Prune ask for another dry run?">
|
||||
The reviewed node roster, reachability, or candidate fingerprint changed before deletion began. Sencho rejected the entire fleet preflight so you can review the current candidates before trying again.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
@@ -239,7 +254,7 @@ Run **Prune Docker resources fleet-wide** with **Images** selected and **Managed
|
||||
Fleet Actions runs admin-only. Confirm the active user has the admin role under **Settings → Users**; operator and viewer roles see every card but cannot apply them.
|
||||
</Accordion>
|
||||
<Accordion title="A node is reported as unreachable">
|
||||
The node is in **Settings → Nodes** but its `api_url` or `api_token` is missing, expired, or unreachable. Stop by label reports it once as a single `<node> (unreachable)` row; Prune reports it per target. Open **Settings → Nodes** on the control instance and test the connection for the remote; fix the credential or the reachability, then re-run the action.
|
||||
The node is in **Settings → Nodes** but its `api_url` or `api_token` is missing, expired, or unreachable. Stop by label reports one `<node> (unreachable)` row. Fleet Prune keeps the node visible as excluded from the reviewed plan; its target rows carry the same reachability error. Open **Settings → Nodes** on the control instance and test the connection for the remote; fix the credential or reachability, then run a new dry run.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ The unit of work is the **bundle**. One bundle has one current `kv` payload; pus
|
||||
</Frame>
|
||||
|
||||
<Note>
|
||||
Fleet Secrets is a limited-availability surface. When it is present on an instance, managing it requires an admin user role.
|
||||
Fleet Secrets is available on every Sencho installation. Managing bundles requires an admin user role.
|
||||
</Note>
|
||||
|
||||
## What Fleet Secrets covers (and what it doesn't)
|
||||
@@ -38,14 +38,14 @@ A **push** is a separate action. It reads the bundle's current version, walks ev
|
||||
|
||||
| Requirement | Why it matters |
|
||||
|---|---|
|
||||
| Admin role on the control instance | Bundle CRUD and push require an administrator when the surface is present; authored-by rows are written into the audit log |
|
||||
| Admin role on the control instance | Bundle CRUD and push require an administrator; authored-by rows are written into the audit log |
|
||||
| At least one stack on at least one node | Pushes target an existing stack directory; the wizard does not create stacks |
|
||||
| The target stack's compose declares the env file via `env_file:` | The env-file dropdown in the push wizard reads `env_file:` entries from a representative node's compose; a stack with only an inline `environment:` block will not show up |
|
||||
| The control instance can reach the remote node's API URL | Each remote write is an HTTP call from the control instance to the remote's `/api/stacks/.../env`; an unreachable remote is reported as a per-node failure, not a transport error for the whole push |
|
||||
|
||||
## Create a bundle
|
||||
|
||||
1. Open **Fleet → Secrets** (when that tab is available on the instance).
|
||||
1. Open **Fleet → Secrets** on the Fleet view.
|
||||
2. Click **New bundle**.
|
||||
3. Give it a name. Names are 2-64 characters, alphanumerics plus space, dot, dash, and underscore, and must start and end with an alphanumeric.
|
||||
4. Optionally add a description; the description is a free-text field and is shown in the bundle list.
|
||||
|
||||
@@ -37,7 +37,7 @@ A single rail summarises the state of every registered node so you can read the
|
||||
|
||||
### Tabs
|
||||
|
||||
The Fleet view is a tab strip. Every tier sees Overview, Status, Map, Docker Labels, Deployments, Federation, and Actions. Snapshots appears for admins. Routing and Secrets are limited-availability fleet surfaces and are not part of the default tab strip. A vertical separator after **Docker Labels** (or after **Map** when Docker Labels is not present) divides the per-node monitoring tabs from the fleet-wide orchestration tabs.
|
||||
The Fleet view is a tab strip. Every tier sees Overview, Status, Map, Docker Labels, Deployments, Federation, and Actions. Snapshots appears for admins. Secrets appears for admins. Routing is a limited-availability fleet surface and is not part of the default tab strip. A vertical separator after **Docker Labels** (or after **Map** when Docker Labels is not present) divides the per-node monitoring tabs from the fleet-wide orchestration tabs.
|
||||
|
||||
| Tab | Tier | What it does |
|
||||
|-----|------|--------------|
|
||||
@@ -50,7 +50,7 @@ The Fleet view is a tab strip. Every tier sees Overview, Status, Map, Docker Lab
|
||||
| **Routing** | Limited availability | Cross-node service routing via Sencho Mesh when that surface is enabled on the instance. See [Sencho Mesh](/features/sencho-mesh). |
|
||||
| **Federation** | Community | Cordon nodes and pin blueprints to specific hosts. See [Fleet Federation](/features/fleet-federation). |
|
||||
| **Actions** | Community (admin role) | Fleet-wide bulk operations: stop stacks by label, bulk-assign labels, prune Docker resources. See [Fleet Actions](/features/fleet-actions). |
|
||||
| **Secrets** | Limited availability | Encrypted env-var bundles you push to labeled nodes when that surface is enabled on the instance. See [Fleet Secrets](/features/fleet-secrets). |
|
||||
| **Secrets** | Community (admin role) | Encrypted env-var bundles you push to labeled nodes across the fleet. See [Fleet Secrets](/features/fleet-secrets). |
|
||||
|
||||
### Action buttons
|
||||
|
||||
@@ -105,18 +105,19 @@ Every node renders as a card. The local node is pinned at the top of the grid wi
|
||||
|
||||
Offline nodes render dimmed, with no stats grid, no usage bars, and no update affordance.
|
||||
|
||||
### Node actions menu (admin)
|
||||
### Node actions menu
|
||||
|
||||
Every card carries a three-dot **Node actions** kebab in the top-right corner. The menu surfaces the same lifecycle actions you would find in **Settings · Infrastructure · Nodes**:
|
||||
Every card carries a three-dot **Node actions** kebab in the top-right corner:
|
||||
|
||||
| Action | Notes |
|
||||
|--------|-------|
|
||||
| **Node details** | Opens an info sheet with the node's connectivity, live capacity, Compose workload, version and update compatibility, and governance info (labels, cordon reason and date, default-node status, Compose directory, registration date). Available to anyone who can see the card; the label picker inside the sheet stays editable only for whoever holds `node:manage` on that node. |
|
||||
| **Edit node** | Opens the Edit dialog prefilled with the node's connection details. For proxy-mode remotes, saving with a changed API URL or token re-runs the connection test automatically. |
|
||||
| **Delete node** | Opens a destructive confirmation. The local (default) node has no Delete option. Deleting a remote only removes it from this console; the remote instance and its containers are untouched. |
|
||||
| **Cordon node** / **Uncordon node** | Marks the node unschedulable so new blueprint deployments skip it. Existing deployments keep running. Requires the `node:manage` permission (admin, or node-admin when scoped to that node). |
|
||||
| **Mute** submenu | Mute node notifications, mute update notifications, mute monitor alerts for this node, or open the full mute-rule manager. Shown to whoever can manage mute rules for the node. See [Alerts & Notifications](/features/alerts-notifications). |
|
||||
|
||||
Edit and delete remain admin-only. Users without `node:manage`, without mute permission, and without edit/delete affordances see no kebab on the card.
|
||||
Edit, delete, cordon, and mute stay gated on `node:manage` or mute permission as before. Every card shows the kebab with at least **Node details**, even for a viewer with no manage permissions.
|
||||
|
||||
### Topology view
|
||||
|
||||
|
||||
@@ -26,7 +26,7 @@ The palette groups results into three sections.
|
||||
|
||||
| Group | What it contains | What happens when you pick one |
|
||||
|-------|------------------|--------------------------------|
|
||||
| **Pages** | The reachable page destinations for your tier and role (the same set Classic / Smart / mobile navigation use). **Home**, **Resources**, **Networking**, **Security**, and **App Store** appear for signed-in operators; **Fleet** appears when your role holds the `node:read` permission; **Logs**, **Update**, **Schedules**, and **Console** appear for admins; **Audit** appears on Admiral for any role with the `system:audit` permission. See [RBAC & User Management](/features/rbac) for the full permission matrix. | Navigates to that page |
|
||||
| **Pages** | The reachable page destinations for your tier and role (the same set Classic / Smart / mobile navigation use). **Home**, **Resources**, **Networking**, **Security**, and **App Store** appear for signed-in operators; **Fleet** appears when your role holds the `node:read` permission; **Logs**, **Update**, **Schedules**, and **Console** appear for admins; **Audit** appears for any role with the `system:audit` permission. See [RBAC & User Management](/features/rbac) for the full permission matrix. | Navigates to that page |
|
||||
| **Nodes** | Every node in your fleet, with a green dot for online and a grey dot for offline. The currently active node carries a small **ACTIVE** chip on the right. | Switches the active node without leaving the current page |
|
||||
| **Stacks** | Every compose stack on every online node, matched on the compose filename (extension included). | Switches to the stack's node and opens it in the editor |
|
||||
|
||||
|
||||
@@ -101,6 +101,21 @@ The Stack Dossier carries a **Rollback readiness** section that answers one ques
|
||||
<img src="/images/health-gated-updates/dossier-rollback-readiness.png" alt="Rollback readiness section in the Stack Dossier showing the overall state chip and the six rows: Previous compose file, Previous env file, Previous image tag, Last successful deploy, Healthchecks, and the Application data row marked not covered" />
|
||||
</Frame>
|
||||
|
||||
## Automatic rollback images
|
||||
|
||||
Before a full-stack update runs, Sencho captures the running image of every service as an opaque, uniquely named copy so it can automatically restore the prior state if the update or its health gate fails. These copies exist in Docker as `sencho-rb/<generation>/<service>:hold`, but they are Sencho-internal recovery state, not part of your image inventory: they are kept out of **Resources → Images** and listed instead in **Resources → Rollback**. If a captured image still carries its original registry tag alongside the hold tag (a compose file pinned to an immutable tag, for example), it stays visible in the Images tab too, badged **Rollback protected** instead of the usual unused label, since it is held on purpose rather than left behind by accident.
|
||||
|
||||
Each capture is one **rollback generation**. The generation currently backing a stack's live deployment is retained for as long as it is current; once a newer update supersedes it, it is retained for a configurable window before Sencho cleans it up automatically. A stack updated repeatedly in a short span can have more than one superseded generation in that window at once.
|
||||
|
||||
**Resources → Rollback** lists every generation on the node: the stack it belongs to, a short generation id, whether it is the current protection or a superseded one awaiting cleanup, and roughly when it clears. An admin can release a generation's protection early from that list, including the current one, which immediately frees its image but means Sencho cannot automatically roll that stack back until its next successful full-stack update; the confirmation dialog says so before you proceed. A generation that is mid-recovery or still being observed by a health gate cannot be released until that finishes.
|
||||
|
||||
Because these images are deliberately held, deleting one directly (by id, including through the API) is refused. Release the generation from **Resources → Rollback** instead, or leave it to clear on its own.
|
||||
|
||||
Two settings under **Settings > Infrastructure > Stacks > Deploy Guardrails** control the automatic cleanup:
|
||||
|
||||
- **Superseded rollback retention** sets how many days a superseded generation is kept before its image is cleaned up. The current generation is unaffected by this window; it stays protected until it is superseded or manually released. Default 7 days.
|
||||
- **Maximum retained rollback generations per stack** caps how many generations a stack keeps at once, current generation included, so the oldest superseded generations beyond the cap are cleaned up ahead of the retention window. 0 (default) leaves the count unlimited and relies on the retention window alone.
|
||||
|
||||
## Classified failures
|
||||
|
||||
When a deploy or update fails, Sencho classifies the failure from the compose output and shows the cause with a suggested next step in the recovery panel: an image pull failure, a missing environment variable, a host port conflict, a missing bind-mount path, a permission problem, a crashed container, a failed healthcheck, an unavailable dependency, an unreachable node or Docker daemon, or an invalid compose file. The classification also lands in **Copy details**, so a bug report carries the cause, not just the raw output.
|
||||
@@ -126,4 +141,10 @@ When a deploy or update fails, Sencho classifies the failure from the compose ou
|
||||
<Accordion title="Updates from the sidebar menu now show a dialog first">
|
||||
The sidebar's per-stack **Update** action runs the same path as the editor toolbar, so it shows the same readiness dialog and deploy progress. One click on **Update now** proceeds. On nodes that do not advertise the capability, updates run directly without the dialog.
|
||||
</Accordion>
|
||||
<Accordion title="Why do I see sencho-rb/... images in docker images on the host">
|
||||
Those are automatic rollback images: an opaque copy of a service's prior image, held so Sencho can restore it if a full-stack update fails. They are not leftovers. Sencho keeps them out of **Resources → Images** on purpose (they are recovery state, not image inventory) and lists them in **Resources → Rollback** instead, showing which stack and generation each one belongs to and how soon it clears on its own. If one still carries a normal registry tag too, it also stays visible in the Images tab with a **Rollback protected** badge.
|
||||
</Accordion>
|
||||
<Accordion title="Deleting a rollback-protected image fails">
|
||||
That failure is intentional: the image is protected by an active or recently superseded rollback generation. Open **Resources → Rollback**, find the matching generation, and use **Release** there if you are sure you do not need it. Releasing the current generation means Sencho cannot automatically roll that stack back until its next successful full-stack update.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
@@ -152,7 +152,7 @@ Configure threshold-based alerts per stack and route notifications to Discord, S
|
||||
|
||||
### Audit log
|
||||
|
||||
Track mutating actions across your Sencho instance with a searchable trail: who deployed, stopped, deleted, or changed settings, with timestamps, user attribution, and node context. Community keeps a rolling 14-day recent-activity audit API window. The Audit navigation view, plus CSV/JSON export, anomaly detection, and configurable retention, is Admiral governance. [Learn more →](/features/audit-log)
|
||||
Track mutating actions across your Sencho instance with a searchable trail: who deployed, stopped, deleted, or changed settings, with timestamps, user attribution, and node context. Users with `system:audit` can open Audit from navigation. Community shows a rolling 14-day window; Admiral adds the 24h signal rail, export, anomaly detection, and configurable retention. [Learn more →](/features/audit-log)
|
||||
|
||||
## Fleet management
|
||||
|
||||
@@ -194,7 +194,7 @@ When several Sencho instances run as a fleet, the control instance is the source
|
||||
|
||||
### Fleet Secrets
|
||||
|
||||
Centralized, encrypted, versioned env-var bundles you push to labeled nodes' stacks. Every save bumps a version, and every push records a per-node diff in the audit log using overlay merge semantics. Limited-availability surface when present; admin role required to manage. [Learn more →](/features/fleet-secrets)
|
||||
Centralized, encrypted, versioned env-var bundles you push to labeled nodes' stacks. Every save bumps a version, and every push records a per-node diff in the audit log using overlay merge semantics. Available on every installation; admin role required to manage. [Learn more →](/features/fleet-secrets)
|
||||
|
||||
### Fleet-wide backups
|
||||
|
||||
|
||||
+19
-15
@@ -20,8 +20,8 @@ Sencho ships with five built-in roles that map to the permissions most operators
|
||||
|------|----------------|------|
|
||||
| **Admin** | Full operator access: deploy, edit compose, manage users, configure nodes, view audit log, every system setting | Community |
|
||||
| **Viewer** | Read-only access to stacks, logs, stats, file contents, and node listings | Community |
|
||||
| **Deployer** | Deploy, restart, stop, and start stacks. Cannot edit compose files, create or delete stacks, or view nodes | Admiral |
|
||||
| **Node Admin** | Full stack and node management across the fleet. No access to system settings, users, or license | Admiral |
|
||||
| **Deployer** | Deploy, restart, stop, and start stacks, and check individual stacks for image updates. Cannot edit compose files, create or delete stacks, view nodes, or manage alert and auto-heal rules | Admiral |
|
||||
| **Node Admin** | Full stack and node management across the fleet, including node-scoped operational Settings. No access to users, licensing, credentials, or system-only Settings | Admiral |
|
||||
| **Auditor** | Read-only access to stacks, nodes, and the audit log. No write access anywhere | Admiral |
|
||||
|
||||
### Permission matrix
|
||||
@@ -31,8 +31,8 @@ Each row is one of the permission keys the backend checks. The matrix below is t
|
||||
| Permission | Admin | Node Admin | Deployer | Auditor | Viewer |
|
||||
|------------|:-----:|:----------:|:--------:|:-------:|:------:|
|
||||
| View stacks, logs, stats (`stack:read`) | Yes | Yes | Yes | Yes | Yes |
|
||||
| Deploy, restart, stop, start, take down (`stack:deploy`) | Yes | Yes | Yes | No | No |
|
||||
| Edit compose and `.env` files (`stack:edit`) | Yes | Yes | No | No | No |
|
||||
| Deploy, restart, stop, start, take down, check a stack for image updates (`stack:deploy`) | Yes | Yes | Yes | No | No |
|
||||
| Edit compose and `.env` files, manage alert rules and auto-heal policies (`stack:edit`) | Yes | Yes | No | No | No |
|
||||
| Create stacks (`stack:create`) | Yes | Yes | No | No | No |
|
||||
| Delete stacks (`stack:delete`) | Yes | Yes | No | No | No |
|
||||
| View nodes (`node:read`) | Yes | Yes | No | Yes | Yes |
|
||||
@@ -115,29 +115,30 @@ Click **Update user** to save. Changing the role takes effect on the next API re
|
||||
|
||||
Scoped permissions let you grant a user a higher role on a specific stack or node without elevating them globally. A Viewer can be granted Deployer on one stack; a Deployer can be granted Node Admin on one server.
|
||||
|
||||
**Stack scopes are node-specific.** The same stack name on two different nodes is two independent grants. Assigning a stack scope means choosing the node first, then picking a stack that exists on that node. Display form conceptually: stack name @ node name (for example `frontend @ prod`).
|
||||
|
||||
**Node scopes are node-wide.** The resource is the node itself. There is no separate node qualifier on a node assignment row. Granting Node Admin (or Deployer, or Admin) on `staging-server` authorizes that role's stack and node operations for every stack on that node, without a separate per-stack grant.
|
||||
|
||||
The box appears below the user form whenever you are editing a user on Admiral.
|
||||
|
||||
<Frame>
|
||||
<img src="/images/rbac/scoped-permissions.png" alt="Edit User form for the viewer account with a Scoped Permissions box below. The box contains an existing assignment row (a Deployer badge with the text on Stack: bazarr and a destructive trash icon on the right) and a three-column add-scope row underneath (Role combobox set to Deployer, Resource Type combobox set to Stack, Resource combobox showing Select..., and a disabled Add button)." />
|
||||
</Frame>
|
||||
|
||||
The add-scope form has three controls and an **Add** button:
|
||||
The add-scope form has these controls and an **Add** button:
|
||||
|
||||
| Control | Options |
|
||||
|---------|---------|
|
||||
| **Role** | Deployer, Node Admin, or Admin. The scoped role picker is narrower than the global role picker. Viewer and Auditor cannot be scoped (they are floor-only roles). |
|
||||
| **Resource Type** | `Stack` or `Node`. |
|
||||
| **Resource** | The picker shows stacks (when type is `Stack`) or remote nodes (when type is `Node`) the gateway knows about. Resource names match what you see in the sidebar. |
|
||||
| **Node** | Shown when Resource Type is `Stack`. Choose the node that hosts the stack before the stack picker unlocks. |
|
||||
| **Stack** or **Node** | When type is `Stack`, the picker lists stacks on the selected node. When type is `Node`, the picker lists nodes the gateway knows about. Names match what you see in the sidebar. |
|
||||
|
||||
Click **Add** to save the assignment. Existing scopes render as a row with the role badge, the line `on <type>: <resource>`, and a trash icon for removal. Removing an assignment is instant; the user's effective permissions are recomputed on their next request.
|
||||
Click **Add** to save the assignment. Existing scopes render as a row with the role badge, the line `on <type>: <resource>`, and for stacks the node name after an `@`, plus a trash icon for removal. Removing an assignment is instant; the user's effective permissions are recomputed on their next request.
|
||||
|
||||
Scoped assignments are **additive only**. A Viewer with a scoped Deployer on `frontend` can deploy `frontend` but stays read-only on every other resource. Scopes never reduce the global role.
|
||||
Scoped assignments are **additive only**. A Viewer with a scoped Deployer on `frontend` at a given node can deploy that stack on that node but stays read-only on every other resource. Scopes never reduce the global role.
|
||||
|
||||
### Example scenarios
|
||||
|
||||
- A **Viewer** with a scoped **Deployer** assignment on the `frontend` stack can deploy, restart, and stop only that stack. They cannot edit compose or delete it.
|
||||
- A **Viewer** with a scoped **Deployer** assignment on the `frontend` stack at node `prod` can deploy, restart, and stop only that stack on `prod`. The same name on another node needs its own grant. They cannot edit compose or delete it.
|
||||
- A **Deployer** with a scoped **Node Admin** assignment on node `staging-server` can manage every stack and node operation on that server, while keeping plain Deployer rights on the rest of the fleet.
|
||||
- A **Node Admin** without any scoped assignments has full stack and node management across every node, but still cannot reach system settings, the user list, or the audit log.
|
||||
- A **Node Admin** without any scoped assignments has full stack and node management across every node, including node-scoped operational Settings, but still cannot reach users, licensing, credentials, or system-only Settings.
|
||||
|
||||
## Two-factor reset
|
||||
|
||||
@@ -224,6 +225,9 @@ Entries include the acting user, IP address, HTTP method and path, response stat
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Permission controls are unavailable">
|
||||
Sencho could not verify your current permissions. Existing pages stay open, but changes remain disabled until verification succeeds. Select **Retry** in the notification bar. If the notice returns, check that the Sencho instance is reachable and sign in again if your session has expired.
|
||||
</Accordion>
|
||||
<Accordion title="The Users entry is missing from the Settings sidebar">
|
||||
The Users entry is hidden in two cases. **One,** you are signed in as a non-admin (Viewer, Deployer, Auditor): the entry is admin-only. **Two,** you have a remote node selected: the panel is hub-only and is hidden in the sidebar when any remote node is active. Switch back to the local node via the node switcher in the masthead.
|
||||
</Accordion>
|
||||
@@ -237,7 +241,7 @@ Entries include the acting user, IP address, HTTP method and path, response stat
|
||||
Check whether **Session policy > Keep active sessions alive** was turned off in **Settings > Users**. With it off, every session hits a strict, fixed 24-hour (or 30-day, with **Stay signed in**) ceiling regardless of activity. Turn it back on so an active session renews itself instead of hard-expiring, or have the user check **Stay signed in** at their next sign-in for a longer session between visits.
|
||||
</Accordion>
|
||||
<Accordion title="A scoped Deployer cannot deploy a stack they were granted">
|
||||
Two causes. **One,** the assignment was created on Admiral but the license has since dropped to Community. The permission resolver only consults scoped assignments when the effective tier is Admiral; on Community the scope is ignored and the user falls back to their global role. **Two,** the resource type or name on the assignment does not match the request's resource. Re-open the user in the edit form and check the existing-scope row matches the stack name (case-sensitive) exactly.
|
||||
Three causes. **One,** the assignment was created on Admiral but the license has since dropped to Community. The permission resolver only consults scoped assignments when the effective tier is Admiral; on Community the scope is ignored and the user falls back to their global role. **Two,** the resource type or stack name on the assignment does not match the request's resource (names are case-sensitive). **Three,** the stack grant is tied to a different node than the one the user is acting on: the same stack name on another node is a separate grant. Re-open the user in the edit form and confirm the existing-scope row shows the expected stack name at the expected node.
|
||||
</Accordion>
|
||||
<Accordion title="The shield (Reset 2FA) icon is missing on a user I expected to see it on">
|
||||
The icon only appears for users with a finished TOTP enrollment. If the user started enrollment but never confirmed their first code, the enrollment is incomplete and the icon stays hidden. Ask the user to finish enrollment from their account settings, or, if they cannot, leave the row alone: there is nothing to reset.
|
||||
|
||||
@@ -83,7 +83,7 @@ A Compose port range such as `8000-8002:8000-8002` is compared conservatively. B
|
||||
|
||||
### Network findings and the Networking tab
|
||||
|
||||
The **network-undeclared** and **network-missing** findings reuse the same comparison the stack's [Networking](/features/compose-networking) tab uses, so the two surfaces never disagree about a network attachment. The difference is history: this tab persists findings in the drift ledger over time, while the Networking tab always shows the current live state with no history. A stack with an open network finding also counts toward the **Drift** chip in the Fleet Overview's Networking filter group, so a network-attachment mismatch is visible both per-stack here and across the fleet there.
|
||||
The **network-undeclared** and **network-missing** findings reuse the same comparison the stack's [Networking](/features/compose-networking) tab uses, so the two surfaces never disagree about a network attachment. Sencho-verified `sencho_mesh` attachments for its own container and Mesh-opted-in stacks are excluded, while a manual attachment from an opted-out stack remains actionable. The difference is history: this tab persists findings in the drift ledger over time, while the Networking tab always shows the current live state with no history. A stack with an open network finding also counts toward the **Drift** chip in the Fleet Overview's Networking filter group, so a network-attachment mismatch is visible both per-stack here and across the fleet there.
|
||||
|
||||
## When drift is recorded
|
||||
|
||||
@@ -93,6 +93,8 @@ The **network-undeclared** and **network-missing** findings reuse the same compa
|
||||
|
||||
**After every deploy or update**, Sencho automatically records a new baseline hash and runs a full reconciliation. You do not need to click re-check after deploying; the ledger is updated as part of the deploy pipeline.
|
||||
|
||||
An open ledger entry for an attachment that is no longer reported clears during the next re-check or post-deploy reconciliation. Opening the tab refreshes the live report but does not change persisted history.
|
||||
|
||||
<img
|
||||
src="/images/stack-drift/drift-history.png"
|
||||
alt="The Drift tab showing both the Findings section and the Drift history section with an open finding marked just now"
|
||||
|
||||
@@ -223,13 +223,17 @@ Each row includes:
|
||||
- **Meta line.** Uptime (`up 12 hours`), the Docker health status when defined (`healthy`, `unhealthy`, or `starting`), and the primary port mapping (`8989 → 8989/tcp`).
|
||||
- **Open link.** When the container publishes a port, the mapping itself is a link (`8989 → 8989/tcp ↗`) that opens the service in a new tab, with a **Copy URL** button beside it. The address uses the active node's host and switches to `https` for port 443. Recognised multi-port apps open their web path automatically (for example, Plex opens `/web`).
|
||||
- **Live stat tiles.** Three tiles show CPU, memory, and network I/O with a rolling sparkline. The sparkline uses the cyan data color and refreshes roughly every 1.5 seconds.
|
||||
- **Action icons.** The image source links button (see above), plus shortcuts to **View logs**, open a bash shell, and (on single-service stacks) the per-container Start / Stop / Restart kebab.
|
||||
- **Action icons.** The image source links button (see above), plus shortcuts to **View logs**, open a bash shell, and the **Service actions** kebab on single-service stacks and single-container multi-service rows (multi-replica services keep the kebab on the shared header; see below).
|
||||
|
||||
### Multi-service stacks
|
||||
|
||||
When a stack declares more than one Compose service, each service gets its own header above its container cards. That header owns **Update service** or **Rebuild service** (when eligible), the update badge when a registry update is confirmed for that service, and **Start / Stop / Restart** for every replica of the service. Child container cards keep logs, shell, ports, and metrics only.
|
||||
When a stack declares more than one Compose service, each service with a single matching container renders as one flat container card. When a registry image update is confirmed for that service, the card shows **Update** immediately left of the image source links control (with an update badge). Build-backed services without a confirmed registry update show **Rebuild** in the same place. The **Service actions** kebab for Start / Stop / Restart stays on the card.
|
||||
|
||||
Build-backed services without a registry update show **Rebuild** wording and no update badge. The editor toolbar **Update** action still updates the entire stack.
|
||||
Services with multiple replicas keep a shared header above their nested container cards. That header owns the same **Update** / **Rebuild** rules, the update badge, and **Start / Stop / Restart** for every replica. Nested cards keep logs, shell, ports, and metrics only.
|
||||
|
||||
A declared service with no running containers shows a compact row with the service name, Update/Rebuild under the same rules, and the Service actions kebab (so you can still Start the service).
|
||||
|
||||
After a service update succeeds and the image check clears, the **Update** button disappears. The editor toolbar **Update** action still updates the entire stack at any time.
|
||||
|
||||
Single-service stacks keep the existing flat container layout; service headers do not appear.
|
||||
|
||||
@@ -425,7 +429,7 @@ Press `B` again or toggle the bulk mode button to leave bulk mode.
|
||||
|
||||
## Controlling a single service
|
||||
|
||||
Inside the stack detail view, each container card has quick actions (View logs, Monitor, Open bash when available) and a **Service actions** kebab on the right side. Use them to act on that service without touching the rest of the stack.
|
||||
Inside the stack detail view, container cards expose quick actions (View logs, Monitor, Open bash when available). On single-service stacks, single-container multi-service rows, and empty declared-service rows, a **Service actions** kebab sits on that row. On multi-replica services the kebab lives on the shared service header only, not on each replica card. Use these actions to act on that service without touching the rest of the stack.
|
||||
|
||||
- **Monitor**: opens the stack **Monitor** sheet with that Compose service preferred in the add forms. Shown only when the container reports a Compose service name.
|
||||
- **Restart service**: stops and starts all containers for that service.
|
||||
|
||||
@@ -53,6 +53,7 @@ These tune optional subsystems. Most deployments never set them; the defaults ar
|
||||
| `SENCHO_COMPOSE_COMMAND_TIMEOUT_MS` | `1800000` | Hard timeout for a single Compose command (pull, up, down) during deploy and update, in milliseconds (30 minutes). Sencho kills the command and reports failure if it runs longer than this, regardless of whether it is still producing output. Raise it only for very large images or slow storage. |
|
||||
| `SENCHO_COMPOSE_STALL_TIMEOUT_MS` | `600000` | Idle-output backstop for deploy and update Compose steps (pull and recreate), separate from the hard timeout above. If a step produces no output for this long while still running, Sencho stops it so a hung image pull surfaces a clear failure and the in-app recovery actions instead of spinning. Raise it on slow links or for heavy local image builds. |
|
||||
| `SENCHO_ZFS_ARCSTATS_PATH` | *(auto)* | Path **inside the container** to the OpenZFS ARC kstat file, for [ZFS ARC-aware host memory](#zfs-arc-aware-host-memory). Sencho checks this path first, then `/host/proc/spl/kstat/zfs/arcstats`, then `/proc/spl/kstat/zfs/arcstats`. Set it only when your ARC stats live at a non-standard path. |
|
||||
| `SENCHO_PROC_MEMINFO_PATH` | *(auto)* | Path **inside the container** to `/proc/meminfo`, for [VM memory ballooning](#vm-memory-ballooning). Sencho checks this path first, then `/host/proc/meminfo`, then `/proc/meminfo`. Set it only when you need a custom meminfo path. |
|
||||
|
||||
Running a remote host as a pilot agent uses four more variables (`SENCHO_MODE`, `SENCHO_PRIMARY_URL`, `SENCHO_ENROLL_TOKEN`, and `SENCHO_PILOT_CA_FILE`), set only on the remote agent container. Sencho bakes them into the enrollment Compose file it generates, so you rarely write them by hand. See [Pilot Agent](/features/pilot-agent) for the full enrollment walkthrough.
|
||||
|
||||
@@ -60,7 +61,7 @@ Running a remote host as a pilot agent uses four more variables (`SENCHO_MODE`,
|
||||
|
||||
On OpenZFS hosts (TrueNAS SCALE, Proxmox, ZFS on Ubuntu or Debian) the ZFS ARC cache can hold a large share of RAM. ARC is reclaimable on demand, but the Linux kernel reports it as unavailable, so a naive reading counts ARC as used memory and can raise false host-memory alerts.
|
||||
|
||||
Sencho reads the ARC kstat when it is available and adds the reclaimable portion back into available memory, so the dashboard memory gauge and host RAM alerts reflect real memory pressure. When no ARC stats are readable the behavior is unchanged.
|
||||
Sencho reads the ARC kstat when it is available and adds the reclaimable portion back into available memory, so the dashboard memory gauge and host RAM alerts reflect real memory pressure. The reclaimable amount is also shown on the dashboard memory tile as a context line. When no ARC stats are readable the behavior is unchanged.
|
||||
|
||||
The ARC kstat is usually visible inside the container at `/proc/spl/kstat/zfs/arcstats` with no extra configuration. If your runtime does not expose it, mount it read-only:
|
||||
|
||||
@@ -71,6 +72,21 @@ volumes:
|
||||
|
||||
Sencho checks `SENCHO_ZFS_ARCSTATS_PATH`, then `/host/proc/spl/kstat/zfs/arcstats`, then `/proc/spl/kstat/zfs/arcstats`. Set `SENCHO_ZFS_ARCSTATS_PATH` only if your ARC stats live somewhere else inside the container.
|
||||
|
||||
## VM memory ballooning
|
||||
|
||||
On Linux virtual machines with memory ballooning enabled (TrueNAS/KVM, Proxmox, VMware), the hypervisor can reclaim guest memory through a balloon driver. The reclaimed amount is tracked in `/proc/meminfo` on the `Balloon:` line but standard memory counters do not account for it, so a ballooned VM can appear memory-critical when the guest workload is actually healthy.
|
||||
|
||||
Sencho reads the `Balloon:` field from `/proc/meminfo` when it is available and shows the ballooned amount on the dashboard memory tile alongside an effective-usage percentage. The memory gauge, health verdict, and host RAM alerts continue to use the standard working-set percentage: unlike ZFS ARC, ballooned memory is reclaimed by the hypervisor and the guest cannot get it back on demand, so balloon data is informational context rather than a factor in alerting or health decisions. When `/proc/meminfo` is unreadable or the `Balloon:` field is absent, the behavior is unchanged.
|
||||
|
||||
`/proc/meminfo` is usually visible inside the container at `/proc/meminfo` with no extra configuration. If your runtime does not expose it, mount it read-only:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
- /proc/meminfo:/host/proc/meminfo:ro
|
||||
```
|
||||
|
||||
Sencho checks `SENCHO_PROC_MEMINFO_PATH`, then `/host/proc/meminfo`, then `/proc/meminfo`. Set `SENCHO_PROC_MEMINFO_PATH` only if your meminfo lives somewhere else inside the container.
|
||||
|
||||
## Listen port
|
||||
|
||||
Sencho always listens on `1852` inside the container. The port is fixed and is not read from an environment variable. To expose Sencho on a different host port, remap with Docker's `-p` flag (or the `ports:` key in your compose file):
|
||||
|
||||
@@ -21,7 +21,7 @@ These are the main terms you will see across the app and docs:
|
||||
- **Resource**: A Docker image, volume, network, or unmanaged container. Sencho classifies resources as managed, external, system, unused, or protected so cleanup decisions are visible.
|
||||
- **Blueprint**: A fleet deployment model for keeping a compose template assigned to matching nodes.
|
||||
- **Pilot Agent**: An outbound connector for remote nodes that cannot accept inbound traffic.
|
||||
- **Limited-availability networking and secrets surfaces**: documented on their own feature pages when enabled on an instance.
|
||||
- **Limited-availability networking surface**: documented on its own feature page when enabled on an instance.
|
||||
|
||||
## What you see after sign-in
|
||||
|
||||
@@ -64,7 +64,7 @@ The **Fleet** view is the multi-node command center. The masthead summarizes onl
|
||||
|
||||
The Fleet toolbar includes **Check Updates**, **Refresh**, and **Add node** for admins. The **Overview** tab supports search, sort, status filters, label filters, and a Grid or Topology view. Node cards show online state, resource use, container counts, version state, update actions, and direct drill-down into stacks on that node.
|
||||
|
||||
Beyond **Overview**, Fleet provides tabs for **Snapshots**, node **Status**, a dependency **Map**, a **Docker Labels** audit, blueprint **Deployments**, **Federation**, and fleet **Actions**. Federation placement (cordon and pin) is available on every tier. Routing and Secrets are limited-availability tabs when enabled on an instance. See [Licensing](/features/licensing) for the full tier breakdown.
|
||||
Beyond **Overview**, Fleet provides tabs for **Snapshots**, node **Status**, a dependency **Map**, a **Docker Labels** audit, blueprint **Deployments**, **Secrets**, **Federation**, and fleet **Actions**. Federation placement (cordon and pin) is available on every tier. Routing is a limited-availability tab when enabled on an instance. See [Licensing](/features/licensing) for the full tier breakdown.
|
||||
|
||||
## Resources, templates, and logs
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 31 KiB |
@@ -61,7 +61,7 @@ Every self-hosted instance includes the full security stack. Some advanced gover
|
||||
</Card>
|
||||
|
||||
<Card title="Fleet Secrets" icon="key-skeleton" href="/features/fleet-secrets">
|
||||
Limited-availability encrypted, versioned env-var bundles pushed to labeled nodes' stacks when the surface is present. Sealed with the same data key as MFA and registry credentials.
|
||||
Encrypted, versioned env-var bundles pushed to labeled nodes' stacks. Sealed with the same data key as MFA and registry credentials.
|
||||
</Card>
|
||||
|
||||
<Card title="Webhook signatures" icon="signature" href="/features/webhooks">
|
||||
@@ -212,7 +212,7 @@ For user management and scoped permissions, see [RBAC & User Management](/featur
|
||||
|
||||
Every POST, PUT, DELETE, and PATCH request to the API is recorded in the audit log with the acting username, IP address, HTTP method, response status, and an auto-generated summary. GET requests are excluded to keep the log focused on mutations.
|
||||
|
||||
The audit log is searchable by keyword (actions, paths, usernames) and filterable by HTTP method and date range. The recent-activity log, scoped to the last 14 days, is available on every tier. With Admiral, results can be exported as CSV or JSON (up to 10,000 entries per export), entries carry anomaly annotations, and retention defaults to 90 days and is configurable from 1 to 365 days in **Settings · Operations · Data Retention**.
|
||||
The audit log is searchable by keyword (actions, paths, usernames) and filterable by HTTP method and date range. Users with the `system:audit` permission can open **Audit** from navigation. Community shows the last 14 days of activity; with Admiral, results can be exported as CSV or JSON (up to 10,000 entries per export), entries carry anomaly annotations, and retention defaults to 90 days and is configurable from 1 to 365 days in **Settings · Operations · Data Retention**.
|
||||
|
||||
The **Auditor** role provides read-only access to the audit log without any other administrative privileges, making it suitable for compliance reviewers who should not have access to system settings.
|
||||
|
||||
|
||||
@@ -207,7 +207,7 @@ Create and manage user accounts with role-based access. The masthead publishes a
|
||||
| **Admin** | Community | Full access to all features |
|
||||
| **Viewer** | Community | Read-only access to stacks and nodes |
|
||||
| **Deployer** | Admiral | Can view stacks and trigger deployments |
|
||||
| **Node Admin** | Admiral | Full stack and node management, no system settings |
|
||||
| **Node Admin** | Admiral | Full stack and node management, including node-scoped operational Settings |
|
||||
| **Auditor** | Admiral | Read-only plus audit log access |
|
||||
|
||||
See [RBAC & User Management](/features/rbac) for details on what each role can access.
|
||||
|
||||
Reference in New Issue
Block a user