mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-06 08:58:05 +00:00
Anso
08caa914ce
* docs(atomic-deployments): refresh page around current UI and behavior
Rewrites the page to match the v1 docs refresh template. Corrects
several factual errors against the current code, fills in missing
detail, and adds a screenshot of the rollback overflow menu.
Notable corrections:
- Scheduled tasks do not run atomically; only stack editor Deploy and
Update, App Store installs, webhook triggers, and image auto-updates
pass the atomic flag through to ComposeService.
- Rollback lives in the stack editor's More actions overflow menu, not
on the action bar directly. The backup timestamp renders as a
sub-line of the menu item.
- Health probe is a 3-second window with an exit-code check on every
container labelled with the compose project name; describe this
exactly rather than as 'waits briefly'.
- Document where backups live (DATA_DIR/backups/<stack>/), why they
are kept outside the compose folder, and that the slot is one per
stack with overwrite semantics.
- Document the four streamed log markers users see in the deploy
progress modal during the atomic flow.
- Add a troubleshooting accordion group covering missing menu entry,
late crashes outside the probe window, manual-intervention message,
and the single-slot retention edge case.
* docs(deploy-enforcement): refresh page for v1 and align with current enforcement paths
Update the page to match the current pre-flight gate behavior, the v1 modal chrome on the
block dialog, and the AccordionGroup troubleshooting pattern used across the v1 docs.
Drift items corrected:
- Replace the broken vulnerability-scanning/deploy-blocked-dialog.png reference with three
fresh captures under docs/images/deploy-enforcement/ (policy list, policy editor, block
dialog).
- Drop "Recreate from the stack actions menu" and the git-source apply pre-flight claim;
neither path runs the gate.
- Add bulk label deploy and the auto-update scheduler to the enforced code paths, with a
dedicated subsection for the auto-update interaction (alert-and-skip, not 409).
- Drop the false claim that severity chips in the block dialog are clickable; the dialog
is informational.
- Document the compose-parse-fails-closed branch with its synthetic violation label.
- Refresh dialog copy to reflect the v1 ModalDestructiveHeader (kicker, title, button
variants).
- Convert the troubleshooting Q&A into AccordionGroup blocks and add accordions for the
compose-parse-error case and the auto-update-skipped case.
- Quote the verbatim audit-log summary format.
* docs(blueprints): refresh against v1 UI and add federation/state-review coverage
* docs(git-sources): refresh page against v1 UI and current behavior
Rewrites the page against the v1 docs refresh template (Note tier-gate,
sectioned anatomy, AccordionGroup troubleshooting), aligning prose with
the live UI labels and the current code paths.
Corrections:
- Authentication toggle reads "Public (no auth)" / "Personal Access
Token" (not "None"), and apply mode "Auto-write files" (not
"Auto-write").
- Diff dialog kicker is GIT . PULL PREVIEW; local-edits state opens an
Overwrite local edits? confirmation modal whose primary button is
Overwrite and apply.
- Sidebar pending indicator is a small GitBranch icon, not a brand-color
dot, and the image-update dot takes priority over it on the same row.
- Pending update banner appears in the panel; Review re-fetches the
commit and opens the diff (no client-side payload caching).
Adds coverage for:
- Anatomy of the panel (pending banner, form, last-applied stat strip,
footer actions).
- 10-second webhook debounce window.
- Pending compose/env content is encrypted at rest in the database, not
just the token.
- Auth/host failures map to HTTP 400, never 401, so they do not sign
the user out.
- Per-stack lock serializes pull, apply, and create-from-git so a
webhook firing during a manual apply waits rather than racing.
- Compose validation has a 10-second budget; clone fetches have a
30-second timeout.
- New troubleshooting accordion for Pending commit has changed since
this pull was fetched.
Recaptures all five screenshots from the v0.74.x production node,
signed in as admin: panel, create-from-git tab, pull-preview diff
dialog, sidebar GitBranch pending icon, webhook Action select with
Git source sync highlighted.
* docs(stack-labels): refresh page for v1 sidebar grouping and fleet-action surface
- Lead with the v1 behavior the previous page did not cover: the sidebar
groups stacks under collapsible label headers (PINNED first, label
buckets sorted by stack count desc then name asc, UNLABELED last)
with a count chip per group. Trailing colored dots on each row
(max 3 + N overflow, paid-only) supplement the headers.
- Drop the stale claim that a label-pill filter bar lives between
search and the stack list; that UI no longer exists.
- Drop the right-click-on-pill bulk actions table (Deploy all / Stop
all / Restart all). The legacy per-node action endpoint stays in
the backend but no longer has a UI binding, so the page documents
only what users can click today.
- Document the two Skipper+ Fleet Action cards: Stop fleet by label
(name match across nodes, autocomplete, per-node breakdown,
HTTP 429 on per-node concurrency) and Bulk label assign (per-node,
replace semantics, clear on empty selection).
- Document the inline 'New label' form inside the stack right-click /
three-dot Labels submenu, the Settings - Advanced - Labels masthead
N/50 stat, the LABELS - NEW / EDIT modal kickers, and the
LABELS - DELETE - IRREVERSIBLE confirmation copy verbatim.
- Document the Fleet Overview Tags multi-select filter (filters by
stack labels aggregated across nodes), with cross-link to fleet-view.
- Capture every screenshot fresh from production signed in as admin:
sidebar-grouping, context-menu-labels, inline-create-form,
settings-labels, create-label-dialog, fleet-tags-filter,
fleet-actions. Drop the now-stale sidebar-with-labels,
sidebar-filtered, and bulk-actions-menu captures.
* docs(dashboard): refresh page for v1 layout (status masthead, gauges, fleet heartbeat, restart map)
Aligns docs/features/dashboard.mdx with the redesigned Home tab. Replaces the obsolete
Recent Activity feed coverage with the actual DashboardActivityCard split (Fleet Heartbeat
when remote nodes are registered, Stack Restarts (7d) otherwise) and recaptures every
screenshot from the v0.74.x production node.
* docs(global-search): refresh page for v1 palette
- Note tier and role gating on the Pages list (Auto-Update, Console,
Schedules, Audit) so the prose matches what the top bar exposes.
- Document the ACTIVE chip on the currently active node row.
- Document the 50-result cap counter and the Searching... loading state.
- Mention the ~250 ms debounce and clarify that filename matching
includes the file extension.
- Replace stack screenshot with a redesigned capture and add empty-state
Pages and Nodes captures showing the ACTIVE chip.
* docs(global-observability): refresh page for v1 layout (masthead, signal rail, filter strip, paused-resume chip)
Full rewrite against the current Logs tab and the v1 docs refresh template
(hero Frame, sectioned anatomy, AccordionGroup troubleshooting, refresh-cadence table).
Replaces the single overview screenshot with seven captures under
docs/images/global-observability/ (overview, masthead, signal-rail,
filter-strip, feed-bands, paused-resume-chip, error-only-filter), all from
the v0.75.x production node signed in as admin with PII scrubbed
(profile chip patched to AD, in-feed LAN IPs and third-party hostnames
substituted via DOM injection while the stream was paused).
Aligns prose with the actual UI labels and code:
- Masthead kicker reads LIVE LOGS · NODE · <NAME> with LOCAL for the
local node; state word toggles Streaming / Idle / Offline; SESSION
uses uppercase letter suffixes (1H 43M / 0M 12S) per formatUptime.
- Signal rail tile counts are scoped to the 2000-entry buffer and reset
with Clear; CONTAINERS is buffer-bound, not a monotonic accumulator.
- Filter strip controls quoted verbatim (Stacks · All / Stacks · n,
segmented controls All / Out / Err and All / Info / Warn / Error).
- Feed row anatomy: severity dot, timestamp, brand-cyan container name
with stack/container tooltip, message tinted by source. Row tint
follows detected level, which is regex-based, so an STDOUT line
containing ERROR: still classifies as ERROR.
- Day bands: NOW, Nm AGO, Nh AGO, calendar date.
- Empty states: two-tier kicker over caption (Awaiting events / No matches).
- Pause keeps the SSE buffer filling up to the 2000-entry cap; resume pill
reads <n> NEW · RESUME and counts the queue, not total arrivals during
the pause.
- Download filename and row format quoted: sencho-logs-<ISO8601>.txt and
[<ISO>] [<stack>/<container>] <LEVEL>: <message>.
Documents behavior the previous page never covered:
- Active-node scoping; node switch resets the stream and the buffer.
- SSE primary transport with 30-second server heartbeat and a 5-second
polling fallback against /api/logs/global (server-capped at 500 lines
per snapshot).
- Initial replay of the last 500 lines per container when the SSE
connection opens, so the feed has context immediately.
- Display limits (2000 client buffer, 300 rendered rows, Showing last
300 of N overflow notice).
- Refresh cadence table covering UI tick, flush cadence, polling
cadence, SSE heartbeat, sparkline window, and the Idle threshold.
Adds a seven-accordion troubleshooting block (Offline state, gray Idle
dot, ERROR-without-tint, growing Resume pill, Clear-cutoff lag,
node-switch buffer drop, fleet-wide aggregation expectations).
Tightens the closing Note so it makes clear that Notification Log
Retention does not govern this live container stream.
* docs(alerts-notifications): refresh page for v1 and absorb notification-routing
Full v1 template rewrite of /features/alerts-notifications. Bundles in
the entire Notification Routing page so a reader sees channels, routing,
per-stack rules, and retention in one place; deletes the standalone
notification-routing.mdx and points all five cross-link sites at the new
in-page anchor.
* docs(alerts-notifications): drop "What's not in scope" section
The page should describe what Sencho does, not enumerate what it does
not ship. Users find missing integrations through the Webhook section
and the routing matcher reference; the explicit disclaimer added noise
without adding guidance.
* docs(audit-log): refresh page for v1 layout, expanded action list, troubleshooting accordion
- Clarify that the search/method/date filter strip lives in Table view only.
Stream view always shows the unfiltered chronological feed.
- Fold the total-entries readout into the card subtitle wording where it
actually renders, instead of describing it as a separate header element.
- Sharpen the Peak hour off-hours window to the literal 08:00 to 17:59
working window the tile keys off, plus the 5% / 20% failure-rate tints.
- Note that the Actors tile names a sample actor alongside the new-IP count.
- Expand the example actions list to cover surfaces that have shipped since
the last edit: per-service stack lifecycle, node cordon/uncordon, fleet
replica role changes, Sencho Cloud Backup operations, Fleet Secrets, and
blueprint federation pin updates.
- Correct the Settings path: Settings · Developer · Data retention card,
Audit log input, Save settings button.
- Add a Troubleshooting AccordionGroup matching the rest of the v1-refresh
pages: missing tab, filter scope, anomaly thresholds, export cap, and
retention pruning.
- Replace all four screenshots with fresh captures of the current UI.
* docs(multi-node): refresh page for v1 layout, pilot agent mode, refreshed table columns
Rewrites docs/features/multi-node.mdx against the current product. The previous page predated the v1 Settings hub redesign and the Pilot Agent enrollment model, so it documented only the Distributed API Proxy add-node flow and missed the new Mode, Endpoint, and Labels columns on the Nodes table.
Restructures the page into 13 sections: intro, How it works, the local node, Choose a remote mode (decision table comparing Pilot Agent vs Distributed API Proxy), Add a remote node: Pilot Agent (three steps plus re-enrollment), Add a remote node: Distributed API Proxy (three steps), Switching between nodes, the Nodes table (full column reference), What Settings apply per node (verified against settings/registry.ts), License enforcement across nodes, Editing and deleting nodes, Security (token security, transport encryption, why no application-layer TLS), and Troubleshooting (AccordionGroup matching the v1 template used on audit-log, atomic-deployments, and deploy-progress pages).
Refreshes seven screenshots against the production node signed in as admin, scrubbing IPs and usernames before capture: full Nodes panel overview, Generate Node Token card with a placeholder token, Add node modal in Pilot Agent mode, Add node modal in Distributed API Proxy mode (with the inline plain-HTTP warning visible), Edit modal showing the Regenerate enrollment token card for a pilot agent, Pilot enrollment modal with the docker run command, refreshed node switcher popover, and a close-up of the table columns. Drops the obsolete add-node-form.png, http-warning.png, and per-node-scheduling/ folder.
* docs(fleet-view): refresh page for v1 layout, expanded tabs, cordon, sheet-based updates
- Aligns the Overview, Status, and Node Updates content with today's UI:
the masthead's `The fleet` headline plus CPU / MEM / CONTAINERS stat tiles,
the eight-tab strip (Overview, Snapshots, Status, Deployments, Traffic,
Federation, Fleet Actions, Secrets) with per-tier visibility, and the
Check Updates surface that is now a system sheet rather than a modal.
- Documents the toolbar (search, sort, filter popover with Status / Type /
Severity / Tags sections) and the Grid / Topology segmented control
including the topology graph's status pill (Online / Critical / Offline),
connector colouring, ReactFlow controls and minimap.
- Documents the per-card surfaces that were missing from the prior page:
Cordoned badge with cross-reference to Fleet Federation, fleet stack
label dots in the drill-down, container drill-down rows (state dot,
badge, image, status, open-in-editor hover button), and the Admiral
three-dot Node actions menu for cordon / uncordon.
- Documents the Node Updates sheet anatomy (Recheck and Update all (n)
header actions, four summary cards, node table columns, Update flow,
reconnecting overlay timing, admin enforcement) and the GitHub Releases
with Docker Hub fallback resolution path with its 30-minute cache.
- Replaces every stale screenshot with a fresh capture (overview,
topology, drill-down, status tab, node updates sheet) and removes the
obsolete files plus the empty docs/images/fleet/ folder.
- Reformats troubleshooting as an AccordionGroup matching dashboard,
multi-node, and audit-log refreshes.
* docs(fleet-backups): refresh page for redesigned fleet and settings UI
Replace all six screenshots with current production captures. Update
content to reflect the new fleet header card, eight-tab layout, full-
page Cloud Backup settings with header stats, and corrected navigation
paths. Add cloud backup rows to the access control table.
* docs(fleet-backups): convert troubleshooting to AccordionGroup pattern
Match the foldable-accordion pattern used across the v1 docs refresh
batch. Merges the standalone Cloud Backup troubleshooting subsection
into a single Troubleshooting section at the bottom of the page with
seven accordions covering skipped nodes, two restore failure modes,
three cloud-upload failure modes, and a diagnostic logging entry.
* docs(remote-updates): refresh page for v1 sheet, accordion troubleshooting, factual fixes
Rewrites the page against the v1 docs refresh template (Note tier gate,
sectioned mechanism deep-dive, Frame screenshots with detailed alt text,
inline AccordionGroup troubleshooting), bringing it in line with the
recently-refreshed fleet-view, fleet-backups, dashboard, and audit-log
pages.
The page is repositioned as the mechanism deep-dive (prerequisites, what
runs on a node during an update, completion and failure detection,
recovery actions). The full UI tour for the Node updates sheet remains in
fleet-view so the two pages stop overlapping; remote-updates now links
into fleet-view#node-updates instead of restating the table anatomy.
Captures three screenshots from the production node, signed in as admin:
fleet-node-updates.png shows the Node updates sheet with eight nodes and
seven remote updates available; local-update-confirm.png shows the
LOCAL · UPDATE alert dialog with the Cancel and Update & restart buttons;
node-card-update-available.png shows the Opsix card with the Update
available pill and the Update to v0.76.7 outline button.
Corrects several factual claims that no longer matched the current code:
- The remote early-fail threshold is about 3 minutes, matching
EARLY_FAIL_MS in backend/src/routes/fleet.ts, not 90 seconds.
- The Recheck button sits in the sheet header, not the footer.
- The component is a SystemSheet, so the page now consistently calls it
the Node updates sheet instead of a dialog, with lowercase "Node
updates" and lowercase "Update all (n)" matching the live UI.
- Reconnecting overlay polls /api/health every 3 seconds, not "every few
seconds".
- The local Failed badge surfaces as soon as the helper writes its error
file, by the 3-minute mark at the latest.
Documents the LocalUpdateConfirmDialog kicker, title, body, and CTA
verbatim, the Triggering... loading state on the Update buttons, the
four completion signals the gateway accepts (version change, process
startedAt change, offline-then-online transition, version at or above
the comparison target after 15 seconds), and the 60-second auto-clear
of the Updated badge.
Drops references to two screenshots that never existed
(fleet-node-updating.png, fleet-node-failed.png); the in-flight and
failed states are described in prose instead, the same way fleet-view
handles them.
* docs(scheduled-operations): refresh page for v1 timeline, fleet-wide update action, sheet-based run history
Rewrites the Scheduled Operations page against the v1 template
(Note tier gate, sectioned anatomy, Frame screenshots, AccordionGroup
troubleshooting) applied to sibling pages in this batch. Captures
seven fresh screenshots against the production node signed in as
admin (timeline, all-tasks, action-picker, create-restart,
create-prune, create-scan, run-history) and removes every legacy
PNG.
Documents the new "Auto-update All Stacks" action that was absent
from the page, extends the Skipper allow-list to all four Skipper
actions (Auto-update Stack, Auto-update All Stacks, Fleet Snapshot,
Vulnerability Scan) and clarifies that the action picker hides
operations the active tier cannot run.
Corrects several factual claims that no longer matched the code:
- Scheduled scan completion is `info`/`scan_finding` on a clean run
and `warning`/`scan_finding` when findings are present (not
`info`/`system` as previously stated). Cross-link now points at
`alerts-notifications#vulnerability-scanning`.
- Lifecycle actions (auto_backup, auto_stop, auto_down, auto_start)
execute against the local Sencho instance only; only Auto-update
Stack / All Stacks have a remote-proxy code path. The page
reinstates the guidance to schedule remote lifecycle operations
from that node's own UI.
- Run history lives in a right-side sheet with a "Schedules ›
<task> › Runs" breadcrumb and a Download CSV secondary action.
- Timeline masthead is described in terms of the v1 visual
(`NEXT 24 HOURS` kicker, italic display heading, monospace date
range, right-anchored Next pill with countdown, glowing cyan now
rail, six-tick bottom axis).
* docs(rbac): refresh RBAC & user management page against v1 template
Bring /features/rbac onto the v1 docs refresh template (Note tier gate,
sectioned anatomy, Frame screenshots, AccordionGroup troubleshooting).
Recapture five screenshots from the production node signed in as admin
and remove the three stale captures under docs/images/rbac/.
Corrections vs. the prior page:
- Deployer no longer claims node:read in the permission matrix; the
backend grants only stack:read and stack:deploy.
- Add the system:registries row (container registry management).
- Document the form as inline below the Add user button (not a modal).
- Note the (you) marker on the signed-in admin's row and the disabled
delete icon on that row.
Additions:
- Settings nav location and hub-only visibility.
- 2FA reset row action with verbatim modal kicker, title, and body.
- Five-failure / 15-minute MFA lockout behavior and admin reset recovery.
- Token-version session-security table covering deletion, role change,
password change, and admin 2FA reset.
- SSO password-fields-hidden line quoted verbatim and the per-provider
Require MFA toggle.
- Audit-log emissions list for every user-management mutation.
- API tokens cross-link explaining the user-vs-machine boundary.
- Scoped permissions section retightened: scoped role picker is
Deployer / Node Admin / Admin only; resource type is Stack or Node.
AccordionGroup with eight troubleshooting entries covering missing nav,
greyed role options, seat-limit errors, unexpected sign-outs, scoped
deployer mismatches, missing shield icon, re-locking MFA accounts, and
SSO role drift at provisioning.
* docs(2fa): refresh two-factor authentication and admin guide against v1 template
Bring /features/two-factor-authentication and /operations/two-factor-admin
onto the v1 docs refresh template (Note tier gate, sectioned anatomy, Frame
screenshots with descriptive alt text, AccordionGroup troubleshooting,
verbatim modal copy with kicker callouts). Recapture every screenshot under
docs/images/two-factor-auth/ from a fresh session and add six new captures
for surfaces the prior page did not document.
Corrections vs the prior pages:
- Panel rename: Settings -> Account & Security is now Settings -> Account,
under the Identity group of the settings sidebar. Replaced every
occurrence on both pages.
- Enrol dialog titles match the current modal: Pair your authenticator,
Confirm the pairing, Save your recovery codes (was: Set up 2FA, Confirm,
Save your backup codes). Step rail 01 PAIR / 02 CONFIRM / 03 ARCHIVE
documented.
- Manual-entry affordance is the always-visible Secret manual entry row
with a copy icon, not the toggleable Can't scan Show secret key link.
- Confirm step auto-submits on the sixth digit; no submit button. Verified
in MfaChallenge.tsx and MfaEnrollDialog.tsx and called out explicitly.
- Authenticator-app list trimmed to match in-app copy (1Password, Bitwarden,
Google Authenticator, or any TOTP app). Authy and Microsoft Authenticator
dropped because the dialog does not mention them.
- Disable dialog: kicker SECURITY MFA DISABLE, title Turn off two-factor,
destructive header, Disable button. Replaces the prior Disable 2FA
paragraph that did not describe the dialog chrome.
- Regenerate dialog: two-step flow with kicker SECURITY BACKUP CODES, Confirm
identity then New recovery codes, with the verbatim PREVIOUS CODES HAVE
BEEN INVALIDATED warn rail on the show step. Documented that the dialog
only accepts a TOTP, not a backup code.
- Per-user SSO toggle label corrected: Require 2FA on SSO sign-in (was:
Require 2FA even when signing in via SSO). Added the per-provider vs
per-user distinction on both pages (admins can also enable Require MFA
on the SSO provider config, which is independent of the per-user toggle).
- Admin reset modal: verbatim USERS RESET 2FA kicker, Reset 2FA for
<username> title, full-body copy reproduced. Documented that the reset
bumps the target's token version and invalidates active sessions.
Additions:
- Sign-in throttle: five failed verifications lock the account for 15
minutes, server returns 423 with Retry-After, UI shows the Retry in MM:SS
countdown plus Rate limited label. Lockout recovery section explains
that the counter only clears on a successful sign-in, so retries after
the window expires re-lock immediately.
- Account panel anatomy section enumerates the three rows (Authenticator
app, Backup codes, Require 2FA on SSO sign-in) plus the destructive
Disable 2FA link, and the masthead 2FA on / BACKUP N left chips.
- Recovery codes section now covers all three count states (3 plus, 1 to 2,
0) with verbatim helper text, tone, and the standalone No backup codes
left callout that renders at zero. New screenshots for the 2-remaining
and 0-remaining states.
- Cross-references to the admin operations page (CLI fallback, token version
rotation, what a reset changes in the DB), the SSO page, and the RBAC
page (per-provider Require MFA toggle, SSO auto-provisioning).
Troubleshooting on the feature page rewritten as an AccordionGroup with
nine entries: clock drift, wrong account selected, QR will not scan, lost
phone with no codes, lost codes with authenticator, ran out of codes,
unexpected SSO prompt (with both toggle causes), repeated lockout after
the window expires, missing shield icon on Users panel.
The admin operations page also gains the SSO + 2FA two-toggles table so
administrators can answer the per-user vs per-provider question without
context-switching between pages.
Six new images added; six existing images replaced. Total 14 captures.
* docs(rbac,host-console): drop enforcement-boundary detail from tier-gate notes
Operator-facing docs should state tier or role requirements once, in plain
customer-facing language, and leave the enforcement chain to the source.
Two surfaces on the v1-refreshed pages over-specified the gate:
- `features/rbac.mdx::Scoped permissions`: the Note enumerated both the UI
hide on Skipper and the `/api/users/:id/roles` write rejection. The first
half ("Scoped permissions require Admiral.") is the operator-relevant
fact; the rest reads as a fence specification, which is awkward for an
open-core product where the gate is readable in source anyway. Trimmed
to just the tier claim.
- `features/host-console.mdx::Availability`: the paragraph already says
who can use the console and that the Console tab is hidden on Community
or Skipper. The trailing "Attempting to access the console endpoint
directly without the correct license or role is rejected" is the same
bypass-prevention coda. Dropped.
No functional behavior change; the gates themselves are untouched.
* docs(sso): refresh SSO & LDAP authentication page against v1 template
Rewrites docs/features/sso.mdx against the v1 docs refresh template (intro
+ tier callout, sectioned Configuration anatomy, Frame screenshots,
AccordionGroup troubleshooting), bringing it in line with the previously
refreshed two-factor-authentication and rbac pages on this branch.
Recaptures all four screenshots from the production node signed in as
admin: sso-settings (overview with the five collapsible provider cards),
sso-settings-ldap (LDAP form expanded), sso-settings-oidc (Google form
expanded), sso-settings-custom-oidc (Custom OIDC form expanded with all
eleven fields).
Refreshes the Settings UI section to match the redesigned panel: each
provider is a collapsible card with an Active badge on the header, an
enable / disable toggle pill, and a footer with Save, Test Connection
(green check or red X next to the button), and Remove (only after a
config has been saved). Documents the static callback-URL helper that
sits below all five cards.
Clarifies that the per-OIDC claim mapping environment variables
(SSO_OIDC_*_ID_CLAIM, *_USERNAME_CLAIM, *_EMAIL_CLAIM) are accepted for
Google, GitHub, and Okta, not just Custom OIDC. The Settings UI hides
those fields on the presets because the defaults match.
Converts the troubleshooting section to an AccordionGroup with five
entries (Test Connection discovery failure, issuer validation error,
wrong username or missing email after sign-in, invalid redirect URI,
SSO buttons missing on the login page). Cross-links the operations
troubleshooting page for setup-time errors.
Tightens the LDAP TLS env var note to spell out the literal string
'false' requirement. Syncs the Combining SSO with 2FA section to use
the live toggle label 'Require 2FA on SSO sign-in'.
* docs(sso): drop the Community-tier Custom OIDC workaround tip
The Tip walked through how a Community-tier operator could integrate
Google, GitHub, or Okta by pointing Custom OIDC at the provider's
discovery URL, bypassing the Skipper preset gate. Operator docs should
state the tier rule once and stop; they should not describe how to
circumvent it.
The tier matrix above the removed block already names which providers
are paid; the Custom OIDC row already lists "any spec-compliant OIDC
provider" as its scope. That is enough.
* docs(vulnerability-scanning): refresh page for v1 UI and corrected tier mapping
The page was last revised before the v1 visual redesign and before the
tier-mapping changes shipped in v0.81.2 (open Community access to
secret scanning, compose misconfig scanning, scan history, and scan
comparison). This refresh:
- Rewrites the tier matrix to match the shipped Community / Skipper /
Admiral split. Secret detection, compose misconfig scanning, scan
history, scan comparison, and misconfig acknowledgements are now
correctly marked as Community. Scheduled fleet scans, scan policies
with block_on_deploy, SBOM, SARIF, and Trivy auto-update stay paid.
- Drops two stale Notes that said secret detection and compose
misconfig scanning required Skipper or Admiral. The page now states
each tier requirement once, in plain language.
- Refreshes all six existing screenshots from the production node:
resources-badges, scan-details-sheet, scan-history-sheet,
scan-compare-sheet, security-settings, app-store-toggle.
- Adds a new scan-config-button screenshot showing the stack-page
overflow menu where Scan config now lives.
- Describes the scan drawer header accurately: Re-scan + Compare + CSV
+ SARIF as top-level buttons, with SBOM as a separate button below
the summary.
- Updates the compose misconfig flow to point at the stack overflow
menu (not the Deploy controls).
- Converts the troubleshooting section to a single AccordionGroup per
the v1 template, and audits each entry for legacy phrasing and the
removed tier claims.
- Adds a TRIVY_BIN reference to the How it works section so operators
know about the host-binary override.
* docs(cve-suppressions): refresh page for v1 UI and corrected suppression specifics
- Recapture all three screenshots from the production node signed in
as admin under `docs/images/cve-suppressions/` (`settings-panel`,
`create-dialog`, `suppressed-row`). The previous file referenced
three image paths that did not exist in the repo.
- Align prose with the actual UI labels:
- Dialog kicker `SUPPRESSIONS . NEW`, title `New suppression`.
- Field labels match the form: `CVE or advisory ID`, `Package
(optional)`, `Image pattern (optional)`, `Reason`, `Expires in
(days, optional)`.
- Remove confirmation reads `Remove suppression` with kicker
`SUPPRESSIONS . REMOVE . IRREVERSIBLE`.
- Factual corrections:
- Fleet sync truncation cap is 5,000 rows (not 10,000).
- State the admin-role requirement once in the lead Note.
- Drop references to a `Fleet . Sync status` page and a `Reanchor`
button; neither exists in the UI. The reanchor flow is an admin
API call and is documented in /features/fleet-sync.
- Sharpen the specificity scoring section (package + image scores
3, package only 2, image only 1, neither 0) so the order matches
the read-time filter logic.
- Note that the image-pattern glob is case-sensitive.
- New coverage:
- Suppressing directly from a scan result, including which fields
are read-only in that inline flow and when to fall back to
Settings to broaden scope.
- The `replicated` and `expired` row badges in the panel.
- Hovering the package column on a suppressed row to surface the
Reason.
- Two distinct read-only modes: viewing a remote node from the hub
(panel hidden, banner shown) versus signing into a replica
instance (panel visible, read-only).
- SARIF export carries suppressions through as
`kind: external, status: accepted`, cross-linked to the
Vulnerability Scanning page.
- Convert troubleshooting to AccordionGroup with six entries; update
the truncation entry to reflect the 5,000-row cap.
* docs(private-registries): refresh page for v1 UI and fleet-wide credential model
Rewrites the page against the v1 docs template (Note tier gate, opening Frame,
sectioned anatomy, AccordionGroup troubleshooting) and replaces every
screenshot with a fresh capture taken against the current product.
Corrects several factual claims that no longer matched the current code:
- Registries are stored once on the control instance and applied fleet-wide,
not configured per node. The old Multi-node behavior section and the
matching troubleshooting entry described a per-node model that the product
no longer has.
- The Registries section is hidden on remote nodes (global scope) and on
Sencho versions that do not surface the feature. New troubleshooting
entries explain both visibility states.
- The feature is admin-only on Admiral. Non-admin operators do not see the
section even on Admiral; previous copy implied any Admiral license user
could manage credentials.
- Registry endpoints are not reachable from API tokens; only an admin
browser session can manage credentials. The Security section now states
this without naming internal route paths.
Documents UI behavior the previous page omitted: the inline form (not modal),
the four type-specific form variants, the Docker Hub read-only URL field, the
destructive delete confirmation with its stack-pull warning, the masthead
REGISTRIES count, and the empty-state callout copy.
Screenshots replaced:
- registries-overview.png: section with one configured GHCR card and the
masthead stat at one.
- registries-empty.png: empty state with the Add registry button and callout.
- registries-add-form.png: inline form with the Docker Hub default and the
read-only URL field.
- registries-ecr-form.png: form switched to ECR, showing the AWS Region
field and the relabelled AWS credential inputs.
- registries-card-detail.png: card close-up with the three action icons and
the metadata row.
- registries-delete-confirm.png: destructive ConfirmModal with the kicker,
title, and stack-pull warning body.
- registries-with-entry.png removed (superseded by registries-overview.png
and registries-card-detail.png).
* docs(auto-update): refresh readiness page for v1 redesign
Bring the Auto-Update Readiness doc in line with the shipped UI:
- Replace the hero screenshot with a fresh capture of the redesigned
board (italic-display hero, brand-cyan accent, per-node groups with
local/remote pills, dashed-border changelog separator).
- Rewrite the card-anatomy list. Drop the rollback-target bullet (the
field exists in the backend payload but is not rendered). Add the
"Rebuild available" inline label and the primary-image / multi-service
count line.
- Rewrite the risk-tags table as a risk-badges table using the actual
badge labels and colors emitted by the UI (Safe / Review / Blocked
with the corresponding icons; Digest rebuild for non-semver tags).
- Add an Empty state section and document the per-node group header.
- Tighten the hero subtitle paragraph to match the actual UI string
(only major-bump count is surfaced separately; preview failures are
not).
- Fix workflow step 4: major-bump apply path is the stack lifecycle
Update action, not the Schedules editor (a scheduled task hits the
same block).
- Add the 2-minute manual-refresh cooldown to the Recheck workflow.
- Remove the broken cross-link to the non-existent
/features/image-update-detection page and inline the 6-hour cadence
fact from ImageUpdateService.INTERVAL_MS.
- Convert troubleshooting to AccordionGroup format per the troubleshoot
ing convention used on /features/deploy-progress.
- Sync the Auto-Update entry in /features/overview.mdx to the new
badge labels and the corrected hero-counter description.
* docs(auto-update): fix Auto-Update entry point in Workflow step 1
Workflow step 1 said "Open the Auto-Update view from the sidebar." The
Auto-Update view is opened from the top nav strip (alongside Home,
Fleet, Resources, App Store, Logs, Schedules, Console, Audit). The
sidebar carries the stack list and the per-stack right-click / kebab
context menu that toggles auto-updates on or off; it does not house
the Auto-Update top-level view.
* docs(auto-update): trim enforcement detail from per-stack control note
State the tier requirement once and stop, per Directive 27. The
"The toggle does not appear on Community" sentence enumerates the
enforcement effect of the gate, which the source already reflects;
operator docs do not need to narrate it.
* docs(auto-heal): refresh page for v1 UI and policy hardening
Rewrite Auto-Heal Policies docs against the current Stack Monitor
sheet: corrects the Max restarts / hr field label, documents the
per-policy enable toggle, the consecutive-failures pill, the full
Recent activity action set (including Docker unavailable), the 30s
evaluation cadence, multi-node behavior, notification dispatches,
and the dashboard Configuration status counter.
Replaces the broken /images/auto-heal-policies/policy-sheet.png
reference with three fresh screenshots captured against a live
node: the sheet on the Auto-heal tab, a single policy row, and
the expanded Recent activity panel.
* docs(webhooks): refresh page for v1 UI, correct tier and add Git source sync
- Fix tier note: gate is Skipper or Admiral, management is admin-only.
- Update Settings path to Settings -> Alerts -> Webhooks; document the
read-only Node field and the green secret-reveal callout.
- Add the missing Git source sync action and the git-pull override value.
- Refresh the configured-webhooks card description: action/stack/node
badges, On/Off toggle, copy URL, and the Recent executions disclosure.
- Tighten the trigger section with a constant-time signature check note
and a status/body/meaning response table.
- Add an Accordion troubleshooting block covering common signature
failures, the 404 case, no-op actions on 202, and git-pull prereqs.
- Re-capture all three screenshots from the v1 UI.
* docs(webhooks): wrap troubleshooting accordions in AccordionGroup
* docs(sidebar): refresh page for v1 redesign with filter chips, bulk mode, row anatomy, and troubleshooting
Rewrites the Stack Sidebar page against the live v1 sidebar and the v1
docs refresh template (Frame screenshots, Note tier callouts,
AccordionGroup troubleshooting). Recaptures all four existing
screenshots and adds three new captures: filter chips, row anatomy,
and bulk mode.
Adds coverage for features the previous page omitted entirely: the
ALL / UP / DOWN / UPDATES filter chips with their counts cap and
collapse toggle; bulk mode (B key, sticky toolbar with Start / Stop /
Restart, and Update gated on Skipper or Admiral); stack-row anatomy
(status pill, label dots with +N overflow, image-update dot vs Git
source icon priority, hover kebab); the Auto-update toggle, Schedule
task, and Open App entries in the context menu; the B shortcut for
bulk mode.
Corrects three claims that no longer matched the code or UI:
Auto-Heal is gated on Skipper or Admiral, not universal; the global
Ctrl+K opens the command palette, not the sidebar search box; the
activity footer kicker reads LIVE / IDLE with the verbatim copy from
SidebarActivityTicker. Documents the in-menu ↗ and L › glyphs as
visual hints rather than global keybindings to match
useStackKeyboardShortcuts.ts.
* docs(sidebar): trim enforcement-effect sentence from context-menu tier note
State the Skipper / Admiral requirement once and stop, per Directive 27.
The "They do not appear in the menu on Community" clause described the
enforcement effect alongside the gate, which the directive bans in
operator-facing docs.
* docs(host-console): refresh page for v1 UI and clarify shell metadata
Rewrite the Host Console page to match the current Cockpit layout
(masthead + terminal well + chip strip), replace the legacy PowerShell
screenshot with a fresh bash capture, and document the masthead tone
states, kicker, metadata pills, and session/heartbeat behavior. Trim
the security section to state the tier and role rule once.
* docs(licensing): refresh page for v1 UI, corrected pricing, and trial flow
Rewrites the Licensing & Billing page to match the redesigned v1
Settings layout. The previous draft still described the legacy
Settings Hub: in-app "Upgrade your plan" Skipper/Admiral cards,
"Start monthly trial" / "Start annual trial" buttons, the
"Have a license key?" field, "Manage Subscription" with a capital S,
"Deactivate License" as the button label, and the license-active.png
asset rendering the literal "Sencho Pro" string in the card title.
None of that exists in the current product.
- Refreshes the Plans table to the live pricing on sencho.io/pricing
and adds an Enterprise mention with the floor price ($3,500/year).
Skipper now $11.99 annual / $14.99 monthly / $449 lifetime, Admiral
now $69.99 annual / $89.99 monthly / $2,499 lifetime.
- Rebuilds the Feature breakdown from a code-level audit of every
requirePaid, requireAdmiral, requireScheduledTaskTier, and
requireTierForSsoProvider call site in backend/src/routes, not
from the marketing page. Notable code-grounded items: CVE
suppressions on Community (no requirePaid guard), manual fleet
snapshots on Community (scheduled snapshots on Skipper),
Sencho Mesh under Admiral (entire mesh.ts router is requireAdmiral),
and scheduled-task tiering names update/scan/snapshot as the
Skipper subset with everything else under Admiral.
- Rewrites the Free trial flow end to end. The previous steps told
operators to click in-app "Start monthly trial" or "Start annual
trial" buttons; no such buttons exist. The new flow starts on
sencho.io/pricing, switches to the Annual or Monthly tab, clicks
"Start 14-day trial" on the Admiral card, completes the Lemon
Squeezy checkout (card-required, no charge before day 14), and
pastes the issued key into Settings -> License -> License key.
- Adds a new "The Plan section" anatomy block describing the masthead
SCOPE / PLAN / DURATION (or RENEWS, TRIAL, STATUS) stat pills and
the Plan card fields (Customer, Product, masked License key, status
helper).
- Adds a new "License states" reference table covering
Community / Trial / Active subscription / Active lifetime /
Expired / Disabled, what each surface renders, and which of the
Plan / Activate / Pricing sections is visible in each state.
- Corrects every UI label that drifted: section heading is Activate,
field label is License key (not "Have a license key?"), buttons are
Manage subscription (lowercase s) and Deactivate (not "Deactivate
License"), and the action-row hint reads "Lemon Squeezy manages
billing".
- Documents the redesigned profile dropdown: identity header with
initials chip, role badge, and tier badge, then Settings,
conditional Billing, Documentation, Feedback, an Appearance
segmented control, and Log Out. Billing only appears when the
license is an active non-lifetime subscription.
- Replaces all four screenshots under docs/images/licensing/:
license-admiral-active.png (production Admiral lifetime view),
profile-menu.png (redesigned popover), and two new captures for
the Community-tier surfaces (license-activate-section.png,
license-community.png). Removes the stale license-active.png
(legacy "Sencho Pro" card) and profile-billing.png (legacy
dropdown).
* docs(settings-reference): refresh page for v1 UI with new sections and masthead
Rewrites docs/reference/settings.mdx against the current Settings Hub so a reader
encounters an accurate map of every section. Adds the previously missing **Cloud
Backup** and **Security** sections, restructures **System Limits** into Host
thresholds and Docker hygiene subsections (GiB units, "Global crash capture"
toggle), fixes the Account password minimum to 8 chars and documents the
two-factor subsection, refreshes License/Routing/Webhooks/App Store with the
field labels actually rendered today, and documents the masthead pills
(SCOPE/NODE, EDITED, plus the per-section stats like 2FA, PLAN, CHANNELS, ROUTES,
WEBHOOKS, LABELS, TRIVY, POLICIES, PROVIDER, USED, SNAPSHOTS, DEV MODE).
Replaces five existing screenshots that predated the v1 redesign and adds five
new captures: Account with the 2FA card, License panel, System Limits with both
subsections, Security with the Trivy installer, and Cloud Backup with Sencho
Cloud Backup provisioned. All shots taken against the production node.
* docs(licensing): drop billing-provider name from operator-facing copy
The previous draft named the third-party billing provider in nine
places (checkout, receipt email, error toast verbatim, Customer /
Product field descriptions, the action-row hint, the billing portal,
and the validation API). Operator docs don't need to advertise which
vendor sits behind the checkout, billing portal, and validation
calls. Rewrite each instance to describe what the operator sees and
does without naming the upstream service.
* docs(node-compatibility): refresh page for v1 UI with lock card visuals and current capability list
- Replaces the legacy "dim + blur + pill overlay" description with the
current CapabilityGate behavior: a centered lock card with an Unplug
icon, title "<feature> is not available on this node", and a body line
that names the node and its running version.
- Corrects the tier-interaction section: on the wrong tier the entry
point is hidden entirely, so the lock card only appears for users who
already cleared the license gate.
- Documents the public /api/meta endpoint, the 5-minute success cache,
the 30-second failure cache, and the lazy-fetch behavior visible in
the switcher (the version pill appears once a node has been visited).
- Refreshes the capability table against the current CapabilityRegistry
list, adding container-exec and vulnerability-scanning, with a note
that vulnerability-scanning is only advertised when Trivy is installed.
- Adds three production screenshots captured on the live fleet:
switcher popover with mixed-version pills (one node on v0.76.9, rest
on v0.81.11), a real lock card on an older pilot agent, and the
Connection Details panel from Settings · Nodes.
* docs(security): refresh security architecture page for v1 UI
Add Fleet Secrets and Webhook signatures cards plus tier-matrix rows for
shipped-but-undocumented features. Rename SSO presets from "one-click" to
"preset providers" (presets still require OAuth-app provisioning on the
upstream IdP). Update settings paths to the v1 middle-dot convention:
Settings · Users, Settings · Account, Settings · Developer · Data retention.
Extend the encryption-at-rest list with registry credentials and Fleet
Secrets bundle payloads (both sealed with the same AES-256-GCM data key)
and clarify the password section with bcrypt cost factor 10.
Add a Webhook signature authentication subsection covering the per-webhook
HMAC-SHA256 secret, one-shot display, masked preview thereafter, and
constant-time comparison on inbound triggers.
Replace the API Tokens screenshot with a fresh capture against the v1
Settings · Identity · API Tokens panel.
* docs(security-advisories): retire reference page
The reference/security-advisories page does not survive the v1 docs
refresh:
- Misuses the term "Security Advisories", which industry-wide refers to
published notices for confirmed product CVEs (ID, severity, affected
versions, fix version, remediation). The retired page was a narrative
changelog of internal hardening work between v0.19 and v0.25.2.
- The narrative is also frozen at v0.25.2 (April 2026) while current
release is v0.81.11. Refreshing it would require backfilling ~56
release entries' worth of hardening copy.
- The framing is uniformly "improved from prior behavior" (minimum 8
characters up from 6, 1-year token expiry previously without expiry,
CORS previously allowed all origins, users should upgrade promptly).
Sencho has not shipped publicly; there are no users to address as
upgraders.
All operationally relevant content already lives elsewhere: the
security architecture page covers the current posture, verifying-images
covers the supply-chain attestations, cve-suppressions covers operator
acknowledgment, vulnerability-scanning covers the in-app scanner, and
contact + the security architecture page both surface the disclosure
path. Published Sencho-product advisories, when any exist, will appear
on the GitHub Security tab, which is already linked from those pages.
Inbound-link audit returned a single hit on the nav entry itself; no
other doc, README, or operator artifact deep-links the slug.
* docs: rewrite Pilot Agent page with deep architecture and operations reference
Reframes docs/features/pilot-agent.mdx as the architecture-and-operations
companion to the operator walkthrough in Multi-Node Management. Adds a
mental model section, an explicit security and trust model, a full agent
env-var reference, an honest limitations list, and a 5-item FAQ. Verifies
every constant and label against the current backend source. Refreshes
four production screenshots (admin login, scrubbed) and resolves the
previously-broken /images/pilot-agent/enrollment-dialog.png reference.
Adjacent edits keep the cross-linking coherent:
- multi-node.mdx adds a one-line forward link to the rewritten page
- security.mdx adds a Pilot Agent tunnel credentials subsection
* docs(fleet-federation): deep rewrite with production screenshots
Rewrites the Fleet Federation page against the v1 docs refresh template
following the recent fleet-view, pilot-agent, and multi-node refreshes.
Doubles the page length (92 to 204 lines) while keeping the cut-line v1
MVP scope: operator-driven placement controls (cordon + pin) for
Blueprints, no expansion into mesh/sync/pilot territory.
What changed:
- Adds four production-captured screenshots under docs/images/fleet-federation/:
the Federation tab with a cordoned node populated, the node-card kebab
menu showing the Cordon node entry, the cordon confirmation dialog
with a reason filled in, and a node card displaying the Cordoned pill.
- Expands the page to eleven sections: opening summary, philosophy
(kept), key capabilities, prerequisites, step-by-step usage with
embedded screenshots, behaviour and lifecycle table, security and
audit, limitations and non-goals (expanded), practical workflows (new:
OS patching, host-to-host migration, gateway pinning), troubleshooting
(eight accordions, up from five), and a Where Federation fits
cross-link table.
- Documents the exact production UI strings observed: the cordon
dialog description, the uncordon confirmation copy, the reason field
cap (256 chars), and the audit log action names (node.cordon,
node.uncordon, blueprint.pin).
- Documents the audit visibility surface so operators know how to
filter the Audit view for cordon and pin history.
- Adds eight cross-links to sibling pages (Fleet View, Multi-Node,
Pilot Agent, Mesh, Fleet Actions, Fleet Sync, Blueprints, Licensing)
with one-line scope contrasts so newcomers can place Federation in
the broader fleet picture.
- Tightens lifecycle table to operator-relevant terms (no DB column
names) and audit section to operator-facing wording (no middleware
names), keeping the page operator-focused rather than
implementation-focused.
Validation:
- Captured screenshots against the production node logged in as admin,
using Playwright MCP. Cordoned and pinned actions reverted; audit log
confirmed the matched cordon/uncordon pair.
- Verified every cross-link target exists in the v1-refresh worktree
(/features/fleet-view, /features/multi-node, /features/pilot-agent,
/features/sencho-mesh, /features/fleet-actions, /features/fleet-sync,
/features/blueprint-model, /features/licensing).
- Compliance: no em dashes, no PII, no "previously"/"used to" framing,
no fence-spec language, tier rule stated once in plain language.
* docs(fleet-federation): drop fence-spec phrasing in the open-core context
Sencho is open-core: anyone can clone the repo and read the tier gate.
Operator docs that name exactly where the UI gate sits ("hidden at the
Community and Skipper tiers", "lower-tier users do not see the toggle",
"only the Federation tab is gated") work as a dig-target for a
tech-savvy reader and undercut the open-core posture. Directive 27
already bans enforcement-chain spelling; the open-core threat model
makes the same phrasings risky even when they describe UI surfaces
rather than route guards.
Removes three instances of the pattern on this page:
- Top Note callout: drops "The tab is hidden at the Community and
Skipper tiers." Keeps the one-line requirement: "Federation is an
Admiral feature. Cordon and pin actions require an admin user role."
- Security and audit section: drops the sentence enumerating which UI
affordances are hidden from which tiers. Keeps the customer-visible
behavior (the Cordoned pill stays visible at every tier as a
read-only signal).
- Troubleshooting "Federation tab is not visible" accordion: rewrites
to lead with the requirement and the role check, drops the
"Federation is hidden by design" and "only the toggle and the
Federation tab are gated" phrasings.
Other claims on the page unchanged; rule is still stated once in plain
language at the top of the page.
* docs(fleet-sync): deep rewrite with production screenshots
Replace fleet-sync.mdx with a verified end-to-end reference. The previous
page named two replicated resources but the code syncs three, described a
sync-status panel and a fleet-vs-node scope picker that do not exist in
the shipped UI, and was missing prerequisites and several edge cases.
Highlights of the rewrite:
- Names all three replicated resources (scan policies, CVE suppressions,
misconfig acknowledgements) and treats them uniformly.
- Drops the sync-status-panel and node-scope-picker UI claims; both move
to the Limitations section as honest caveats.
- Adds prerequisites covering the paid-tier requirement on the control,
admin-role requirement, proxy-mode remotes, and reachability.
- Expands lifecycle coverage: per-node serialised pushes, add-node
backfill, monotonic pushedAt, per-resource watermarks, identity-drift
notifications, the 5000-row truncation cap, stale-target warnings,
audit-log entries on the replica.
- New "Where Fleet Sync fits" closing table cross-linking to Fleet View,
Multi-Node Management, Pilot Agent, Vulnerability Scanning, CVE
Suppressions, Fleet Federation, Fleet Actions, and Licensing.
- Two fresh production screenshots: control Security panel and the
"Scanner is per-node" callout shown when proxying to a remote.
* docs(fleet-actions): deep rewrite with production screenshots
Three cards are documented end to end: Stop fleet by label, Bulk label
assign, and Prune Docker resources fleet-wide. Adds the execution-path
distinction (control-orchestrated fan-out vs single-node proxy), per-card
behaviour and partial-failure semantics, prerequisites, limitations,
practical workflows, an Accordion troubleshooting section, and a Where
Fleet Actions fits comparison table linking the surrounding Fleet view
features.
Corrects the prior page's tab-neighborhood claim, confirm-dialog wording,
autocomplete-vs-request scope, and missing batch ceiling. Replaces the
ten-day-old single screenshot with five fresh production captures under
docs/images/fleet-actions/.
* docs(fleet-secrets): deep rewrite with production screenshots
Full rewrite of /features/fleet-secrets matching the fleet-actions
structure. Replaces the sparse v1 page (no Frames, inline Q&A) with a
gold-standard layout: opening Frame, single Note for the tier gate,
'What it covers' table, mental model, prerequisites, create + edit +
versions + push (Target / Preview / Results) sections each with a
production Frame, Import from stack section, behaviour and lifecycle
table, audit-trail mapping with the six exact audit strings,
limitations and non-goals, practical workflows, AccordionGroup
troubleshooting, and a Where-it-fits cross-link table.
Adds six fresh production screenshots under
docs/images/fleet-secrets/ : overview, create, versions, target,
preview, and results.
Documents the Import-from-stack flow (depends on the bundle editor's
new Import action) and uses the post-rename 'Send' wording on the
bundle-row action (depends on the aria-label fix).
Corrects three factual drifts vs the code: env-key regex described as
'letter or underscore, then letters/digits/underscores; case-
sensitive' to match ^[A-Za-z_][A-Za-z0-9_]*$ ; documents only the
'ok' and 'failed' status pills (the 'skipped' enum value is unused);
replaces the bogus 'stack not found' troubleshooting entry with the
real 'env file not declared' cause.
Drops the fence-spec phrasing 'The tab is hidden on Community.' per
Directive 31; the tier requirement is now stated once in plain
language.
* docs(sencho-mesh): deep rewrite with mental model, lifecycle, security, screenshots
Replace the feature-reference page with a deep product + technical guide.
Adds:
- Opening hook framing audience and problem (cross-node service-to-service
without a separate VPN or service-mesh sidecar).
- Mental model: three moving parts (sencho_mesh bridge, alias registry,
cross-node transport) with direction-of-flow described in prose.
- Key capabilities, prerequisites, step-by-step usage with inline screenshots.
- Full lifecycle section covering opt-in, opt-out, sticky stack-stopped state,
peer reconnect, and the proxy-mode bridge with its real default (persistent,
env-override for idle).
- Security and trust boundaries split into authentication, inbound exposure,
encryption, audit, and app-layer caveats.
- Limitations and non-goals: one-alias-per-port, port 1852 reserved,
central-relay for remote-to-remote, shared 1024-stream pool with the Pilot
tunnel, no L7, host-network unsupported, in-memory activity log.
- Three concrete workflow examples and a complete troubleshooting accordion
(every data-plane reason, every probe stage, every unreachable cause) plus
a Common questions FAQ.
- Where Mesh fits CardGroup linking Pilot Agent, Multi-Node, Federation,
Licensing.
Corrections vs prior text:
- Tab is labelled Traffic in the UI (not Routing); all navigation references
updated.
- Proxy-mode bridge default is no idle close (env-overridable to opt into idle
teardown); prior 5-minute-teardown claim removed.
- Audit trail scope tightened: only opt-in / opt-out write durable rows;
tunnel-state and probe events live in the in-memory activity log.
Adds seven production screenshots under docs/images/sencho-mesh covering
Table view, opt-in sheet, graph (Tunnels and Aliases edge modes), Diagnostics,
activity log, and per-stack topology.
* docs(blueprints): add missing detail-state-review screenshot
Captures the Blueprint detail sheet with a deployment row in the
"Awaiting confirmation" status (stateful first-deploy gate), to fix the
broken image referenced at blueprint-model.mdx:132. mint broken-links
now reports zero broken references.
* docs(blueprints): deep rewrite with mental model, lifecycle, security, prerequisites
Restructures the Blueprints page against the v1-refresh template used by the
recently-refreshed mesh, secrets, and atomic-deployments pages. Adds a mental
model, prerequisites table, lifecycle and status-transition map, security and
trust boundaries section, practical workflows, common questions accordion,
and a Where Blueprints fits CardGroup. Removes the internal-style rollout
and watch-plan section. Replaces all nine production screenshots with fresh
captures against the production node signed in as admin, and adds two new
captures (federation pin policy table, stateless eviction dialog). Rewrites
the tier-gate Note to drop the fence-spec phrasing that violated Directive
31. Every retained claim is anchored to current backend or frontend code.
* docs(pilot-agent): recapture enrollment dialog with compose payload
Replaces the pre-0.84 docker-run capture with the current dialog (Compose
file, two-step instructions, "Copy compose file" button) and refines the
alt text to describe the captured content. URL and token redacted to
placeholder values during capture.
425 lines
34 KiB
Plaintext
425 lines
34 KiB
Plaintext
---
|
||
title: Alerts & Notifications
|
||
description: Threshold and event alerts for your fleet, dispatched to Discord, Slack, or any webhook, with per-stack rules and Admiral routing.
|
||
---
|
||
|
||
Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and one of three external channels you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with Admiral routing rules, and tuning retention.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/notifications-settings.png" alt="Settings · Notifications panel showing the Discord, Slack, and Webhook tabs with the masthead breadcrumb, the CHANNELS 3/3 stat, the active Discord tab with its Enabled toggle on, the Webhook URL input, and the Test and Save actions." />
|
||
</Frame>
|
||
|
||
## Notification channels
|
||
|
||
Open **Settings · Notifications** to configure the three channel types. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the three slots are enabled.
|
||
|
||
Each tab carries the same controls: an **Enabled** toggle (helper: `Send Sencho events to this <name> channel.`), a **Webhook URL** input (placeholder `https://...`, helper: `Sencho posts JSON payloads here. Use a private channel.`), and the **Test** and **Save** buttons. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up.
|
||
|
||
<Note>
|
||
Webhook URLs must use HTTPS. Sencho rejects `http://` and any string that does not parse as a URL. The same rule applies to test sends and to routing-rule URLs.
|
||
</Note>
|
||
|
||
### Discord
|
||
|
||
Paste an incoming webhook URL from your Discord channel's **Edit Channel · Integrations · Webhooks** view. Sencho posts a single embed per alert, with the title `Sencho Alert [<LEVEL>]`, the message in the description, and the embed color set by severity (info blue, warning yellow, error red).
|
||
|
||
### Slack
|
||
|
||
Paste an incoming webhook URL from a Slack app installed in your workspace (URL shape `https://hooks.slack.com/services/...`). Sencho posts a single text message: `<emoji> *Sencho Alert [<LEVEL>]*` followed by the message body, with the emoji set per severity (`ℹ️` info, `⚠️` warning, `🚨` error).
|
||
|
||
### Generic webhook
|
||
|
||
Use the Generic Webhook tab when you have your own receiver: a Mattermost or Teams adapter, a serverless function that fans out to email or SMS, or a logging endpoint. Sencho posts JSON of the shape:
|
||
|
||
```json
|
||
{
|
||
"level": "warning",
|
||
"message": "[Node: Local] The cpu_percent for plex has exceeded your threshold of 80% (Currently: 91%).",
|
||
"timestamp": "2026-05-08T22:14:09.812Z",
|
||
"source": "sencho"
|
||
}
|
||
```
|
||
|
||
`level` is one of `info`, `warning`, `error`. `source` is always the literal string `sencho`. `timestamp` is ISO-8601 with millisecond precision.
|
||
|
||
### Test sends and delivery semantics
|
||
|
||
The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take. Test sends require the admin role; the server returns 403 if an operator or viewer submits one.
|
||
|
||
Each dispatch is a single-shot HTTP POST with a 10-second `AbortSignal.timeout`. There is no retry queue. If your endpoint is down at the moment of the dispatch, the alert is recorded in the bell with an internal `dispatch_error` field set and is not redelivered.
|
||
|
||
## Notification Routing
|
||
|
||
<Note>
|
||
Notification Routing requires a **Sencho Admiral** license. Admin role is required to create, edit, or delete routes.
|
||
</Note>
|
||
|
||
Routing lets you direct alerts that match specific criteria to dedicated channels. Production crashes can land in `#prod-incidents` on Slack while staging notifications go to a less urgent Discord channel, all without juggling per-channel webhook URLs across teams.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/routing-list.png" alt="Settings · Routing card list showing two rules ('Critical to Slack' and 'Production alerts'), each with a Discord badge, an ON pill, the 'Matches all alerts' summary line, a truncated channel URL, a Priority chip, and lightning-test, edit, and delete icon actions on the right." />
|
||
</Frame>
|
||
|
||
### How routing fits into dispatch
|
||
|
||
For every alert Sencho dispatches, the routing engine evaluates every enabled route. A route matches when **all of its non-empty matchers** match the alert: the **Node**, **Stacks**, **Labels**, and **Categories** filters compose with AND. An empty matcher is treated as match-anything.
|
||
|
||
If at least one route matches, every matching route fires and the **global channels are skipped** for that alert. If zero routes match, the alert falls back to the global channels configured in the previous section.
|
||
|
||
A route with all four matchers left empty matches every alert and intercepts global delivery entirely. Stack-less alerts (host CPU, RAM, disk, fleet-sync warnings, and scheduled-task results without a stack target) almost always miss any populated `Stacks` filter, so they fall back to global by default.
|
||
|
||
### Creating a routing rule
|
||
|
||
Open **Settings · Routing** and click **+ Add Route**.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/routing-modal.png" alt="The 'New routing rule' modal with the 'ROUTING · NEW RULE' kicker, a Name input filled with 'Production alerts', a Node scope select reading 'Any node', empty Stacks, Labels, and Categories combobox pickers, the helper line 'Leave blank to match all categories. All non-empty filters must match (AND).', a Channel tab strip with Discord selected and a webhook URL filled in, and Priority and Enabled fields with Cancel and CREATE actions in the footer." />
|
||
</Frame>
|
||
|
||
| Field | Purpose |
|
||
|-------|---------|
|
||
| **Name** | A human label, up to 100 characters. Shown on the rule card. |
|
||
| **Node scope** | Either `Any node` or a specific node. When set to a node, the rule only matches alerts originating from that node. |
|
||
| **Stacks** *(optional)* | A combobox of stacks on the active node. Selected stacks appear as removable pills. Empty matches any stack. |
|
||
| **Labels** *(optional)* | A combobox of stack labels on the active node. Empty matches any label. |
|
||
| **Categories** *(optional)* | A combobox of notification categories. The helper line reads `Leave blank to match all categories. All non-empty filters must match (AND).` |
|
||
| **Channel** | Tabs for Discord, Slack, and Webhook with a URL input below. URL must use HTTPS. |
|
||
| **Priority** | A number used to sort the rule list. Lower numbers appear higher up. Priority does not gate dispatch: when multiple rules match the same alert, every matching rule fires concurrently. |
|
||
| **Enabled** | Toggle the rule on or off without deleting it. |
|
||
|
||
The modal kicker reads `ROUTING · NEW RULE` when adding and `ROUTING · EDIT RULE` when modifying.
|
||
|
||
### Managing rules
|
||
|
||
Each rule renders as a card on the Routing page with the rule name, channel-type badge, an `ON` / `OFF` pill, then a row of small badges naming each matcher: one mono badge per **Stack**, one outline badge per **Label**, one outline mono badge per **Category** label. When all matchers are empty, a single muted `Matches all alerts` line replaces the badge row. After the badges, a vertical bar separator is followed by the truncated channel URL, then a second separator and a `Priority: N` chip when priority is non-zero.
|
||
|
||
Three icon actions appear on the right edge of each card:
|
||
|
||
- **Lightning** sends a test message through the rule's channel using the same payload shape as the global Test button.
|
||
- **Pencil** opens the edit modal pre-filled with the current values.
|
||
- **Trash** deletes the rule after a confirmation dialog.
|
||
|
||
The masthead carries `SCOPE` (`global`), `ROUTES` (total), and `ENABLED` (count).
|
||
|
||
## Notification categories
|
||
|
||
Every alert Sencho dispatches carries a category that you can filter on in the bell, target with a routing rule's **Categories** matcher, or reason about when reading audit history.
|
||
|
||
| Category | Label in the bell | Trigger |
|
||
|----------|-------------------|---------|
|
||
| `deploy_success` | Deploy success | Stack deploy completed without error |
|
||
| `deploy_failure` | Deploy failure | Stack deploy returned a non-zero exit |
|
||
| `stack_started` | Stack started | Stack started via the dashboard or API |
|
||
| `stack_stopped` | Stack stopped | Stack stopped via the dashboard or API |
|
||
| `stack_restarted` | Stack restarted | Stack restarted via the dashboard or API |
|
||
| `image_update_available` | Update available | Image-update poll found a newer digest |
|
||
| `image_update_applied` | Update applied | Manual or scheduled auto-update applied new images |
|
||
| `autoheal_triggered` | Auto-heal | Auto-heal restarted, failed to restart, or auto-disabled a policy |
|
||
| `monitor_alert` | Monitor alert | Per-stack threshold breach, host CPU/RAM/disk warning, or healthcheck failure |
|
||
| `scan_finding` | Scan finding | Vulnerability-scan completion, per-violation alert, post-deploy scan failure, or auto-update gate block |
|
||
| `system` | System | Sencho version, Trivy auto-update, fleet sync, daemon connectivity, scheduled-task lifecycle, cloud-backup upload failure |
|
||
| `blueprint_deployed` | `blueprint_deployed` | Blueprint provisioned a new deployment (Admiral) |
|
||
| `blueprint_deployment_failed` | `blueprint_deployment_failed` | Blueprint deployment errored out (Admiral) |
|
||
| `blueprint_drift_detected` | `blueprint_drift_detected` | Blueprint drift detected in `suggest` or `enforce` mode (Admiral) |
|
||
| `blueprint_drift_correction_failed` | `blueprint_drift_correction_failed` | Blueprint enforce-mode redeploy failed (Admiral) |
|
||
|
||
The four `blueprint_*` categories are accepted by routing rules but render as raw category strings in the bell because the frontend label map omits them.
|
||
|
||
## Per-stack alert rules
|
||
|
||
Each stack carries its own set of threshold rules that fire when a metric stays above (or below) a value for a configurable window. Rules live on the node where the stack runs and are evaluated locally on a 30-second tick.
|
||
|
||
Open the rules editor by right-clicking a stack in the sidebar and choosing **Alerts**, or by pressing `A` while focused on a stack.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/alert-panel.png" alt="The Stack › PLEX › MONITOR sheet with the Alerts tab active, a green 'Notifications active via Discord, Slack, Webhook' banner under the NOTIFICATION CHANNELS heading, an ACTIVE RULES section listing 'CPU Usage (%) > 80' with the secondary line 'Trigger after 5m • Cooldown 60m', and an ADD NEW RULE form with Metric, Operator, Threshold, Duration, Cooldown fields and an Add Rule submit button." />
|
||
</Frame>
|
||
|
||
### Alert fields
|
||
|
||
| Field | Purpose |
|
||
|-------|---------|
|
||
| **Metric** | The system resource or metric to monitor. |
|
||
| **Operator** | Comparison: `Greater than`, `Greater or eq`, `Less than`, `Less or eq`, `Equals`. |
|
||
| **Threshold** | A number ≥ 0. The unit follows the chosen metric. |
|
||
| **Duration (mins)** | How long the breach must persist before firing. Default `5`, range 0 to 1440. |
|
||
| **Cooldown (mins)** | Silence window between fires after a rule triggers. Default `60`, range 0 to 10080. |
|
||
|
||
Sencho tracks the start of each breach in memory; the rule fires only after the breach has lasted for the full **Duration**, and only if the previous fire is older than **Cooldown**.
|
||
|
||
### Available metrics
|
||
|
||
| Metric | UI label | Unit |
|
||
|--------|----------|------|
|
||
| `cpu_percent` | CPU Usage (%) | percent of all cores |
|
||
| `memory_percent` | Memory Usage (%) | percent of container memory limit |
|
||
| `memory_mb` | Memory Usage (MB) | MB of resident set size, minus cache |
|
||
| `net_rx` | Network In (MB/s) | MB per second, computed as a delta between consecutive samples |
|
||
| `net_tx` | Network Out (MB/s) | MB per second, computed as a delta between consecutive samples |
|
||
| `restart_count` | Restart Count | integer count of Docker-reported restarts |
|
||
|
||
### Channel banner states
|
||
|
||
The **NOTIFICATION CHANNELS** banner above the rules list reflects what dispatch will look like for this stack:
|
||
|
||
- **Loading** is a spinner with `Checking notification channels...` while Sencho asks the target node for its agent state.
|
||
- **Remote node** is a blue banner reading `Remote node: <name>`, with the body `Alert rules are stored and evaluated on this remote instance. Notifications are dispatched using that node's configured channels.` A sub-line reports whether the remote has any channels configured.
|
||
- **No channels** is an amber banner reading `No notification channels configured`, body `Alert rules will be saved and evaluated, but no notifications will be dispatched. Configure Discord, Slack, or a webhook in Settings → Notifications.`
|
||
- **Active** is a green banner reading `Notifications active via Discord, Slack, …` with the configured channels listed.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/alert-panel-remote-banner.png" alt="The Stack › SAELIX-DB › MONITOR sheet on a remote node, with the blue 'Remote node: node-a' banner explaining that rules are stored and evaluated on the remote, plus a follow-up amber line noting that no notification channels are configured on that remote." />
|
||
</Frame>
|
||
|
||
### Example: alert on high CPU
|
||
|
||
To page when CPU on a stack stays above 80% for at least five minutes, with no more than one alert per hour:
|
||
|
||
| Field | Value |
|
||
|-------|-------|
|
||
| **Metric** | CPU Usage (%) |
|
||
| **Operator** | Greater than |
|
||
| **Threshold** | `80` |
|
||
| **Duration (mins)** | `5` |
|
||
| **Cooldown (mins)** | `60` |
|
||
|
||
### Permissions and validation
|
||
|
||
Operators and viewers see existing rules read-only; only admins see the add and delete affordances. Submitting an empty threshold surfaces `Please enter a threshold.` Successful saves toast `Alert rule added.` and `Alert rule deleted.` A failed POST surfaces `Network error. Could not reach the node.`
|
||
|
||
The delete confirmation dialog reads `Delete Alert Rule` / `This will permanently remove this alert rule. Notifications for this condition will no longer be sent.` with a destructive **Delete** button.
|
||
|
||
## Notification history (the bell)
|
||
|
||
The bell icon at the top of the shell is the live feed of every alert across the fleet. A pulsing red dot surfaces unread items.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/notification-popover.png" alt="The notification bell open, with a 'Notifications' italic title, '9 UNREAD' uppercase mono caption, an All / Unread / Alerts segmented control, the filter-toggle, mark-all-read, and clear-all icon actions, a 'TODAY' day-band header, and a stack of severity-tinted rows describing Sencho version updates and a recent stack stop, each with a node-name pill, a relative timestamp, and a per-row dismiss target." />
|
||
</Frame>
|
||
|
||
### Anatomy
|
||
|
||
A title bar with `Notifications` in the italic display face, the unread count to the right (`<n> UNREAD`), and a row of controls below.
|
||
|
||
The segmented control switches between `All`, `Unread` (with badge), and `Alerts` (rows where level is `warning` or `error`). To the right, three icon buttons:
|
||
|
||
- **SlidersHorizontal** toggles a hidden filter row. A small dot on the icon means at least one filter is active.
|
||
- **CheckCheck** is `Mark all read`.
|
||
- **Trash2** is `Clear all`. The action issues `DELETE /api/notifications` against every node that contributed a row.
|
||
|
||
The hidden filter row has two combobox dropdowns: **Filter by node** (only rendered with two or more nodes registered) and **Filter by category**, listing all the user-facing labels from the categories table above.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/bell-filter-row.png" alt="The bell with its filter row expanded, the 'ALL TYPES' combobox open over a long popover listing every notification category from 'All types' through 'System', and the underlying notification rows visible beneath." />
|
||
</Frame>
|
||
|
||
### Row anatomy
|
||
|
||
Each row carries:
|
||
|
||
- A **3 px left rail** colored by severity: brand cyan for `info`, amber for `warning`, destructive rose for `error`. Rail saturation drops to about 30% once the row is read.
|
||
- A **severity icon** in the same hue: `Info`, `AlertTriangle`, or `AlertOctagon`.
|
||
- The **message body**, slightly dimmer once read.
|
||
- A **kicker line** below the message with an optional **node-name pill** (only rendered for multi-node fleets), a `·` separator, and a relative timestamp (`just now`, `Nm ago`, `Nh ago`, `yesterday`, then a short locale date).
|
||
- A **dismiss** target on hover, top-right.
|
||
|
||
There is no level chip in the row; the rail color and icon do that work. Rows whose payload carries a `stack_name` become click targets that jump to the stack and, when a `container_name` is present, surface that container's logs.
|
||
|
||
### Day groupings
|
||
|
||
Rows are grouped under uppercase day-band headers: `TODAY`, `YESTERDAY`, `THIS WEEK`, `EARLIER`.
|
||
|
||
### Empty states
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/bell-empty.png" alt="The bell open in its empty state with a strikethrough bell icon, the headline 'You're all caught up', and the mono caption 'New notifications appear here in real time.'" />
|
||
</Frame>
|
||
|
||
The popover renders three different empty states:
|
||
|
||
- All filters, no rows ever: `You're all caught up` over `New notifications appear here in real time.`
|
||
- Unread filter, all read: `No unread notifications` over `Everything in your feed has been read.`
|
||
- Alerts filter, no warnings or errors: `No active alerts` over `Warnings and errors will surface here when they occur.`
|
||
|
||
### What you won't see in the bell
|
||
|
||
Sencho deliberately suppresses one class of notification from the popover: rows where the category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied`, AND the row carries an `actor_username` other than `system`. These are confirmations of an action you just clicked, surfaced as a toast at the moment of the action; replaying them in the bell would be noise. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
|
||
|
||
### Limits
|
||
|
||
The popover holds the latest 50 rows per node per fetch. The backend caps `notification_history` at 100 rows per node, regardless of the user-tunable retention setting; the cap evicts the oldest rows on every insert. There is no `Load more` affordance; older rows roll off the cap.
|
||
|
||
## Cross-node delivery
|
||
|
||
The bell aggregates across the entire fleet by hitting `/api/notifications` against every registered node and opening one WebSocket per node: `/ws/notifications` for the local instance and `/ws/notifications?nodeId=<id>` for every remote. A new alert on any remote arrives in the bell within about a second; a 60-second safety-net poll catches anything the WebSocket dropped.
|
||
|
||
Per-stack alert rules and channel configuration are **stored on the node where the stack runs**. To configure a rule on a remote, switch the active node via the picker, open the stack's **Monitor** sheet, and the form's POST is forwarded to the remote.
|
||
|
||
Crash detection runs only on local Docker; remote nodes run their own copy of `DockerEventService` and emit through the proxy. Each emitted message is prefixed with `[Node: <nodeName>]` so the source is unambiguous in the channel.
|
||
|
||
Switching the active node only affects per-stack rule editing and the Settings panels. The bell aggregates every node regardless of which one is active.
|
||
|
||
## Alerts emitted by the system
|
||
|
||
Sencho dispatches notifications from many code paths. The list below covers every emitter in current code, organized by source. Each message is prefixed with `[Node: <nodeName>]` when emitted by a node-scoped service.
|
||
|
||
### Container crash and health
|
||
|
||
Real-time on the Docker event stream:
|
||
|
||
- **Crash** at `error`/`monitor_alert`: `Container Crash Detected: <name> exited unexpectedly (Code: <N>).`
|
||
- **OOM kill** at `error`/`monitor_alert`: `Container OOM Kill: <name> was killed by the OOM killer (out of memory).` When a `die` arrives with exit code 137 but no preceding `oom` event, Sencho inspects the container and reclassifies as OOM if the `OOMKilled` flag is set.
|
||
- **Healthcheck failure** at `error`/`monitor_alert`: `Healthcheck failed: <name> is unhealthy.`
|
||
- **Mass exit** kicks in when a daemon-disconnect gap is followed by 20% or more of containers exiting on reconnect, in which case a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` is emitted instead of N crash alerts.
|
||
- **Rate limit** caps crash dispatches at 20 per 60-second window per node, then a single `warning`/`monitor_alert` `N additional containers crashed in the last minute.`
|
||
- **Dedup** is 60 minutes per container after a non-rate-suppressed dispatch.
|
||
|
||
### Daemon connectivity
|
||
|
||
- `warning`/`system`: `Lost connection to Docker daemon; monitoring paused.` (one-shot until reconnect)
|
||
- `info`/`system`: `Reconnected to Docker daemon.`
|
||
- `warning`/`system`: `Received malformed Docker event payloads. Monitoring continues but some events may be skipped.` (when more than 10 parse errors in a minute)
|
||
|
||
### Host thresholds
|
||
|
||
`warning`/`monitor_alert` for host CPU, RAM, and disk when the configured threshold is exceeded. 5-minute cooldown per signal. Example: `Host CPU utilization is critically high: 92% (Threshold: 90%)`. Evaluated on the 30-second monitor tick.
|
||
|
||
### Docker janitor
|
||
|
||
`info`/`system`: `Node "<name>" has accumulated <N> GB of unused Docker data. Consider using the Janitor tool.` 24-hour cooldown.
|
||
|
||
### Sencho version availability
|
||
|
||
`info`/`system`: `Sencho X.Y.Z is available (currently running A.B.C). Visit the Fleet dashboard to update.` Polled every 6 hours; one-shot dedup until the running version reaches or passes the notified version.
|
||
|
||
### Image update availability
|
||
|
||
`info`/`image_update_available`: `Stack "<name>" has image updates available.` Polled every 6 hours, with a 2-minute startup delay and a 2-minute cooldown on manual refresh. Notifies on **state transition only**; pre-existing `has_update` rows are backfilled silently on first run.
|
||
|
||
### Auto-update execution
|
||
|
||
`info`/`image_update_applied`: `Auto-update: stack "<name>" updated with new images`. If a block-on-deploy policy gates the auto-update: `warning`/`scan_finding` `Policy "<name>" blocked auto-update: N image(s) exceed <severity>`. See [Auto-update policies](/features/auto-update-policies).
|
||
|
||
### Auto-heal
|
||
|
||
- `info`/`autoheal_triggered`: `Auto-Heal: Restarted <container> on stack <stack> after being unhealthy for <N> minute(s).`
|
||
- `warning`/`autoheal_triggered`: `Auto-Heal: Failed to restart <container> on stack <stack>. Error: <err>`
|
||
- `warning`/`autoheal_triggered`: `Auto-Heal: Policy for <stack>[/<svc>] has been auto-disabled after <N> consecutive failures.`
|
||
|
||
See [Auto-heal policies](/features/auto-heal-policies).
|
||
|
||
### Vulnerability scanning
|
||
|
||
- **Per-violation, during a scheduled scan** is `warning`/`scan_finding`. One alert per violation: `Policy "<name>" violated by <imageRef>: <severity> exceeds <maxSeverity>`.
|
||
- **Scan completion** is `info`/`scan_finding` (or `warning` if any image failed): `Scheduled scan "<task name>" completed: <output>` where `<output>` is one of `Scanned N image(s); X skipped (cached). Found A critical, B high, C medium.`, `No critical, high, or medium findings.`, `No images to scan`, or `All N image(s) already scanned recently (cache hit)`.
|
||
- **Pre-deploy gate when Trivy is missing** is `warning`/`scan_finding`: `Pre-deploy scan for "<stack>" skipped: Trivy not installed on this node`.
|
||
- **Post-deploy scan finding** is `error`/`scan_finding` if criticals are present, `warning` otherwise: `Vulnerability scan for <imageRef>: <N> critical, <M> high`.
|
||
- **Post-deploy scan failure** is `warning`/`scan_finding`: `Post-deploy scan failed for <imageRef> (<stack>): <msg>`.
|
||
- **Trivy auto-update** is `info`/`system`: `Trivy updated from vX to vY` or `Trivy update available: vX (currently vY)`.
|
||
|
||
See [Vulnerability scanning](/features/vulnerability-scanning).
|
||
|
||
### Scheduled tasks
|
||
|
||
- `error`/`system`: `Scheduled task "<name>" (<action>) failed: <err>`
|
||
- `info`/`system` recovery: `Scheduled task "<name>" (<action>) recovered successfully`
|
||
|
||
### Cloud Backup upload failure
|
||
|
||
`warning`/`system`: `Cloud backup failed for scheduled snapshot <id>: <message>`. See [Fleet backups](/features/fleet-backups).
|
||
|
||
### Blueprints (Admiral)
|
||
|
||
- `warning`/`blueprint_drift_detected`: `Blueprint "<n>" drifted on node "<n2>": <reason>` (suggest mode), with stateful-safeguard variants when enforce mode declines to redeploy.
|
||
- `error`/`blueprint_drift_correction_failed`: `Auto-fix for "<n>" on node "<n2>" failed: <err>`.
|
||
|
||
See [Blueprints](/features/blueprint-model).
|
||
|
||
### Fleet sync (Admiral)
|
||
|
||
- `warning`/`system` identity drift: `Fleet self-identity changed from "X" to "Y". Identity-scoped policies are reapplied on the next sync.`
|
||
- `warning`/`system` truncation: `Fleet sync truncated <resource> to <N> rows (<dropped> not replicated). Reduce the local set or contact support.`
|
||
- `warning`/`system` stale: `Fleet sync to node "<n>" has been failing for over an hour for <resource>. Check the node's connectivity and API token.`
|
||
|
||
## Retention and limits
|
||
|
||
Three retention controls live under **Settings · Developer · Data retention**.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/developer-retention.png" alt="Settings · Developer · Data retention card showing three rows: Container metrics with a 24 HRS field, Notification log with a 30 DAYS field, and Audit log (Admiral) with a 90 DAYS field, plus a SAVE SETTINGS button." />
|
||
</Frame>
|
||
|
||
| Control | Range | Default | What it prunes |
|
||
|---------|-------|---------|----------------|
|
||
| **Container metrics** | 1 to 8760 hours | 24 hours | Per-container CPU, RAM, and network history used by the dashboard sparklines |
|
||
| **Notification log** | 1 to 365 days | 30 days | The bell's `notification_history` table |
|
||
| **Audit log** *(Admiral)* | 1 to 365 days | 90 days | Audit trail entries |
|
||
|
||
A hard 100-row per-node cap inside `notification_history` always applies on top of the user-tunable retention; new inserts evict the oldest rows beyond 100 even if your retention window is longer.
|
||
|
||
A separate rate limit applies to crash and health alerts only: 20 emits per 60-second window per node, with a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` issued at the end of the window.
|
||
|
||
## Crash detection toggle
|
||
|
||
The global crash-capture switch lives under **Settings · System · Docker hygiene**.
|
||
|
||
<Frame>
|
||
<img src="/images/alerts-notifications/system-crash-toggle.png" alt="Settings · System · System Limits panel showing a HOST THRESHOLDS section with CPU limit, RAM limit, and Disk limit at 100%, then a DOCKER HYGIENE section with a Janitor threshold of 5 GiB and a Global crash capture toggle in the ON state with the helper line 'Watch every managed container for unexpected exits.'" />
|
||
</Frame>
|
||
|
||
The **Global crash capture** toggle controls whether `DockerEventService` raises crash, OOM, and healthcheck alerts on the active node. Helper text: `Watch every managed container for unexpected exits.` Defaults to on; if the database read fails, Sencho falls back to default-deny so the system never leaks alerts you cannot turn off.
|
||
|
||
The same panel carries the **Host thresholds** rows (CPU limit, RAM limit, Disk limit, all expressed as percent) that drive the host-level monitor warnings, and the **Janitor threshold** (in GiB) that drives the unused-Docker-data alert.
|
||
|
||
## Refresh cadence
|
||
|
||
| Surface | Cadence |
|
||
|---------|---------|
|
||
| Crash, OOM, and healthcheck events | Real-time over the Docker event stream |
|
||
| Host CPU / RAM / disk threshold checks | 30 seconds |
|
||
| Per-stack alert rule evaluation | 30 seconds |
|
||
| Image update poll | 6 hours, with a 2-minute startup delay and a 2-minute cooldown on manual refresh |
|
||
| Sencho version check | 6 hours |
|
||
| Notification fanout to channels | Single shot per dispatch, 10-second timeout, no retries |
|
||
| Bell live updates | Pushed live over the notifications WebSocket per node |
|
||
| Bell safety-net reconcile | 60 seconds |
|
||
| Crash dedup window per container | 60 minutes |
|
||
| Crash rate-limit window | 20 alerts per 60 seconds per node, then a single roll-up |
|
||
|
||
Switching the active node tears down per-stack rule editors and reloads channel state, but does not affect the bell, which keeps every node's WebSocket open in parallel.
|
||
|
||
## Troubleshooting
|
||
|
||
<AccordionGroup>
|
||
<Accordion title="Notifications never arrive in Discord, Slack, or my webhook">
|
||
Check three things in order. First, the channel toggle in **Settings · Notifications** must be on; the kicker on each tab reads `enabled` or `off`. Second, the URL must use HTTPS; the form rejects plain `http://` outright. Third, an Admiral routing rule with empty `Stacks`, `Labels`, and `Categories` matchers will intercept every alert and skip the global channels. Use the per-channel **Test** button to issue a one-shot dispatch and watch your endpoint for the literal message `🔌 Test Notification from Sencho!` Sencho records the failure reason in `notification_history.dispatch_error` when delivery throws, so a row that appears in the bell with no follow-up at the endpoint usually means a 4xx or timeout at the receiver.
|
||
</Accordion>
|
||
<Accordion title="An alert rule never fires even when the threshold is breached">
|
||
Three causes account for almost every case. First, the rule's **Duration** has not elapsed yet: the breach must persist for the full duration before the rule fires. Second, the rule is still in cooldown after a previous fire. Third, the panel's banner is not green: a remote-node banner means the rule was saved on a remote whose channels you may not have configured, and an amber `No notification channels configured` banner means the rule evaluates fine but Sencho has nowhere to send the alert. The evaluator runs on a 30-second tick, so expect up to 30 seconds of latency between the breach starting and the timer engaging.
|
||
</Accordion>
|
||
<Accordion title="I stopped a stack but got a crash alert anyway">
|
||
The Docker event service defers `die` classification by 500 ms to absorb out-of-order `kill` events from the daemon, then asks the lifecycle classifier whether the exit was intentional, clean (exit 0), a crash, or an OOM kill. If your stop happened far enough outside that window, or the daemon emitted the events without the kill marker the classifier looks for, the exit can be classified as a crash. The classifier favors avoiding silent crashes over avoiding noisy false positives. Compare the alert timestamp against your `docker compose down` time; entries within a second of each other are usually the same event seen from two angles.
|
||
</Accordion>
|
||
<Accordion title="After a Docker daemon restart, the bell only has one summary line">
|
||
When the daemon disconnects and reconnects, Sencho snapshots every container at the moment of reconnect. If at least 20% exited during the gap, the service emits a single `info`/`system` summary `Docker daemon interruption detected: N containers exited during connection gap.` instead of one crash alert per container. Below the threshold, every gap exit is classified individually.
|
||
</Accordion>
|
||
<Accordion title="Crash alerts stopped arriving and nothing else looks wrong">
|
||
Check **Settings · System · Docker hygiene · Global crash capture**. The toggle is the master switch for the Docker event service. The panel cache reads the database every 500 ms; if the database read errors, Sencho defaults the toggle to off so the failure mode is silent rather than spammy. Repair the toggle, save, and the next Docker event reaches the dispatcher.
|
||
</Accordion>
|
||
<Accordion title="A burst of crashes happened but I only see about twenty alerts">
|
||
Sencho rate-limits crash and health dispatches to 20 emits per rolling 60-second window per node, then emits a single `warning`/`monitor_alert` roll-up `N additional containers crashed in the last minute.` once the window closes. Every alert is still persisted to `notification_history` and visible in the bell up to the 100-row per-node cap; only the channel fanout is throttled.
|
||
</Accordion>
|
||
<Accordion title="My deploy success doesn't appear in the bell, only as a toast">
|
||
Sencho deliberately hides rows whose category is one of `deploy_success`, `stack_started`, `stack_stopped`, `stack_restarted`, or `image_update_applied` AND whose `actor_username` is set to a real user. The reasoning: those are confirmations of the action you just clicked and are already shown as a toast. The rows are still persisted to `notification_history` and still dispatched to global channels and matching routes; only the bell render hides them.
|
||
</Accordion>
|
||
<Accordion title="A routing rule is set up but the global channel still fires">
|
||
Routing matchers AND together: every non-empty matcher must match the alert. A rule with **Stacks** set to `prod-api` will not match a `monitor_alert` for a different stack, and a rule with a populated **Stacks** matcher will not match host-level alerts (which carry no stack target). When zero rules match, Sencho falls back to global channels. To intercept everything, leave all four matcher fields empty on the rule. Also confirm the rule's **Enabled** pill is `ON`.
|
||
</Accordion>
|
||
<Accordion title="Fleet shows an update button but I never received a Sencho version notification">
|
||
The version poll runs every 6 hours and dedups against the `last_sencho_update_notified_version` system-state key, so each newly-released version produces exactly one alert per node. If you upgraded Sencho between the release and the poll, the dedup self-heals; the alert simply does not fire because the running version already matches. To force a fresh check, restart the Sencho container; the version poll runs once on startup after a 2-minute delay.
|
||
</Accordion>
|
||
<Accordion title="A stack shows the blue update indicator but no notification was received">
|
||
The image-update service emits on **state transitions only**. The first run of the service backfills a flag against rows that already had `has_update: true` so they do not all re-fire on first install. Once the flag is set, only the false-to-true transition triggers an alert. If a stack was already showing the indicator before backfill ran, a notification will only appear the next time it goes from up-to-date back to behind.
|
||
</Accordion>
|
||
</AccordionGroup>
|