* fix(notifications): version mute replica retractions for soft-cleanup restore
Soft cleanup and authoritative delete shared an unversioned permanent
tombstone, so a later hub re-save could not restore scheduled mutes on a
remote. Carry hub-authored kind and source_updated_at on replica DELETE,
allow recoverable recreate when updated_at is newer, keep permanent deletes
fail-closed, and reject stale recoverable DELETEs against newer rows.
* fix(notifications): durable mute retractions across mixed-version fleets
Gate recoverable replica DELETEs on a new capability, durable-queue failures
and incompatible remotes for retry, fan permanent deletes to every known
remote, and return applied vs ignored outcomes on replica writes.
* feat: add weekly UTC maintenance windows to mute rules
Let mute rules suppress only during recurring UTC windows, normalize
replica node identity, and fail-open when remotes lack schedule support
so older nodes never keep an all-day scheduled mute after a successful cleanup DELETE.
* fix: fail closed on corrupt mute schedules and clean invalid replicas
Empty or whitespace stored schedules no longer act as all-day mutes. Invalid schedules trigger remote DELETE cleanup, and the weekly-window form gains accessibility attributes plus component coverage.
* fix: require explicit repair before clearing a corrupt mute schedule
The suppression engine already fails closed on an unreadable stored
schedule (scheduleInvalid), but the frontend never surfaced that flag:
a corrupt rule looked identical to an ordinary unscheduled one, and
opening Edit then clicking Update sent an explicit schedule: null,
silently turning the corruption into a valid all-day mute. Add the
flag to the rule type, show an Invalid schedule badge on the card, and
block saving in the edit form until the operator explicitly touches
the weekly window (configures a new one, or toggles it to confirm the
clear).
* fix: correct contradictory toggle-sequence copy in schedule-repair toast
The blocking toast told operators to toggle the weekly window "off then
on" to confirm clearing a corrupt schedule, but the toggle starts off
for a corrupt rule, so that sequence leaves it on and trips the
no-selected-day validation instead. The correct, tested sequence is on
then off, matching the inline hint below the toggle. Also add a
regression test confirming the invalid-schedule save gate resets
cleanly across edit sessions on different rules.
* fix: enforce replica node_id and guard fleet sync against stale writes
Two hardenings to the suppression-rule fleet sync path found during
review: the /replica endpoint trusted the payload's node_id instead of
forcing it to null server-side, so a direct proxy-authenticated caller
could persist a scoped replica; and upsertNotificationSuppressionRuleReplica
overwrote unconditionally with no ordering check, so a delayed older
POST arriving after a newer one could downgrade the stored rule. Force
node_id to null on every replica write, and skip (with a warning log)
any incoming write whose updated_at is not newer than what's stored.
* test: assert the exact-tie updated_at case in the fleet sync stale-write guard
The staleness guard added in c31458a1 uses >= (ties are ignored, not
just strictly older writes); add the missing assertion for that
boundary and make the comment explicit about it.
* fix: bump vulnerable transitive backend dependencies
npm audit flagged body-parser, fast-uri, and protobufjs (one high
severity: fast-uri host confusion via failed IDN canonicalization).
All three have patch/minor fixes within existing semver ranges;
npm audit fix resolves all three with no package.json changes.
* fix: sanitize suppression replica fields before logging
Log entries built from fleet-sync replica payloads embedded rule id
and timestamp values directly, allowing a compromised peer to forge
log lines via control characters.
* fix: prevent delayed replica writes from resurrecting deleted mute rules
A network-reordered replica POST arriving after a DELETE fell into the
insert-when-absent branch with no protection, since the staleness guard
only compares against a row that still exists. Add a permanent
per-id tombstone (safe because rule ids are AUTOINCREMENT and never
reused): every delete records one, and the replica upsert refuses to
recreate a tombstoned id regardless of the incoming updated_at.
* feat: add Apprise as a fourth notification channel
Support keyed and stateless Apprise endpoints with secret-safe public DTOs, fail-closed malformed config, and mode-specific Settings UI. Docs and screenshots updated for four-channel Channels and routing.
* fix: harden Apprise secrets at rest and preserve-on-write saves
Encrypt Apprise endpoint and config with CryptoService so a downgrade cannot leak via SELECT *. Align channel and routing saves so blank destination fields omit config on same-mode URL edits, enforce keyed notify IDs, and keep secrets_redacted truthful.
* fix: harden Apprise route type changes and mixed-version config UI
Require a raw channel_url when switching notification-route types so ciphertext cannot strand under Discord/Slack/webhook. Default missing remote apprise status, replace Channels state on node switch, and exercise the production config-column migrator.
* fix: tolerate stub fleet configuration payloads without agents
Normalize remote Apprise agent status only when notifications.agents is present so successful Pilot/stub fetches stay online instead of throwing into the offline catch path.
* fix: correct TypeScript in configuration normalize tests
* fix: ignore stale Channels agent bodies after node switch
Compare the active node after response JSON parsing so a slow body
cannot overwrite the newly selected node's channel state.
* fix: isolate corrupt Apprise crypto and keep keyed Tags visible
Decrypt failures on one Apprise row no longer 500 agent/route lists or
suppress sibling channel dispatch. Treat public /notify/<redacted> as keyed
so Tags remain editable after reload.
* feat: add developer-mode startup and stack hydration timing
Instrument boot-to-list and detail hydration with commit-aligned milestones, truthful request stages, and destination/gateway debug duration logs so performance work is guided by measurements.
* fix: redact stack names and complete hydration request stages
Stop logging stack identifiers in containers debug timing, and record state_dispatch (plus detail fetch spans) so copied reports match the advertised stage breakdown.
* feat: add notification suppression rules
* fix: restore label routing and routing test mocks for suppression
* fix: allow bell mute shortcuts for history-only notification categories
Suppression rule validation used the routable category whitelist, which rejected history-only categories such as update_started that appear in the bell during stack updates.
* feat: expand Mute Rules UX with compose-first entry points and activity badges
* fix: add missing NodeContext mocks for notification suppression tests
* feat(pricing): collapse to two tiers (Community + Admiral)
Collapse Sencho's pricing from three tiers (Community / Skipper / Admiral)
to two: a generous free Community tier and a single paid Admiral tier. The
Skipper tier is removed.
Now free in Community: auto-heal, auto-update, scheduled operations,
webhooks, notification routing, Fleet Actions and bulk operations, SSO
preset providers (Google / GitHub / Okta), unlimited users with admin and
viewer roles, and deploy safety (atomic deploys, auto-rollback, and
one-click rollback).
Admiral (paid) is focused on running and governing a fleet: blueprints,
Fleet Secrets, deploy enforcement, vulnerability report export, audit log,
host console, private registries, mesh networking, node cordon, managed
cloud backup, LDAP / Active Directory SSO, and the advanced RBAC roles
(deployer, node-admin, auditor) with per-resource scoped assignments.
Internally the license variant distinction is removed so tier is binary
(community / paid). License validation still verifies the Lemon Squeezy
store and product before granting paid status.
Docs and the contributor guide are updated to the two-tier model.
* docs(pricing): correct licensing page to two-tier pricing and tidy stale tier wording
The licensing docs page kept the old Admiral pricing plus a Founder
Lifetime column and an Enterprise paragraph after the two-tier collapse.
Update it to $12/month or $99/year, drop the lifetime and Enterprise
content, and link to the pricing page for current pricing.
Also fix stale "Skipper" wording in CLA.md, SUPPORT.md, one test title,
and three test comments. Historical CHANGELOG entries and the
retired-Skipper license-guard test are intentionally left as-is.
* docs: align licensing and SSO pages with the two-tier model
Correct the SSO overview so the Google, GitHub, and Okta presets read as
available on every tier, matching the provider table; only LDAP and Active
Directory require Sencho Admiral. Remove the lifetime-plan references from the
licensing, settings, and troubleshooting pages so they reflect subscription-only
Admiral pricing.
* fix(rbac): omit scoped permissions from /me on the Community tier
Scoped role assignments only take effect on the paid tier, but GET /api/permissions/me returned them unconditionally, so a downgraded instance with leftover assignments rendered per-resource affordances the API then rejected with 403. The endpoint now mirrors the permission middleware and includes scoped permissions only on the paid tier. Adds a regression test covering the downgrade case.
* docs: use custom-pricing wording on the contact page
The two-tier model has no Enterprise tier; reword the contact page's enterprise pricing/deals to custom pricing/deals so it does not imply a tier that no longer exists.
* fix(notifications): harden alert dispatch crash-safety and redact webhook secrets in logs
Make NotificationService.dispatchAlert never reject so the many
fire-and-forget callers (monitors, event streams, policy and image-update
paths) cannot trigger an unhandledRejection on an unhealthy database. The
whole dispatch body now sits inside a guard covering node resolution, the
history insert, channel-table reads, and the WebSocket broadcast; a
failure logs and drops the notification instead of crashing the process.
An inner guard still splits the write-success and write-failure metrics.
Also:
- Redact webhook URLs in diagnostic logs via a new maskWebhookUrl helper;
Discord/Slack/custom webhook URLs embed their token in the path, so only
the origin is safe to emit.
- Add error logging to four notification-history route handlers that
previously swallowed database errors silently before returning 500.
- Snapshot the subscriber set before broadcasting so a close/error handler
firing mid-send cannot mutate the set during iteration.
- Sanitize the admin-supplied route name in dispatch log lines.
Adds tests for dispatch crash-safety (write failure, post-write routing
failure, broadcast send failure), the success-path write metric, webhook
URL masking, and history-route error logging.
* fix(notifications): sanitize route name and patterns in create logs
Apply sanitizeForLog to the admin-supplied route name and stack patterns
in the route-creation log lines, matching the dispatch-site sanitization
and closing the remaining log-injection path on this feature.
Also extend tests: userinfo-stripping in maskWebhookUrl, subscriber-set
snapshot behavior under an unsubscribe-during-send, and error logging on
the mark-read, delete-one, and clear-all notification-history handlers.
* feat(notifications): move Notification Routing to Skipper tier
Notification routing is automation (route alerts to channels by rules),
not enterprise compliance. Aligning the gate with Skipper makes the tier
boundary read consistently with the rest of the automation surface
(webhooks, auto-update, auto-heal, scheduled tasks).
Backend: requireAdmiral -> requirePaid on the five /api/notification-routes
endpoints. Dashboard configuration-status now exposes the routing-rules
row to any paid tier.
Frontend: settings registry tier flipped to skipper; the Admiral wrapper
around NotificationRoutingSection is removed (the inner CapabilityGate
stays, preserving forward-compat with older remote nodes).
Tests: added a tier-enforcement describe block covering Skipper (200) and
Community (403 PAID_REQUIRED on all five endpoints).
Docs: refreshed alerts-notifications, licensing, overview, dashboard,
troubleshooting, and reference/settings; cleaned one fence-spec line per
Directive 31.
* fix(notifications): address audit findings on tier-move PR
Docs: rewrite three lines that survived the initial sweep. The dashboard
"you do not see a locked placeholder" clause and the settings.mdx
"hidden on Community and Skipper" phrase were Directive 31 fence-spec.
The alerts-notifications troubleshooting note still said "an Admiral
routing rule" and contradicted the tier move.
Tests: the Community-negative cases on POST/PUT/DELETE/POST :id/test
could not distinguish requirePaid from a stray requireAdmiral, because
Community fails on the tier check before variant is read. Adding
Skipper-positive coverage per endpoint locks the gate identity in.
Replace the leaky mockReturnValueOnce with a per-test mockReturnValue
plus an afterEach restore so spies cannot bleed across tests.
Adds backend/src/utils/parseIntParam.ts with a shared parseIntParam
helper that writes a 400 'Invalid <label>' response and returns null on
non-numeric route params. Consolidates the parseInt + isNaN + 400 shape
that was inlined or duplicated across multiple routers.
Updated:
- routes/fleet.ts: replaced the local parseIdParam wrapper.
- routes/autoHeal.ts: replaced the local parsePolicyId wrapper.
- routes/notifications.ts: replaced parseRouteId wrapper plus an inline
notification-id site.
- routes/apiTokens.ts, routes/labels.ts, routes/registries.ts,
routes/scheduledTasks.ts, routes/users.ts: replaced inline copies.
Out of scope (route handlers without an existing isNaN check, kept
intentionally untouched to avoid introducing new 400 responses): alerts,
nodes, webhooks, and several user-routes handlers that rely on a
downstream 404 instead.
Closes#748
- Add label_ids and categories columns to notification_routes via idempotent migration
- Matcher logic always evaluates routes (AND semantics across all non-empty matchers)
- getStackLabelIds skips DB call when no enabled route uses label filtering
- Extract ALL_NOTIFICATION_CATEGORIES array from NotificationService as single source of truth
- Derive VALID_CATEGORIES set from the array in the route handler
- Extract validateLabelIds and validateCategories helpers to remove POST/PUT duplication
- Extract tryAddColumn as a private DatabaseService class method (removes 5 local re-declarations)
- Extract CATEGORY_LABELS to frontend/src/lib/notificationCategories.ts (shared by NotificationPanel and NotificationRoutingSection)
- Frontend form adds label and category multiselects with AND-filter hint
- Route cards show label and category badges; empty-matcher routes show 'Matches all alerts'
- Add tests for category-only, label-only, and combined AND-semantics routing
Adds a nullable node_id column to notification_routes (null = any
node, integer = fire only when the alert originates from that specific
node). This fixes a multi-node fleet defect where a route scoped to
"my-app" would fire on every node that hosts a stack with that name.
Backend changes:
- DatabaseService: idempotent migration adds node_id INTEGER NULL and
a composite index on (node_id, enabled, priority); the two statements
are in separate try-catch blocks so the index is always created even
when the column was added in an earlier run
- NotificationService: route matcher now pre-filters by node_id before
checking stack_patterns (== null matches any node)
- notifications route: POST/PUT accept optional node_id, validated to
be null or the local node's ID; NodeRegistry guards against
cross-node misroutes
Frontend changes:
- NotificationRoutingSection: node scope Select field uses useNodes()
from NodeContext (no extra API call) to populate the local node option
- Route cards show a node badge when node_id is set
Tests: 3 new tests covering node-match, node-mismatch, and null-scope;
all 75 files (1413 tests) passing.
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
Move four small Round C route groups out of index.ts:
- /api/auto-heal/* -> routes/autoHeal.ts
- /api/notifications/*, /api/notification-routes/* -> routes/notifications.ts
- /api/system/console-token -> routes/console.ts
- /api/sso/config/* -> routes/ssoConfig.ts
Share NOTIFICATION_CHANNEL_TYPES, cleanStackPatterns, and validateHttpsUrl
via a new helpers/notificationChannels.ts so agents.ts and notifications.ts
consume the same allowlist and URL validator.
Handlers moved verbatim. Middleware chains and response shapes preserved.
index.ts drops from 3231 to 2739 lines.
Anso