diff --git a/README.md b/README.md index e4c6d1c2..6953500f 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,12 @@ [![Latest release](https://img.shields.io/github/v/release/studio-saelix/sencho?label=release)](https://github.com/studio-saelix/sencho/releases) [![Docker Pulls](https://img.shields.io/docker/pulls/saelix/sencho)](https://hub.docker.com/r/saelix/sencho) [![CI](https://github.com/studio-saelix/sencho/actions/workflows/ci.yml/badge.svg)](https://github.com/studio-saelix/sencho/actions/workflows/ci.yml) + [![CodeQL](https://github.com/studio-saelix/sencho/actions/workflows/codeql.yml/badge.svg)](https://github.com/studio-saelix/sencho/actions/workflows/codeql.yml) [![License](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE) - [![Discussions](https://img.shields.io/github/discussions/studio-saelix/sencho)](https://github.com/studio-saelix/sencho/discussions) + [![Last commit](https://img.shields.io/github/last-commit/studio-saelix/sencho)](https://github.com/studio-saelix/sencho/commits/main) + [![Open issues](https://img.shields.io/github/issues/studio-saelix/sencho)](https://github.com/studio-saelix/sencho/issues) + [![Website](https://img.shields.io/website?url=https%3A%2F%2Fsencho.io&label=website)](https://sencho.io) + [![Docs](https://img.shields.io/website?url=https%3A%2F%2Fdocs.sencho.io&label=docs)](https://docs.sencho.io)
@@ -35,11 +39,11 @@ ## What Sencho is -Sencho is for homelab operators, small DevOps teams, and platform engineers who run services on Docker Compose, want a graphical interface without giving up file-on-disk workflows, and need to manage more than one machine without SSH gymnastics or a VPN. +Sencho is a Docker Compose control plane for DevOps engineers, platform teams, and system administrators who run services on Compose and need a real operational surface: a graphical interface that does not give up file-on-disk workflows, and the ability to manage more than one machine without SSH gymnastics or a VPN. -It runs as a single container on your hardware and gives you a UI for the work you currently do over SSH on compose stacks: deploying, editing files, watching logs, restarting containers, browsing volumes, and recovering from failures. Your compose files stay on the host filesystem and remain the source of truth. +It runs as a single container on your hardware and provides a UI for common Compose operations: deploying, editing files, watching logs, restarting containers, browsing volumes, and recovering from failures. Your compose files stay on the host filesystem and remain the source of truth. -A Sencho instance is autonomous. To manage another machine, you install a second Sencho on it and connect them with a long-lived API token; the primary dashboard then acts as an authenticated HTTP and WebSocket proxy across your fleet. Use TLS, a VPN, or a private network for any untrusted link. Each node still uses its local Docker socket (see Quick start), but Sencho does not require SSH and does not expose a remote Docker socket on the network. For nodes behind NAT or strict firewalls, the Pilot Agent establishes a single outbound WebSocket tunnel to the primary, so the remote host opens no inbound port at all. +Multi-node was part of the architecture from the start, not bolted on later: every Sencho instance is the same autonomous node, whether it runs alone or as one of many in a fleet. To manage another machine, you install a second Sencho on it and connect them with a long-lived API token; the primary dashboard then acts as an authenticated HTTP and WebSocket proxy across your fleet. Use TLS, a VPN, or a private network for any untrusted link. Each node still uses its local Docker socket (see Quick start), but Sencho does not require SSH and does not expose a remote Docker socket on the network. For nodes behind NAT or strict firewalls, the Pilot Agent establishes a single outbound WebSocket tunnel to the primary, so the remote host opens no inbound port at all. Sencho is free, open-source software under AGPLv3. Everything below is included in the Community tier with unlimited nodes and users. @@ -48,15 +52,18 @@ Sencho is free, open-source software under AGPLv3. Everything below is included ## Capabilities ### Stacks -- Full Compose lifecycle: create, deploy, restart, stop, pull +- Full Compose lifecycle: create, deploy, restart, stop, take down, pull - Atomic deployments with automatic rollback on failure - Monaco editor with diff preview before save and one-click rollback to any prior deploy - [Health-gated updates](https://docs.sencho.io/features/health-gated-updates) that hold a rollout until health checks pass, with stalled-update detection and in-app recovery - [Git-sourced stacks](https://docs.sencho.io/features/git-sources) pulled and synced from any repository, with ordered multi-file Compose - [File explorer](https://docs.sencho.io/features/stack-file-explorer) for compose, env, and supporting files, with move and rename across directories +- [Drift detection](https://docs.sencho.io/features/stack-drift) that compares running containers against the effective Compose model and flags exactly what changed +- [Environment and secrets guardrails](https://docs.sencho.io/features/environment-guardrails) that inventory every variable a stack uses and flag missing or duplicate values, without ever exposing a value +- [Storage portability](https://docs.sencho.io/features/compose-storage) checks that show whether a stack's mounts can move cleanly to another node before you move it - [Compose Doctor](https://docs.sencho.io/features/compose-doctor) preflight checks that catch compose problems before deploy - [Stack labels](https://docs.sencho.io/features/stack-labels) for grouping and bulk operations -- [App Store](https://docs.sencho.io/features/app-store) with LinuxServer.io templates +- [App Store](https://docs.sencho.io/features/app-store) with LinuxServer.io templates by default, or any custom Portainer-compatible registry ### Observability - Aggregated [log search and stream](https://docs.sencho.io/features/global-observability) across every container in the fleet @@ -73,6 +80,8 @@ Sencho is free, open-source software under AGPLv3. Everything below is included - [Fleet Federation](https://docs.sencho.io/features/fleet-federation): cordon nodes and pin Blueprints to specific hosts - [Fleet Actions](https://docs.sencho.io/features/fleet-actions): bulk label operations, fleet-wide stop-by-label, and fleet-wide prune - [Fleet Dossier](https://docs.sencho.io/features/fleet-dossier): export the whole fleet as a single browsable Markdown archive +- [Docker Label Audit](https://docs.sencho.io/features/docker-label-audit) across every node, for labels that drive external automation +- [Remote updates](https://docs.sencho.io/features/remote-updates): pull the latest image and recreate any node in the fleet from the Fleet view, no SSH session required - Node labels and grouping - [Pilot Agent](https://docs.sencho.io/features/pilot-agent) for nodes behind NAT or strict firewalls - Node compatibility checks before deploying @@ -91,21 +100,22 @@ Sencho is free, open-source software under AGPLv3. Everything below is included - [Security overview](https://docs.sencho.io/features/security) with a chart-led scan summary, sortable images, and searchable scan history - [Vulnerability scanning](https://docs.sencho.io/features/vulnerability-scanning) via Trivy, with on-demand node-wide scans, VEX-based suppression, SARIF export, and SBOM upload - [Compose network inspector](https://docs.sencho.io/features/compose-networking) with an exposure-intent guard for unintended published ports -- Scan policy packs for reusable security rules +- Node-wide network inventory, topology, and exposure findings across every stack on a node +- [Scan policies](https://docs.sencho.io/features/vulnerability-scanning#scan-policies) that set severity thresholds and can block a deploy - [Private registries](https://docs.sencho.io/features/private-registries) for Docker Hub, GHCR, and custom registries, plus [deploy enforcement](https://docs.sencho.io/features/deploy-enforcement) for non-compliant images - [API tokens](https://docs.sencho.io/features/api-tokens) for automation ### Operations - Off-site stack archives via [custom S3-compatible storage](https://docs.sencho.io/operations/backup) -- [Notification routing](https://docs.sencho.io/features/alerts-notifications#notification-routing) to Slack, Discord, email, and webhooks -- [Global search](https://docs.sencho.io/features/global-search) across stacks, containers, and services +- [Notification routing](https://docs.sencho.io/features/alerts-notifications#notification-routing) to Slack, Discord, and any generic webhook +- [Global search](https://docs.sencho.io/features/global-search) across pages, nodes, and every stack in the fleet - [Resources view](https://docs.sencho.io/features/resources) for images, volumes, and networks with scoped prune actions --- ### Before you install -Sencho talks to Docker through the host's `/var/run/docker.sock`. Mounting this socket grants Sencho the same privilege as `sudo docker` on the host. This is the same model used by Portainer, Dockge, Komodo, and other Compose dashboards. If your threat model requires stricter isolation, see [running with a non-root container user](https://docs.sencho.io/operations/self-hosting#non-root-user) and front Sencho with a reverse proxy that enforces authentication. +Sencho talks to Docker through the host's `/var/run/docker.sock`. Mounting this socket grants Sencho the same privilege as `sudo docker` on the host. This is the same model used by Portainer, Dockge, Komodo, and other Compose dashboards. If your threat model requires stricter isolation, see [running with a non-root container user](https://docs.sencho.io/getting-started/configuration#container-user) and front Sencho with a reverse proxy that enforces authentication. ## Quick start @@ -191,6 +201,7 @@ Sencho does not emit telemetry, analytics, or crash reports, and makes no outbou ## Documentation, community, and license - **Documentation:** [docs.sencho.io](https://docs.sencho.io) +- **Blog:** [sencho.io/blog](https://sencho.io/blog) - **Known limitations:** [KNOWN_LIMITATIONS.md](KNOWN_LIMITATIONS.md) - **If something breaks:** the [Recovery guide](https://docs.sencho.io/operations/recovery) covers getting back to a working state when Sencho, a deploy, sign-in, Docker, or a node fails. - **Community:** [GitHub Discussions](https://github.com/studio-saelix/sencho/discussions) diff --git a/backend/package-lock.json b/backend/package-lock.json index 07c403a9..4e74aa99 100644 --- a/backend/package-lock.json +++ b/backend/package-lock.json @@ -2487,9 +2487,9 @@ "optional": true }, "node_modules/brace-expansion": { - "version": "5.0.6", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz", - "integrity": "sha512-kLpxurY4Z4r9sgMsyG0Z9uzsBlgiU/EFKhj/h91/8yHu0edo7XuixOIH3VcJ8kkxs6/jPzoI6U9Vj3WqbMQ94g==", + "version": "5.0.7", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.7.tgz", + "integrity": "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA==", "dev": true, "license": "MIT", "dependencies": { diff --git a/backend/package.json b/backend/package.json index 99641897..bdb61960 100644 --- a/backend/package.json +++ b/backend/package.json @@ -28,6 +28,9 @@ "engines": { "node": ">=26.0.0" }, + "overrides": { + "brace-expansion": "^5.0.7" + }, "devDependencies": { "@eslint/js": "^10.0.1", "@types/bcrypt": "^6.0.0", diff --git a/backend/src/__tests__/notification-apprise.test.ts b/backend/src/__tests__/notification-apprise.test.ts index 503846dd..c6e27da1 100644 --- a/backend/src/__tests__/notification-apprise.test.ts +++ b/backend/src/__tests__/notification-apprise.test.ts @@ -11,6 +11,7 @@ const { mockGetStackLabelIds, mockAddNotificationHistory, mockUpdateNotificationDispatchError, + mockGetGlobalSettings, } = vi.hoisted(() => ({ mockGetEnabledNotificationRoutes: vi.fn().mockReturnValue([]), mockGetEnabledNotificationSuppressionRules: vi.fn().mockReturnValue([]), @@ -24,6 +25,7 @@ const { is_read: 0, }), mockUpdateNotificationDispatchError: vi.fn(), + mockGetGlobalSettings: vi.fn().mockReturnValue({ notification_dispatch_retries: '0' }), })); vi.mock('../services/DatabaseService', () => ({ @@ -35,6 +37,7 @@ vi.mock('../services/DatabaseService', () => ({ getStackLabelIds: mockGetStackLabelIds, addNotificationHistory: mockAddNotificationHistory, updateNotificationDispatchError: mockUpdateNotificationDispatchError, + getGlobalSettings: mockGetGlobalSettings, }), }, })); diff --git a/backend/src/__tests__/notification-retries.test.ts b/backend/src/__tests__/notification-retries.test.ts new file mode 100644 index 00000000..9eebbbaf --- /dev/null +++ b/backend/src/__tests__/notification-retries.test.ts @@ -0,0 +1,337 @@ +/** + * Configurable in-process notification dispatch retries. + */ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +const { + mockGetEnabledNotificationRoutes, + mockGetEnabledNotificationSuppressionRules, + mockGetEnabledAgents, + mockGetStackLabelIds, + mockAddNotificationHistory, + mockUpdateNotificationDispatchError, + mockGetGlobalSettings, +} = vi.hoisted(() => ({ + mockGetEnabledNotificationRoutes: vi.fn().mockReturnValue([]), + mockGetEnabledNotificationSuppressionRules: vi.fn().mockReturnValue([]), + mockGetEnabledAgents: vi.fn().mockReturnValue([]), + mockGetStackLabelIds: vi.fn().mockReturnValue([]), + mockAddNotificationHistory: vi.fn().mockReturnValue({ + id: 42, + level: 'error', + message: 'test', + timestamp: Date.now(), + is_read: 0, + }), + mockUpdateNotificationDispatchError: vi.fn(), + mockGetGlobalSettings: vi.fn().mockReturnValue({ notification_dispatch_retries: '0' }), +})); + +vi.mock('../services/DatabaseService', () => ({ + DatabaseService: { + getInstance: () => ({ + getEnabledNotificationRoutes: mockGetEnabledNotificationRoutes, + getEnabledNotificationSuppressionRules: mockGetEnabledNotificationSuppressionRules, + getEnabledAgents: mockGetEnabledAgents, + getStackLabelIds: mockGetStackLabelIds, + addNotificationHistory: mockAddNotificationHistory, + updateNotificationDispatchError: mockUpdateNotificationDispatchError, + getGlobalSettings: mockGetGlobalSettings, + }), + }, +})); + +vi.mock('../services/NodeRegistry', () => ({ + NodeRegistry: { + getInstance: () => ({ + getDefaultNodeId: () => 1, + getComposeDir: () => '/app/compose', + }), + }, +})); + +vi.mock('../services/StackActivityMetricsService', () => ({ + StackActivityMetricsService: { + getInstance: () => ({ record: vi.fn() }), + }, +})); + +import { NotificationService } from '../services/NotificationService'; + +const DISCORD = 'https://discord.com/api/webhooks/1/token'; + +function makeRoute(overrides: Record = {}) { + return { + id: 1, + name: 'Prod Discord', + node_id: null as number | null, + stack_patterns: [] as string[], + label_ids: null as number[] | null, + categories: null as string[] | null, + levels: null as ('info' | 'warning' | 'error')[] | null, + channel_type: 'discord' as const, + channel_url: DISCORD, + priority: 0, + enabled: true, + created_at: Date.now(), + updated_at: Date.now(), + ...overrides, + }; +} + +describe('notification dispatch retries', () => { + let svc: NotificationService; + let mockFetch: ReturnType; + + beforeEach(() => { + (NotificationService as unknown as { instance?: NotificationService }).instance = undefined; + NotificationService.setRetryDelayMsForTests(0); + svc = NotificationService.getInstance(); + mockFetch = vi.fn().mockResolvedValue({ ok: true, status: 200 }); + vi.stubGlobal('fetch', mockFetch); + mockGetEnabledNotificationRoutes.mockReturnValue([]); + mockGetEnabledNotificationSuppressionRules.mockReturnValue([]); + mockGetEnabledAgents.mockReturnValue([]); + mockGetStackLabelIds.mockReturnValue([]); + mockUpdateNotificationDispatchError.mockClear(); + mockAddNotificationHistory.mockClear(); + mockGetGlobalSettings.mockReset(); + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '0' }); + }); + + afterEach(() => { + NotificationService.setRetryDelayMsForTests(1000); + vi.unstubAllGlobals(); + }); + + it('retries=0 performs a single fetch', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '0' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch.mockResolvedValue({ ok: false, status: 500 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(1); + expect(mockGetGlobalSettings).toHaveBeenCalledTimes(1); + }); + + it('retries=2 on persistent 5xx performs three attempts', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '2' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch.mockResolvedValue({ ok: false, status: 502 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(3); + expect(mockUpdateNotificationDispatchError).toHaveBeenCalledWith( + 42, + expect.stringContaining('HTTP 502'), + ); + }); + + it('uses a fixed 1s delay between retryable attempts in production config', async () => { + NotificationService.setRetryDelayMsForTests(1000); + vi.useFakeTimers(); + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '1' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch + .mockResolvedValueOnce({ ok: false, status: 503 }) + .mockResolvedValueOnce({ ok: true, status: 200 }); + + try { + const p = svc.dispatchAlert('error', 'monitor_alert', 'down'); + // First attempt runs immediately; do not advance AbortSignal.timeout (10s). + await vi.advanceTimersByTimeAsync(0); + await Promise.resolve(); + expect(mockFetch).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(999); + expect(mockFetch).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(1); + await p; + expect(mockFetch).toHaveBeenCalledTimes(2); + } finally { + vi.useRealTimers(); + } + }); + + it('stops after success on the second attempt', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '2' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch + .mockResolvedValueOnce({ ok: false, status: 503 }) + .mockResolvedValueOnce({ ok: true, status: 200 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(2); + expect(mockUpdateNotificationDispatchError).not.toHaveBeenCalled(); + expect(mockAddNotificationHistory).toHaveBeenCalledTimes(1); + }); + + it('does not retry non-retryable 4xx', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '3' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch.mockResolvedValue({ ok: false, status: 404 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(1); + }); + + it('reads settings once for multi-destination fanout', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '1' }); + mockGetEnabledNotificationRoutes.mockReturnValue([ + makeRoute({ id: 1, name: 'A', channel_url: `${DISCORD}-a` }), + makeRoute({ id: 2, name: 'B', channel_url: `${DISCORD}-b` }), + ]); + mockFetch.mockResolvedValue({ ok: true, status: 200 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockGetGlobalSettings).toHaveBeenCalledTimes(1); + expect(mockFetch).toHaveBeenCalledTimes(2); + }); + + it('falls back to zero retries when settings throw and still sends once', async () => { + mockGetGlobalSettings.mockImplementation(() => { + throw new Error('db down'); + }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch.mockResolvedValue({ ok: false, status: 500 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(1); + }); + + it('falls back to zero retries for corrupt stored values', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '9' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute()]); + mockFetch.mockResolvedValue({ ok: false, status: 500 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(1); + }); + + it('does not retry unsupported channel types', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '2' }); + mockGetEnabledNotificationRoutes.mockReturnValue([ + makeRoute({ channel_type: 'sms' as 'discord', channel_url: 'https://example.com' }), + ]); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).not.toHaveBeenCalled(); + expect(mockUpdateNotificationDispatchError).toHaveBeenCalledWith( + 42, + expect.stringContaining('Unsupported channel type'), + ); + }); + + it('aggregates final errors from multiple failed destinations', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '0' }); + mockGetEnabledNotificationRoutes.mockReturnValue([ + makeRoute({ id: 1, name: 'One', channel_url: `${DISCORD}-1` }), + makeRoute({ id: 2, name: 'Two', channel_url: `${DISCORD}-2` }), + ]); + mockFetch + .mockResolvedValueOnce({ ok: false, status: 500 }) + .mockResolvedValueOnce({ ok: false, status: 503 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + const joined = mockUpdateNotificationDispatchError.mock.calls[0][1] as string; + expect(joined).toContain('Route "One"'); + expect(joined).toContain('Route "Two"'); + }); + + it('records the last attempt message for a destination after retries', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '1' }); + mockGetEnabledNotificationRoutes.mockReturnValue([makeRoute({ name: 'Flaky' })]); + mockFetch + .mockResolvedValueOnce({ ok: false, status: 500 }) + .mockResolvedValueOnce({ ok: false, status: 503 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + const joined = mockUpdateNotificationDispatchError.mock.calls[0][1] as string; + expect(joined).toContain('HTTP 503'); + expect(joined).not.toContain('HTTP 500'); + }); + + describe('channel retry classification matrix', () => { + const channels: Array<{ type: 'discord' | 'slack' | 'webhook'; url: string }> = [ + { type: 'discord', url: DISCORD }, + { type: 'slack', url: 'https://hooks.slack.com/services/T/B/X' }, + { type: 'webhook', url: 'https://example.com/hooks/sencho' }, + ]; + + for (const channel of channels) { + it(`${channel.type}: does not retry non-retryable 4xx`, async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '3' }); + mockGetEnabledNotificationRoutes.mockReturnValue([ + makeRoute({ channel_type: channel.type, channel_url: channel.url }), + ]); + mockFetch.mockResolvedValue({ ok: false, status: 404 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(1); + }); + + it(`${channel.type}: retries persistent 5xx for configured extras`, async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '2' }); + mockGetEnabledNotificationRoutes.mockReturnValue([ + makeRoute({ channel_type: channel.type, channel_url: channel.url }), + ]); + mockFetch.mockResolvedValue({ ok: false, status: 502 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(3); + }); + + it(`${channel.type}: stops after a successful retry`, async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '2' }); + mockGetEnabledNotificationRoutes.mockReturnValue([ + makeRoute({ channel_type: channel.type, channel_url: channel.url }), + ]); + mockFetch + .mockResolvedValueOnce({ ok: false, status: 503 }) + .mockResolvedValueOnce({ ok: true, status: 200 }); + + await svc.dispatchAlert('error', 'monitor_alert', 'down'); + + expect(mockFetch).toHaveBeenCalledTimes(2); + expect(mockUpdateNotificationDispatchError).not.toHaveBeenCalled(); + }); + } + }); + + + describe('testDispatch parity', () => { + it('retries a retryable failure then succeeds', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '1' }); + mockFetch + .mockResolvedValueOnce({ ok: false, status: 502 }) + .mockResolvedValueOnce({ ok: true, status: 200 }); + + await svc.testDispatch('discord', DISCORD); + + expect(mockGetGlobalSettings).toHaveBeenCalledTimes(1); + expect(mockFetch).toHaveBeenCalledTimes(2); + }); + + it('does not retry a non-retryable test failure', async () => { + mockGetGlobalSettings.mockReturnValue({ notification_dispatch_retries: '3' }); + mockFetch.mockResolvedValue({ ok: false, status: 401 }); + + await expect(svc.testDispatch('discord', DISCORD)).rejects.toMatchObject({ + status: 401, + retryable: false, + }); + expect(mockFetch).toHaveBeenCalledTimes(1); + }); + }); +}); diff --git a/backend/src/__tests__/notification-routing.test.ts b/backend/src/__tests__/notification-routing.test.ts index 275fbc6d..1bb7dc6a 100644 --- a/backend/src/__tests__/notification-routing.test.ts +++ b/backend/src/__tests__/notification-routing.test.ts @@ -13,6 +13,7 @@ const { mockGetStackLabelIds, mockAddNotificationHistory, mockUpdateNotificationDispatchError, + mockGetGlobalSettings, } = vi.hoisted(() => ({ mockGetEnabledNotificationRoutes: vi.fn().mockReturnValue([]), mockGetEnabledNotificationSuppressionRules: vi.fn().mockReturnValue([]), @@ -26,6 +27,7 @@ const { is_read: 0, }), mockUpdateNotificationDispatchError: vi.fn(), + mockGetGlobalSettings: vi.fn().mockReturnValue({ notification_dispatch_retries: '0' }), })); vi.mock('../services/DatabaseService', () => ({ @@ -37,6 +39,7 @@ vi.mock('../services/DatabaseService', () => ({ getStackLabelIds: mockGetStackLabelIds, addNotificationHistory: mockAddNotificationHistory, updateNotificationDispatchError: mockUpdateNotificationDispatchError, + getGlobalSettings: mockGetGlobalSettings, }), }, })); @@ -301,7 +304,7 @@ describe('NotificationService - routing logic', () => { expect(mockUpdateNotificationDispatchError).toHaveBeenCalledWith( 1, // notification id from mock - expect.stringContaining('Connection refused') + expect.stringContaining('Discord webhook request failed') ); }); @@ -314,7 +317,7 @@ describe('NotificationService - routing logic', () => { expect(mockUpdateNotificationDispatchError).toHaveBeenCalledWith( 1, - expect.stringContaining('Timeout') + expect.stringContaining('Slack webhook request failed') ); }); diff --git a/backend/src/__tests__/notification-suppression.test.ts b/backend/src/__tests__/notification-suppression.test.ts index fade91e1..5c03d0da 100644 --- a/backend/src/__tests__/notification-suppression.test.ts +++ b/backend/src/__tests__/notification-suppression.test.ts @@ -12,6 +12,7 @@ const { mockGetEnabledNotificationSuppressionRules, mockUpdateNotificationSuppressionMatch, mockBroadcast, + mockGetGlobalSettings, } = vi.hoisted(() => ({ mockGetEnabledNotificationRoutes: vi.fn().mockReturnValue([]), mockGetEnabledAgents: vi.fn().mockReturnValue([]), @@ -27,6 +28,7 @@ const { mockGetEnabledNotificationSuppressionRules: vi.fn().mockReturnValue([]), mockUpdateNotificationSuppressionMatch: vi.fn(), mockBroadcast: vi.fn(), + mockGetGlobalSettings: vi.fn().mockReturnValue({ notification_dispatch_retries: '0' }), })); vi.mock('../services/DatabaseService', () => ({ @@ -39,6 +41,7 @@ vi.mock('../services/DatabaseService', () => ({ updateNotificationDispatchError: mockUpdateNotificationDispatchError, getEnabledNotificationSuppressionRules: mockGetEnabledNotificationSuppressionRules, updateNotificationSuppressionMatch: mockUpdateNotificationSuppressionMatch, + getGlobalSettings: mockGetGlobalSettings, }), }, })); diff --git a/backend/src/__tests__/settings-routes.test.ts b/backend/src/__tests__/settings-routes.test.ts index 2afd30ac..83c11dc6 100644 --- a/backend/src/__tests__/settings-routes.test.ts +++ b/backend/src/__tests__/settings-routes.test.ts @@ -338,6 +338,64 @@ describe('health gate settings', () => { }); }); +describe('notification_dispatch_retries setting', () => { + it('seeds to "0" in a fresh database', () => { + expect(DatabaseService.getInstance().getGlobalSettings().notification_dispatch_retries).toBe('0'); + }); + + it('is exposed through the settings GET projection', async () => { + const res = await request(app).get('/api/settings').set('Cookie', adminCookie); + expect(res.status).toBe(200); + expect(res.body.notification_dispatch_retries).toBe('0'); + }); + + it('accepts integer number and digit-string writes via POST and PATCH', async () => { + const postNum = await request(app) + .post('/api/settings') + .set('Cookie', adminCookie) + .send({ key: 'notification_dispatch_retries', value: 2 }); + expect(postNum.status).toBe(200); + expect(DatabaseService.getInstance().getGlobalSettings().notification_dispatch_retries).toBe('2'); + + const patchStr = await request(app) + .patch('/api/settings') + .set('Cookie', adminCookie) + .send({ notification_dispatch_retries: '3' }); + expect(patchStr.status).toBe(200); + expect(DatabaseService.getInstance().getGlobalSettings().notification_dispatch_retries).toBe('3'); + + DatabaseService.getInstance().updateGlobalSetting('notification_dispatch_retries', '0'); + }); + + it('rejects malformed values on POST and PATCH without persisting', async () => { + const before = DatabaseService.getInstance().getGlobalSettings().notification_dispatch_retries; + + for (const value of [null, true, false, '', ' ', '1.5', '-1', 4, '4', 'abc']) { + const post = await request(app) + .post('/api/settings') + .set('Cookie', adminCookie) + .send({ key: 'notification_dispatch_retries', value }); + expect(post.status).toBe(400); + + const patch = await request(app) + .patch('/api/settings') + .set('Cookie', adminCookie) + .send({ notification_dispatch_retries: value }); + expect(patch.status).toBe(400); + } + + expect(DatabaseService.getInstance().getGlobalSettings().notification_dispatch_retries).toBe(before); + }); + + it('rejects a non-admin write with 403', async () => { + const res = await request(app) + .post('/api/settings') + .set('Cookie', viewerCookie) + .send({ key: 'notification_dispatch_retries', value: '1' }); + expect(res.status).toBe(403); + }); +}); + describe('env_block_deploy_on_missing_required setting', () => { it('seeds to "0" (opt-in) in a fresh database', () => { expect(DatabaseService.getInstance().getGlobalSettings().env_block_deploy_on_missing_required).toBe('0'); diff --git a/backend/src/helpers/notificationDispatchRetries.ts b/backend/src/helpers/notificationDispatchRetries.ts new file mode 100644 index 00000000..550fc5cd --- /dev/null +++ b/backend/src/helpers/notificationDispatchRetries.ts @@ -0,0 +1,16 @@ +/** + * Strict parser for notification_dispatch_retries (extra attempts, 0..3). + * Accepts JSON number integers or single-digit strings "0".."3" only. + * Rejects null, booleans, empty/whitespace, decimals, and out-of-range values. + */ +export function parseNotificationDispatchRetries(raw: unknown): number | null { + if (typeof raw === 'number') { + if (!Number.isInteger(raw) || raw < 0 || raw > 3) return null; + return raw; + } + if (typeof raw === 'string') { + if (!/^[0-3]$/.test(raw)) return null; + return Number(raw); + } + return null; +} diff --git a/backend/src/routes/settings.ts b/backend/src/routes/settings.ts index 7919ebfc..782b13c2 100644 --- a/backend/src/routes/settings.ts +++ b/backend/src/routes/settings.ts @@ -3,6 +3,7 @@ import { z } from 'zod'; import { DatabaseService } from '../services/DatabaseService'; import { authMiddleware } from '../middleware/auth'; import { requireAdmin, requirePaid } from '../middleware/tierGates'; +import { parseNotificationDispatchRetries } from '../helpers/notificationDispatchRetries'; // Strict allowlist of keys readable and writable via the generic settings // API. This is the single source of truth for what the endpoint exposes: @@ -34,6 +35,7 @@ const ALLOWED_SETTING_KEYS = new Set([ 'env_block_deploy_on_missing_required', 'auto_create_missing_external_networks', 'image_update_sidebar_indicators', + 'notification_dispatch_retries', ]); // Keys whose write requires a paid license, not just an admin role. @@ -66,6 +68,15 @@ const SettingsPatchSchema = z.object({ env_block_deploy_on_missing_required: z.enum(['0', '1']), auto_create_missing_external_networks: z.enum(['0', '1']), image_update_sidebar_indicators: z.enum(['0', '1']), + // Strict: do not use bare z.coerce.number() (null/false/'' become 0; true becomes 1). + notification_dispatch_retries: z.unknown().superRefine((v, ctx) => { + if (parseNotificationDispatchRetries(v) === null) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + message: 'Must be an integer from 0 to 3', + }); + } + }).transform((v) => String(parseNotificationDispatchRetries(v)!)), }).partial(); export const settingsRouter = Router(); diff --git a/backend/src/services/DatabaseService.ts b/backend/src/services/DatabaseService.ts index 973c032d..5f7cce45 100644 --- a/backend/src/services/DatabaseService.ts +++ b/backend/src/services/DatabaseService.ts @@ -1826,6 +1826,7 @@ export class DatabaseService { stmt.run('image_update_check_mode', 'interval'); stmt.run('image_update_check_cron', ''); stmt.run('image_update_sidebar_indicators', '1'); + stmt.run('notification_dispatch_retries', '0'); stmt.run('env_block_deploy_on_missing_required', '0'); stmt.run('auto_create_missing_external_networks', '0'); diff --git a/backend/src/services/NotificationService.ts b/backend/src/services/NotificationService.ts index d1ea2542..b1f139ac 100644 --- a/backend/src/services/NotificationService.ts +++ b/backend/src/services/NotificationService.ts @@ -19,6 +19,7 @@ import { parseStoredAppriseConfig, validateNotificationChannel, } from '../helpers/notificationChannels'; +import { parseNotificationDispatchRetries } from '../helpers/notificationDispatchRetries'; export type NotificationCategory = | 'deploy_success' @@ -71,6 +72,13 @@ export const ALL_SUPPRESSIBLE_CATEGORIES: readonly NotificationCategory[] = [ /** Webhook timeout: 10 seconds per external dispatch call. */ const WEBHOOK_TIMEOUT_MS = 10_000; +/** Fixed delay between retryable delivery attempts (extra attempts only). */ +const RETRY_DELAY_MS_DEFAULT = 1_000; + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + /** Valid notification channel types for defense-in-depth validation. */ const ALLOWED_CHANNEL_TYPES = new Set(['discord', 'slack', 'webhook', 'apprise']); @@ -84,6 +92,8 @@ export class NotificationService { private static instance: NotificationService; private dbService: DatabaseService; private readonly subscribers = new Set(); + /** Overridable in tests so retry loops need not wait a real second. */ + private static retryDelayMs = RETRY_DELAY_MS_DEFAULT; private constructor() { this.dbService = DatabaseService.getInstance(); @@ -96,6 +106,11 @@ export class NotificationService { return NotificationService.instance; } + /** @internal Test-only: set the inter-attempt delay (production uses 1000). */ + public static setRetryDelayMsForTests(ms: number): void { + NotificationService.retryDelayMs = ms; + } + /** * Register a WebSocket as a live-notification subscriber. Returns an * unsubscribe function the caller should invoke on `'close'` / `'error'` @@ -256,6 +271,9 @@ export class NotificationService { return; } + // Resolve retry extras once for this dispatch (shared by all destinations). + const retries = this.resolveDispatchRetries(); + // 3. Check notification routing rules — always evaluated, matchers compose AND const errors: string[] = []; @@ -264,7 +282,7 @@ export class NotificationService { if (isDebugEnabled()) console.log(`[Notify:diag] Matched ${matched.length} route(s) for stack "${sanitizeForLog(stackName ?? '(none)')}", category="${sanitizeForLog(category)}"`); await Promise.allSettled( matched.map(route => - this.sendToChannel(route.channel_type, route.channel_url, level, sanitized, route.config) + this.sendWithRetries(route.channel_type, route.channel_url, level, sanitized, route.config, retries) .then(() => { if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via route "${sanitizeForLog(route.name)}" (${route.channel_type})`); }) @@ -288,7 +306,7 @@ export class NotificationService { if (isDebugEnabled()) console.log(`[Notify:diag] Falling back to ${agents.length} global agent(s)`); await Promise.allSettled( agents.map(agent => - this.sendToChannel(agent.type, agent.url, level, sanitized, agent.config) + this.sendWithRetries(agent.type, agent.url, level, sanitized, agent.config, retries) .then(() => { if (isDebugEnabled()) console.log(`[Notify:diag] Dispatched ${level} via global agent (${agent.type})`); }) @@ -304,6 +322,25 @@ export class NotificationService { } } + /** + * Read notification_dispatch_retries once. Missing, malformed, out-of-range, + * or thrown settings reads fall back to 0 so the initial send still happens. + */ + private resolveDispatchRetries(): number { + try { + const raw = this.dbService.getGlobalSettings().notification_dispatch_retries; + const parsed = parseNotificationDispatchRetries(raw); + if (parsed === null) { + console.warn('[Notify] Invalid notification_dispatch_retries; using 0'); + return 0; + } + return parsed; + } catch (err) { + console.warn('[Notify] Failed to read notification_dispatch_retries; using 0:', err); + return 0; + } + } + /** Persist dispatch errors to the notification record for user visibility. */ private recordDispatchErrors(notificationId: number, errors: string[]) { if (errors.length > 0) { @@ -315,6 +352,43 @@ export class NotificationService { } } + /** + * Deliver with up to `retries` extra attempts after the first try. + * Waits a fixed 1s between attempts only when a retryable failure leaves attempts remaining. + */ + private async sendWithRetries( + type: string, + url: string, + level: 'info' | 'warning' | 'error', + message: string, + config: string | null | undefined, + retries: number, + ): Promise { + const totalAttempts = 1 + retries; + let lastError: NotificationDeliveryError | undefined; + for (let attempt = 0; attempt < totalAttempts; attempt++) { + try { + await this.sendToChannel(type, url, level, message, config); + return; + } catch (error) { + const deliveryError = error instanceof NotificationDeliveryError + ? error + : new NotificationDeliveryError( + getErrorMessage(error, 'Notification delivery failed'), + null, + false, + ); + lastError = deliveryError; + const attemptsRemain = attempt < totalAttempts - 1; + if (!deliveryError.retryable || !attemptsRemain) { + throw deliveryError; + } + await sleep(NotificationService.retryDelayMs); + } + } + throw lastError ?? new NotificationDeliveryError('Notification delivery failed', null, false); + } + private async sendToChannel(type: string, url: string, level: 'info' | 'warning' | 'error', message: string, config?: string | null): Promise { if (type === 'discord') { await this.sendDiscordWebhook(url, level, message); @@ -329,7 +403,7 @@ export class NotificationService { } await this.sendAppriseNotify(url, level, message, parsed); } else { - throw new Error(`Unsupported channel type: ${type}`); + throw new NotificationDeliveryError(`Unsupported channel type: ${type}`, null, false); } } @@ -338,7 +412,8 @@ export class NotificationService { const validation = validateNotificationChannel(type, url, config); if (validation) throw new Error(`URL ${validation}`); const stored = type === 'apprise' ? normalizeAppriseStoredJson(url, config) : (config == null ? null : JSON.stringify(config)); - await this.sendToChannel(type, url, 'info', '🔌 Test Notification from Sencho!', stored); + const retries = this.resolveDispatchRetries(); + await this.sendWithRetries(type, url, 'info', '🔌 Test Notification from Sencho!', stored, retries); } private async sendAppriseNotify( @@ -393,15 +468,28 @@ export class NotificationService { }] }; - const response = await fetch(url, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - signal: AbortSignal.timeout(WEBHOOK_TIMEOUT_MS), - }); + try { + const response = await fetch(url, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + signal: AbortSignal.timeout(WEBHOOK_TIMEOUT_MS), + }); - if (!response.ok) { - throw new Error(`Discord Webhook responded with ${response.status}`); + if (response.status >= 400 && response.status < 500) { + throw new NotificationDeliveryError(`Discord webhook responded with HTTP ${response.status}`, response.status, false); + } + if (!response.ok) { + throw new NotificationDeliveryError(`Discord webhook responded with HTTP ${response.status}`, response.status, true); + } + } catch (error) { + if (error instanceof NotificationDeliveryError) throw error; + const aborted = error instanceof Error && (error.name === 'AbortError' || error.name === 'TimeoutError'); + throw new NotificationDeliveryError( + aborted ? 'Discord webhook request timed out' : 'Discord webhook request failed', + null, + true, + ); } } @@ -416,15 +504,28 @@ export class NotificationService { text: `${emojiMap[level]} *Sencho Alert [${level.toUpperCase()}]*\n${message}` }; - const response = await fetch(url, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - signal: AbortSignal.timeout(WEBHOOK_TIMEOUT_MS), - }); + try { + const response = await fetch(url, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + signal: AbortSignal.timeout(WEBHOOK_TIMEOUT_MS), + }); - if (!response.ok) { - throw new Error(`Slack Webhook responded with ${response.status}`); + if (response.status >= 400 && response.status < 500) { + throw new NotificationDeliveryError(`Slack webhook responded with HTTP ${response.status}`, response.status, false); + } + if (!response.ok) { + throw new NotificationDeliveryError(`Slack webhook responded with HTTP ${response.status}`, response.status, true); + } + } catch (error) { + if (error instanceof NotificationDeliveryError) throw error; + const aborted = error instanceof Error && (error.name === 'AbortError' || error.name === 'TimeoutError'); + throw new NotificationDeliveryError( + aborted ? 'Slack webhook request timed out' : 'Slack webhook request failed', + null, + true, + ); } } @@ -436,15 +537,28 @@ export class NotificationService { source: 'sencho' }; - const response = await fetch(url, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - signal: AbortSignal.timeout(WEBHOOK_TIMEOUT_MS), - }); + try { + const response = await fetch(url, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify(payload), + signal: AbortSignal.timeout(WEBHOOK_TIMEOUT_MS), + }); - if (!response.ok) { - throw new Error(`Custom Webhook responded with ${response.status}`); + if (response.status >= 400 && response.status < 500) { + throw new NotificationDeliveryError(`Custom webhook responded with HTTP ${response.status}`, response.status, false); + } + if (!response.ok) { + throw new NotificationDeliveryError(`Custom webhook responded with HTTP ${response.status}`, response.status, true); + } + } catch (error) { + if (error instanceof NotificationDeliveryError) throw error; + const aborted = error instanceof Error && (error.name === 'AbortError' || error.name === 'TimeoutError'); + throw new NotificationDeliveryError( + aborted ? 'Custom webhook request timed out' : 'Custom webhook request failed', + null, + true, + ); } } } diff --git a/docs/docs.json b/docs/docs.json index e53eb7b9..0205718c 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -76,6 +76,9 @@ "display": "simple" } }, + "interaction": { + "drilldown": false + }, "navigation": { "tabs": [ { @@ -83,17 +86,16 @@ "groups": [ { "group": "Start here", - "root": "getting-started/introduction", - "directory": "card", "pages": [ + "getting-started/introduction", "getting-started/quickstart", "getting-started/configuration" ] }, { "group": "Product guide", - "root": "features/overview", "pages": [ + "features/overview", { "group": "Core workflow", "expanded": true, @@ -184,9 +186,8 @@ }, { "group": "Operations", - "root": "operations/self-hosting", - "directory": "accordion", "pages": [ + "operations/self-hosting", "operations/upgrade", "operations/backup", "operations/recovery", @@ -199,9 +200,8 @@ }, { "group": "Reference", - "root": "reference/settings", - "directory": "accordion", "pages": [ + "reference/settings", "features/licensing", "reference/security", "reference/contact" diff --git a/docs/features/alerts-notifications.mdx b/docs/features/alerts-notifications.mdx index 4987b5a9..ff612474 100644 --- a/docs/features/alerts-notifications.mdx +++ b/docs/features/alerts-notifications.mdx @@ -7,13 +7,15 @@ description: Threshold and event alerts for your fleet, dispatched to Discord, S Sencho watches each node it manages for container crashes, host pressure, scheduled-task results, and update availability, then surfaces every signal in two places: the in-app notification bell at the top of the shell and an external channel you configure. This page covers everything from configuring channels to writing per-stack threshold rules, routing alerts to dedicated channels with routing rules, and tuning retention. - Settings · Notifications · Channels panel with Discord, Slack, Webhook, and Apprise tabs, CHANNELS 1/4 in the masthead, the Apprise tab active with Enabled on, a redacted Apprise endpoint, an empty Tags field, and Test disabled beside Save. + Settings · Notifications · Channels panel with NODE Local in the header, a Delivery retries row showing Extra attempts 0 and a Save retries button, Discord Slack Webhook and Apprise tabs with Apprise selected, Enabled off, an empty Apprise endpoint placeholder, and Test beside Save. ## Notification channels Open **Settings · Notifications · Channels** to configure Discord, Slack, custom webhook, and Apprise channels. Each channel is per-node, so switching the active node via the node picker reloads the panel against that node's stored settings. The masthead carries a `CHANNELS` stat showing how many of the four slots are enabled. +Above the channel tabs, **Delivery retries** sets how many extra in-process attempts (0 to 3) Sencho makes after a transient delivery failure on that node. The default is `0` (single-shot). Extra attempts wait a fixed one second between tries. Admin role is required to change the value. + Each Discord, Slack, and Webhook tab carries an **Enabled** toggle, a **Webhook URL** input (HTTPS only), and **Test** / **Save**. The Apprise tab uses an **Apprise endpoint** instead: keyed `/notify/{key}` shows optional **Tags**; stateless `/notify` shows **Destination URLs**. The kicker on each tab toggles between `enabled` and `off` so you can see at a glance which slots are wired up. @@ -51,9 +53,9 @@ Sencho sends the alert title, body, severity, and either tags or destination URL ### Test sends and delivery semantics -The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take. Test sends require the admin role; the server returns 403 if an operator or viewer submits one. +The **Test** button on each tab dispatches the literal message `🔌 Test Notification from Sencho!` at level `info` through the same path a real alert would take, including the node's delivery-retries setting. Test sends require the admin role; the server returns 403 if an operator or viewer submits one. -Each dispatch is a single-shot HTTP POST with a 10-second `AbortSignal.timeout`. There is no retry queue. If your endpoint is down at the moment of the dispatch, the alert is recorded in the bell with an internal `dispatch_error` field set and is not redelivered. +Each delivery attempt is an HTTP POST with a 10-second `AbortSignal.timeout`. By default (`Delivery retries` = 0) Sencho makes one attempt. You can allow up to three extra in-process attempts with a fixed one-second delay between them. Retries apply only to classified transient failures (for example HTTP 5xx or network timeouts). Client errors such as HTTP 4xx and Apprise HTTP 204 are not retried. There is no durable retry queue: if the process exits mid-dispatch, remaining attempts are not persisted. Delivery is at-least-once under ambiguous timeouts or connection resets, so a receiver that accepted a request whose response was lost can receive a duplicate. If every attempt fails, the alert remains in the bell with `dispatch_error` set. ## Notification Routing @@ -421,7 +423,7 @@ The **Host Alerts** panel carries the **Host thresholds** rows (CPU limit, RAM l | Per-stack alert rule evaluation | 30 seconds | | Image update poll | Configurable (default every 2 hours), with a 2-minute startup delay and a 2-minute cooldown on manual refresh | | Sencho version check | Monitor evaluation every 30 seconds; shared version cache refreshes every 30 minutes when published or every 3 minutes while publish is pending | -| Notification fanout to channels | Single shot per dispatch, 10-second timeout, no retries | +| Notification fanout to channels | One attempt by default; optional 0-3 extra in-process attempts with a fixed 1s delay; 10-second timeout per attempt; no durable queue | | Bell live updates | Pushed live over the notifications WebSocket per node | | Bell safety-net reconcile | 60 seconds | | Crash dedup window per container | 60 minutes | diff --git a/docs/features/api-tokens.mdx b/docs/features/api-tokens.mdx index 59467e2e..d81a5056 100644 --- a/docs/features/api-tokens.mdx +++ b/docs/features/api-tokens.mdx @@ -21,7 +21,7 @@ An API token is an opaque secret value (not a JWT), 56 characters long, that Sen The three scopes form a ladder: - **Read Only** authorises every `GET` request the underlying API exposes. -- **Deploy Only** adds the six stack-lifecycle POSTs: deploy, down, restart, stop, start, update. +- **Deploy Only** adds the eight stack-lifecycle POSTs: deploy, down, restart, stop, start, update, plus per-service update and restore. - **Full Admin** adds every other read and write on the underlying API. A separate layer sits on top: a set of universal restrictions that **no** API token can reach, regardless of scope or HTTP method. These cover identity, trust roots, billing, and interactive shell access; even a Read Only `GET` against `/api/users` or `/api/sso/config` is rejected. They require a human user session. @@ -145,7 +145,7 @@ Authorises **every** `GET` against the API. Representative uses: list stacks, fe ### Deploy Only -Authorises every `GET`, plus exactly six `POST` patterns that operate on a stack by name: +Authorises every `GET`, plus eight `POST` patterns that operate on a stack (or one of its declared services) by name: | Method | Path pattern | Action | |--------|--------------|--------| @@ -155,6 +155,8 @@ Authorises every `GET`, plus exactly six `POST` patterns that operate on a stack | `POST` | `/api/stacks/:name/stop` | Stop without removing. | | `POST` | `/api/stacks/:name/start` | Start a stopped stack. | | `POST` | `/api/stacks/:name/update` | Pull newer images and redeploy. | +| `POST` | `/api/stacks/:name/services/:service/update` | Update or rebuild one declared Compose service without recreating its siblings. | +| `POST` | `/api/stacks/:name/services/:service/restore` | Roll a single service back to a prior recovery snapshot. | Any other write (creating a stack, editing a stack file, calling a non-lifecycle action) returns `403 SCOPE_DENIED`. @@ -190,6 +192,7 @@ Regardless of scope, every API token is rejected from the following routes with | **Host-console session token** | Mint the short-lived ticket the browser console uses | | **API token self-management** | Create, list, revoke API tokens | | **Recovery Vault** | Configure or trigger Recovery Vault | +| **Image channel management** | Preflight or switch the instance's image channel, acknowledge a failed switch | The rationale is the same in every case: a programmatic credential should not be able to grant itself more authority, change the trust roots that issued it, or open an interactive shell. Those actions require a live human user logged into a browser. @@ -337,8 +340,8 @@ Read Only is sufficient for the notification stream. The scope does not authorise the method or path you called. Either widen the scope by issuing a new token, or change the call. Common cases: - Read Only attempting a `POST` of any kind. - - Deploy Only attempting a write that is not one of the six lifecycle actions (`deploy`, `down`, `restart`, `stop`, `start`, `update`). - - Any scope attempting a [universal restriction](#universal-restrictions) (user management, SSO, MFA, license, registries, console, node management, cloud backup, API-token self-management). + - Deploy Only attempting a write that is not one of the eight lifecycle actions (`deploy`, `down`, `restart`, `stop`, `start`, `update`, and the per-service `update`/`restore`). + - Any scope attempting a [universal restriction](#universal-restrictions) (user management, SSO, MFA, license, registries, console, node management, cloud backup, image channel management, API-token self-management). diff --git a/docs/features/app-store.mdx b/docs/features/app-store.mdx index ee455341..eafdb36c 100644 --- a/docs/features/app-store.mdx +++ b/docs/features/app-store.mdx @@ -45,23 +45,23 @@ Tiles are sorted by GitHub star count, descending. Click any tile to open the de On small screens, the App Store adapts to a single-column layout. The masthead shows the store label, the total app count, the category count, and an action menu for navigation. Templates are listed alphabetically rather than by star count, and the category sidebar and featured hero are not shown on mobile. - App Store on a mobile device: status masthead showing APP STORE, 202 apps, and 12 categories, with an alphabetical single-column template list below + App Store on a mobile device: status masthead showing APP STORE, 201 apps, and 12 categories, with an alphabetical single-column template list below ## Deploying a template -The deployment sheet slides in from the right. The header carries a breadcrumb (`App store ›