Remote-node alerts now appear in the local notification bell alongside
local ones, each tagged with the originating node name.
- backend: reorder WS upgrade handler so /ws/notifications?nodeId=<remote>
falls through to the existing proxy path instead of short-circuiting
- frontend/api.ts: add fetchForNode() helper for explicit node-targeted
requests without touching the localStorage active-node key
- frontend/EditorLayout: fetch notification history from all registered
nodes in parallel on mount and on node-list changes; open a per-remote
WebSocket connection for real-time push; route mark-read / delete / clear
actions back to the originating node; show node-name badge on remote alerts
When the gateway proxied a WebSocket upgrade for a remote node's interactive
terminal (host console or container exec), it injected the long-lived api_token
(scope: 'node_proxy') as the Bearer token. The remote's WS upgrade handler
correctly blocked this token via its isProxyToken guard (added in a prior
security hardening commit), causing a 403 → socket destroy → "connection error,
session ended" on the client.
Fix: introduce a POST /api/system/console-token endpoint that issues a
short-lived JWT (scope: 'console_session', 60 s TTL). Before forwarding an
interactive WS upgrade to a remote node, the gateway calls this endpoint using
the api_token, then substitutes the returned console_session token as the Bearer
header for the WS upgrade. The remote's isProxyToken guard (scope === 'node_proxy')
continues to block the long-lived api_token from spawning shells directly while
allowing gateway-delegated console sessions through.
Non-interactive WS paths (stack logs) continue to use the api_token unchanged.
The previous fix attempted to re-stream req.body via proxyReq.write() in the
proxyReq event handler. This raced against http-proxy's synchronous
req.pipe(proxyReq) call: when express.json() consumed the IncomingMessage stream,
Node.js's pipe() detected readableEnded and scheduled process.nextTick(proxyReq.end),
which fired before the proxyReq socket event. Any subsequent write() threw
"write after end", leaving the remote server stalled waiting for body bytes that
never arrived (the "Add Rule spins forever" bug).
Root fix: a conditional middleware now skips express.json() entirely for requests
targeting a remote node on non-auth, non-nodes API paths. The raw stream is left
intact, allowing http-proxy's req.pipe(proxyReq) to forward the body correctly.
express.json() fully drains the incoming request stream. http-proxy-middleware then
pipes an already-consumed empty stream to the remote Sencho instance, which holds the
connection open waiting for body bytes that never arrive (Content-Length says N but 0
bytes are forwarded). This caused all POST/PUT/PATCH requests to remote nodes — most
visibly 'Add Alert Rule' — to hang indefinitely with an infinite loading spinner.
Fix: in the proxyReq handler, write JSON.stringify(req.body) with correct Content-Type
and Content-Length headers before forwarding, restoring the full request body.
- StackAlertSheet: fetch agent status on open and show contextual banner
(amber warning when no channels configured, green when active, blue info
callout on remote nodes explaining remote-instance evaluation semantics)
- StackAlertSheet: surface actual server error message on addAlert failure
instead of generic string; add console logging for all failure paths
- SettingsModal: expose Notifications tab for remote nodes so agents can be
configured directly on any Sencho instance; section header shows node name
and Remote badge with tooltip when proxying to a remote node
- SettingsModal: re-fetch agents/settings when active node changes
- SettingsModal: add hidden DialogTitle/DialogDescription to satisfy Radix
UI accessibility requirements (eliminates console errors)
- backend POST /api/alerts: add Zod validation schema; rejects unknown
metric/operator values, negative thresholds, and missing fields with 400
- EditorLayout WS: upgrade reconnect from flat 5s retry to exponential
backoff (1s→2s→4s→8s→16s→30s max); onerror now logs the event; cleanup
only closes OPEN sockets — CONNECTING sockets are closed in onopen after
isMounted check, eliminating "closed before established" StrictMode error
- Reject node_proxy scoped JWT tokens with 403 on host-console and
container exec (/ws) upgrades; machine-to-machine credentials must
not open interactive shells
- Validate stackParam against path.resolve + startsWith(baseDir) to
prevent directory traversal on the PTY cwd (Rule 9 pattern)
- Strip JWT_SECRET, AUTH_PASSWORD, AUTH_PASSWORD_HASH, DATABASE_URL
from the environment passed to node-pty spawned shells
- fix(editor): Monaco compose↔.env tab switching no longer accumulates height
each switch — root cause was Monaco's position:static internal child creating
a circular CSS dependency with the CSS grid; broken by calling layout({0,0})
to force reflow before re-measuring. Added overflow-hidden to Monaco container.
- fix(ui): sliding tab indicator on compose/env tabs via motion.div layoutId
- fix(ui): sliding tab indicator on Notification tabs (Discord/Slack/Webhook)
- fix(ui): switch thumb now animates position (Radix data-attribute CSS translation)
- fix(ui): 'Always Local' tooltip no longer crashes — replaced animate-ui tooltip
(getStrictContext throws outside provider) with pure Radix primitives + CSS animation
- feat(settings): Auto theme option added (light/dark/auto with matchMedia listener)
- fix(ui): NodeManager dialog buttons no longer stuck together (DialogFooter gap)
- fix(ui): AlertDialog spring animation + Quick Clean button text wraps correctly
- fix(ui): Resources/App Store/Logs buttons toggle off on second click
Install motion + animate-ui, overhaul design tokens with brand cyan accent,
and replace CSS keyframe animations in Dialog, Tabs, Switch, and Tooltip
with spring-physics and blur-fade transitions via animate-ui Radix primitives.
Eliminates the 5-second setInterval polling loop for notifications in
EditorLayout. A persistent /ws/notifications WebSocket connection is
opened on mount; the backend pushes each alert the instant dispatchAlert
fires, with 5s auto-reconnect on close.
Key changes:
- NotificationService: injectable broadcaster callback (setBroadcaster)
- DatabaseService.addNotificationHistory: returns full NotificationHistory
record (with id/is_read) instead of void
- index.ts: notificationSubscribers Set + /ws/notifications upgrade handler
(JWT-verified, placed before remote proxy path)
- EditorLayout: polling removed, WS connect/reconnect replaces it
- Add localOnly option to apiFetch — omits x-node-id header so the
request bypasses the proxy and always hits the local Sencho instance
- fetchSettings now performs two fetches when a remote node is active:
active node fetch for per-node settings (CPU/RAM/disk limits, janitor,
crash detection), and a localOnly fetch for UI preferences
(developer_mode, global_logs_refresh, metrics_retention_hours,
log_retention_days)
- saveDeveloperSettings passes localOnly: true — developer preferences
can no longer be written into a remote node's database
- Update scope badges: System Limits shows "Configuring: [node]",
Developer shows "Always Local" when a remote node is active
Security:
- Strip auth credential keys (auth_username, auth_password_hash,
auth_jwt_secret) from GET /api/settings response
- Add allowlist guard to POST /api/settings — rejects unknown or
auth-namespace keys with a 400
Backend:
- Add PATCH /api/settings bulk endpoint with Zod schema validation
(type coercion, range checks, URL format) and atomic SQLite transaction
- Add system_state table — moves last_janitor_alert_timestamp out of
global_settings; adds getSystemState/setSystemState on DatabaseService
- Add metrics_retention_hours and log_retention_days configurable settings;
MonitorService reads both dynamically each evaluation cycle
- Add cleanupOldNotifications(days) to DatabaseService, called each cycle
Frontend:
- Replace single isLoading flag with per-operation states
(isSavingSystem, isSavingDeveloper, isSavingPassword, isSavingRegistry,
isSavingAgent/isTestingAgent per agent type)
- Add skeleton loader that blocks interaction until fetchSettings resolves
- Explicit key-picking in fetchSettings — auth keys cannot enter state
- Unsaved-changes amber dot on System Limits and Developer sidebar items
- Separate saveSystemSettings / saveDeveloperSettings — no cross-tab clobber
- Developer tab gains Data Retention section (metrics hours, log days)
- All settings saves use new PATCH /api/settings endpoint
- TemplateService: static LSIO_CATEGORY_MAP covers ~80 well-known apps
across Automation, Downloaders, Media, Monitoring, Networking,
Security, Development, Productivity, Utilities, and Other. Category
lookup is O(1) and runs once at cache-fill time. Adds source field
('linuxserver' | 'custom') to Template for future per-source UX.
Portainer v2 registries pass their native categories through unchanged.
Adds clearCache() method wired to POST /api/templates/refresh-cache.
- AppStoreView: category pill bar (All + sorted dynamic categories)
rendered between search and grid; active pill fills on click; clicking
a category badge on a card also sets the active filter; app count
updates reactively; filter composed with existing search query.
- SettingsModal: new App Store section (local-node only) exposes a
custom Portainer v2 JSON URL input with Save & Refresh (saves setting
then busts the 24h template cache) and Reset to Default.
Each container's onmessage handler was calling setContainerStats()
independently, causing React to schedule up to N separate reconciliation
passes per second (one per container). With 20 containers streaming Docker
stats at ~1 update/s, EditorLayout was re-rendering up to 20 times/s.
Fix: incoming stats are written into pendingStatsRef (no re-render cost),
then flushed to React state in one batched setContainerStats call every 1.5s.
Two bugs in the original proposal are addressed:
- rawBytesRef (never cleared) owns rx/tx tracking so net I/O rate is always
accurate; avoids the stale containerStats closure that would have shown
0 B/s after every flush cycle
- pending snapshot is captured and cleared BEFORE calling setState so the
functional updater stays pure (no side-effects inside it)
- pendingStatsRef is cleared in the effect cleanup so stale entries from
the previous stack don't briefly appear on stack switch
key={idx} with a shifting .slice(-300) window meant every new log arrival
caused React to update all 300 existing DOM nodes (index 1 became 0, 2
became 1, etc.), even though only one entry actually changed.
The proposed content-hash fix (timestamp + containerName + message.substring)
carries a real collision risk: chatty containers emitting identical lines at
the same millisecond produce duplicate keys, triggering undefined React
reconciliation behaviour.
Fix: add a _id: number field to LogEntry, stamped client-side with a
monotonic counter (logIdRef) at the point of ingestion — once in the SSE
onmessage handler and once in the polling setLogs path. React now tracks
data identity rather than position, so the slice window can shift without
touching the 299 unchanged DOM nodes.
Playwright investigation revealed the Logs view (GlobalObservabilityView)
rendered 1,859+ log entries as real DOM nodes (9,616 total DOM nodes) with
no virtualization. Combined with a 5-second polling cycle replacing all
React elements each time and smooth-scroll animations stacking on every
update, the renderer process grew rapidly on a host running at 97% RAM
usage, crashing the browser tab with Out of Memory within minutes.
Fixes:
- Cap rendered DOM rows to MAX_DISPLAY_ROWS (300) via .slice(-300) so the
browser only ever holds ~1,500 log-related DOM nodes regardless of how
many entries are in state
- Add a truncation notice when log count exceeds the display cap
- Reduce SSE-mode in-memory log cap from 10,000 to MAX_LOG_ENTRIES (2,000)
- Switch auto-scroll from behavior:'smooth' to behavior:'instant' to stop
stacking layout animations on every 5-second poll
- Reduce /api/logs/global response limit from 2,000 to 500 lines since the
client renders at most 300 rows, making the extra payload wasteful
- Destroy Docker stats stream on WS close to prevent orphaned daemon polling
- Guard all ws.send() calls with readyState === OPEN check
- Add .catch() to unawaited streamStats/execContainer calls to prevent
unhandled rejections crashing the process (Node >= 15)
- Close per-connection WebSocket.Server instances after handleUpgrade to
prevent listener accumulation over many connections
- Invert auth token precedence to bearerToken || cookieToken in both
authMiddleware and the WS upgrade handler so node-to-node Bearer tokens
are never shadowed by a stale browser cookie
- Narrow proxyRes type in remoteNodeProxy error handler before calling
.status() to avoid throwing on raw Socket (WS/TCP-level proxy errors)
Two related auth/routing bugs broke Terminal logs, container stats, the
LogViewer SSE stream, and exec bash when a remote node was selected:
1. WebSocket proxy (upgrade handler) was forwarding the browser's
`cookie` header to the remote Sencho instance. The remote's
`authMiddleware` evaluates `cookieToken || bearerToken`, so it
picked the cookie first — signed with the *local* JWT secret — and
returned 401 before even seeing the valid Bearer token.
Fix: `delete req.headers['cookie']` before `wsProxyServer.ws()`,
matching the `proxyReq.removeHeader('cookie')` already present in
the HTTP proxy.
2. WebSocket proxy was also forwarding `?nodeId=<gatewayId>` in the
URL. The remote's `nodeContextMiddleware` rejected it with 404
("Node X not found") because gateway node IDs don't exist on the
remote instance.
Fix: strip `nodeId` from `req.url` before proxying, so the remote
defaults cleanly to its own local node.
3. HTTP proxy (`remoteNodeProxy`) was forwarding `?nodeId=<gatewayId>`
in `proxyReq.path` to the remote. Affected EventSource endpoints
like `/api/containers/:id/logs?nodeId=9` that pass nodeId as a
query param rather than the `x-node-id` header.
Fix: strip `nodeId` from `proxyReq.path` in `onProxyReq`.
- Generic WebSockets (stats, exec) now connect to /ws?nodeId= instead of
the bare root so the upgrade handler detects the remote node and proxies
the WS connection to the correct remote Sencho instance.
- Vite dev proxy gains ws:true on /api and a new /ws entry so WebSocket
upgrades reach localhost:3000 during npm run dev.
- Backend WS message handler falls back to the default local node when the
nodeId in a proxied message doesn't exist in the local DB.
- Open App button now extracts the hostname from the remote node's api_url
instead of using window.location.hostname.
The remoteNodeProxy middleware (line 373) is already positioned before
all API route definitions, so remote requests are correctly proxied to
the target Sencho instance before any route handler executes.
However, /api/system/stats had a dead remote branch that called
NodeRegistry.getDocker() for remote nodes. This method throws by design
("remote nodes are not directly accessible") since no direct Docker TCP
socket is used for remote nodes. The thrown error propagated through the
catch block, returning a 500 to the frontend and causing system stats to
show as loading/unavailable for remote nodes.
Removed the dead branch. Remote requests for /api/system/stats are now
correctly proxied, returning real CPU/RAM/disk data from the remote host.
Adds an ImageUpdateService that queries OCI registry manifest endpoints
every 6 hours to compare remote digests against local RepoDigests.
Results are cached in a new stack_update_status SQLite table. A pulsing
blue dot badge appears in the stack sidebar for stacks with updates
available. Manual refresh available via POST /api/image-updates/refresh
with a 10-minute rate limit.
Task 1 - Network layer fixes:
- AppStoreView: replace raw fetch() with apiFetch() on /templates and
/templates/deploy so x-node-id header is injected for proxy routing
- GlobalObservabilityView: replace raw fetch() with apiFetch() on
/stacks and /logs/global
- HostConsole: append ?nodeId= to WebSocket URL so the upgrade handler
correctly routes the PTY session to the active remote node
Task 2 - Scoped context two-tier UX (Option A / Portainer-style):
- EditorLayout: add a context pill in the top header bar showing the
active node name (pulsing blue for remote, green for local)
- HostConsole: header now reads "Host Console - [Node Name]"
- ResourcesView: title now reads "Resources Hub - [Node Name]" for remote
- GlobalObservabilityView: floating node badge in top-left corner for remote
- AppStoreView: deploy sheet shows "Deploying to: [Node Name]" badge
- SettingsModal: remote nodes only see System Limits + Developer tabs;
global-only tabs (Account, Appearance, Notifications, Nodes) are hidden;
header subtitle shows remote node name
Two backend fixes:
1. proxyReq was forwarding the browser's sencho_token cookie to the remote
Sencho. The remote's authMiddleware uses cookieToken || bearerToken, so
the invalid local-signed cookie was verified before the valid Bearer token,
returning 401 on every proxied API call. Strip the cookie header in
proxyReq so remote auth uses only the Bearer token.
2. nodeContextMiddleware blocked /api/nodes when x-node-id referenced a
deleted node, trapping the frontend in an unrecoverable 404 loop (it
could not fetch the nodes list to discover the node was gone). Exempt
/api/nodes alongside /api/auth/ so the app can always self-heal.
Anso