* feat(scheduler): add auto_backup, auto_stop, auto_down, auto_start actions and delete_after_run one-shot mode
Extends the scheduler with four new stack-targeted actions:
- auto_backup: backs up stack compose files and .env using the existing
FileSystemService.backupStackFiles primitive
- auto_stop: runs compose stop (containers preserved)
- auto_down: runs compose down (containers removed)
- auto_start: runs compose up -d via deployStack (universal start for both
stopped and down stacks)
Adds delete_after_run boolean column to scheduled_tasks. When enabled,
the task self-deletes after its first successful execution; failures keep
the task so the user can debug and retry.
All four new actions gate at Admiral tier, consistent with restart/snapshot/prune.
Migration is idempotent (maybeAddCol).
* docs(scheduler): update scheduled-operations doc with new lifecycle actions and delete-after-run
Adds the four new actions (Backup Stack Files, Stop Stack, Take Stack Down,
Start Stack) to the action table. Documents the delete-after-run one-shot mode
with its success-only deletion semantics. Adds the Stack Lifecycle Scheduling
section explaining stop-vs-down semantics and the local-execution boundary.
Adds three troubleshooting entries: auto-start on a missing compose folder,
auto-backup single-slot overwrite by design, and one-shot task disappearing
after successful run.
Updates the timeline description from four to five lanes. Refreshes
screenshots to show the new dialog layout with the Lifecycle lane visible.
- Add label_ids and categories columns to notification_routes via idempotent migration
- Matcher logic always evaluates routes (AND semantics across all non-empty matchers)
- getStackLabelIds skips DB call when no enabled route uses label filtering
- Extract ALL_NOTIFICATION_CATEGORIES array from NotificationService as single source of truth
- Derive VALID_CATEGORIES set from the array in the route handler
- Extract validateLabelIds and validateCategories helpers to remove POST/PUT duplication
- Extract tryAddColumn as a private DatabaseService class method (removes 5 local re-declarations)
- Extract CATEGORY_LABELS to frontend/src/lib/notificationCategories.ts (shared by NotificationPanel and NotificationRoutingSection)
- Frontend form adds label and category multiselects with AND-filter hint
- Route cards show label and category badges; empty-matcher routes show 'Matches all alerts'
- Add tests for category-only, label-only, and combined AND-semantics routing
Adds a nullable node_id column to notification_routes (null = any
node, integer = fire only when the alert originates from that specific
node). This fixes a multi-node fleet defect where a route scoped to
"my-app" would fire on every node that hosts a stack with that name.
Backend changes:
- DatabaseService: idempotent migration adds node_id INTEGER NULL and
a composite index on (node_id, enabled, priority); the two statements
are in separate try-catch blocks so the index is always created even
when the column was added in an earlier run
- NotificationService: route matcher now pre-filters by node_id before
checking stack_patterns (== null matches any node)
- notifications route: POST/PUT accept optional node_id, validated to
be null or the local node's ID; NodeRegistry guards against
cross-node misroutes
Frontend changes:
- NotificationRoutingSection: node scope Select field uses useNodes()
from NodeContext (no extra API call) to populate the local node option
- Route cards show a node badge when node_id is set
Tests: 3 new tests covering node-match, node-mismatch, and null-scope;
all 75 files (1413 tests) passing.
Introduce a NotificationCategory string-literal union (11 values) and
thread it through dispatchAlert as a required second argument. All
callers (DockerEventService, AutoHealService, ImageUpdateService,
MonitorService, PolicyEnforcement, policyGate, SchedulerService,
imageUpdates route) pass an explicit category at every call site,
giving TypeScript compile-time enforcement that no new emit site can
be added without choosing a category.
DatabaseService gains an idempotent migration that adds a nullable
category TEXT column to notification_history; existing rows keep
category=NULL (displayed as Uncategorized in the UI). The
getNotificationHistory method accepts an optional category filter
that is forwarded from the GET /api/notifications/history route via
a ?category= query param.
NotificationPanel gains a category Select dropdown so users can
filter history by category. The frontend types mirror the backend
union so API responses are type-safe end-to-end.
All 75 test files (1410 tests) updated to the new 4-arg dispatchAlert
signature and passing.
Allow a scheduled task with action='update' and target_type='fleet'
to update every eligible stack on a node in a single schedule entry.
The executor respects each stack's per-stack auto-update policy via a
single batch query, skipping stacks that have opted out. For remote
nodes the request proxies to the remote Sencho instance, which already
enforces the same policy in its /api/auto-update/execute endpoint.
Backend route validation now accepts update+fleet as a valid combo
(previously only update+stack was allowed) and requires node_id.
Frontend adds an "Auto-update All Stacks" option to the scheduled-task
creation form with a node selector and descriptive helper text.
* feat(auto-update): add per-stack auto-update enable/disable toggle
Paid users (Skipper and Admiral) can now opt individual stacks out of
scheduled auto-updates from the stack context menu without disabling
the global feature.
- Add stack_auto_update_settings table (node_id, stack_name) with
default enabled=true; four typed DatabaseService accessors with
parameterized queries.
- Add GET /stacks/auto-update-settings, GET /stacks/:name/auto-update,
and PUT /stacks/:name/auto-update (requirePaid + requireAdmin).
PUT broadcasts state-invalidate with action auto-update-settings-changed
so all open tabs refresh immediately.
- Stack DELETE clears the auto-update setting row alongside stack_update_status.
- autoUpdateRouter /execute skips disabled stacks before any registry
call; skip is recorded in the results array. Manual Update actions
are not affected.
- Add Auto-update: Enabled/Disabled toggle in the stack inspect group
(paid tiers only, hidden for Community, consistent with Auto-Heal).
Toggle uses optimistic update with revert-on-error toast.
- AutoUpdateReadinessView shows an Auto: Off pill and disables the
Apply now button for stacks with auto-updates off. Detection still
runs so the readiness card remains visible.
- Add 21 backend Vitest tests covering DB round-trips, endpoint auth
and tier gates, execute skip for both wildcard and named targets.
Add 3 frontend hook tests for toggle visibility and callback behavior.
* docs(auto-update): document per-stack auto-update control
Add a Per-stack control section to the auto-update readiness page
explaining how to disable and re-enable auto-updates for individual
stacks, what disabling means (scheduled apply skipped; detection still
runs; manual update unaffected), and a troubleshooting entry for
scheduled runs not applying to a specific stack.
Group readiness cards by node so updates pending on every reachable node
are visible without having to switch the active node. Apply now targets
the owning node directly, and Recheck fans out to every reachable node
in parallel; per-node cooldowns are surfaced in the toast.
Adds POST /image-updates/fleet/refresh and invalidates the fleet
aggregation cache after auto-update execute so the next read reflects
the new state immediately. A small banner appears under the hero when
some online nodes did not respond within the request timeout.
Previously, fetching the .env file for a stack with no env files at all
returned a 404 with a JSON error body. The frontend's secondary loader
(changeEnvFile) called res.text() without checking res.ok, which caused
the error body to be stuffed directly into the editor as if it were
file content.
Two-part fix:
Backend (routes/stacks.ts):
- For the default GET /stacks/:name/env (no ?file= query) when the
stack has no env files, respond 200 with an empty body and an
X-Env-Exists: false header instead of 404.
- For an explicit ?file= query that resolves to a missing file, keep
the 404 (the caller asked for something specific).
- Catch a TOCTOU ENOENT between access() and readFile() and return the
same friendly empty-body shape, not a generic 500.
Frontend (EditorLayout.tsx::changeEnvFile):
- Check res.ok before reading the body. On a non-OK response, clear the
editor content and surface a friendly toast instead of pasting the
server's JSON error string into the file.
Updates the backend listen port, Vite dev proxy target, Docker EXPOSE,
compose port mapping, .env.example default, GitHub Actions smoke-test
default, healthcheck URLs, and every doc/example reference. Test fixtures
that include example URLs were updated for consistency, though their
assertions are port-agnostic.
The rate-limit value of 3000 in middleware/rateLimiters.ts and the
3000 entry in WEB_UI_PORTS (which detects user containers like Grafana)
are intentionally untouched.
Custom OIDC stays on Community so self-hosters can wire any spec-compliant
OIDC identity provider (Authelia, Keycloak, Authentik, Zitadel, and others).
Google, GitHub, and Okta one-click presets move to Skipper. LDAP / Active
Directory and scoped RBAC are Admiral-only.
Backend enforces the split via a new requireTierForSsoProvider helper in
middleware/tierGates.ts, applied after requireAdmin in all four ssoConfig
mutation handlers. GET /sso/config (list) stays ungated so downgraded
admins can still see previously-configured providers. Invalid provider ids
now 400 before the tier check to avoid leaking tier information.
Frontend adds a compact mode to PaidGate and AdmiralGate for inline
list-item locks, and SSOSection reorders the provider cards as
Custom OIDC > Google > GitHub > Okta > LDAP to reinforce the
free-to-paid progression.
Stale 'SSO is Admiral' copy in AdmiralGate, PaidGate, and the Admiral
upgrade card on the License settings page has been replaced to reflect the
new split. User-facing licensing, SSO, overview, quickstart, and security
docs have been updated with the per-tier provider matrix.
Move the nine /api/nodes/* endpoints out of index.ts into routes/nodes.ts
(list, scheduling-summary, get, create, pilot-enroll, update, delete,
test, meta). mintPilotEnrollment and the REMOTE_META_* constants move
with the router as local helpers.
Handlers moved verbatim. Two safe cleanups applied during the move:
- Inline req.apiTokenScope 403 blocks replaced with the shared
rejectApiTokenScope helper; payload shape unchanged.
- catch (error: any) rewritten to catch (error: unknown) with explicit
instanceof Error narrowing to satisfy the no-any strictness rule.
Response body shapes unchanged.
Removes now-unused imports from index.ts: jwt, crypto, authMiddleware,
isValidRemoteUrl, PilotTunnelManager, PilotCloseCode, CAPABILITIES,
getSenchoVersion, fetchRemoteMeta, RemoteMeta, FleetUpdateTrackerService,
plus the module-scope updateTracker alias.
index.ts drops from 1364 to 1021 lines. Only the stacks group remains
inline for the final phase 4c slice.
Move the six /api/containers/* and /api/ports/in-use endpoints out of
index.ts. Handlers moved verbatim. routes/containers.ts exports two
routers:
- containersRouter (mounted at /api/containers): list, stream logs,
start, stop, restart.
- portsRouter (mounted at /api/ports): /in-use host port inventory.
Removes the now-unused requireAdmin import from index.ts. Middleware
chains, response shapes, and error messages are all preserved.
index.ts drops from 1437 to 1364 lines. Remaining inline groups:
stacks and nodes.
Move the entire /api/security/* surface out of index.ts:
- Trivy lifecycle: status, install, uninstall, update-check, update, auto-update toggle
- Scanning: POST /scan (image), POST /scan/stack (compose)
- Scan queries: list, get, vulnerabilities, secrets, misconfigs, image-summaries, SARIF export
- SBOM generation
- Scan policies CRUD (fleet-replicated; replica writes rejected)
- CVE suppressions CRUD (fleet-replicated; replica writes rejected)
- GET /compare: diff two scans
Handlers moved verbatim. Local CVE_ID_RE, parseScannersInput, and
shapeScanForResponse helpers move with the router. The previously-closure
fetchAll inside the SARIF handler is hoisted to module scope as
fetchAllPages so it is not re-allocated per request.
Removes nine now-unused imports from index.ts (parsePolicyEvaluation,
VulnerabilityScan, FleetSyncService, requireAdmiral, trivyInstallLimiter,
SbomFormat, TrivyInstaller, validateImageRef, applySuppressions,
generateSarif).
index.ts drops from 2130 to 1437 lines. Remaining inline groups:
containers, stacks, and nodes.
Move three more Round C route groups out of index.ts:
- /api/registries/* -> routes/registries.ts
- /api/system/{orphans,prune,docker-df,resources,images,volumes,networks,...} -> routes/systemMaintenance.ts
- /api/templates/* -> routes/templates.ts
Handlers moved verbatim. Middleware chains, response shapes, and error
messages are preserved. Registry scope-denial now uses the shared
rejectApiTokenScope helper (same payload shape).
index.ts drops from 2739 to 2130 lines. Remaining inline groups:
containers, stacks, security (Trivy/scans/SBOM/policies), and nodes.
Move four small Round C route groups out of index.ts:
- /api/auto-heal/* -> routes/autoHeal.ts
- /api/notifications/*, /api/notification-routes/* -> routes/notifications.ts
- /api/system/console-token -> routes/console.ts
- /api/sso/config/* -> routes/ssoConfig.ts
Share NOTIFICATION_CHANNEL_TYPES, cleanStackPatterns, and validateHttpsUrl
via a new helpers/notificationChannels.ts so agents.ts and notifications.ts
consume the same allowlist and URL validator.
Handlers moved verbatim. Middleware chains and response shapes preserved.
index.ts drops from 3231 to 2739 lines.
Wraps up Phase 4 Round B by tackling the two deferred groups. 25 new
integration tests land first and run green against the inline monolith,
then each group is extracted byte-for-byte.
index.ts drops from ~3,678 to ~3,231 lines; test count rises 1,320 → 1,345.
New coverage:
- metrics-routes.test.ts (11) — auth + shape checks for /api/stats,
/api/metrics/historical, /api/system/stats, /api/system/cache-stats
(admin-only), and SSE headers for /api/logs/global/stream
- image-updates-routes.test.ts (14) — auth, admin gating, rate-limit
tolerance on /refresh, fleet aggregation, /auto-update/execute input
validation and no-stacks short-circuit
New route files:
- routes/metrics.ts — /stats, /metrics/historical, /logs/global (+ SSE
/stream), /system/stats, /system/cache-stats. Mounted at /api so the
mixed sub-paths line up.
- routes/imageUpdates.ts — /api/image-updates CRUD + fleet aggregation,
plus a separate autoUpdateRouter mounted at /api/auto-update that
owns the /execute handler. Same split pattern as license.ts +
systemUpdateRouter.
index.ts trims unused imports left behind by the extraction:
globalDockerNetwork, si, STATS_CACHE_TTL_MS, SYSTEM_STATS_CACHE_TTL_MS,
GlobalLogEntry + log-parsing helpers.
Round B of Phase 4. Writes integration tests for three under-covered
route groups BEFORE extracting them, then does the extraction once the
new tests pass against the monolith. index.ts drops from ~4,206 to
~3,678 lines.
New test coverage (42 new assertions):
- settings-routes.test.ts (14) — auth, admin gating, private-key stripping,
allowlist, single-key write, bulk PATCH validation + partial update
- scheduled-tasks-routes.test.ts (18) — list/create/get/toggle/delete/runs,
action+target_type matrix, cron validation, tier gating on non-admin
- agents-routes.test.ts (10) — GET/POST, admin gating, channel type +
HTTPS URL validation, boolean enabled check, upsert semantics
Each suite was verified against the inline monolith first, then the
route extraction was performed byte-for-byte and all suites re-run to
ensure no regression.
New route files:
- routes/settings.ts — GET/POST/PATCH with PRIVATE_SETTINGS_KEYS strip,
ALLOWED_SETTING_KEYS allowlist, and SettingsPatchSchema zod bulk schema
- routes/scheduledTasks.ts — 9 endpoints (list, create, get, update,
delete, toggle, run-now, runs history, runs CSV export). File-local
helpers parseTaskId, validateActionTarget, validateOptionalFields
collapse duplication across create+update handlers. Uses shared
escapeCsvField from utils/csv.ts.
- routes/agents.ts — notification-channel GET/POST. Owns
NOTIFICATION_CHANNEL_TYPES and validateHttpsUrl locally because the
notification-routes block still inlines identical copies; the helpers
will converge once those routes extract in a later slice.
Final slice of Phase 4 Round A. Pulls the four remaining well-tested route
groups out of index.ts. index.ts drops from ~5,930 to ~4,206 lines.
New route files:
- routes/webhooks.ts: /api/webhooks CRUD + HMAC-authenticated trigger.
Uses shared webhookTriggerLimiter. Trigger preserves the raw-body path
established by the conditional JSON parser for HMAC validation.
- routes/users.ts: /api/users CRUD + /:id/mfa/reset + /:id/roles
scoped-assignment surface. Uses rejectApiTokenScope across every
handler, validateUsername helper, BCRYPT_SALT_ROUNDS, and
isSqliteUniqueViolation for the role-assignment UNIQUE guard.
- routes/gitSources.ts: /api/git-sources + /api/stacks/:name/git-source/*.
Exports two routers (gitSourcesRouter + stackGitSourceRouter) because
the per-stack paths need to mount at /api/stacks alongside the label
routes extracted in phase 4a-1. String length limits are now named
constants so the 400 responses stay truthful if the bounds change.
- routes/fleet.ts: /api/fleet role, sync, overview, node drill-down,
update-status + trigger (single + fleet-wide), and snapshot CRUD +
restore. Local parseIdParam helper collapses seven copies of the
parseInt/isNaN route-param pattern.
Bugs fixed during review:
- users.ts :id/roles POST — replace the fragile
(err as Error).message?.includes('UNIQUE constraint') check with
isSqliteUniqueViolation from utils/errors.ts.
index.ts carries forward three symbols (updateTracker alias,
CVE_ID_RE, parseScannersInput) until the corresponding security /
nodes / scan routes get extracted in a later slice.
Second slice of Phase 4. Pulls the three auth-family route groups out of
index.ts into focused routers. All handlers move verbatim; index.ts drops
~845 lines.
New route files:
- routes/auth.ts: /api/auth core (status, setup, login, password, logout,
check, generate-node-token)
- routes/mfa.ts: /api/auth/login/mfa + full /api/auth/mfa/* surface
(status, enroll start/confirm, disable, backup-codes/regenerate,
sso-bypass)
- routes/sso.ts: /api/auth/sso/{providers,ldap,oidc/:provider/authorize,
oidc/:provider/callback} + getSSOBaseUrl helper. Module-load calls
SSOService.getInstance().seedFromEnv() so env-seeded providers are
available on the first request.
Shared lifts:
- helpers/constants.ts: MFA_REPLAY_TTL_MS + MFA_REPLAY_PURGE_INTERVAL_MS
(used by mfa.ts and the startup purge timer in index.ts) and
BCRYPT_SALT_ROUNDS (shared between setup and password-change handlers).
- middleware/auth.ts: new reissueSessionAfterTokenBump(req, res, userId)
helper collapses three copies of "bump → fetch user → re-sign cookie"
across auth.ts (password change) and mfa.ts (enrol confirm, disable).
Code review fixes:
- File-local requireEnrolledMfaUser helper in mfa.ts eliminates four
copies of "auth check + rejectApiTokenScope + load enrolled MFA" with
near-identical shape.
- Applied BCRYPT_SALT_ROUNDS to auth.ts setup + password handlers.
Mount order in index.ts: authRouter / mfaRouter / ssoRouter sit before
authGate because login / setup / SSO-callback are public; handlers that
need auth use authMiddleware directly on the route.
First slice of Phase 4 (route extraction). Pulls 8 well-tested, mostly
independent route groups out of index.ts into focused Router files. No
behavior change; every handler body moves verbatim.
New route files under backend/src/routes/:
- meta.ts /api/health, /api/meta (mounted before authGate)
- license.ts /api/license/* + /api/system/update,
exports scheduleLocalUpdate for the fleet route
- permissions.ts /api/permissions/me
- convert.ts POST /api/convert
- alerts.ts /api/alerts/*
- labels.ts /api/labels/* + PUT /api/stacks/:name/labels
(exported as stackLabelsRouter)
- apiTokens.ts /api/api-tokens/*
- auditLog.ts /api/audit-log/*
Shared helper lifts:
- helpers/cacheInvalidation.ts: invalidateNodeCaches()
- middleware/tierGates.ts: requireBody (was inline in index.ts)
- utils/errors.ts: isSqliteUniqueViolation (was inline in index.ts)
- middleware/apiTokenScope.ts: rejectApiTokenScope() helper (new)
- utils/csv.ts: escapeCsvField() (new)
index.ts drops from ~7520 to ~6775 lines and now mounts the routers right
after enforceApiTokenScope. The remote proxy and fleet/auth/webhooks/users
routes remain inline in index.ts pending later Phase 4 slices.
Code review fixes: rejectApiTokenScope helper replaces duplicated
`if (req.apiTokenScope) 403 SCOPE_DENIED` blocks in apiTokens.ts and
license.ts; escapeCsvField replaces the inline CSV escape in auditLog.ts.
Anso