Commit Graph

8 Commits

Author SHA1 Message Date
Anso 5f1baa7522 fix: harden deploy/update concurrency and node-targeting safety (#1390)
* fix: harden deploy/update concurrency and node-targeting safety

Release stabilization for deploy/update operational safety.

Per-stack operation locking is now global. Background lifecycle paths
(scheduler auto stop/down/start/backup/update, webhook execute, Git source
auto-deploy, image auto-update, label bulk actions, fleet snapshot redeploy,
and mesh redeploy) acquire the per-node, per-stack lock through a new
StackOpLockService.runExclusive helper and skip rather than race a manual
deploy/update/rollback/backup on the same stack and node. Skips surface
honestly (a failed scheduled run, a recorded webhook failure, a per-stack
batch result, or a thrown error) instead of a silent no-op.

Update readiness and policy-bypass now run against the node captured when the
dialog opened, not the live active node, so switching nodes while a dialog is
open cannot retarget the update or the bypass retry.

Rollback readiness no longer presents a moving-tag or unpinned image as a ready
image revert. Restoring files does not revert a moving tag, so those stacks
read as partial, and the rollback success message states that the compose and
env files were restored.

* fix: lock blueprint reconcile against manual ops and correct rollback wording

Follow-up to the deploy/update safety hardening, closing two more gaps from a
verification pass.

BlueprintService.deployLocal and withdrawLocal called ComposeService directly,
so blueprint reconciliation could race a manual deploy/update/rollback/backup on
an owned stack. Both now run their compose lifecycle call through
StackOpLockService.runExclusive and skip (recorded as a failed reconcile,
retried on the next cycle) on conflict. The withdraw holds the lock across both
the compose down and the directory delete so neither races a manual operation.

The runtime rollback messages overstated recovery: a rollback restores the
compose and env files and recreates containers, but does not revert an image
behind a moving tag. The auto-rollback deploy-progress output, the recovery
panel and chip, the failure toasts, and the manual rollback route message now
state that the compose and env files were restored, with the matching OpenAPI
example and atomic-deployments doc updated.

* fix: acquire stack lock before blueprint deploy mutates compose and marker files

Local blueprint deploy wrote the compose and marker files and ran the policy
assert before acquiring the per-stack lock; the lock only wrapped the deploy
itself. A reconcile could therefore rewrite an owned stack's files while a
manual deploy/update/rollback/backup was running. The lock now wraps the whole
critical section (create, write compose, write marker, policy assert, deploy),
so on conflict nothing is written and the reconcile records a failed outcome.

Adds a test asserting a deploy under a held lock records failed, writes no
marker file, and leaves the manual lock untouched.

* fix: make remote blueprint apply atomic under the receiving node's stack lock

Remote blueprint deploy wrote the compose and marker files to the target node
via separate HTTP calls and only locked on the final deploy, so the file writes
could race a manual operation on that node. A node's operation lock is
process-local and cannot be held by the hub across HTTP calls, so the locked
create/write/deploy now runs on the receiving node.

The locked critical section is extracted into BlueprintService.applyLocalUnderLock
and exposed via POST /api/blueprints/apply-local. The hub posts the blueprint to
that endpoint in one call; the receiving node runs create + write compose+marker
+ deploy under its own per-stack lock. Older nodes without the route answer 404
and fall back to the legacy multi-call flow. The endpoint is gated by paid tier
and the same per-stack stack:edit and stack:deploy permissions as the
PUT-compose + deploy it bundles, validates the stack name, compose size, and
marker structure, and returns 409 on a lock conflict without writing anything.

Adds tests for the atomic single-call path, the 404 legacy fallback, the 409
lock-conflict mapping, the route validation and permission paths, and the
write-compose-then-marker-then-deploy ordering of the shared locked apply.

* fix(deps): bump undici to 7.28.0 to clear high-severity advisory

The frontend CI npm audit gate (--audit-level=high) failed on a transitive
undici 7.25.0 (a dev-only dependency via jsdom): TLS certificate validation
bypass (GHSA-vmh5-mc38-953g) and cross-user cache information disclosure
(GHSA-pr7r-676h-xcf6). Bumping undici within jsdom's existing ^7.25.0 range to
7.28.0 clears the high-severity advisory and unblocks the frontend job. Lockfile
only; no direct dependency or source change.
2026-06-18 13:38:19 -04:00
Anso b4cca9b995 fix(blueprints): allow deleting blueprints stuck on awaiting-confirmation deployments (#1313)
* fix(blueprints): allow deleting blueprints stuck on awaiting-confirmation deployments

Withdrawing a stateful deployment removed its row, but the reconciler
re-created it as "Awaiting confirmation" for any node still matching the
selector. The delete guard counted that recreated row as blocking, so a
stateful blueprint could never be deleted: every withdraw came back as a
pending review and delete kept refusing.

The delete guard now blocks only on deployments that have a live stack on
a node (active, drifted, correcting, evict_blocked, or a pending review
that was previously deployed). A never-deployed "Awaiting confirmation"
deployment no longer blocks; the delete path's best-effort withdraw-all
loop and the foreign-key cascade clear those rows. Live stateful
deployments still require explicit withdrawal first, so the
snapshot-vs-destroy choice is always made by the operator.

Updates the delete dialog copy to match and adds delete-guard coverage.

* fix(blueprints): harden delete against destroying unmanaged or live stacks

Address two issues found while auditing the delete guard:

- A never-deployed deployment (for example a reconciler-created pending
  review) no longer blocked delete, but the route's withdraw-all loop
  still ran the withdraw primitive for it. withdrawFromNode proceeds on a
  missing marker, so deleting the blueprint could down and delete an
  unmanaged same-name stack on the node. The loop now withdraws only the
  stacks Sencho deployed and still owns, and skips never-deployed,
  name_conflict, and withdrawn rows.

- A stack that was deployed then failed keeps its last_deployed_at, but
  the guard's status list did not include failed, so it could be deleted
  without the snapshot-vs-destroy choice. The guard now keys on
  last_deployed_at (a stack we deployed and own) rather than a status
  list, so any owned live stack blocks regardless of its current status,
  while never-deployed, name_conflict, and withdrawn rows do not.

Adds delete-guard coverage for these paths, including that withdraw is
never invoked for an unmanaged row and is invoked for an owned one.
2026-06-05 14:29:09 -04:00
Anso 865d792874 feat(pricing): collapse to two tiers (#1309)
* feat(pricing): collapse to two tiers (Community + Admiral)

Collapse Sencho's pricing from three tiers (Community / Skipper / Admiral)
to two: a generous free Community tier and a single paid Admiral tier. The
Skipper tier is removed.

Now free in Community: auto-heal, auto-update, scheduled operations,
webhooks, notification routing, Fleet Actions and bulk operations, SSO
preset providers (Google / GitHub / Okta), unlimited users with admin and
viewer roles, and deploy safety (atomic deploys, auto-rollback, and
one-click rollback).

Admiral (paid) is focused on running and governing a fleet: blueprints,
Fleet Secrets, deploy enforcement, vulnerability report export, audit log,
host console, private registries, mesh networking, node cordon, managed
cloud backup, LDAP / Active Directory SSO, and the advanced RBAC roles
(deployer, node-admin, auditor) with per-resource scoped assignments.

Internally the license variant distinction is removed so tier is binary
(community / paid). License validation still verifies the Lemon Squeezy
store and product before granting paid status.

Docs and the contributor guide are updated to the two-tier model.

* docs(pricing): correct licensing page to two-tier pricing and tidy stale tier wording

The licensing docs page kept the old Admiral pricing plus a Founder
Lifetime column and an Enterprise paragraph after the two-tier collapse.
Update it to $12/month or $99/year, drop the lifetime and Enterprise
content, and link to the pricing page for current pricing.

Also fix stale "Skipper" wording in CLA.md, SUPPORT.md, one test title,
and three test comments. Historical CHANGELOG entries and the
retired-Skipper license-guard test are intentionally left as-is.

* docs: align licensing and SSO pages with the two-tier model

Correct the SSO overview so the Google, GitHub, and Okta presets read as
available on every tier, matching the provider table; only LDAP and Active
Directory require Sencho Admiral. Remove the lifetime-plan references from the
licensing, settings, and troubleshooting pages so they reflect subscription-only
Admiral pricing.

* fix(rbac): omit scoped permissions from /me on the Community tier

Scoped role assignments only take effect on the paid tier, but GET /api/permissions/me returned them unconditionally, so a downgraded instance with leftover assignments rendered per-resource affordances the API then rejected with 403. The endpoint now mirrors the permission middleware and includes scoped permissions only on the paid tier. Adds a regression test covering the downgrade case.

* docs: use custom-pricing wording on the contact page

The two-tier model has no Enterprise tier; reword the contact page's enterprise pricing/deals to custom pricing/deals so it does not imply a tier that no longer exists.
2026-06-04 17:45:53 -04:00
Anso dd2c2b22ec fix(federation): gate node cordon control on node:manage to match backend (#1277)
* fix(federation): gate node cordon control on node:manage to match backend

The cordon and uncordon control on the node card rendered whenever the
instance held an Admiral license, but the backend route also requires the
node:manage permission. Non-admin users without that permission (deployer,
viewer, auditor) saw a Cordon action the API rejected with 403. Gate the
control on the same permission the route enforces, so it renders only for
users who can use it.

Add developer-mode diagnostic logging to the cordon and pin handlers, and
route-level tests covering the cordon and uncordon permission, tier,
API-token, and input-validation boundaries.

* fix(federation): reject non-numeric node ids in cordon/uncordon

The cordon and uncordon routes validated the node id with parseInt, which
accepts numeric-prefix strings (parseInt('1abc', 10) === 1), so a request to
/api/nodes/1abc/cordon would operate on node 1. Require a strict positive
integer before parsing, and add tests for both routes.

* fix(federation): sanitize user-derived values in cordon and pin diagnostics

Route the node id, blueprint id, target node id, and reason length through
the shared log sanitizer before they reach the developer-mode diagnostic
log lines, and switch the format specifiers to %s to match. The values are
already validated integers, so this is defense in depth at the log sink and
keeps the diagnostics on the same sanitize-every-interpolated-value pattern
used elsewhere. No runtime behavior change: the lines stay gated behind the
developer_mode setting, off by default.
2026-06-02 09:48:19 -04:00
Anso 19cdb3681d fix: harden blueprint deployment guardrails (#1027)
* fix: harden blueprint deployment guardrails

* fix: update Docker toolchain to Go 1.26.3

* fix: repair Dockerfile tr argument split across lines

* fix: bump protobufjs to clear npm audit high-severity advisories
2026-05-12 15:49:19 -04:00
Anso 52b46753af feat(fleet): add Federation tab with cordon and pin policy (Admiral) (#964)
Ships the v1 MVP for the Federation tab as placement control, not
placement automation:

- Cordon a node: marks the node unschedulable so the BlueprintReconciler
  skips it for new placements only. Existing deployments continue to
  drift-check and redeploy on revision changes; cordon never triggers
  withdraw or eviction. Toggle on the NodeCard kebab (Admiral, admin
  role); Cordoned pill renders for all tiers.
- Pin a blueprint to a node: stores blueprints.pinned_node_id, replacing
  the desired set with the pinned node regardless of selector. Pin
  overrides cordon by design. Action lives only in the Federation tab;
  BlueprintDetail and the deployment table show read-only Pinned
  indicators.

Backend: idempotent migrations add nodes.cordoned/cordoned_at/cordoned_reason
and blueprints.pinned_node_id. New routes POST /api/nodes/:id/cordon,
POST /api/nodes/:id/uncordon, PUT /api/blueprints/:id/pin, all gated by
requireAdmiral plus requireAdmin. Audit summaries added so the existing
auditLog middleware records every operator action. deleteNode clears
dangling pins.

Reconciler: pin override evaluated before selector match; cordon filter
applied only to the new-placement branch (deploy/stateReview without an
existing deployment). 11 new Vitest cases cover cordon filter, pin
override, pin-overrides-cordon, missing pin target, pin shrinks
desired set (stateless withdraw + stateful evict_blocked), and pin
clearing on node delete.

Frontend: new FederationTab.tsx with cordoned-nodes summary and
pin-policy table. Federation moved out of the experimental flag into
{isAdmiral && (...)} + AdmiralGate, mirroring the Routing tab pattern.
Secrets stays under experimental.

Tests pass: backend tsc, full Vitest suite (1704 passed), frontend
tsc -b, ESLint (0 errors). Manual verification via the local dev
instance confirmed the tab is hidden at Community, the kebab and pill
render at Admiral, and cordon and pin endpoints round-trip end to end.

Refs cut-line-1.0.md Federation v1 MVP.
2026-05-07 05:55:00 -04:00
Anso 7fe90d9f3a feat(blueprints): capture compose snapshot before stateful eviction (#957)
Wire the snapshot_then_evict withdraw mode to actually persist the
blueprint's compose YAML to fleet_snapshots before running the
eviction. The mode previously recorded intent only. Capture failure
aborts the eviction with HTTP 500 rather than silently falling
through to a destructive withdraw.

Volume bytes remain out of scope: the snapshot holds the compose
definition only. UI copy and the Blueprints docs (Withdraw note,
Migrating stateful data section, two new Troubleshooting entries)
clarify that operators must move volumes by hand if they need the
data on another node.

Adds 9 route-level tests covering the success path, snapshot DB
write failure, orphan-row cleanup when insertSnapshotFiles fails,
empty compose_content, evict_and_destroy unchanged, stateless
unchanged, evict_blocked gate, omitted confirm field, and bad
confirm value.
2026-05-06 21:58:45 -04:00
Anso 685d5d729e feat(blueprints): backend foundation for fleet-wide compose templates (#860)
* feat(blueprints): add backend foundation for fleet-wide compose templates

Introduces the Blueprint Model: a docker-compose.yml plus a node selector
(labels or explicit IDs) that Sencho reconciles across the fleet. Backend
foundation only; the frontend tab and documentation follow.

Schema (DatabaseService):
- node_labels table for fleet-level orchestration tagging
- blueprints table with compose content, selector, drift_mode, classification
- blueprint_deployments table for per-node materialized state
- New idempotent migrate methods following the existing pattern

Services:
- BlueprintAnalyzer: pure compose-YAML classifier (stateless / stateful /
  unknown) with 17 covered cases including named volumes, bind mounts,
  external volumes, and tmpfs
- NodeLabelService: label CRUD plus selector matching helper (any/all/ids)
- BlueprintService: local + remote deploy/withdraw orchestration, marker
  file management, name-conflict guard, per-(blueprint,node) lock
- BlueprintReconciler: 60-second loop with three-mode drift policy
  (observe/suggest/enforce), state-aware guards, and Enforce-downgrade for
  volume-destroying drift

Routes (gated requirePaid + requireAdmin on mutations):
- /api/blueprints (CRUD + apply + withdraw + accept + preview + analyze)
- /api/node-labels (CRUD + listAll + listDistinct)

Notifications: four new categories registered in NotificationService for
deploy/failure/drift events.

Bootstrap: reconciler start/stop wired in startup and shutdown.

Tests: 45 new Vitest cases covering selector matching, classifier rules,
state-aware guards, drift-mode branching, and marker parsing. Full backend
suite (1625 tests) passes; tsc clean.

* fix(lint): replace bare Function type in blueprint reconciler tests

Replace 8 occurrences of `as unknown as { computeDecision: Function }`
with a properly typed `ReconcilerWithCompute` alias that mirrors the
real method signature. Export `ReconcileDecision` from
BlueprintReconciler so the test can reference it.

Resolves @typescript-eslint/no-unsafe-function-type errors that were
failing the Backend (Lint) CI step.
2026-05-01 18:57:44 -04:00