feat: add Admiral Hardened Build channel and business assurance surfaces (#1629)

* feat: add Admiral Hardened Build channel and business assurance surfaces

Introduce Studio Saelix entitlement-backed Hardened Build switching, a
single-flight image operation coordinator, Recovery Vault naming, Admiral
Account settings, and typed Fleet update failures while preserving Community
custom-repo and targetless pull-current updates.

* fix: harden image-op paths and clear CI CodeQL/pilot flake

Validate operation IDs before filesystem use, use hostname checks in Fleet
fetch mocks, sanitize registry probe logs, and swallow expected TCP teardown
errors in the pilot reverse-route post-handshake test.

* fix: sanitize image-op docker config write and probe logs

Allowlist-copy registry host keys and base64 auth before writing the
temp DOCKER_CONFIG, and log registry probe failures with a fixed message
so CodeQL no longer flags network-to-file and log-injection mediums.

* fix: address Admiral Hardened Build audit blockers

Expose imageChannel so hardened Fleet peers still POST for typed rejection, claim community updates before 202, terminalize helper failures, gate Hardened on paid, and align support/docs/e2e wording.

* fix: terminalize image ops on helper survival and aborted claims

* fix: prevent recreating persist from overwriting helper-exit failure

* test: assert helper-exit failure lands before recreating persist

* fix: keep current pointer when acknowledging a stale image operation
This commit is contained in:
Anso
2026-07-14 10:47:54 -04:00
committed by GitHub
parent 8ca8ebaa24
commit 381ed2a91f
54 changed files with 2302 additions and 125 deletions
+1 -1
View File
@@ -361,7 +361,7 @@ See [Vulnerability scanning](/features/vulnerability-scanning).
- `error`/`system`: `Scheduled task "<name>" (<action>) failed: <err>`
- `info`/`system` recovery: `Scheduled task "<name>" (<action>) recovered successfully`
### Cloud Backup upload failure
### Recovery Vault upload failure
`warning`/`system`: `Cloud backup failed for scheduled snapshot <id>: <message>`. See [Fleet backups](/features/fleet-backups).
+1 -1
View File
@@ -189,7 +189,7 @@ Regardless of scope, every API token is rejected from the following routes with
| **Registry credentials** | View, create, update, delete, test registry creds |
| **Host-console session token** | Mint the short-lived ticket the browser console uses |
| **API token self-management** | Create, list, revoke API tokens |
| **Cloud Backup** | Configure or trigger Sencho Cloud Backup |
| **Recovery Vault** | Configure or trigger Recovery Vault |
The rationale is the same in every case: a programmatic credential should not be able to grant itself more authority, change the trust roots that issued it, or open an interactive shell. Those actions require a live human user logged into a browser.
+1 -1
View File
@@ -46,7 +46,7 @@ Expanding a row in the Table view reveals additional detail:
- Fleet replica role changes (re-anchor, demote to control)
- Fleet Secrets: create, update, delete, import-from-stack, push (and push preview)
- Blueprint federation pin updates
- Sencho Cloud Backup: config update, connection test, provisioning, snapshot upload, snapshot deletion
- Recovery Vault: config update, connection test, provisioning, snapshot upload, snapshot deletion
- User management: create, update, delete, role assignment and removal
- Password changes and node token generation
- License activation and deactivation
+2 -2
View File
@@ -78,7 +78,7 @@ The table paginates at eight rows; chevrons appear in the header along with an `
The Configuration Status card is the at-a-glance audit of every toggleable automation and security feature on the active node, so nothing is silently off when you expect it to be on.
<Frame>
<img src="/images/dashboard/configuration-status.png" alt="Configuration Status card with four sections: Notifications (Notification agents, Alert rules, Notification routing all reading None), Automation (Auto-heal None, Auto-update '1 / 1', Webhooks None, Scheduled tasks '2 actives'), Security (MFA Off, SSO Off, Vulnerability scanning None), and Backups & Thresholds (Cloud Backup 'Sencho Cloud', Alert thresholds 'CPU 100% · RAM 100% · Disk 100%', Crash detection On)." />
<img src="/images/dashboard/configuration-status.png" alt="Configuration Status card with four sections: Notifications (Notification agents, Alert rules, Notification routing all reading None), Automation (Auto-heal None, Auto-update '1 / 1', Webhooks None, Scheduled tasks '2 actives'), Security (MFA Off, SSO Off, Vulnerability scanning None), and Backups & Thresholds (Recovery Vault active, Alert thresholds 'CPU 100% · RAM 100% · Disk 100%', Crash detection On)." />
</Frame>
The card is divided into four sections.
@@ -112,7 +112,7 @@ The card is divided into four sections.
| Row | What it shows |
|-----|---------------|
| **Cloud Backup** | The active cloud backup target: `Sencho Cloud` (Admiral only), `Custom S3` (with ` (auto)` appended when auto-upload is enabled), or `Disabled` |
| **Recovery Vault** | The active recovery target: `Recovery Vault` (Admiral), `Custom S3` (with ` (auto)` appended when auto-upload is enabled), or `Disabled` |
| **Alert thresholds** | The current host thresholds, formatted `CPU x% · RAM y% · Disk z%`, or `Off` when host threshold alerts are disabled |
| **Crash detection** | `On` when global container-crash notifications are enabled, `Off` otherwise |
+10 -10
View File
@@ -99,22 +99,22 @@ Each stack is restored independently: if one stack cannot be restored (for examp
Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.
## Cloud Backup
## Recovery Vault
<Note>
Custom S3-compatible storage is available on every tier. Sencho Cloud Backup is an Admiral feature. Configure either in **Settings → Infrastructure → Cloud Backup**.
Custom S3-compatible storage is available on every tier. Recovery Vault is an Admiral feature. Configure either in **Settings → Infrastructure → Recovery Vault**.
</Note>
Cloud Backup mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. The Cloud Backup settings page (reached via **Settings → Infrastructure → Cloud Backup**) shows a header with your current scope, provider, storage used, and total snapshot count in the cloud. Two storage modes are supported.
Recovery Vault mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. The Recovery Vault settings page (reached via **Settings → Infrastructure → Recovery Vault**) shows a header with your current scope, provider, storage used, and total snapshot count in the cloud. Two storage modes are supported.
### Sencho Cloud Backup (included)
### Recovery Vault (included)
A managed 500 MB allowance backed by Cloudflare R2, included with every Admiral license. Open **Settings → Infrastructure → Cloud Backup**, choose **Sencho Cloud Backup (Included)**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically.
A managed 500 MB allowance, included with every Admiral subscription. Open **Settings → Infrastructure → Recovery Vault**, choose **Recovery Vault (Included)**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically.
Once active, the settings page shows a storage gauge (used / 500 MB and object count), a status message confirming auto-upload is on, and a **Reprovision** button to refresh credentials if needed. You can verify connectivity at any time with the **Test** button.
<Frame>
<img src="/images/cloud-backup/cloud-backup-active.png" alt="Cloud Backup settings page with Sencho Cloud Backup active, showing the storage gauge, auto-upload status, and the Cloud Snapshots list" />
<img src="/images/cloud-backup/cloud-backup-active.png" alt="Recovery Vault settings page with Recovery Vault active, showing the storage gauge, auto-upload status, and the Cloud Snapshots list" />
</Frame>
### Custom S3 (BYOB)
@@ -142,7 +142,7 @@ To upload a single snapshot on demand, open the **Snapshots** tab in Fleet View.
### Browsing and downloading cloud snapshots
The **Cloud Snapshots** panel in **Settings → Infrastructure → Cloud Backup** lists every archive currently in your bucket, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery. Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack. When the snapshot preserved Dossier notes, the archive also includes a `documentation.json` with those notes.
The **Cloud Snapshots** panel in **Settings → Infrastructure → Recovery Vault** lists every archive currently in your bucket, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery. Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack. When the snapshot preserved Dossier notes, the archive also includes a `documentation.json` with those notes.
### Restoring from a cloud snapshot
@@ -152,7 +152,7 @@ For in-place rollback, use the **Restore** action on the snapshot detail view as
Every fleet snapshot action requires the **admin** role: viewing the snapshot list, browsing snapshot contents, creating, restoring, deleting, and uploading to the cloud. Because a snapshot captures the `.env` file of every stack, the snapshot list and detail views are restricted to administrators rather than read-only roles.
Cloud Backup configuration is also admin-only. Mirroring to Sencho Cloud Backup additionally requires an Admiral license; a Custom S3-compatible target works on every tier.
Recovery Vault configuration is also admin-only. Mirroring to Recovery Vault additionally requires an Admiral subscription; a Custom S3-compatible target works on every tier.
## Storage
@@ -173,11 +173,11 @@ Snapshots are stored in Sencho's SQLite database. Captured file contents, includ
<Accordion title="Restore all reports that some stacks failed">
Restore all applies each stack independently, so a failure on one stack does not stop the others. A stack is reported as failed when its node has been removed from the fleet since the snapshot was taken, when a remote node is offline or unreachable, or when its existing files cannot be written. The successful stacks are fully restored regardless. Resolve the underlying cause (re-add a removed node, bring an offline node back online) and run Restore all again, or restore the remaining stacks individually from the same snapshot.
</Accordion>
<Accordion title="Cloud Backup Test reports an authentication error">
<Accordion title="Recovery Vault Test reports an authentication error">
Double-check the Access Key ID, Secret Access Key, and bucket name; one wrong character is the most common cause. Some providers require S3-compatible API access to be enabled on the bucket separately from the credentials. For MinIO, confirm the user has read/write permission on the target bucket. After correcting the values, click **Test** again before saving.
</Accordion>
<Accordion title="Cloud uploads fail after hitting the 500 MB quota">
Sencho Cloud Backup carries a 500 MB allowance per license. Once you hit the cap, new uploads fail with a quota error and the storage gauge in the settings page reads at or near 500 MB. Free space by deleting older archives from the **Cloud Snapshots** panel in **Settings → Infrastructure → Cloud Backup**. Local snapshots are unaffected by cloud deletions, so the in-place restore path stays intact. To raise the ceiling, switch the storage mode to **Custom S3 (BYOB)** and point at a bucket you control.
Recovery Vault carries a 500 MB allowance per subscription. Once you hit the cap, new uploads fail with a quota error and the storage gauge in the settings page reads at or near 500 MB. Free space by deleting older archives from the **Cloud Snapshots** panel in **Settings → Infrastructure → Recovery Vault**. Local snapshots are unaffected by cloud deletions, so the in-place restore path stays intact. To raise the ceiling, switch the storage mode to **Custom S3 (BYOB)** and point at a bucket you control.
</Accordion>
<Accordion title="Cloud upload returns a network timeout or 5xx error">
Transient errors surface as a notification and leave the local snapshot in place. Retry by clicking the cloud-upload action on the snapshot row, or wait for the next scheduled snapshot which retries on its own. Persistent failures usually indicate an endpoint outage; verify the storage provider is reachable from your Sencho host (custom S3 endpoints often sit behind a different DNS or firewall path than the rest of your traffic).
+8 -7
View File
@@ -49,15 +49,16 @@ See [the pricing page](https://sencho.io/pricing) for current pricing.
**Admiral** is the official business assurance plan from Studio Saelix. It includes everything in Community, plus:
- **Assurance and support:** priority email support and Studio Saelixbacked continuity for production fleets
- **Assurance and support:** priority email support and Studio Saelix-backed continuity for production fleets
- **Governance:** advanced RBAC roles (Deployer, Node Admin, Auditor), scoped permissions per stack or node, and audit log export (CSV, JSON), anomaly detection, and configurable retention beyond the recent window
- **Managed continuity:** Sencho Cloud Backup (a managed, off-site snapshot allowance)
- **Managed continuity:** Recovery Vault (a managed, off-site snapshot allowance)
- **Hardened Build:** an Admiral image channel with published supply-chain assurance artifacts
- **Directory integration:** LDAP / Active Directory authentication
- **Current plan availability:** some product surfaces (including AWS ECR credentials) still require an Admiral plan today. That access rule is temporary availability, not the reason Admiral exists. Other operator surfaces may be limited-availability on a given instance and are documented on their own feature pages when enabled.
## Free trial
Sencho offers a **14-day Admiral trial** so you can evaluate Admiral assurance (priority support path, managed Cloud Backup, governance depth including advanced RBAC, LDAP / Active Directory, and audit export) with your real infrastructure before committing. The trial is offered on the monthly and annual Admiral plans.
Sencho offers a **14-day Admiral trial** so you can evaluate Admiral assurance (priority support path, Recovery Vault, Hardened Build, governance depth including advanced RBAC, LDAP / Active Directory, and audit export) with your real infrastructure before committing. The trial is offered on the monthly and annual Admiral plans.
To start a trial:
@@ -68,7 +69,7 @@ To start a trial:
5. Your license key is emailed to you within a few minutes.
6. Activate the key in the Sencho dashboard as described in [Activating your license](#activating-your-license).
When your trial ends, the card you provided is automatically charged and your plan continues as a paid Admiral subscription. To avoid being charged, cancel from the **Manage subscription** button in **Settings → License** (or from your receipt email) any time before day 14.
When your trial ends, the card you provided is automatically charged and your plan continues as a paid Admiral subscription. To avoid being charged, cancel from the **Manage subscription** button in **Settings → Admiral Account** (or from your receipt email) any time before day 14.
<Tip>
Fresh installs land on the **Community** tier until you activate a license key. All Community features (unlimited nodes, compose editor, global logs, alerts, Custom OIDC, and more) are available immediately.
@@ -76,7 +77,7 @@ When your trial ends, the card you provided is automatically charged and your pl
## Activating your license
Open **Settings → License** in the Sencho dashboard. When the license is not active, the **Activate** section sits below the Plan card:
Open **Settings → Admiral Account** in the Sencho dashboard. When the license is not active, the **Activate** section sits below the Plan card:
<Frame>
<img src="/images/licensing/license-activate-section.png" alt="The Activate section of the License page on a Community-tier instance, with a License key input field and an Activate button on the right" />
@@ -89,7 +90,7 @@ Sencho validates the key and activates your plan. If activation fails, the toast
## The Plan section
When a license is active, **Settings → License** opens on the **Plan** section. The page masthead at the top exposes three stat pills, and the section below lists the metadata for the active license.
When a license is active, **Settings → Admiral Account** opens on the **Plan** section. The page masthead at the top exposes three stat pills, and the section below lists the metadata for the active license.
<Frame>
<img src="/images/licensing/license-admiral-active.png" alt="License page showing an active Sencho Admiral license, with the Plan section listing the Customer, Product, and masked License key fields and a Deactivate button" />
@@ -154,7 +155,7 @@ For details on how this works, see [License enforcement across nodes](/features/
To transfer your license to a different instance:
1. Go to **Settings → License**.
1. Go to **Settings → Admiral Account**.
2. Click **Deactivate** in the Plan section.
3. On your new instance, paste the same license key into **License key** and click **Activate**.
+2 -2
View File
@@ -197,7 +197,7 @@ When you select a remote node in the switcher, the Settings hub filters to the p
| Support | Per browser | Diagnostics bundle, docs links, contact channels. |
| About | Per browser | Build metadata for whichever instance the page is loaded from. |
Panels that manage control-plane concerns (Account, License, Users, SSO, API Tokens, Registries, Cloud Backup, Nodes, Routing, Webhooks) are hidden when a remote node is active.
Panels that manage control-plane concerns (Account, Admiral Account, Users, SSO, API Tokens, Registries, Recovery Vault, Nodes, Routing, Webhooks) are hidden when a remote node is active.
## License enforcement across nodes
@@ -325,6 +325,6 @@ Sencho takes the opposite approach: infrastructure-level encryption (VPN, revers
The control instance's license tier is authoritative for proxied requests, so a Community control plane gates Admiral features on every remote, even if the remote itself has its own Admiral license. Activate an Admiral license on the control instance to lift the gate fleet-wide. The reverse case (Admiral control plane, Community remote) works automatically because the control plane's tier is what the remote trusts.
</Accordion>
<Accordion title="Settings panels disappear when I switch to a remote">
That is intentional. Account, License, Users, SSO, API Tokens, Registries, Cloud Backup, Nodes, Routing, and Webhooks are control-plane concerns and are hidden while a remote node is selected. Switch back to **Local** from the node switcher to manage them. The full list of which panels are per-node, per-browser, and control-plane-only lives in the [What Settings apply per node](#what-settings-apply-per-node) table above.
That is intentional. Account, Admiral Account, Users, SSO, API Tokens, Registries, Recovery Vault, Nodes, Routing, and Webhooks are control-plane concerns and are hidden while a remote node is selected. Switch back to **Local** from the node switcher to manage them. The full list of which panels are per-node, per-browser, and control-plane-only lives in the [What Settings apply per node](#what-settings-apply-per-node) table above.
</Accordion>
</AccordionGroup>
+8 -8
View File
@@ -16,7 +16,7 @@ Open the Settings Hub by clicking the **Profile** icon in the top bar and select
|-------|----------------|
| **Personal** | Account, Appearance |
| **Access** | License, Users, SSO, API Tokens |
| **Infrastructure** | Nodes, Fleet, Registries, Cloud Backup, App Store, Stacks |
| **Infrastructure** | Nodes, Fleet, Registries, Recovery Vault, App Store, Stacks |
| **Monitoring** | Host Alerts, Container Alerts, Docker & Storage |
| **Notifications** | Channels, Notification Routing |
| **Automation** | Image update checks, Webhooks |
@@ -37,7 +37,7 @@ Every section renders inside the same masthead-and-sidebar layout. The masthead
| **SCOPE** `operator` / `browser` / `global` | Setting applies to your account (`operator`), to this browser only (`browser`, for browser-local sections such as Appearance), or to the whole instance (`global`, every other non-node group) |
| **NODE** `<node name>` | Setting is per-node and is currently being edited against this node |
| **EDITED** `<count>` pending / `saved` | The current section has unsaved changes |
| Section-specific stats | Each section can publish its own pills: `2FA on`/`off` and `BACKUP <n> left` (Account); `PLAN`, `TRIAL <n>d left`, `RENEWS`, `STATUS` (License); `OPERATORS` (Users); `CHANNELS` (Channels); `ROUTES` (Notification Routing); `WEBHOOKS` and `ENABLED` (Webhooks); `LABELS` (Labels); `PROVIDER`, `USED`, `SNAPSHOTS` (Cloud Backup); `DEV MODE` (Developer Diagnostics) |
| Section-specific stats | Each section can publish its own pills: `2FA on`/`off` and `BACKUP <n> left` (Account); `PLAN`, `TRIAL <n>d left`, `RENEWS`, `STATUS` (Admiral Account); `OPERATORS` (Users); `CHANNELS` (Channels); `ROUTES` (Notification Routing); `WEBHOOKS` and `ENABLED` (Webhooks); `LABELS` (Labels); `PROVIDER`, `USED`, `SNAPSHOTS` (Recovery Vault); `DEV MODE` (Developer Diagnostics) |
### Quick search
@@ -328,18 +328,18 @@ See [Private Registries](/features/private-registries) for the full walkthrough.
---
## Cloud Backup
## Recovery Vault
<Note>
Custom S3-compatible storage is available on every tier. Sencho Cloud Backup is an Admiral feature.
Custom S3-compatible storage is available on every tier. Recovery Vault is an Admiral feature.
</Note>
**Scope:** Global, admin-only
Mirror fleet snapshots to Sencho Cloud Backup or any S3-compatible storage so a control-plane loss does not take the recovery history with it. The masthead publishes a **PROVIDER** pill (`sencho` / `s3` / `disabled`), a **USED** pill showing storage consumption, and a **SNAPSHOTS** pill with the cloud object count.
Mirror fleet snapshots to Recovery Vault or any S3-compatible storage so a control-plane loss does not take the recovery history with it. The masthead publishes a **PROVIDER** pill (`sencho` / `s3` / `disabled`), a **USED** pill showing storage consumption, and a **SNAPSHOTS** pill with the cloud object count.
<Frame>
<img src="/images/settings/settings-cloud-backup.png" alt="Cloud Backup section with Sencho Cloud Backup mode selected" />
<img src="/images/settings/settings-cloud-backup.png" alt="Recovery Vault section with Recovery Vault mode selected" />
</Frame>
### Storage modes
@@ -347,7 +347,7 @@ Mirror fleet snapshots to Sencho Cloud Backup or any S3-compatible storage so a
| Mode | When to pick it |
|------|-----------------|
| **Disabled** | Snapshots stay on the control plane's local disk only. Default for fresh installs. |
| **Sencho Cloud Backup (included)** | Replicate snapshots to managed storage that ships with the Admiral subscription. On first selection, click **Activate** to provision the 500 MB allowance (backed by Cloudflare R2, scoped to this Admiral license). Auto-upload is always on for this mode. The **Test** and **Reprovision** buttons appear on the activated card if you need to verify connectivity or replace the credentials. |
| **Recovery Vault (included)** | Replicate snapshots to managed storage that ships with the Admiral subscription. On first selection, click **Activate** to provision the 500 MB allowance. Auto-upload is always on for this mode. The **Test** and **Reprovision** buttons appear on the activated card if you need to verify connectivity or replace the credentials. |
| **Custom S3 (BYOB)** | Point Sencho at your own S3-compatible bucket (AWS S3, Cloudflare R2, MinIO, Backblaze B2). Required fields: **Endpoint URL**, **Region**, **Bucket**, **Path Prefix** (default `sencho/`), **Access Key ID**, **Secret Access Key**. The secret access key is masked after save. Use **Test connection** to verify credentials, then toggle **Auto-upload** to start replicating snapshots. |
The **Cloud Snapshots** list at the bottom of the section is an inventory of objects currently in the configured destination. Each row exposes a **Download** icon and a **Delete** icon.
@@ -691,7 +691,7 @@ Links to help resources, with an additional channel for Admiral operators.
| Channel | Tier | Description |
|---------|------|-------------|
| **Priority Email Support** | Admiral | Direct email support with responses within 24 hours. |
| **Priority Email Support** | Admiral | Direct email support during business hours (MonFri 09:0017:00 America/New_York). We aim to first-respond within one business day. This is an operational target, not an SLA. |
Community operators see an upgrade callout with a link to the pricing page in place of the support channels.