feat(cloud-backup): mirror fleet snapshots to S3-compatible storage (#782)

* feat(cloud-backup): mirror fleet snapshots to S3-compatible storage

Add an Admiral-tier Cloud Backup feature that replicates every fleet
snapshot to off-site storage, with two provider modes that share the
same `@aws-sdk/client-s3` code path:

- Sencho Cloud Backup: zero-config, 500 MB allowance backed by
  Cloudflare R2, provisioned via the sencho.io worker against the
  user's Lemon Squeezy license.
- Custom S3 (BYOB): any S3-compatible bucket (AWS, MinIO, Backblaze
  B2, Wasabi, R2 with own keys), with credentials encrypted via
  `CryptoService` before storage.

API-triggered snapshots upload fire-and-forget so the UI returns
immediately; scheduled snapshots block on the upload so the task's
success/failure reflects cloud durability. Object keys include the
instance_id segment to prevent collisions when the same Admiral
license is activated on multiple Sencho instances.

* fix(cloud-backup): drop ES2022-only Error cause arg breaking ES2020 build

The backend tsconfig pins lib to ES2020. The two-argument
`Error(message, { cause })` form requires ES2022, so tsc rejected it
with TS2554. Revert to single-argument throw to match the
convention used elsewhere in the backend services.
This commit is contained in:
Anso
2026-04-26 15:42:21 -04:00
committed by GitHub
parent 801a098a5b
commit 03f91cd5bb
20 changed files with 2805 additions and 401 deletions
+53
View File
@@ -82,6 +82,59 @@ Sencho writes the snapshot's files back to the target node:
Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.
## Cloud Backup
<Note>
Cloud Backup requires an Admiral license. Configure it in **Settings → Cloud Backup**.
</Note>
Cloud Backup mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. Two storage modes are supported.
### Sencho Cloud Backup (included)
A managed 500 MB allowance backed by Cloudflare R2, included with every Admiral license. Open **Settings → Cloud Backup**, choose **Sencho Cloud Backup**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically. The settings panel shows your storage usage and lets you reprovision credentials if needed.
### Custom S3 (BYOB)
Bring any S3-compatible bucket: AWS S3, MinIO, Backblaze B2, Wasabi, or your own Cloudflare R2 token. Choose **Custom S3** in the storage-mode dropdown and fill in:
- **Endpoint URL** (e.g. `https://s3.us-east-1.amazonaws.com`, `https://my-minio.example.com:9000`)
- **Region** (e.g. `us-east-1`, or `auto` for R2)
- **Bucket** name
- **Path Prefix** (default `sencho/`)
- **Access Key ID** and **Secret Access Key**
- **Auto-upload** toggle
Click **Test** to verify connectivity, then **Save**. Secret keys are encrypted at rest. Sencho only sends them to your configured endpoint.
### Manual upload vs auto-upload
When auto-upload is on, every fleet snapshot is replicated as soon as it is created. Manual snapshots from the **Fleet → Snapshots** view upload asynchronously so the UI returns immediately; scheduled snapshots block on the upload so the task's success status reflects cloud durability.
To upload a single snapshot on demand, open the **Snapshots** tab in Fleet View. Each row that hasn't been mirrored yet shows a cloud-upload action next to the **View** button. Once a snapshot is in the cloud, a small cloud icon appears next to its description.
### Browsing and downloading cloud snapshots
The **Cloud Snapshots** panel in **Settings → Cloud Backup** lists every archive currently in your bucket, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery. Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack.
### Restoring from a cloud snapshot
For in-place rollback, use the **Restore** action on the snapshot detail view as described above; the local copy is the source of truth for live restore. Cloud snapshots cover the disaster-recovery case where the local disk is gone: download the archive, extract it, and bring up a fresh Sencho instance pointed at the recovered files.
### Cloud Backup troubleshooting
#### Bad credentials
If **Test** reports an authentication error, double-check the Access Key ID, Secret Access Key, and bucket. Some providers require you to enable S3-compatible API access on the bucket separately. For MinIO, verify the user has read/write permission on the target bucket.
#### Over quota (Sencho Cloud Backup)
Sencho Cloud Backup has a 500 MB allowance per license. When you hit the cap, new uploads fail with a quota error. Delete older cloud snapshots from the **Cloud Snapshots** panel to free space. The local copies are unaffected.
#### Network timeout or 5xx error
Transient errors surface as a notification. Retry by clicking the cloud-upload action on the snapshot row, or wait for the next scheduled snapshot which will retry on its own. Persistent failures usually indicate an endpoint outage; verify the storage provider is reachable from your Sencho host.
## Access control
| Action | Admin | Node Admin | Deployer | Auditor | Viewer |
Binary file not shown.

After

Width:  |  Height:  |  Size: 200 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 166 KiB