mirror of
https://github.com/Studio-Saelix/sencho.git
synced 2026-08-14 20:53:14 +00:00
Anso
f5178889eb
* feat(recovery): capture complete authored Compose project for atomic rollback Replace the root-compose-only backup slot with staged recovery generations that record the managed inventory, exact Compose invocation, and prior image identity, and wire the same engine through deploy, update, manual rollback, and Git apply. * fix(recovery): satisfy CodeQL path barriers and update-guard mock Inline resolve+startsWith checks at generation/inventory fs sinks and stub getCurrentStackUpdateRecovery in UpdateGuardService tests. * fix(recovery): drop unused FileSystemService import in generation store test * fix(recovery): harden authored-project rollback for upgrade and restore safety Preserve legacy UUID backup rows, restore Git deploy state with files, make multi-file restore recoverable, evaluate policy on the restored target, and fail closed when Git capture cannot cover an apply. * fix(recovery): unblock Git apply unit tests and CodeQL pre-restore TOCTOU Mock recovery capture in git-source-service tests after fail-closed apply capture, and re-resolve live paths immediately before pre-restore snapshot reads. * fix(recovery): fall back to authored inventory when Git manifesto is missing First Git apply captures before promote, so a missing managed-project manifesto must not block rollback capture when the live stack already has authored files. * fix(recovery): make authored-project rollback atomic across Git state Restore the managed-project manifesto with files, keep nullable Git identity on first-apply captures, persist Git side-state in restore intents for startup reconcile, compensate legacy materialize failures, and refuse directory collisions before mutation. * fix(recovery): satisfy CodeQL path and TOCTOU barriers on manifesto restore Add inline resolve barriers for manifesto read/clear sinks and remove the access-then-read race when restoring a generation manifesto snapshot. * fix(recovery): close third-audit rollback generation blockers Fail closed on incomplete Git inventory fallbacks, execute captured Compose invocation during recovery, refuse startup and mutations while restore intents remain unresolved, propagate legacy stale-delete failures, and add Docker-level exact prior-image coverage plus regression tests. * fix(recovery): mark acquired before handoff in prior-image Docker test Match the production updateStack CAS sequence so the exact prior-image integration test does not fail handoff from the captured phase. * fix(recovery): close fourth-audit rollback safety blockers Evaluate policy against held images, use index-based pre-restore snapshots, hold the shared stack lock across Git apply, replay Mesh and empty captured invocations exactly, restore POSIX modes with fail-closed sensitive permissions, keep case-sensitive paths, and link Git auto-deploy health gates. Add regression coverage for these cases. * test(recovery): fix mocks for health-gate link and authored compose args Add linkGateOrRetain to the Git apply recovery mock, and mock authoredComposeArgs so the case-collision inventory test is not masked by a missing getComposeDir stub. * fix(recovery): close fifth-audit rollback safety blockers Share git_apply locking for webhook auto-apply, fail closed on malformed recovery service records, refuse mixed-image capture, and require exact probe counts with hold-tag eligibility checks. * fix(recovery): close sixth-audit rollback safety blockers Preserve the legacy backup slot during generation capture, encrypt sensitive pre-restore snapshots, revert files on a failed health probe without committing Git, fail closed when an absent-file revert would delete a directory, skip Compose one-offs, route manual and scheduled backup through the current generation, and persist runtime image platform identity. * fix(recovery): close seventh-audit rollback safety blockers Fleet snapshot restore and restore-all now capture a recovery generation under the stack lock before any authored file write, including on remote nodes. * fix(recovery): keep pre-deploy generations during health-gate observe Link deploy recovery generations to the observing gate so backup cannot replace them mid-observe. Distinguish missing hold tags from probe failures, refuse generation release when services metadata is corrupt, classify mixed-replica and coverage refusals, and toast the backend rollback message. * fix(recovery): wrap webhook deploy case for eslint const bindings in an unbraced switch case trip no-case-declarations. Match the pull case block.
216 lines
22 KiB
Plaintext
216 lines
22 KiB
Plaintext
---
|
|
title: Fleet-Wide Backups
|
|
sidebarTitle: Fleet backups
|
|
description: Snapshot compose files across all nodes for disaster recovery and auditing.
|
|
---
|
|
|
|
<Note>
|
|
Manual and scheduled fleet snapshots are available on every tier (admin role required).
|
|
</Note>
|
|
|
|
Create point-in-time snapshots of every `compose.yaml` and `.env` file across your entire fleet, local and remote nodes alike. Snapshots are stored centrally in Sencho's database and can be browsed, previewed, and restored at any time.
|
|
|
|
## Creating a snapshot
|
|
|
|
1. Navigate to **Fleet** and select the **Snapshots** tab
|
|
2. Click **Create Snapshot**
|
|
3. An inline form appears above the snapshot list with an optional description field (e.g. "Before v2 migration")
|
|
4. Click **Create** to capture files from every reachable node
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/create-snapshot.png" alt="Fleet Snapshots tab with the inline create form open above the table: a description textbox and Create and Cancel buttons" />
|
|
</Frame>
|
|
|
|
During creation, Sencho connects to each node in parallel:
|
|
- **Local nodes** are read directly from the compose directory
|
|
- **Remote nodes** are fetched via the Distributed API proxy using the node's API token
|
|
|
|
If a remote node is offline or unreachable, it is **skipped gracefully**. The snapshot still captures data from all reachable nodes, and skipped nodes are recorded with the reason for the failure. Individual stacks that could not be captured (for example, a compose file that failed to read, or a file too large to store) are recorded the same way, so a snapshot is never silently incomplete.
|
|
|
|
### Scheduled snapshots
|
|
|
|
Fleet snapshots can also be created automatically on a recurring schedule. Navigate to the **Schedules** view, create a new scheduled operation, and select **Create Fleet Snapshot** (grouped under the **Backups** category) as the action type. Scheduled snapshots appear in the snapshot list with a "Scheduled snapshot" prefix in their description.
|
|
|
|
### Triggered from an update
|
|
|
|
The stack update flow surfaces fleet snapshot coverage too. Before an admin updates or rebuilds a stack, the [Update Readiness](/features/health-gated-updates) dialog shows how long ago the most recent fleet snapshot covering that stack was taken, or that none exists yet, and offers a **Create a fleet snapshot before updating** checkbox. Checking it captures a fresh snapshot as part of proceeding; if the snapshot fails, the update does not start. These pre-update snapshots appear in the snapshot list with a "Pre-update snapshot: `<stack>`" description, alongside manual and scheduled ones.
|
|
|
|
The [Storage Portability](/features/compose-storage) tab surfaces the same coverage for any stack with persistent volumes: a warning card and a **Take a fleet snapshot →** link appear when the stack has no fleet snapshot in the last 7 days, or the relative time since the last one otherwise.
|
|
|
|
### Capturing stack documentation
|
|
|
|
Snapshots can also preserve each stack's [Dossier](/features/stack-dossier) notes (purpose, owner, access URLs, firewall and recovery notes, and the rest) alongside the captured files, so a recovery restores the operational knowledge around a stack, not just its configuration.
|
|
|
|
This is off by default. Turn it on in **Settings → Infrastructure → Fleet** with **Capture stack documentation in snapshots**. Once enabled, every manual and scheduled snapshot records the current Dossier notes for each captured stack. Stacks with no notes are simply omitted, and a stack's notes are never required. Secret values are never included: a Dossier only ever holds the notes an operator typed, never the contents of a `.env`.
|
|
|
|
## Browsing snapshots
|
|
|
|
The snapshot list shows each snapshot in a table with the following columns:
|
|
|
|
- **Date** - when the snapshot was taken
|
|
- **Description** - your optional label, or a prefix like "Scheduled snapshot" for automated ones. If Recovery Vault (or a custom off-site target) is configured, an upload icon in this column marks snapshots that have been mirrored off-site.
|
|
- **Scope** - how many nodes and stacks were captured (e.g. "3 nodes, 21 stacks")
|
|
- **Warnings** - a warning icon with a count if any nodes or stacks were skipped, or "None"
|
|
- **Actions** - **View** to open the detail view, a cloud-upload icon for snapshots not yet mirrored to a configured Recovery Vault or custom off-site target, and a delete button for admins
|
|
|
|
The list paginates at 10 snapshots per page; use the arrows next to **Create Snapshot** to page through history.
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/browse-snapshots.png" alt="Fleet Snapshots list paginated 1 of 5, with rows for a manual snapshot, a Pre-update snapshot: dozzle row, and several Scheduled snapshot: Nightly Snapshot rows, each showing date, scope, warnings, and View, cloud-upload, and delete actions" />
|
|
</Frame>
|
|
|
|
## Snapshot detail view
|
|
|
|
Click **View** on any snapshot to open the detail view. Use **Back to Snapshots** in the top-left to return to the list.
|
|
|
|
The header shows the snapshot's title (or "Untitled Snapshot"), who created it, when, and badge counts for nodes and stacks captured. When the snapshot preserved Dossier notes, a **Documentation captured** badge appears alongside those counts.
|
|
|
|
Below the header, each node appears as a collapsible card. Expand a node to see its stacks, then expand a stack to see individual files. If documentation was captured for a stack, its preserved Dossier notes are shown read-only beneath the file list. Each file has a **Preview** button that renders the full file contents inline in a scrollable panel, and a **Download** button that saves that single file to your machine (named `<stack>-<filename>`, for example `web-compose.yaml`).
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/snapshot-detail.png" alt="Snapshot detail view for an Untitled Snapshot, with the Local node expanded to its stack list and the plex stack further expanded to show its compose.yaml file with Preview and Download buttons, plus a Restore button per stack and Restore all in the header" />
|
|
</Frame>
|
|
|
|
Up to four warning banners can appear at the top of the detail view, each only when it applies: skipped nodes that were unreachable during creation, snapshot files that could not be decrypted (see [Storage](#storage) below), stacks that were only partially captured, and, when documentation capture was on, stacks whose Dossier notes could not be fetched. Each banner names the affected node, stack, and reason. A clean snapshot with no problems shows none of them.
|
|
|
|
## Restoring from a snapshot
|
|
|
|
Admins can restore individual stacks from any snapshot:
|
|
|
|
1. Open a snapshot's detail view
|
|
2. Expand the node and stack you want to restore
|
|
3. Click **Restore** below the stack's file list
|
|
4. A confirmation dialog appears. Optionally check **Redeploy stack after restore** to immediately apply the restored configuration. When the snapshot preserved Dossier notes for this stack, a **Restore documentation notes** option also appears.
|
|
5. Click **Restore** to confirm
|
|
|
|
By default, restoring a stack only writes back its files and leaves the stack's current Dossier notes untouched. Notes are restored only when you explicitly check **Restore documentation notes**, which overwrites the current notes with the captured ones. The same option appears on **Restore all** when the snapshot has documentation.
|
|
|
|
<Frame>
|
|
<img src="/images/fleet-backups/restore-dialog.png" alt="Restore plex on Local confirmation dialog with the warning that it overwrites the current compose files with the snapshot version, a Redeploy stack after restore checkbox, and Cancel and Restore buttons" />
|
|
</Frame>
|
|
|
|
Sencho writes the snapshot's files back to the target node. Restoring over an existing stack captures a recovery generation of the current authored project before any snapshot file is written. If that capture fails, the restore stops and the live files are left unchanged. A stack that does not yet exist is created from the snapshot files. Local and remote restores run that capture-then-write sequence on the target node. If a later file write fails after capture succeeded, live files may already have changed; the captured generation remains so you can use Rollback to return to the pre-restore project.
|
|
|
|
<Warning>
|
|
Restoring overwrites the current compose and environment files on the target node. For an existing stack, Sencho captures a recovery generation first so you can use Rollback if the restore stops after that capture.
|
|
</Warning>
|
|
|
|
### Restoring an entire snapshot
|
|
|
|
To roll back every captured stack at once, click **Restore all** in the snapshot detail header. A confirmation dialog appears with an optional **Redeploy all stacks after restore** checkbox. On confirm, Sencho restores each stack across the fleet and reports how many succeeded.
|
|
|
|
Each stack is restored independently: if one stack cannot be restored (for example, its node has since been removed, or a remote node is offline), that stack is reported as failed and the rest still proceed. The result is summarised as a success, a partial result naming how many failed, or a full failure.
|
|
|
|
## Deleting snapshots
|
|
|
|
Admins can delete snapshots from the list view by clicking the trash icon on the right side of each row. A confirmation dialog asks you to confirm before the snapshot is permanently removed. Deleting a snapshot removes all captured file data from the database. This action cannot be undone.
|
|
|
|
## Recovery Vault
|
|
|
|
<Note>
|
|
Custom S3-compatible storage is available on every tier. Recovery Vault is an Admiral feature. Configure either in **Settings → Infrastructure → Recovery Vault**.
|
|
</Note>
|
|
|
|
Recovery Vault mirrors every fleet snapshot to off-site storage so your snapshots survive local disk failure. The Recovery Vault settings page (reached via **Settings → Infrastructure → Recovery Vault**) shows a header with your current scope, provider, storage used, and total snapshot count in the cloud. Two storage modes are supported.
|
|
|
|
### Recovery Vault (included)
|
|
|
|
A managed 500 MB allowance backed by Cloudflare R2, included with every Admiral subscription. Open **Settings → Infrastructure → Recovery Vault**, choose **Recovery Vault (included)**, and click **Activate**. Sencho exchanges your license key for scoped storage credentials and starts replicating new snapshots automatically.
|
|
|
|
Once active, the settings page shows a storage gauge (used / 500 MB and object count), a status message confirming auto-upload is on, and a **Reprovision** button to refresh credentials if needed. You can verify connectivity at any time with the **Test** button.
|
|
|
|
<Frame>
|
|
<img src="/images/cloud-backup/cloud-backup-active.png" alt="Recovery Vault settings page with Storage Mode set to Recovery Vault (included), Test and Reprovision buttons, a storage-used gauge reading 0 Bytes of 500 MB, the auto-upload confirmation message, and an empty Cloud Snapshots panel" />
|
|
</Frame>
|
|
|
|
### Custom S3 (BYOB)
|
|
|
|
Bring any S3-compatible bucket: AWS S3, MinIO, Backblaze B2, Wasabi, or your own Cloudflare R2 token. Choose **Custom S3 (BYOB)** in the storage-mode dropdown and fill in:
|
|
|
|
- **Endpoint URL** (e.g. `https://s3.us-east-1.amazonaws.com`, `https://my-minio.example.com:9000`)
|
|
- **Region** (e.g. `us-east-1`, or `auto` for R2)
|
|
- **Bucket** name
|
|
- **Path Prefix** (default `sencho/`)
|
|
- **Access Key ID** and **Secret Access Key**
|
|
- **Auto-upload** toggle
|
|
|
|
Click **Test** to verify connectivity, then **Save**. Secret keys are encrypted at rest. Sencho only sends them to your configured endpoint.
|
|
|
|
<Frame>
|
|
<img src="/images/cloud-backup/cloud-backup-custom.png" alt="Custom S3 Configuration form with Storage Mode set to Custom S3 (BYOB) and empty fields for Endpoint URL, Region, Bucket, Path Prefix, Access Key ID, and Secret Access Key, plus an Auto-upload toggle set to off" />
|
|
</Frame>
|
|
|
|
### Manual upload vs auto-upload
|
|
|
|
When auto-upload is on, every fleet snapshot is replicated as soon as it is created. Manual snapshots from the **Fleet → Snapshots** view upload asynchronously so the UI returns immediately; scheduled snapshots block on the upload so the task's success status reflects cloud durability.
|
|
|
|
To upload a single snapshot on demand, open the **Snapshots** tab in the Fleet view. Each row that hasn't been mirrored yet shows a cloud-upload action in the Actions column. Once a snapshot is in the cloud, an upload icon appears next to its description.
|
|
|
|
### Browsing and downloading cloud snapshots
|
|
|
|
The **Cloud Snapshots** panel in **Settings → Infrastructure → Recovery Vault** lists every archive currently in your bucket, paginated at 10 per page, with size and last-modified timestamp. Click the download icon to save a `.tar.gz` archive locally for off-host disaster recovery, or the trash icon to permanently delete the archive from your bucket (the corresponding local snapshot in the **Fleet → Snapshots** list is unaffected). Each archive contains a `metadata.json` describing the snapshot and a `nodes/` tree with the captured compose and environment files, organised by node and stack. When the snapshot preserved Dossier notes, the archive also includes a `documentation.json` with those notes.
|
|
|
|
### Restoring from a cloud snapshot
|
|
|
|
For in-place rollback, use the **Restore** action on the snapshot detail view as described above; the local copy is the source of truth for live restore. Cloud snapshots cover the disaster-recovery case where the local disk is gone: download the archive, extract it, and bring up a fresh Sencho instance pointed at the recovered files.
|
|
|
|
## Access control
|
|
|
|
Every fleet snapshot action requires the **admin** role: viewing the snapshot list, browsing snapshot contents, creating, restoring, deleting, and uploading to the cloud. Because a snapshot captures the `.env` file of every stack, the snapshot list and detail views are restricted to administrators rather than read-only roles. The **Snapshots** tab itself does not appear in the Fleet view for non-admin roles, so there is no read-only path into a stack's `.env` contents.
|
|
|
|
Recovery Vault configuration is also admin-only. Mirroring to Recovery Vault additionally requires an Admiral subscription; a Custom S3-compatible target works on every tier.
|
|
|
|
Fleet Snapshots and Recovery Vault settings are both hub-only: the **Snapshots** tab lives inside the **Fleet** view and Recovery Vault lives in **Settings → Infrastructure**, neither of which appears when a remote node is the active selection. See [Multi-Node Management](/features/multi-node) for the full list of hub-only views.
|
|
|
|
## Storage
|
|
|
|
Snapshots are stored in Sencho's SQLite database. Captured file contents, including `.env` files, are encrypted at rest with the instance key, so a database copy never exposes stack secrets in plaintext. Compose files are typically small (under 10 KB each), so even hundreds of snapshots consume minimal disk space. Each captured file is capped at 1 MB to keep snapshots bounded: an oversized `compose.yaml` skips that stack entirely for the snapshot, while an oversized `.env` is dropped and the rest of the stack (including its `compose.yaml`) is still captured. Either case is recorded as a warning, never silently dropped. For very large fleets, consider periodically deleting old snapshots to keep the database lean.
|
|
|
|
If a stored encrypted file cannot be decrypted (for example after on-disk corruption, including damaged ciphertext that still looks encryption-shaped), the snapshot detail view marks that file unavailable and lists it in a warning. Restore for that stack is blocked so live compose and environment files are never overwritten with that damaged data. Configured off-site upload (Recovery Vault or Custom S3) also refuses to publish an archive that would omit or fabricate those files. Intact stacks in the same snapshot remain readable and restorable. Only short, clearly non-ciphertext values that happen to start with `enc:` (for example `enc:hello`, `enc:hello-world`, or `enc:FOO_BAR=baz`) stay readable as legacy plaintext. Encryption-shaped damage remains unavailable.
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="A snapshot file shows as Unavailable">
|
|
The stored ciphertext for that file could not be decrypted. Sencho keeps the rest of the snapshot readable and blocks restore for the affected stack so live compose and `.env` files are not overwritten. Configured off-site upload (Recovery Vault or Custom S3) also refuses that snapshot. Create a fresh snapshot from healthy nodes if you need a complete archive again.
|
|
</Accordion>
|
|
<Accordion title="A snapshot shows skipped nodes">
|
|
If a remote node is offline, unreachable, or its API token has expired, the node is skipped during snapshot creation. The list shows a warning icon with a count of skipped nodes, and the snapshot's detail view names each one along with the reason. Common causes are the remote Sencho instance being stopped or restarting, the node's API URL or token having been changed after it was added, or a firewall or network issue blocking the connection. Verify the remote is running and reachable, update the node's API URL and token in **Settings → Infrastructure → Nodes** if needed, then create a new snapshot.
|
|
</Accordion>
|
|
<Accordion title="Restore fails with 'Target node no longer exists'">
|
|
The node recorded in the snapshot has been removed from the fleet since the snapshot was taken. Snapshots reference nodes by registry ID, so a node that was deleted and re-added is treated as a different target. Re-add the node first, then retry the restore against the freshly registered row.
|
|
</Accordion>
|
|
<Accordion title="Restore fails with 'No files found for this stack'">
|
|
The stack was not captured in the snapshot, usually because its compose file was missing or unreadable on disk at the time the snapshot was taken. Open the snapshot's detail view to verify which stacks and files are available, and pick a different snapshot if the one you have is incomplete.
|
|
</Accordion>
|
|
<Accordion title="Restore all reports that some stacks failed">
|
|
Restore all applies each stack independently, so a failure on one stack does not stop the others. A stack is reported as failed when its node has been removed from the fleet since the snapshot was taken, when a remote node is offline or unreachable, when a snapshot file for that stack could not be decrypted, or when its existing files cannot be written. The successful stacks are fully restored regardless. Resolve the underlying cause (re-add a removed node, bring an offline node back online, or pick a healthy snapshot) and run Restore all again, or restore the remaining stacks individually from the same snapshot.
|
|
</Accordion>
|
|
<Accordion title="Recovery Vault Test reports an authentication error">
|
|
Double-check the Access Key ID, Secret Access Key, and bucket name; one wrong character is the most common cause. Some providers require S3-compatible API access to be enabled on the bucket separately from the credentials. For MinIO, confirm the user has read/write permission on the target bucket. After correcting the values, click **Test** again before saving.
|
|
</Accordion>
|
|
<Accordion title="Cloud uploads fail after hitting the 500 MB quota">
|
|
Recovery Vault carries a 500 MB allowance per subscription. Once you hit the cap, new uploads fail with a quota error and the storage gauge in the settings page reads at or near 500 MB. Free space by deleting older archives from the **Cloud Snapshots** panel in **Settings → Infrastructure → Recovery Vault**. Local snapshots are unaffected by cloud deletions, so the in-place restore path stays intact. To raise the ceiling, switch the storage mode to **Custom S3 (BYOB)** and point at a bucket you control.
|
|
</Accordion>
|
|
<Accordion title="Cloud upload returns a network timeout or 5xx error">
|
|
Transient errors surface as a notification and leave the local snapshot in place. Retry by clicking the cloud-upload action on the snapshot row, or wait for the next scheduled snapshot which retries on its own. Persistent failures usually indicate an endpoint outage; verify the storage provider is reachable from your Sencho host (custom S3 endpoints often sit behind a different DNS or firewall path than the rest of your traffic).
|
|
</Accordion>
|
|
<Accordion title="I need more detail about a snapshot or restore operation">
|
|
Enable **Developer Mode** under **Settings → Operations → Developer Diagnostics** to activate diagnostic logging for snapshot creation (per-node capture timing and file counts), restore operations, and scheduled snapshot execution. Diagnostic logs appear in the server's standard output with a `:debug` suffix.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Where Fleet Backups fits
|
|
|
|
Fleet snapshots show up in several other views. Each adjacent surface answers a different question.
|
|
|
|
| Related feature | What it covers | Why it is not Fleet Backups |
|
|
|---|---|---|
|
|
| [Health-Gated Updates](/features/health-gated-updates) | The readiness check before a stack update or rebuild. | Health-Gated Updates gates the update itself; Fleet Backups is the point-in-time capture it can optionally trigger first. |
|
|
| [Storage Portability](/features/compose-storage) | Per-stack mount inventory and portability verdict. | Storage Portability surfaces snapshot recency for stacks with persistent volumes; Fleet Backups owns the actual capture and restore. |
|
|
| [Scheduled Operations](/features/scheduled-operations) | Recurring or one-shot scheduled stack operations. | Scheduled Operations owns recurrence and dispatch; Fleet Backups defines what a snapshot captures and how to restore it. |
|
|
| [Stack Dossier](/features/stack-dossier) | Operator-authored notes about a stack (purpose, owner, access, recovery). | Stack Dossier owns the notes; Fleet Backups only preserves a read-only copy of them inside a snapshot when documentation capture is on. |
|
|
| [Atomic Deployments](/features/atomic-deployments) | The rollback-point mechanism created on every local deploy. | Atomic Deployments backs up automatically on every deploy; Fleet Backups is a separate, on-demand, fleet-wide capture you trigger yourself. |
|
|
| [Multi-Node Management](/features/multi-node) | How nodes join the fleet and which top-level views are hub-only. | Multi-Node Management is why Snapshots and Recovery Vault are hub-only; Fleet Backups just inherits that rule. |
|
|
| [Licensing](/features/licensing) | Community and Admiral plans. | Licensing covers plan coverage across the product; Fleet Backups only cites the Recovery Vault tier requirement above. |
|