mirror of https://github.com/rustfs/rustfs.git synced 2026-10-27 06:18:43 +00:00

Files

T

Henry Guo 3b92708011 feat(table-catalog): add durable catalog backups (#8266 )

* feat(table-catalog): add durable catalog backups

* fix(table-catalog): preserve migration lock ordering

* fix(table-catalog): preserve backup recovery fences

* fix(table-catalog): order backup publication locks

* fix(table-catalog): preserve recovery publication fences

* test(table-catalog): use the storage facade for recovery fencing

* test(ecstore): repair catalog publication fixtures

* fix(table-catalog): address catalog backup clippy failures

* test(table-catalog): adapt generation races to publication fencing

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>

2026-10-06 23:53:53 -07:00

10 KiB

Raw Permalink Blame History

S3 Tables Durable Backing Cutover Runbook

Use this when: moving a table-catalog warehouse from object-backed catalog state to the durable strong snapshot backing (RUSTFS_TABLE_CATALOG_BACKING=durable-strong), or rolling the strong snapshot format from version 1 to version 2. Source of truth: the {warehouse}/catalog/migration routes registered in rustfs/src/admin/handlers/table_catalog/routes.rs; the env constants named below; claims and status labels in docs/architecture/s3-tables-support-matrix.md.

Preconditions

Requirement	Why
A principal with `GetTableCatalogAction` on each table bucket (preflight) and `admin:MigrateTableCatalog` (migration `POST` / `DELETE`).	The mutations are admin-gated.
Every catalog writer runs a release that recognizes the durable-backing migration fence.	An older writer does not see the persisted fence and can mutate the object-backed source after the snapshot inventory is captured.
An object-backed catalog backup, plus the current metadata pointer and version token for representative tables.	Recovery after a failed cutover is an operator-selected restore, not a restart against the stale pointer.
Every mutating object-only operation (maintenance workers, catalog recovery, export, diagnostics, external catalog bridge writes) is inventoried and confirmed supported in durable-strong mode.	Unsupported operations fail closed after cutover rather than continuing against object-backed state.

Cutover Procedure

Take the object-backed backup and record pointer and version token for representative tables.
Run the preflight for each warehouse and treat every blockers entry as fail-closed. Repair commit recovery state and backfill the warehouse prefix index before continuing. Requests are SigV4-signed with the catalog's REST signing name; the /_iceberg/v1 alias accepts the same paths.
```
GET /iceberg/v1/{warehouse}/catalog/migration
```
Drain every catalog writer that predates the migration fence and restart it on a fence-aware release. Keep all writers on that release until cutover completes.
Quiesce mutating object-only operations (step 4 of Preconditions).
Run the migration POST with admin:MigrateTableCatalog. It acquires the exclusive migration fence to drain in-flight fence-aware mutations, persists the source fence while exclusivity is held, then copies catalog state and reports ready_to_enable_durable_strong.
```
POST /iceberg/v1/{warehouse}/catalog/migration
```
Repeat preflight and materialization for every table bucket. Do not proceed until the preflight reports SNAPSHOT_MATERIALIZED, no blockers, and ready_to_enable_durable_strong: true for all of them.
Restart with RUSTFS_TABLE_CATALOG_BACKING=durable-strong, then verify catalog config, table and view loads, commit idempotency, and table data-plane policy resolution before admitting writers.
Preserve the object-backed backup until durable strong backing has passed the operator's retention window.

Cancelling Before Cutover

Before the restart in step 7, DELETE on the migration endpoint removes a migration-created target bucket snapshot and releases the source fence. It releases the bucket fence only while the target state has not advanced, and releases the registry fence after the last bucket is cancelled. Retries and DELETE may restore a known-absent initial target after an ambiguous first write, but fail closed if a previously existing or materialized global snapshot disappears.

DELETE /iceberg/v1/{warehouse}/catalog/migration

After the durable-strong state advances, cancellation fails closed; recovery requires an operator-selected restore or reverse migration.

Controlled Catalog Backup And Restore

The durable catalog backup and restore endpoints are a controlled, same-table-bucket recovery mechanism. They are available only with RUSTFS_TABLE_CATALOG_BACKING=durable-strong; they protect the catalog snapshot and maintenance state, but do not copy or replicate table data, metadata, manifest, or delete objects.

Backup artifacts and restore intents use a strict internal format version. A binary rejects artifacts or pending intents from another format version; create a new backup after upgrading instead of assuming cross-version restore compatibility.

Stop or drain catalog writers and maintenance workers for the table bucket. The backup operation acquires the table-bucket publication fence and the durable migration fence, rejects pending commit recovery and pending restore recovery, rejects active maintenance leases, and records the current snapshot ETag, format version, catalog fingerprint, and referenced object watermarks.
Create a backup with a principal authorized for MigrateTableCatalogAction. Backup creation persists an immutable recovery artifact and is therefore not covered by read-only catalog permission. The optional expected-snapshot-etag provides an additional compare-and-capture guard.
```
POST /iceberg/v1/{warehouse}/catalog/backup
{"expected-snapshot-etag":"<current-etag>"}
```
Keep every referenced table object available for the retention period. A successful backup is immutable and idempotent by content-derived backup ID; it is not a copy of those objects and cannot restore an object that was deleted or changed after capture.
Before restoring, stop writers and maintenance workers again. Verify the target table bucket is still table-enabled, obtain its current snapshot ETag, and use expected-snapshot-etag whenever replacing an existing catalog. allow-replace must be explicitly true to replace a non-identical existing bucket snapshot.
```
POST /iceberg/v1/{warehouse}/catalog/restore
{"backup-id":"<backup-id>","expected-snapshot-etag":"<current-etag>","allow-replace":true}
```
Treat a restore conflict as a stale-target or changed-object condition, not as permission to retry blindly. The service verifies all recorded object watermarks before applying an ETag-CAS snapshot replacement. If the process fails after catalog replacement, retrying the same backup ID is safe only while maintenance state is exactly the persisted source state or already exactly the backup target state. A partial multi-object reconciliation fails closed with a conflict and requires operator inspection before manual cleanup.

This procedure is not cross-region failover. Cross-region recovery requires an independent object replication/backup system and an operator-selected catalog import or restore procedure.

Strong Snapshot Version 1 to Version 2

Keep snapshot writes on version 1 during a rolling binary upgrade. Current binaries read both versions.
After every catalog writer can read version 2, set both RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2=true and RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2_FLEET_CONFIRMED=true and restart the catalog writers. Setting only one gate does not change the write format.
Perform a controlled catalog write or migration materialization and confirm the persisted snapshot is version 2 before serving table data-plane traffic. Once v2 is fleet-confirmed, data-plane resolution fails closed until the persisted snapshot is v2.
After any version 2 snapshot is persisted, do not roll writers back to a binary that only reads version 1. Current binaries preserve version 2 even when the gates are later disabled.

Rollback And Collision Repair Rules

A running process rejects restored version 1 content after observing version 2, but cannot distinguish an older snapshot with the same format version from a deliberate restore. The format high-water mark is process-local: restoring any older snapshot and restarting every writer is a privileged disaster-recovery rollback that must restore a compatible binary and an operator-selected snapshot together.
Migration preflight rejects an active table/view identifier collision before writing a migration fence. A pre-existing version 1 strong snapshot with such a collision loads in cleanup-only quarantine: ambiguous reads fail closed, each cleanup mutation must reduce the collision set, and unrelated writes stay blocked until all collisions are removed. Drain writers that predate cleanup quarantine before starting the repair, and finish cleanup before the first version 2 write.

S3 Tables support matrix
Table catalog conformance scripts (failure_coverage.py --print-disaster-recovery-rehearsal generates the rehearsal for this procedure)

Disable an Empty Table Catalog

Use GET /iceberg/v1/buckets/{warehouse} to check enabled and disable-supported. When supported, DELETE on the same route requires admin:SetTableBucket and returns the bucket discovery response with enabled: false. The /_iceberg/v1 alias behaves identically. Existing enable and query requests remain unchanged.

Review the bucket lifecycle configuration before disabling: clearing table-bucket protection allows existing object expiration rules to run again. Objects are preserved by the disable operation itself. Catalog disablement requires no namespaces, retained table/view resources, or objects under .rustfs-table/; an active rename or backing migration also blocks it. Clean up through supported catalog operations; do not manually delete internal metadata to bypass these checks.

The server persists an inactive catalog entry before clearing the bucket marker. If marker publication fails, retry the same DELETE; the marker retains object protections until the retry succeeds. A subsequent PUT explicitly enables the bucket again. Stale requests cannot initialize an inactive catalog entry. No table metadata or object data is deleted by either state change.

Before rolling back to a server version without this endpoint, re-enable disabled catalogs with PUT while the supporting version is running. Older enable handlers do not reactivate inactive catalog entries. Complete the server upgrade across the cluster before using disablement.

10 KiB Raw Permalink Blame History