feat(table-catalog): harden table row-level commit conflicts (#3517)

* feat(table-catalog): harden snapshot conflict validation

* feat: harden table row-level commit conflicts

* fix: accept additional row-level snapshot shapes

* test(table-catalog): expand smoke catalog probes

* fix(table-catalog): satisfy row-level clippy checks

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Henry Guo
2026-06-18 07:32:20 +08:00
committed by GitHub
parent 24fa03e04b
commit 604379d62f
5 changed files with 1831 additions and 53 deletions
+19 -3
View File
@@ -32,6 +32,9 @@ The smoke test covers:
- create namespace and table
- append two rows through PyIceberg
- reload and scan the table
- probe direct REST catalog endpoints for metadata-location, table refs,
Iceberg views, maintenance config, metadata maintenance, worker run, and
catalog diagnostics
- optionally drop the table and namespace
The default profile uses the canonical RustFS catalog URI:
@@ -104,6 +107,15 @@ temporary credentials:
- `PutObject` and `GetObject` to the same bucket outside that prefix must be
rejected.
The direct REST catalog probes run by default after the PyIceberg append and
scan. For deployments that intentionally expose only the core Iceberg REST
Catalog table path, skip those probes explicitly:
```bash
python3 scripts/table-catalog/pyiceberg_smoke.py \
--skip-catalog-api-probes
```
## Machine-Readable Inventories
The script can print the current conformance inventories without importing
@@ -120,20 +132,23 @@ work items. They are intentionally conservative: only PyIceberg is automated by
this script today; other engines are documented until a repeatable harness is
added.
RustFS also exposes catalog-backed advanced Iceberg surfaces that are not part
of the PyIceberg append smoke path yet:
The smoke test also probes catalog-backed advanced Iceberg surfaces:
- table refs can be listed, created or replaced, and deleted through catalog
commits; refs with explicit retention policy require a forced delete, and
`main` cannot be deleted
- Iceberg views support basic create, list, load, replace, existence check, and
drop routes with persisted view metadata and view-scoped authorization
- metadata maintenance supports safe dry-run planning and controlled worker
execution checks
- catalog diagnostics exposes the table recovery and consistency state used by
operators
## Client Matrix
| Client | Current status | Claim |
|---|---|---|
| PyIceberg | Automated smoke target | create namespace, create table, append, reload, scan, optional catalog-vended table credentials with exact-prefix data-plane scope probe |
| PyIceberg | Automated smoke target | create namespace, create table, append, reload, scan, metadata-location, refs, views, maintenance, diagnostics, optional catalog-vended table credentials with exact-prefix data-plane scope probe |
| Spark Iceberg REST catalog | Manual-ready | create/load/append/reload should be verified against a running RustFS endpoint |
| Trino Iceberg REST catalog | Documented, not automated | no write compatibility claim yet |
| DuckDB Iceberg | Documented, not automated | read-path reference only |
@@ -163,6 +178,7 @@ current unsupported inventory is:
- snapshot expiration dry-run planning and manual catalog commit: supported through metadata maintenance reports
- automatic maintenance scheduling: external scheduler hook supported through the worker run endpoint; built-in periodic scheduling is not claimed
- compaction rewrite: controlled run-once support for unpartitioned Parquet binpack through metadata maintenance; built-in periodic scheduling, sort compaction, delete-file rewrite, and row-level compaction are not claimed
- row-level delete/update/merge commits: standard catalog commit validates append, overwrite, delete, and replace snapshot manifests for table-warehouse scope, referenced object existence, current-live-file deletes, and stale add/delete conflicts; end-to-end SQL DML client coverage remains a compatibility validation item
- external catalog bridges: metadata import/register is supported, but Polaris/Glue/DLF/Hive synchronization is unsupported
- multi-table transactions: not a short-term production claim