test(table-catalog): add Iceberg compatibility and failure coverage (#3581)

* test(table-catalog): add Iceberg client compatibility coverage

* test(table-catalog): add production failure coverage

* fix(table-catalog): generate valid failure probe requests

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>

scripts/table-catalog/README.md

@@ -123,6 +123,8 @@ PyIceberg, PyArrow, or boto3:
 
 ```bash
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-client-matrix
+python3 scripts/table-catalog/pyiceberg_smoke.py --print-engine-compatibility
+python3 scripts/table-catalog/pyiceberg_smoke.py --print-production-failure-coverage
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-vendor-profiles
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-unsupported-inventory
 python3 scripts/table-catalog/pyiceberg_smoke.py --print-production-readiness
@@ -130,8 +132,41 @@ python3 scripts/table-catalog/pyiceberg_smoke.py --print-production-readiness
 
 Use these outputs when updating release notes, PR descriptions, or follow-up
 work items. They are intentionally conservative: only PyIceberg is automated by
-this script today; other engines are documented until a repeatable harness is
-added.
+this script today. Spark now has generated configuration and SQL smoke input;
+other engines are documented until a repeatable harness is added.
+
+The standalone engine helper prints the same compatibility matrix and can also
+generate Spark REST catalog input without importing PyIceberg:
+
+```bash
+python3 scripts/table-catalog/engine_compatibility.py --print-engine-matrix
+python3 scripts/table-catalog/engine_compatibility.py --print-spark-config
+python3 scripts/table-catalog/engine_compatibility.py --print-spark-sql --cleanup
+```
+
+The production failure helper records the negative coverage required before
+calling a release production-ready and can generate REST probe steps for a live
+RustFS endpoint:
+
+```bash
+python3 scripts/table-catalog/failure_coverage.py --print-failure-matrix
+python3 scripts/table-catalog/failure_coverage.py \
+  --warehouse rustfs-s3table-smoke \
+  --namespace smoke \
+  --table events \
+  --rest-path /iceberg \
+  --print-failure-probes
+```
+
+The generated probe plan covers stale-token commit conflicts, missing metadata
+object rejection, diagnostics/recovery for finalization gaps, maintenance stale
+plan rejection, and external catalog sync conflicts. These steps are meant to be
+run against a prepared live table and should be recorded with the exact RustFS
+build and client versions used.
+
+`--rest-path` defaults to `/iceberg` and generated probe paths include the
+mounted catalog prefix, for example `/iceberg/v1/{warehouse}/...`. Use
+`--rest-path /_iceberg` to generate paths for the compatibility alias.
 
 The smoke test also probes catalog-backed advanced Iceberg surfaces:
 
@@ -153,12 +188,41 @@ The smoke test also probes catalog-backed advanced Iceberg surfaces:
 | Client | Current status | Claim |
 |---|---|---|
 | PyIceberg | Automated smoke target | create namespace, create table, append, reload, scan, metadata-location, refs, views, maintenance, diagnostics, optional catalog-vended table credentials with exact-prefix data-plane scope probe |
-| Spark Iceberg REST catalog | Manual-ready | create/load/append/reload should be verified against a running RustFS endpoint |
+| Spark Iceberg REST catalog | Generated smoke harness | configuration and SQL can be generated for create namespace, create table, append, reload, count, and cleanup against a running RustFS endpoint |
 | Trino Iceberg REST catalog | Documented, not automated | no write compatibility claim yet |
 | DuckDB Iceberg | Documented, not automated | read-path reference only |
+| StarRocks Iceberg REST catalog | Documented, not automated | external catalog read-path reference only |
 | Databend | Documented, not automated | S3 data-plane reference only; Iceberg REST catalog integration is not claimed |
 | Snowflake/Open Catalog integrations | Documented, not automated | reference only |
 
+## Production Failure Coverage
+
+Production failure coverage is tracked separately from positive client
+conformance. Positive smoke tests prove a client can create and use a table;
+failure probes prove RustFS does not silently advance table state when something
+goes wrong.
+
+The current failure matrix covers:
+
+- stale commit tokens and stale base metadata returning a conflict without
+  advancing the table pointer
+- post-CAS finalization gaps surfacing through diagnostics and safe recovery
+  repair without pointer movement
+- missing metadata, manifest, data, or delete objects failing closed before a
+  commit or maintenance operation advances state
+- concurrent writers producing a single winning CAS and retryable conflicts
+- table catalog and ordinary S3 object permission denials preventing data-plane
+  bypass
+- stale maintenance plans failing closed before object deletion or catalog
+  commit
+- external catalog sync conflicts leaving pointer, token, and generation
+  unchanged
+- backing migration remaining blocked until WAL/recovery replay is clean
+
+Do not promote a failure case from `probe-required` or `load-test-required` to
+an automated claim until the live probe or stress harness is repeatable and its
+RustFS build, client version, and expected response shape are recorded.
+
 ## Vendor Profile References
 
 | Profile | Catalog shape | Signing name | Credential model | RustFS claim |
@@ -221,10 +285,21 @@ The TTL is clamped to the supported short-lived range by the server.
 ## Spark Manual Baseline
 
 Spark validation should use the same RustFS endpoint and warehouse bucket as the
-PyIceberg smoke test. The exact package version should be recorded in the client
-matrix after each run.
+PyIceberg smoke test. The exact Spark and Iceberg package versions should be
+recorded in the client matrix after each run.
 
-Minimum configuration shape:
+Generate the configuration properties:
+
+```bash
+python3 scripts/table-catalog/engine_compatibility.py \
+  --endpoint http://127.0.0.1:9000 \
+  --warehouse rustfs-s3table-smoke \
+  --access-key rustfsadmin \
+  --secret-key rustfsadmin \
+  --print-spark-config
+```
+
+The generated configuration shape is:
 
 ```properties
 spark.sql.catalog.rustfs=org.apache.iceberg.spark.SparkCatalog
@@ -237,7 +312,22 @@ spark.sql.catalog.rustfs.s3.path-style-access=true
 spark.sql.catalog.rustfs.rest.sigv4-enabled=true
 spark.sql.catalog.rustfs.rest.signing-name=s3
 spark.sql.catalog.rustfs.rest.signing-region=us-east-1
+spark.sql.catalog.rustfs.s3.access-key-id=rustfsadmin
+spark.sql.catalog.rustfs.s3.secret-access-key=rustfsadmin
 ```
 
-Until Spark is automated, do not claim Spark support beyond a manually verified
-run with the exact Spark and Iceberg versions recorded.
+Generate the SQL smoke input:
+
+```bash
+python3 scripts/table-catalog/engine_compatibility.py \
+  --catalog-name rustfs \
+  --namespace smoke \
+  --table events \
+  --print-spark-sql \
+  --cleanup
+```
+
+The generated SQL covers namespace creation, table creation, append, refresh,
+count, and optional cleanup. Until Spark execution is automated in CI, do not
+claim Spark support beyond a manually verified run with the exact Spark and
+Iceberg versions recorded.