gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-19 11:06:17 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(table-catalog): add distributed maintenance scheduling (#4257)

Browse Source 
* feat(table-catalog): add distributed maintenance scheduling

* fix(table-catalog): address scheduler review feedback

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
This commit is contained in:
Henry Guo
2026-07-04 23:44:08 +08:00
committed by GitHub
parent f3147c5f8c
commit be45b35472
7 changed files with 973 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/table-catalog/README.md
+5 -5
View File
@@ -230,9 +230,9 @@ The smoke test also probes catalog-backed advanced Iceberg surfaces:
`main` cannot be deleted
- Iceberg views support basic create, list, load, replace, existence check, and
drop routes with persisted view metadata and view-scoped authorization
- metadata maintenance supports safe dry-run planning, controlled worker
execution checks, and a scheduler status report with disabled, paused,
backpressure, retry, quarantine, and audit-timeline state; job reports expose
- metadata maintenance supports safe dry-run planning, controlled scheduler
queueing, worker execution checks, and a scheduler status report with disabled,
paused, queued, backpressure, retry, quarantine, and audit-timeline state; job reports expose
structured audit events for planning, worker transitions, heartbeat updates,
lease expiry, and mutating quarantine operations; quarantined jobs can be
inspected, released, retried, or abandoned through an operator endpoint
@@ -338,10 +338,10 @@ Unsupported behavior is documented instead of hidden behind internal errors. The
current unsupported inventory is:
- credential vending: automated after table bootstrap with exact-prefix validation and a data-plane scope probe; full no-long-term-data-credential bootstrap is not claimed
- background maintenance worker: controlled run-once, heartbeat, quarantine operation, and scheduler status endpoints are registered; disabled/paused/backpressure/retry/quarantine/audit-timeline state and per-job audit events are machine-readable; continuous in-process scheduling is not claimed
- background maintenance worker: controlled scheduler run, worker run-once, heartbeat, quarantine operation, and scheduler status endpoints are registered; disabled/paused/queued/backpressure/retry/quarantine/audit-timeline state and per-job audit events are machine-readable; continuous in-process scheduling is not claimed
- manifest/data reachability cleanup: metadata maintenance reads manifest-list and manifest Avro references, reports manifest/data/delete reachability, and deletes only unreferenced table objects that pass the safety window
- snapshot expiration dry-run planning and manual catalog commit: supported through metadata maintenance reports
- automatic maintenance scheduling: external scheduler hook supported through the worker run endpoint and scheduler status report; built-in periodic scheduling is not claimed
- automatic maintenance scheduling: external scheduler hook supported through the scheduler run endpoint, worker run endpoint, and scheduler status report; built-in periodic scheduling is not claimed
- compaction rewrite: controlled run-once support for partition-local and sort-order-preserving Parquet binpack through metadata maintenance; manifests with position or equality delete files produce machine-readable row-level planning and fail closed before rewrite; built-in periodic scheduling, delete-file rewrite, and row-level compaction execution are not claimed
- row-level delete/update/merge commits: standard catalog commit validates append, overwrite, delete, and replace snapshot manifests for table-warehouse scope, referenced object existence, current-live-file deletes, and stale add/delete conflicts; end-to-end SQL DML client coverage remains a compatibility validation item
- external catalog bridges: metadata import/register and operator-supplied metadata pointer sync are supported for Polaris/Glue/DLF/Hive identity boundaries; online vendor SDK polling and policy mirroring are not claimed