Commit Graph

415 Commits

Author SHA1 Message Date
Henry Guo 52529403a6 test(table-catalog): record live conformance evidence (#4606)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-09 20:15:11 +08:00
Zhengchao An 05833063c7 refactor(concurrency): remove zero-caller facade modules, fix feature build (#4530)
refactor(concurrency): remove zero-caller facade modules and fix no-default-features build (backlog#1025)

The audit in rustfs/backlog#1010 (consistent with #805) established that most of crates/concurrency was a decorative facade with zero production callers; the real runtime concurrency control lives in rustfs/src/storage/*. This deletes the dead facades and keeps only what the workspace actually consumes.

Deleted (zero callers verified by workspace-wide grep):
- manager.rs: ConcurrencyManager, lifecycle start/stop, misleading 'started' lifecycle logs
- config.rs: ConcurrencyConfig, ConcurrencyFeatures, from_env
- timeout.rs: TimeoutManager, TimeoutGuard, TimeoutManagerPolicy
- lock.rs: LockManager, LockScopeGuard, OptimizedLockGuard
- scheduler.rs: SchedulerManager, SchedulerPolicy, IoStrategy
- deadlock.rs facade: DeadlockManager, RequestTracker
- backpressure.rs facade: BackpressureManager, BackpressurePipe
- the prelude module, unused io-core re-exports, and all feature flags

Kept (real callers in ecstore/heal/rustfs):
- workload.rs admission contract types (unchanged)
- workers.rs Workers pool (unchanged, retained per #4498)
- GetObjectQueueSnapshot (moved from manager.rs to new queue.rs)
- PipeBackpressurePolicy (used by rustfs/src/storage/backpressure.rs)
- DeadlockMonitorPolicy (used by rustfs/src/storage/deadlock_detector.rs)
- OperationProgress re-export (used by rustfs/src/storage/timeout_wrapper.rs)

Removing the feature flags fixes the previously broken cargo check -p rustfs-concurrency --no-default-features (E0432). Docs and the logging guardrail file list are updated to match.

Ref: rustfs/backlog#1025
2026-07-09 01:12:43 +08:00
Zhengchao An a91d9cefc6 test(interop): add real MinIO read and migration parity tests (#4377)
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580)

Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with
versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3
encryption, a webhook notification target, and a replication rule, plus inline /
versioned / multipart objects and a delete marker) and prove RustFS reads and
migrates them losslessly:

- filemeta parses_real_minio_object_xlmeta: small inline, two-object-version +
  delete marker, and multipart object xl.meta parse to the expected FileInfo.
- ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO
  .metadata.bin msgpack decodes via the PascalCase field names and
  parse_all_configs loads all ten config types present (policy, lifecycle incl.
  <ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification,
  encryption/SSE-S3, replication incl. DeleteMarkerReplication /
  ExistingObjectReplication) without loss.
- ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object
  body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S
  verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix).
- ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive
  local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated
  into .rustfs.sys byte-identically for every config, exercising the Phase 2
  source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer.

All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a
RustFS drive) is documented as out of scope for one-way migration.

Refs rustfs/backlog#580
2026-07-08 01:12:49 +08:00
Henry Guo 0e61ba7c63 test(table-catalog): add scale fault rehearsal (#4359)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-07 16:37:58 +08:00
Zhengchao An 0271abc14d docs: add MinIO compatibility router + file-format docs (#4328)
Add two durable architecture references grounded in current code:

- minio-rustfs-router-compatibility.md: MinIO cmd/api-router.go (S3
  object/bucket) and cmd/admin-router.go (admin /v3, /v4) vs RustFS
  implementation status, with per-row landing points and a gaps-only
  checklist of still-missing admin endpoints (profiling, healthinfo,
  LDAP IDP config, replication diff, MRF, batch, locks, speedtest,
  log stream, top, trace).
- minio-file-format-compat.md: xl.meta (meta_ver 3 write, <=3 read,
  XL2 magic, rs-vandermonde, HighwayHash256 bitrot, inline data) and
  .metadata.bin bucket-metadata interop matrix for the backlog#580
  items, plus old-RustFS -> new migration and a phased plan.

Cross-links s3-compatibility-matrix.md and admin-route-action-snapshot.md
and indexes both docs in docs/architecture/README.md.

Refs rustfs/backlog#596 rustfs/backlog#603 rustfs/backlog#580
2026-07-06 23:34:31 +08:00
Henry Guo be52e35a1f test(table-catalog): add vendor compatibility audit (#4299)
* test(table-catalog): add vendor compatibility audit

* fix(table-catalog): include OSS data-plane endpoint

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-06 14:12:19 +08:00
Henry Guo 5f2359de45 test(table-catalog): expand live conformance guide (#4277)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-05 15:41:08 +08:00
Zhengchao An 0271d7aa2b refactor(ecstore): narrow api facade exports (#4270)
* refactor(ecstore): narrow bucket api facade

* refactor(ecstore): narrow more api facades

* test(ecstore): stabilize multipart cleanup test
2026-07-05 06:06:39 +08:00
Zhengchao An a6b3e4f5d6 refactor: use canonical Rust module paths (#4269) 2026-07-05 03:01:14 +08:00
Henry Guo be45b35472 feat(table-catalog): add distributed maintenance scheduling (#4257)
* feat(table-catalog): add distributed maintenance scheduling

* fix(table-catalog): address scheduler review feedback

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-04 23:44:08 +08:00
Zhengchao An d0792b87be refactor(lifecycle): extract core rule contracts (#4258) 2026-07-04 20:05:56 +08:00
Zhengchao An 123552d32b refactor(replication): own utility wire contracts (#4255) 2026-07-04 08:00:37 +08:00
Zhengchao An 3d4fb532e5 refactor(replication): own filemeta wire contracts (#4254) 2026-07-04 06:39:54 +08:00
Zhengchao An 125c228832 refactor(replication): own delete DTO contracts (#4253) 2026-07-04 04:00:27 +08:00
Zhengchao An 8a0617865b refactor(replication): route app storage through ecstore (#4251) 2026-07-04 02:00:28 +08:00
Henry Guo 2214f67fba feat(table-catalog): deepen row-level maintenance planning (#4249)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-03 23:49:40 +08:00
Zhengchao An 4883d3eb50 refactor(replication): route runtime facades through ecstore (#4248) 2026-07-03 23:49:24 +08:00
Zhengchao An 7cc470cb08 refactor(replication): isolate ecstore boundary imports (#4247) 2026-07-03 23:18:27 +08:00
Zhengchao An d19ee6c51c refactor(replication): isolate storage api contracts (#4244) 2026-07-03 22:48:21 +08:00
Zhengchao An 24e88a2cf4 refactor(replication): isolate filemeta facade contracts (#4243) 2026-07-03 22:20:46 +08:00
Zhengchao An 6839e57c96 refactor(replication): isolate storage api contracts (#4240)
* refactor(replication): isolate storage api contracts

* docs(replication): record storage api contract boundary
2026-07-03 22:10:16 +08:00
Henry Guo 782e73ca92 feat(table-catalog): add maintenance audit timeline (#4207)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-03 00:36:17 +08:00
Henry Guo 31c026dd4f feat(table-catalog): add maintenance quarantine operations (#4201)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-02 23:30:44 +08:00
Zhengchao An d1db9a10cd chore(docs): refresh agent docs, guard doc paths, archive plans (#4203)
Agent-instruction and architecture docs had drifted from the code:

- CLAUDE.md: slim to commands + pointers; fix wrong claim that
  `make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
  drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
  hand-maintained (already stale) scoped-AGENTS index; link
  architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
  with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
  removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
  (rebalance-decommission phases, migration-progress ledger, PR
  template) to docs/superpowers/plans with archive headers; fix stale
  source paths in kept inventories (core/sets.rs, core/pools.rs,
  store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
  playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
  instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
  .gitignore now keeps .claude/skills and docs/operations committable

Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-02 23:30:13 +08:00
Henry Guo 962c20bf47 feat(table-catalog): deepen maintenance rewrite support (#4192)
* feat(table-catalog): deepen maintenance rewrite support

* fix(ecstore): satisfy replication boundary clippy lint

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-02 22:35:07 +08:00
Henry Guo 14c0fca576 test(table-catalog): extend live engine conformance probes (#4184)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-02 16:34:37 +08:00
Zhengchao An e80391f1aa refactor(replication): extract resync contracts crate (#4154) 2026-07-02 01:30:01 +08:00
Zhengchao An 00e93f8267 fix(storage): tighten replication handle visibility (#4153) 2026-07-02 00:57:08 +08:00
Zhengchao An 572a001f93 refactor(storage): hide replication runtime handles (#4152) 2026-07-02 00:40:42 +08:00
Zhengchao An 14016dbe8c refactor(scanner): hide replication queue DTOs (#4150) 2026-07-02 00:17:03 +08:00
Zhengchao An 644a472e52 refactor(obs): hide replication stats handle (#4149) 2026-07-01 23:37:23 +08:00
Zhengchao An b43655ccdd refactor(app): hide replication object DTOs (#4147) 2026-07-01 23:02:00 +08:00
Henry Guo a305160087 feat(table-catalog): add maintenance scheduler guardrails (#4123) 2026-07-01 22:31:19 +08:00
Zhengchao An b94e7d514d refactor(admin): hide replication resync DTOs (#4146) 2026-07-01 22:18:06 +08:00
Zhengchao An 22cbd41ee9 refactor(ecstore): add replication owner bridges (#4141)
* refactor(ecstore): add replication owner bridges

* fix(ecstore): tighten replication owner guards
2026-07-01 21:51:26 +08:00
Zhengchao An c0dfc260ca refactor(ecstore): add replication object bridge (#4140) 2026-07-01 19:24:20 +08:00
Zhengchao An e5ab6ee8e4 refactor(ecstore): make replication facade explicit (#4139) 2026-07-01 18:54:05 +08:00
Zhengchao An fc806679fb refactor(ecstore): add replication work bridges (#4137) 2026-07-01 18:37:01 +08:00
Zhengchao An 28e87c1cab refactor(ecstore): name replication boundary contracts (#4134) 2026-07-01 17:55:12 +08:00
Zhengchao An b0f491549a refactor(ecstore): hide replication runtime types (#4131) 2026-07-01 15:39:02 +08:00
Zhengchao An 8fd532ebb7 refactor(ecstore): use local replication paths (#4130) 2026-07-01 15:08:26 +08:00
Zhengchao An 7cb42529fc refactor(ecstore): route replication config access (#4128) 2026-07-01 09:30:25 +08:00
Zhengchao An 722c0805ce refactor(ecstore): route replication errors (#4127) 2026-07-01 08:13:57 +08:00
Zhengchao An 50a3ed8bf3 refactor(ecstore): route replication event host (#4126) 2026-07-01 06:00:50 +08:00
Zhengchao An e917f1f4be refactor(ecstore): route replication filemeta contracts (#4122) 2026-07-01 01:01:01 +08:00
Zhengchao An 942a8b505d refactor(ecstore): route replication metadata paths (#4121) 2026-06-30 22:48:34 +08:00
Henry Guo 5e68455eed test(table-catalog): add disaster recovery rehearsal probes (#4119)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-30 21:27:06 +08:00
Zhengchao An da7f421f69 refactor(ecstore): route replication storage contracts (#4118) 2026-06-30 20:12:06 +08:00
Henry Guo cfb24b31ea test(table-catalog): add live client conformance harness (#4110)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-30 16:33:14 +08:00
Zhengchao An 70a1f37508 refactor(ecstore): route replication target types (#4114) 2026-06-30 16:29:05 +08:00