Commit Graph

11 Commits

Author SHA1 Message Date
Zhengchao An 8f0d4a20d1 refactor(ecstore): extract the embedded S3 client into rustfs-s3-client (#6627)
The storage engine embedded a ~8.4K-line hand-written S3 HTTP client under crates/ecstore/src/client (rustfs/backlog#1842). That client is a legitimate engine capability — it consumes remote S3-compatible endpoints for ILM tier warm backends and transition targets — but it was misfiled inside the engine, dragging s3s/hyper wire types into ecstore and blocking ARCHITECTURE.md invariant 4.

This PR is the pure-move step: 21 modules move verbatim to the new crates/s3-client crate (rustfs-s3-client), and crates/ecstore/src/client/mod.rs becomes a re-export shim so every in-crate crate::client:: path keeps working. The two server-side modules that were historically misfiled under client/ — object_api_utils.rs and object_handlers_common.rs — stay in ecstore.

Three reverse dependencies from the client into engine internals are severed so the move can be pure:

- transition_api::ReaderImpl::ObjectBody held ecstore's GetObjectReader; the client only ever reads the body, so the variant now holds an ObjectReader newtype over Box<dyn AsyncRead + Send + Sync + Unpin> with the same read_all() surface. The single production construction site (set_disk transition upload) and the two engine-side consumers were adjusted.
- api_list/api_remove used ecstore's storage_api_contracts / object_api types; api_list now imports BucketInfo from rustfs-storage-api directly, and api_remove uses the client's own transition_api::ObjectInfo (only .name/.version_id were read; the error-path bucket name is now threaded as a parameter instead of read from the deleted objects).
- the api_put_object_streaming regression tests built a GetObjectReader by hand; they now wrap the duplex stream in ObjectReader::new.

Guard updates: the s3s footprint ratchet gains an ecstore-scoped counter (42 files, shrink-only, per rustfs/backlog#1842), the ecstore module-lint-blanket register follows the moved files into crates/s3-client so the blanket ratchet keeps covering them, the logging guardrail path pin follows transition_api.rs, and the ::other(format!) baseline is regenerated (moved call sites left ecstore).

Verification: cargo check -p rustfs-s3-client -p rustfs-ecstore; cargo nextest run -p rustfs-s3-client (43 passed) and -p rustfs-ecstore (4515/4523; the 8 failures reproduce identically on pristine origin/main on the same machine); cargo clippy --all-targets; scripts/check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_s3s_footprint.sh, check_logging_guardrails.sh, check_error_other_format_ratchet.sh, check_doc_paths.sh, check_ci_paths_sync.sh all pass.
2026-08-26 12:38:52 +08:00
houseme a5800033bd feat(heal): incremental status cursors and typed overlap policy (HS-06) (#6206)
* feat(heal): incremental heal status cursors and typed overlap policy (HS-06)

Incremental results: every retained result item now carries a monotonic
sequence number. The status query accepts a client cursor (sinceSeq on
the admin wire, Option<u64> internally) and returns only newer items,
plus nextSeq (the next cursor) and minSeq (the oldest retained
sequence). A cursor that fell behind the 1024-item retention window is
flagged through the existing truncated signal together with minSeq so
the client can restart from it. Sequencing survives task completion:
the completion archive stores the seq-stamped window. None keeps the
exact legacy full-snapshot behavior, so existing clients see no change.

Typed overlap handling for admin starts: RUSTFS_HEAL_OVERLAP_POLICY
(merge default | minio_error). Under minio_error, an admin start whose
path overlaps an active or queued task rejects with typed
already-running / overlapping-paths admission reasons (surfaced through
reason_label in the admin error body, sharing the existing
OperationAborted site because the s3s footprint ratchet forbids new
s3_error! sites); an exact duplicate start rejects with
already-running instead of silently merging. Scanner/autoheal/
read-repair sources never take the rejection path.

forceStart semantics now match MinIO for admin requests: an admin
forceStart first cancels the overlapping active admin task, then
admits the replacement.

Wire: the heal-control Query command grows an optional sinceSeq
(defaulted and skipped when absent, so older peers stay compatible);
the admin handler accepts the sinceSeq query parameter; the local
channel query gains the same cursor.

Tests: seq monotonicity and incremental slicing, window slide moving
minSeq with lagging-cursor flags, overlap matrix (same/containing/
contained/disjoint x policy x source), forceStart cancel-then-admit,
and the completion-archive window handoff.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: fmt after main merge

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-08-18 16:09:30 +08:00
cxymds d60a77b750 fix(quota): enforce durable hard quota reservations (#6058)
* fix(quota): enforce durable hard quota reservations

* fix(quota): close reservation bypasses

* fix(quota): isolate tests and box object futures

* fix(quota): close legacy and deferred settlement bypasses

* fix(app): keep object futures off caller stacks

* fix(metrics): preserve object operation labels

* fix(logging): retain GET trace guard contract
2026-08-14 06:26:00 +00:00
Zhengchao An 882ad4c113 fix: update s3s footprint baseline after table-catalog hardening (#5964) 2026-08-12 06:44:43 +08:00
Zhengchao An 31cb720471 fix: exclude e2e_test from s3s footprint ratchet baseline (#5949) 2026-08-11 05:48:27 +00:00
cxymds a7de957eb8 fix(rebalance): fence peers before activation (#5842) 2026-08-08 11:55:50 +00:00
cxymds dd2e0328fd fix(ci): ignore strings in s3s footprint ratchet (#5787) 2026-08-07 01:53:24 +00:00
Zhengchao An efd5481b35 fix(auth): log structured denial reasons for generic AccessDenied responses (#5761) 2026-08-06 02:45:54 +00:00
Zhengchao An 5e0fdaa247 fix(table-catalog): route read_bounded_json_body errors through ApiError to hold s3s ratchet (#5759)
The namespace REST contracts PR (#5745) added 7 new s3_error! invocations in
read_bounded_json_body, pushing the s3s footprint counter from 1687 to 1693
and violating the ratchet baseline.

Replace those calls with ApiError::invalid_request() (gateway-side error
abstraction, rustfs/backlog#1677 F1, rustfs/backlog#1733) and lower the
baseline from 1687 to 1686.

- Add ApiError::invalid_request(message) constructor to rustfs/src/error.rs
- Replace 7 s3_error! calls in read_bounded_json_body with S3Error::from(ApiError)
- Lower S3_ERROR_LINES_BASELINE from 1687 to 1686

Verification:
- make pre-commit: all guard scripts + fmt-check + quick-check passed
- make clippy-check: passed
- cargo test table_catalog + admin handler tests: 406/406 passed
- s3s-e2e: 27/27 passed
2026-08-06 00:34:46 +00:00
Zhengchao An 5f3bc617fe fix: bump s3_error! footprint baseline to 1687 for auth ListBuckets fix (#5747) 2026-08-06 00:05:01 +08:00
Zhengchao An db1daaece2 ci(scripts): add s3s footprint ratchet ahead of s3gate migration (#5739)
Freeze the direct s3s dependency surface with a lower-only ratchet so it
cannot grow while the s3gate/gateway migration shrinks it
(rustfs/backlog#1677 review finding F1; acceptance criteria in
rustfs/backlog#1733). Baselines verified on 2026-08-05: 236 files
importing s3s, 1686 s3_error! invocation lines. Wired into make
pre-commit / pre-pr / dev-check and the Quick Checks job in ci.yml and
its ci-docs-only.yml mirror.
2026-08-05 15:21:35 +08:00