mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-11 15:46:53 +00:00
58d4bdc79f5e7b00c7e2d2d49205e7a7d87bccfa
91 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zhengchao An
|
8c9e884cf2 |
fix(ecstore): make inline-rollback reclamation file-precise to keep #5703's child-key safety (#5732)
#5724 reclaimed the synthetic inline-rollback dir after a committed rename with delete_data_dir(recursive: true), which has no notion of object metadata: for unversioned objects the synthetic UUID is a fixed, publicly-known constant, so object/<rollback-dir> can simultaneously be a legitimate child key's directory, and recursively deleting it reopens the authorization bypass #5703 closed (PutObject on K destroying K/<uuid> without DeleteObject permission). Replace the recursive pass with a file-precise one: after quorum commit, delete exactly object/<rollback>/xl.meta.bkp with a non-recursive delete on every disk whose rollback dir is not also the cleanup dir. The parent-rmdir walk removes the dir only when the backup was its sole content, so the BucketNotEmpty leak fix is preserved (#5724's regression test passes unchanged) while a child key at the same path keeps its metadata. The undo path's restore_metadata_backup now also reclaims the emptied synthetic dir, mirroring restore_delete_rollback. |
||
|
anthonymartin
|
ec135f8c4c |
fix(heal): bound per-object logging (#5719)
Co-authored-by: Anthony Martin <949506+anthonymartin@users.noreply.github.com> |
||
|
Zhengchao An
|
f3eba31aee |
fix(ecstore): retain namespace read locks for streaming GETs to prevent quota-bypass (#5699)
* fix(ecstore): retain locks for streaming GETs * fix(rpc): keep disabled snapshot leases lint-clean |
||
|
houseme
|
b71483b1c8 |
fix(ecstore): split internal get metadata metrics (#5687)
Classify expected metadata-missing errors separately from unknown get pipeline failures and attribute internal meta-bucket reader failures to an internal_meta path instead of legacy_duplex. This keeps scanner/data-usage metadata probes from polluting user GET/mixed failure attribution while preserving the existing read error behavior. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
98d3619613 |
fix: address rc.1 release blockers (#5648)
* fix: address rc.1 release blockers
* fix: route release guards through architecture boundaries
* fix: close remaining rc.1 regression gaps
* refactor: group multipart listing options
* fix: resolve rc.1 CI regressions
* fix(ecstore): keep bucket-config writes off the caller's stack
A bucket-config write nests incarnation resolution (which can drive legacy
migration and a peer fan-out), a full metadata load, and `save` — itself an
object PUT that pulls in the whole erasure write path. Every request that
mutates bucket config is already several futures deep, so inlining all of
that into one state machine overflows the 2MiB worker stack in debug builds.
Two CI lanes aborted with SIGABRT on this:
ILM Integration (serial)
rustfs app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions
Test and Lint (swift)
rustfs-protocols::swift_metadata_persistence::
swift_metadata_writes_are_durable
Neither test file is touched by this branch and both lanes are green on
main. Stack-pointer probing showed ~780KiB consumed between
`metadata_sys::update` and the config read alone, with single hops of
363KiB (`update` -> `acquire_config_write_guard_for_incarnation`), 125KiB
and 105KiB.
Box the deep sub-futures on both read-modify-write paths (`update` /
`update_checked` and `update_config_with` / `update_config_with_checked`)
so each guard's own state machine stays small. Behaviour is unchanged;
`update` -> guard drops to 253KiB and both tests pass on the default stack.
* fix(lifecycle): unbreak restore under the bucket generation fence
The ILM lane aborted on a stack overflow before reaching these, so they
were never reported; with that fixed, four restore tests fail. All four
are green on main and none of their test files are touched by this branch.
1. RestoreObject and ListMultipartUploads hard-required
`opts.expected_bucket_incarnation_id`, but `apply_bucket_generation_guard`
deliberately leaves it unset when no guard extension is present — only the
S3 access layer installs one. Every direct caller therefore got
`InternalError: ... bucket generation guard is missing`. Resolve the
current generation instead, the way the copy path already does. The fence
is unaffected: RestoreObject still re-reads the incarnation from disk and
compares before admitting the restore, and the multipart listing is
filtered by the value it resolves.
2. `restore_expiry_snapshot_matches` (new on this branch) rejected every
restored-copy expiry whose `restore_expires` had not already elapsed.
Whether the restored copy is due to expire is the ILM evaluator's
decision, made when it emitted DeleteRestoredAction; re-deriving it in
the set layer only adds a way for a legitimate action to be rejected.
The stale-event risk it appears to guard is already covered by the
surrounding snapshot match — a re-restore rewrites `restore_expires`,
so a replayed event fails the equality check. Drop the clause; the
fifteen identity clauses are unchanged.
Fixed:
rustfs app::lifecycle_transition_api_test::
restore_object_usecase_accepts_exactly_one_of_two_concurrent_restores
restore_object_usecase_completes_suspended_null_version_in_place
restore_object_usecase_reports_ongoing_conflict
rustfs-scanner::lifecycle_integration_test serial_tests::
test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore
Verification: the CI ILM lane filter now runs 53/53 green locally.
* chore: address review follow-ups on this branch
Four items from the adversarial review that were still open.
- Restore the assertion `test_bucket_replication_replayed_delete_marker_
preserves_source_mtime_without_source_restart` is named for. The branch
had replaced the backlog#867 mtime check with `assert_replication_
converged`, which any successful replication satisfies, and deleted the
two helpers it needed — so the regression the test exists to catch would
now pass. This matters here specifically because the branch changes the
flag feeding `replication_delete_remove_options` and routes replay
through a new file and ordering.
- Drop `read_config_no_lock_preserve_empty`: zero production callers (the
one real consumer calls the `_with_metadata` variant directly). Its test
stanza now exercises that variant, so the coverage moves to live code
rather than being deleted.
- Revert the `bytesize` bump. It is a no-op: `Cargo.lock` already pinned
2.7.0 before this branch and is untouched, so the caret range already
resolved there. Nothing in the diff uses the crate.
- Split the AGENTS.md "Adversarial Validation" policy change out of this
branch. The edit is defensible on its own, but it relaxes the review gate
that this branch has to pass, so it should land as its own PR reviewed on
its own merits rather than bundled with the change that benefits from it.
The reverted hunks are unchanged and ready to re-apply.
Not changed, deliberately: the missing-sidecar path still fails closed.
`missing_bucket_incarnation_sidecar_for_new_metadata_fails_closed` pins
that on purpose, and serving a non-authoritative Object Lock state would
be the wrong trade. The residual concern stands and is recorded in review
— a crash between the two writes in `persist_new_and_set` leaves the
bucket unloadable until DeleteBucket+CreateBucket, and the repair branches
in `migrate_legacy_metadata` and `make_bucket` are unreachable dead code
for that case. Resolving it needs the read path and the (transaction-lock
holding) repair path to be separated, which is more than a follow-up edit.
* test(ci): serialize the new bucket-incarnation tests
The five tests this branch adds around the incarnation / lifecycle fence
drive `init_bucket_metadata_sys` and `bucket_metadata_sys_of` — process-global
OnceLock state that `serial_test`'s `#[serial]` cannot protect across
nextest's process boundary — and they delete+recreate buckets, the shape that
raced into InsufficientWriteQuorum in backlog#937.
Add them to the `ecstore-serial-flaky` group in both the default and ci
profiles (nextest evaluates a named profile's own overrides list, so the
ci mirror is required). Preventive serialization only, no retries.
Not a full fix for the review comment: `bucket_delete_waits_for_config_
mutation_fence` still proves liveness with a fixed 200ms sleep plus
`assert!(!delete.is_finished())`. Turning that into readiness polling needs
a production-side signal to wait on — asserting "still blocked" is inherently
a negative. Serializing the group removes the parallel-load pressure that
makes the window fragile; the sleep itself is left for a follow-up.
* test(ecstore): pin that a drained bucket is actually deletable
`DeleteBucket`'s emptiness check is `has_xlmeta_files`, a raw scan of the
bucket directory on local disks — not an S3-level listing. So "the client
drained the bucket" and "the bucket is deletable" are two different
contracts, and only the first one was covered.
That gap is what the `S3 Implemented Tests` lane is failing on: 219 cases,
all `BucketNotEmpty` on `nuke_prefixed_buckets`, with every test body
passing. The first one is `test_versioning_obj_suspend_versions`, reported
by pytest as PASSED followed by ERROR at teardown.
Add the missing assertion for the unversioned path: PUT, client DELETE,
then assert no `xl.meta` survives and `DeleteBucket` succeeds. It passes —
which is itself a result: the plain delete path leaves no residue, so the
s3-tests failure is not there.
The versioning-suspended path is the remaining suspect (the client DELETE
leaves a null delete marker, and draining means purging it by
`versionId=null`). It is not covered here: `BucketVersioningSys` resolves
through the ambient `get_bucket_metadata_sys()` OnceLock, which this unit
env cannot set, so the bucket never actually reports as suspended. That
repro belongs at the e2e layer where a real server owns the versioning
state.
* fix(ecstore): let an explicit null-version delete purge its delete marker
Root cause of the `S3 Implemented Tests` lane: 219 cases, all
`BucketNotEmpty` on `nuke_prefixed_buckets`, every test body passing.
On a versioning-suspended bucket a client DELETE leaves a null delete
marker — correct S3 semantics, and an `xl.meta` on disk. Draining the
bucket therefore means purging that marker as `?versionId=null`, which is
what `nuke_bucket` does before `DeleteBucket`. That purge was rejected:
explicit null-version purge of the null delete marker must succeed,
got [Some(MethodNotAllowed)]
so the marker survived, and `DeleteBucket`'s emptiness check — a raw
`has_xlmeta_files` scan of the bucket directory, not an S3 listing — kept
reporting the bucket as non-empty.
The two sides of the version comparison in the batch delete loop are in
different namespaces. `goi.version_id` is the client-facing identity, where
`from_file_info` synthesizes `Some(Uuid::nil())` for a null version on a
versioned *or versioning-suspended* bucket. `version_id` is the storage
identity, where `delete_file_info_version_id` maps an explicit
`?versionId=null` to `None`. Comparing them raw makes the purge look like a
version mismatch, so `explicit_delete_marker` is false and the
`MethodNotAllowed` from the lookup is recorded as a delete failure.
This only became reachable on this branch: previously `check_opts` did not
carry `dobj.version_id`, so `set_disk_delete_creates_delete_marker` was
true, `object_lock_check_required` was false, and the lookup that produces
`MethodNotAllowed` never ran. Adding the version id to `check_opts` lit up
a comparison that was already wrong.
Normalize both sides through `delete_file_info_version_id`.
The regression test injects a real Suspended bucket-config snapshot — the
delete path reads versioned/suspended from that snapshot, not from `opts`,
so without it `from_file_info` never synthesizes the null version id and
the branch is not reached. Mutation-checked: restoring the raw comparison
fails the test with the exact `MethodNotAllowed` above.
* fix(app): drop the now-needless struct update
Reverting `crates/replication` to main removed the extra `MrfReplicateEntry`
fields, so this literal specifies every field again and `..Default::default()`
trips `clippy::needless_update` under `-D warnings`.
Caught by CI, not locally: I had run `cargo check --workspace --all-targets`,
which does not see clippy-only lints. Ran `cargo clippy --workspace
--all-targets -- -D warnings` here — clean.
* test(e2e): assert the fresh-volume classification
four_node_empty_legacy_volumes_start_as_fresh only started the cluster and
listed buckets — no assertion, so any classification path that still permits
startup left it green without proving the pre-created empty `.minio.sys`
directories were treated as fresh volumes.
Pin what that classification actually leaves behind: no buckets adopted into
the namespace, `.rustfs.sys/format.json` written on every drive, and the empty
legacy directory left untouched rather than migrated into.
* fix(bucket): apply the requested Object Lock to existing buckets
Site replication replays make-with-versioning against the destination,
carrying the source's `lockEnabled`. When the destination bucket already
exists it takes `force_create`, and the whole option-application block was
gated on `confirmed_missing` — so the call returned success while the replica
stayed unlocked. Replicated versions could then be deleted without the
retention the source enforces.
Object Lock enable is one-way, so applying it to an existing bucket is safe:
move it out of the creation-only gate, keeping `created` and versioning-only
options creation-scoped as before.
An existing authoritative bucket takes the `cache_bucket_metadata_in` branch,
which only caches, so the enable would have been dropped on restart. Persist
instead when the enable actually changed something.
Mutation-checked: restoring the creation-only gate fails the new
`force_create_enables_object_lock_on_an_existing_bucket` with "Object Lock
must be enabled on the existing bucket".
cargo nextest run -p rustfs-ecstore --lib: 3633 passed.
* fix(ecstore): box the generation-checked config mutation paths too
The earlier stack fix boxed `update` and `delete`, but an authorized
bucket-config mutation carrying an incarnation takes `update_if_incarnation`
/ `delete_if_incarnation` instead — which were still inlining the whole
resolve/load/save chain into an already-deep request future. Same overflow,
sibling path.
* fix(restore): keep the nil-version normalization the strip removed
Reverting the replication subsystem to main took `set_disk/replication.rs`
with it, but one line in that file was this branch's own fix rather than
replication work:
- self.version_id.filter(|v| !v.is_nil()) == fi.version_id.filter(|v| !v.is_nil())
+ self.version_id == fi.version_id
For a versioning-suspended object the expected version is `Some(Uuid::nil())`
while the read-back `FileInfo` carries `None`, so the raw compare reports
every suspended restore as "restored object changed before restore metadata
finalization" and the copy-back never commits. Same nil-vs-None mismatch as
the null delete-marker purge fixed earlier on this branch.
Caught by `Test and Lint (rio-v2)`, not by my local runs: the test lives in
`transition_commit_failure_tests`, gated behind `feature = "test-util"`, so
the 3633-test suite I had been running never included it. Re-ran with
`--features rio-v2,test-util`: 3722 passed.
|
||
|
houseme
|
a8574d0104 |
fix(metrics): close dimension review gaps (#5656)
* fix(metrics): close dimension review gaps Co-Authored-By: heihutu <heihutu@gmail.com> * test(metrics): cover dimension review gaps Co-Authored-By: heihutu <heihutu@gmail.com> * test(metrics): cover failed disk info UUID fallback Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
035ce5d784 |
feat(obs): add bounded metrics dimensions (#5645)
* feat(obs): add drive topology detail metrics Expose additive drive info, topology, state, and per-drive API metrics while preserving the existing drive metric label sets. Backlog: rustfs/backlog#1655 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): preserve suspect drive runtime state Keep suspect as a bounded drive runtime state and avoid all-zero runtime_state samples for that storage health state. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): skip unknown drive inode samples Avoid exporting zero inode gauges for missing or stale drive snapshots and ignore zero-count API latency buckets. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner source work detail metrics Expose additive scanner source and cycle work metrics with bounded server/source/state labels while leaving the existing aggregate scanner metrics unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add ilm action detail metrics Expose additive ILM action/state task metrics with a server label while preserving the existing aggregate ILM series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add delivery target server metrics Expose additive audit and notification delivery target metrics with server labels and extend removed-target tombstones for the server-aware series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication target flow metrics Expose additive bucket replication target sent and failed-flow metrics while preserving existing bucket aggregates and target backlog series. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add request server metrics Expose additive API request metrics with server labels while preserving the existing request and traffic metric label sets. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): apply rustfmt to metrics changes Apply rustfmt output to the metrics dimension changes without altering behavior. Co-Authored-By: heihutu <heihutu@gmail.com> * style(obs): reuse audit target label constant Use the exported audit target_id label constant for legacy audit target metrics. Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): populate drive disk metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add scanner bucket drive result metrics Co-Authored-By: heihutu <heihutu@gmail.com> * feat(obs): add replication proxy server metrics Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric liveness review Use checked division for drive API latency aggregation and keep recovered drive, scanner current-cycle, replication flow, audit target, and notification target series from retaining stale values. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metric dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address additional metric review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): count drive calls at start Co-Authored-By: heihutu <heihutu@gmail.com> * fix(obs): address metrics dimension review Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address dimension review gaps Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address scanner review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): reduce disk metric contention Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): address runtime review follow-ups Co-Authored-By: heihutu <heihutu@gmail.com> * fix(metrics): retire stale dimension series Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
c09d11ff3b |
fix(config): fence persisted config updates and reloads (#5512)
* fix(config): fence persisted config updates and reloads * fix(ci): unblock config and e2e checks Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
唐小鸭
|
ad7663afd1 |
refactor(sse): decouple ecstore and harden KMS lifecycle (#5435)
* refactor(sse): decouple encryption from ecstore * feat(kms): enhance KMS service manager with runtime state and persistence support * feat(kms): add local key export functionality for SSE-S3 migration tests * fix(kms): keep local key export narrowly scoped * fix(sse): validate copy source customer algorithm --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
cxymds
|
67904a6c18 |
fix(ecstore): start with unresolved Kubernetes peers (#5460)
* fix(ecstore): start with unresolved Kubernetes peers * fix(ecstore): infer Kubernetes endpoint identity safely * fix(ecstore): fail closed on unsafe format migration * fix(ecstore): reject poisoned format heal candidates * fix(ecstore): reject unsafe legacy migration outliers * fix(ecstore): resume interrupted format migrations * fix(ecstore): preserve Kubernetes startup compatibility |
||
|
Zhengchao An
|
3991a1d73c |
test(ecstore): stabilize late snapshot lease cleanup (#5456)
test(ecstore): wait for late snapshot lease cleanup |
||
|
cxymds
|
cc24ef173c |
fix(ecstore): defer delete cleanup for snapshot reads (#5408)
* feat(ecstore): add local snapshot leases * feat(ecstore): add remote snapshot lease RPCs * feat(ecstore): protect streaming GETs with snapshot leases * fix(ecstore): defer version cleanup for snapshot reads * fix(ecstore): cover batch snapshot cleanup safely * fix(e2e): stub snapshot lease RPCs in lock mock * fix(e2e): stub snapshot lease RPCs in lock mock * fix(ecstore): bind deferred delete cleanup intents * fix(rpc): keep snapshot lease checks CI-compatible |
||
|
cxymds
|
91597db9d2 |
feat(ecstore): protect streaming GETs with snapshot leases (#5391)
* feat(ecstore): add local snapshot leases * feat(ecstore): add remote snapshot lease RPCs * feat(ecstore): protect streaming GETs with snapshot leases * fix(e2e): stub snapshot lease RPCs in lock mock * fix(rpc): keep snapshot lease checks CI-compatible * fix(ecstore): retain GET lock for missing lease disk * chore(proto): preserve node service formatting * fix(ecstore): harden snapshot lease deadlines * fix(ecstore): bound late lease cleanup |
||
|
cxymds
|
294c79c156 |
fix(tiering): gate remote version state safely (#5374)
* feat(tiering): model provider version capabilities * feat(tiering): persist opaque remote versions * fix(tiering): gate remote version state safely * fix(tiering): preserve remote version state on delete * fix(tiering): accept unversioned transition responses * fix(tiering): replay exact cleanup journals * test(tiering): pin empty exact cleanup guard * test(tiering): accept strict missing journal errors * test(tiering): exercise free-version identity guard * test(tiering): reach destination identity guard * test(tiering): persist version identity drift * test(tiering): bind version drift fixture --------- Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
5af56cbb02 |
test(ci): stabilize lifecycle timeout coverage (#5404)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
63e57378d6 |
fix(ecstore): never cache fabricated bucket metadata as authoritative (#5307)
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata (object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and cached it in the map that the map-only, fail-closed metadata_sys::get() serves. The object-lock batch-delete gate (object_lock_delete_check_required, backlog#929 / #4297) treats that map as authoritative, so a metadata miss became a cached "no lock" answer: a versioning peek could poison the cache and let delete_objects skip the per-object retention/legal-hold stat. The same fabrication raced make_bucket (lost update overwriting freshly persisted lock-enabled metadata) and let the 15-minute refresh loop replace good cached metadata on a transient quorum dip. Production changes: - get_config caches only metadata actually read from disk; misses are recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by set()) so repeated lookups for metadata-less names cost no extra namespace-lock + erasure-set fanout (reachable pre-auth via CORS preflight and per-key in DeleteObjects). - concurrent_load never lets a fabricated default REPLACE an existing map entry; startup insert-if-vacant behavior for legacy buckets is preserved. - delete_objects and new_ns_lock resolve dist-erasure, versioning, and the object-lock gate from the set's own instance context (backlog#1052) instead of the ambient facade, so a second in-process instance (or, in tests, another test's transient DistErasure window) cannot reroute locking onto an empty dist locker list or answer with the wrong instance's bucket state. Test-isolation changes (the bug that surfaced all of the above: the delete_objects lock-gating test failed deterministically when sharing a process with the lifecycle env tests): - The MinIO-migration test builds on an isolated InstanceContext instead of registering soon-deleted disks in the shared bootstrap registry. - The cached lifecycle env re-registers its disks on every use, surviving other serial tests' reset_local_disk_test_state. - Hermetic SetDisks helpers gain isolated-context variants pinned to plain erasure; tier-free non-serial test modules use them, guard-based SetupTypeGuard tests stay on the bootstrap context. - Three deterministic pin tests (nextest-safe) cover the caching contract, the delete gate resolution source, and the ns-lock resolution source. Verification: - cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the report> (previously failing, now green) - cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across repeated runs; cargo fmt --check and cargo clippy --lib --tests clean - Adversarial validation (high-risk tier, all seven roles) run per AGENTS.md; all findings fixed or rebutted with evidence |
||
|
houseme
|
09a991e735 |
test(perf): cover GET fallback metric probes (#5291)
* test(perf): cover GET fallback metric probes Co-Authored-By: heihutu <heihutu@gmail.com> * test(perf): prefer format fallback labels Co-Authored-By: heihutu <heihutu@gmail.com> * test(ilm): remove duplicate manual transition import Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Henry Guo
|
a63b79004c |
fix(scanner): make distributed usage convergence authoritative (#5151)
* fix(scanner): make distributed usage cycles authoritative * fix(scanner): close distributed refresh races * fix(config): align scanner reload integration * fix(admin): scope config test helpers * fix(scanner): harden distributed usage convergence * fix(scanner): preserve rolling activity compatibility * fix(admin): expose non-secret optional config values * fix(scanner): acknowledge distributed dirty usage * fix(ecstore): make bucket mutations cancellation safe * fix(scanner): preserve pending dirty acknowledgements * test(obs): account for superseded scanner metric * fix(api): reject excess detached bucket mutations * test: close scanner convergence coverage gaps * fix(scanner): make path tracking cleanup one-shot --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
houseme
|
2dc4d0b651 |
refactor(getobject): scope downstream close tagging (#5224)
Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
9ddb30139d |
fix(getobject): distinguish downstream output closure (#5220)
fix(getobject): preserve internal broken pipe failures Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
dd46de0945 |
fix(heal): report no-parity bitrot as unrecoverable (#5192)
Keep corrupted no-parity heal results on the integrity-failure path, preserve the object geometry in operator output, and document the recovery boundary for historical bad shards. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
358caa23cb | fix(storage): expose truthful storage class capabilities (#5172) | ||
|
cxymds
|
28fdcc87be |
fix(tiering): make rejected upload cleanup durable (#5059)
* fix(tiering): make rejected upload cleanup durable * fix(tiering): close transition upload cancellation gap * test(tiering): cover failed upload without candidate * test(tiering): synchronize cancelled cleanup recovery * test(tiering): stabilize cancelled cleanup recovery Prefer cancellation when the tier delete journal recovery worker is racing an immediate tick, and build the cancelled-cleanup regression store with an already-cancelled token so production recovery cannot consume the test journal. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: houseme <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
35f3599992 |
fix(ecstore): fence restore final commit by operation id (#5062)
Refs rustfs/backlog#1356 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
9e4c5e949f |
fix(ecstore): fence restore cleanup by operation id (#5058)
Refs rustfs/backlog#1356 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
cxymds
|
b0c6c4cbce |
fix(storage): resolve erasure parity per pool (#4977)
* fix(filemeta): add state-aware file info validation
* fix(filemeta): validate shard arithmetic and delete paths
* fix(ecstore): add fallible erasure construction
* fix(ecstore): resolve storage parity per pool
* fix(storage): report heterogeneous erasure layouts
* fix(admin): publish prepared storage config atomically
* fix(storage): harden per-pool parity boundaries
* fix(storage): address pre-PR validation findings
* test(ci): fix strict-topology validation fixtures
* fix(heal): preserve delete markers during repair
* refactor(filemeta): drop unused ValidatedFileInfo witness
ValidatedFileInfo wrapped an unread `_file_info` reference alongside an `Option<ValidatedErasureLayout>`, but only the layout was ever consumed. Return the layout directly from `FileInfo::validate` so the sole production consumer (`LocalDisk::check_parts`) and the two unit tests read it without the extra witness type and lifetime.
No behavior change.
* fix(filemeta): keep compressed and MinIO-migrated tiered objects readable
The new decode-path validation rejected several legitimate on-disk shapes that older RustFS and MinIO-migrated data carry, turning readable objects into FileCorrupt:
- Compressed objects written with an unknown upload size persist a negative per-part actual_size (the documented "unknown size" sentinel that ObjectInfo::get_actual_size already tolerates). validate_collection_contents rejected it via usize::try_from; now a negative actual_size skips shard validation and only real, non-negative sizes are checked.
- MinIO-migrated objects transitioned to a versioned remote tier store the tier version id as a UUID string, not 16 raw bytes. MetaObject::into_fileinfo returned FileCorrupt (main tolerated it as None), making all versions of the object unreadable; MetaDeleteMarker free-version records took a Some(nil) sentinel path with the same effect, which also breaks free-version expiry (remote-tier leak). Both now decode through a shared transitioned_version_id_from_meta_sys helper: 16 raw bytes or a UUID string are accepted, anything else is tolerated as None instead of failing the read.
Regression tests updated to assert the readable/compat behavior, with new tests covering MinIO string-form recovery.
* fix(scanner): build the delete-marker test fixture without erasure geometry
get_size_counts_delete_markers_separately_from_versions built its delete marker with `FileInfo::new(object, 1, 1)`, which attaches erasure geometry (data=1/parity=1/distribution). This PR classifies versions by shape via `is_storage_delete_marker()` (no geometry) rather than the raw `deleted` flag, so a geometry-bearing "delete marker" is correctly serialized as a purge-pending payload Object and counted as a version — CI saw summary.versions=3, expected 2.
Real delete markers carry no erasure geometry (delete paths build them as `FileInfo { deleted: true, ..Default::default() }`), so construct the fixture the same way. It then classifies as a storage delete marker and the counts (versions=2, delete_markers=1) hold. This keeps the PR's more-correct classification, which prevents a purge-pending object's geometry from being dropped when serialized as a bare delete marker.
* docs(changelog): note per-pool parity fix and storage-class startup upgrade caveat
Records the #4801 per-pool erasure parity fix under Fixed, and documents the upgrade behavior where a persisted storage class that a small or heterogeneous pool cannot satisfy now fails startup — with the RUSTFS_STORAGE_CLASS_STANDARD recovery steps. Docs-only; covers R4 from the on-disk compatibility audit.
* fix(heal): report parity from erasure geometry, not is_valid()
heal_object set HealResultItem.parity_blocks via `if lfi.is_valid()`, which was missed by the migration of the other quorum/metadata predicates. With the new `is_valid()` semantics (full payload validation; delete markers now return false), a delete marker or a geometry-bearing version with a benign collection quirk would misreport parity as the pool default instead of its own. Use `has_valid_erasure_geometry()` — the narrow "does this carry erasure geometry" predicate the rest of the migration uses — so reporting matches the object's actual layout. Reporting-only; no data-path change.
* fix(filemeta): do not silently serialize a non-canonical deleted FileInfo as an Object
`From<FileInfo> for FileMetaVersion` classifies by `is_storage_delete_marker()` (shape), which correctly routes canonical delete markers to Delete and purge-pending payloads (deleted=true with real erasure geometry) to Object. But a `deleted` FileInfo that is neither a canonical marker nor a valid erasure payload would silently serialize as a zero-geometry MetaObject that later fails `validate_for_metadata_read`. Write paths validate first (`validate_for_erasure_write` / `validate_for_metadata_read`), so this is a caller bug; `From` is infallible, so surface it with a structured `warn!` on the malformed branch instead of writing corrupt metadata silently. Legitimate purge-pending objects (valid geometry) are unaffected — the guard only fires for `deleted && !has_valid_erasure_geometry()`.
* test(filemeta): assert real historical xl.meta versions pass metadata-read validation
Empirical companion to the code-reasoned decode-tolerance invariants (docs/architecture/erasure-coding.md §11) and the rolling-upgrade / MinIO-migration compatibility concern: the tightened `validate_for_metadata_read` runs on every local disk read and peer-RPC-decoded FileInfo, so it must accept every version of real historically-written xl.meta, never reject it as FileCorrupt.
Loads five real fixtures — MinIO small-inline, MinIO versioned (two object versions + a delete marker), MinIO large multipart, a legacy V1 (xl.json-derived) object, and a legacy meta_ver 2 object — decodes every version with parts materialized, and asserts validate_for_metadata_read() is Ok for each. Reverting the tolerant handling (delete-marker shape, legacy per-part checksums, string/short transitioned-versionID, negative actual_size) turns this red.
* fix(ci): remove duplicate storage test re-exports
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
|
||
|
houseme
|
21049401fa |
fix(ilm): harden tier transition failure boundaries (#5031)
* fix(tier): fence generation-scoped operations Refs rustfs/backlog#1354 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ilm): verify transition upload streams Refs rustfs/backlog#1353 Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): expand transition fault matrix Refs rustfs/backlog#1355 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
15f4e75870 |
fix(cache): harden object data cache coordination (#5004)
* fix(cache): enforce projected entry capacity Refs: rustfs/backlog#1335 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): fence identity budget eviction by generation Refs rustfs/backlog#1334. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): fence clear against concurrent fills Refs rustfs/backlog#1333 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): linearize memory reservation claims Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): retain allocation memory claims Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): publish memory snapshots by epoch Co-Authored-By: heihutu <heihutu@gmail.com> * fix(cache): coordinate cold object fills Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): fence metadata cache transition races Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
4d22ed4465 |
perf(capacity): drop per-PUT global lock and per-disk allocation from write dirty-scope (#4933)
perf(capacity): remove per-PUT global lock and per-disk allocation from write dirty-scope Every successful write recorded its capacity dirty scope by allocating an endpoint/path String per online disk, deduplicating through a HashSet, entering the global dirty-scope Mutex, and — in the app response path — taking a global async RwLock to record the write frequency. Under small-object high concurrency this created a global serialization point and O(disks) allocation on the hot path (https://github.com/rustfs/backlog/issues/1315). This change makes the steady-state write path allocation-free and lock-free without altering capacity accounting semantics: - Memoize the per-set dirty scope. Each set resolves its disks' immutable endpoint/path identity lazily into a slot-indexed cache and reuses a shared `Arc<CapacityScope>`; steady-state writes clone the Arc under a read lock instead of rebuilding String/HashSet. The heal path keeps an ad-hoc scope builder because it passes disks in erasure-distribution order rather than physical-slot order. - Add a monotonic generation to the global dirty-scope registry, advanced only when a non-empty drain removes disks. A set upgrades the global registry mutex only on the first write of each generation and then skips it while the generation is unchanged; the observed generation is read under the registry lock so a concurrent drain forces a re-mark, preventing lost updates. The write commits its bytes before recording the scope, so any drain that could remove the mark is ordered after the commit and the following refresh reads the committed bytes. - Replace the write-frequency `RwLock<WriteRecord>` with lock-free atomics: per-second CAS buckets, an atomic last-write timestamp, and an atomic total counter. The frequency window and debounce semantics the refresh scheduler relies on are unchanged. Capacity marking remains a conservative superset of the disks actually written, so admin/scan totals are byte-for-byte identical: extra dirty marks only trigger a re-read of a disk whose usage is unchanged. White-box tests assert the memoized scope equals the previous ad-hoc construction, that the global registry is upgraded exactly once per generation and re-marked after a drain, and that the lock-free write record is exact under concurrent contention. Ref: https://github.com/rustfs/backlog/issues/1315 |
||
|
Zhengchao An
|
b41bbe2db4 |
fix(ecstore): split rename_data signature from heal-convergence decision (#4926)
CompleteMultipartUpload enqueued a normal-priority heal whenever `rename_data` returned a `Some(versions)` signature. But the per-disk signature is produced for every object with <=10 versions, and a healthy quorum reduces to `Some` as well, so the `Option<Vec<u8>>` return value conflated two distinct facts — "a version signature exists" and "the committed replicas need heal". The result: nearly every healthy MPU completion self-enqueued a heal, while >10-version objects (signature `None`) did not — an algorithmic heal amplification on the healthy path (rustfs/backlog#1321). Replace the overloaded `Option<Vec<u8>>` second element of `SetDisks::rename_data` with an explicit `RenameConvergence` classification computed after the write-quorum gate: - AllSuccessIdentical — every attempted disk committed with an identical, known signature (no heal). - PartialCommit — write quorum met but a disk failed/offline; a committed replica is missing or stale (heal). - SignatureDivergent — all committed but signatures diverge, or mix signed (<=10-version) with unsigned (>10-version) disks (heal). - Unknown — all committed, no signature produced (>10 versions); latent divergence is left to the scanner backstop, not self-enqueued. `RenameConvergence::needs_heal()` is the single decision point. The version signature is now only comparison material; it no longer doubles as a heal flag. The old `select_rename_data_versions` / `reduce_common_versions` / `rename_data_versions_key` machinery that carried the conflation is removed. The heal submission in `complete_multipart_upload` moves off the ACK critical path into a detached task: it runs after the object lock is dropped and after the durable `rename_data` commit, survives cancellation of the completion future, and coalesces through the existing bounded / deduplicated / observable heal-channel admission (one submit per degraded completion, at most). A completion cancelled in the narrow window between the durable commit and reaching the enqueue is scanner-backstopped, as is the Unknown (>10-version) case. The PUT path (`object.rs`) binds the second element as `_` and is unchanged. The change is orthogonal to and composes with the #1312 commit fence on the same `rename_data` path (epoch rejection is a commit-gate failure surfaced through `Result::Err`, convergence is a post-commit signal); documented in docs/architecture/unified-object-generation.md. Tests: `classify_rename_convergence` white-box cases cover the full acceptance matrix (healthy 4/4 and 8/8, 3-same-1-divergent, failed/offline disk, no-common-quorum split, >10-version all-success and with-failure, mixed signed/unsigned) and fail on revert to the old "signature exists => heal" semantics. The decision function is tested directly rather than through the process-global heal channel, whose receiver is owned exclusively by the blackbox serial test (init_heal_channel is once per binary). Refs: https://github.com/rustfs/backlog/issues/1321 |
||
|
houseme
|
750e5d15eb |
feat(checksums): add native S3 additional checksum support (#4805)
* feat(rio): wire XXHash3/64/128 and SHA-512 into ChecksumType (S2) Add the AWS 2026-04 additional checksum algorithms as base types in rustfs-rio's ChecksumType, covering every dispatch site (key, raw_byte_len, hasher, Display, from_string_with_obj_type, BASE_CHECKSUM_TYPES) so no path silently strips them. Derive BASE_TYPE_MASK from BASE_CHECKSUM_TYPES as the single source of truth, allocate the new base-type bits append-only above bit 9 to preserve the on-disk varint format, and add streaming hashers whose digest uses the S3 canonical big-endian encoding (seed 0). The new algorithms are COMPOSITE-only: an explicit FULL_OBJECT request is rejected and they are never routed through add_part()/can_merge(). A round-trip guardrail test asserts every base type survives all dispatch sites, failing loudly if a future algorithm is added but a match arm or the mask is forgotten. Refs rustfs/backlog#1254 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(rio): pin XXHash/SHA-512 digests to official vectors, big-endian (S3) Lock the byte order and seed of the new algorithms against the OFFICIAL upstream xxHash / SHA-512 empty-input test vectors (XXH3-64, XXH64, XXH3-128, SHA-512), in big-endian, so the stored and echoed checksum is byte-for-byte identical to what AWS SDKs (awscrt) compute — the interop correctness this feature hinges on. Add a non-empty regression lock (official "fox" vectors) that also asserts the encoded field is the standard-base64 of the raw digest. Refs rustfs/backlog#1255 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(rio): lock on-disk checksum round-trip and forward-compat degrade (S8) Cover the xl.meta varint (de)serialization for the new algorithms: to_bytes() -> read_checksums() must recover the value under the Display key for XXHASH3/64/128 and SHA512. Pin the rolling-upgrade contract that a node reading a future, unknown base-type bit degrades safely — skips the entry and returns without panicking or mis-decoding a length. Combined with the append-only bit allocation from S2, this protects mixed-version clusters. Refs rustfs/backlog#1260 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(head): echo XXHash/SHA-512 additional checksums on HeadObject (S5) HeadObject with x-amz-checksum-mode: ENABLED now returns the XXHash3/64/128 and SHA-512 checksums that S3 stored, closing the head_object gap in #4800. s3s HeadObjectOutput has no typed field for these, so they are emitted as raw response headers via response.headers (the same mechanism RustFS already uses for tagging-count), keyed by ChecksumType::key(). The existing five typed algorithms are unchanged. Also carries the Cargo.lock update for the xxhash-rust dependency introduced in S2. Refs rustfs/backlog#1257 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * fix(checksums): fail-closed on unknown checksum algorithm (S7) A. Harden unknown/unsupported checksum algorithms to fail closed instead of panicking. ChecksumMode::base() in the outbound S3 client (crates/ecstore/src/client/checksum.rs) previously did `panic!("enum err.")` for any mode without a concrete base algorithm (e.g. a bare ChecksumFullObject flag); it now falls back to ChecksumNone. Added unit tests proving base() never panics and hasher() returns Err for unsupported modes. rustfs-checksums FromStr already returns Err on unknown names; added a regression test asserting garbage/unknown names fail closed. B. Extend rustfs-checksums ChecksumAlgorithm with the AWS 2026-04 additional algorithms Sha512/Xxhash3/Xxhash64/Xxhash128. Updated FromStr, as_str, into_impl, name constants, the x-amz-checksum-* header constants and the HttpChecksum impls. Byte order/seed matches the server-side rustfs-rio spec: xxh3/xxh64 as u64 big-endian (8 bytes, seed 0), xxh128 as u128 big-endian (16 bytes), sha512 via sha2::Sha512. Added tests validating each digest against a direct library computation. MD5 stays intentionally rejected (PR #4513) and is left untouched. C. crates/ecstore/src/client/checksum.rs ChecksumMode is enumset repr="u8" with 7 variants already consuming 7 bits; adding the 4 new algorithms would overflow u8 and require a breaking repr change, so ChecksumMode is left unchanged. The new algorithms are available through the rustfs-checksums ChecksumAlgorithm path. Refs rustfs/backlog#1259 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(get,put): echo XXHash/SHA-512 checksums on GetObject and PutObject (S5-GET, S4) Complete the additional-checksum round-trip so AWS SDKs can verify integrity on download and confirm it on upload: - GetObject with x-amz-checksum-mode: ENABLED now returns XXHash3/64/128 and SHA-512 checksums (the download-side path SDKs auto-verify). The values flow from build_get_object_checksums through GetObjectOutputContext into finalize_get_object_response and are emitted after wrap_response_with_cors. - PutObject echoes the server-computed additional checksum on its response, captured at the want_checksum set points before opts is moved. Both reuse a single centralized helper, inject_additional_checksum_headers, which HeadObject now also uses. This is the ONLY place that emits these headers, so when s3s gains typed fields for these algorithms the migration is one spot (fill the typed field, drop the insert) with no risk of duplicate headers. The five s3s-typed algorithms are unchanged. Trailing-checksum PUT echo (value lands after the body) is left for e2e coverage in S10. Refs rustfs/backlog#1257 rustfs/backlog#1256 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(multipart): support XXHash/SHA-512 composite multipart checksums (S9) Make multipart uploads work end-to-end for the composite-only algorithms (XXHash3/64/128, SHA-512): - complete_part_checksum previously returned the outer None for any algorithm outside the five typed ones, which failed CompleteMultipartUpload with InvalidPart. It now accepts any valid base type with no double-check value (Some(None)) — mirroring the missing-value path of the typed algorithms — since s3s CompletePart has no field to carry a client-supplied per-part value and the part was already verified server-side at UploadPart. Genuinely unset/invalid types are still rejected. - The existing COMPOSITE assembly (Checksum::new_from_data over the concatenated per-part raw digests; full_object_requested() is false so add_part() is correctly bypassed) already works for these algorithms via the S2 wiring. A rio test locks the assembly and that add_part refuses them. - UploadPart and CompleteMultipartUpload echo the new-algorithm checksum on their responses via the shared inject_additional_checksum_headers helper (now pub(crate)), since s3s has no typed output field. Refs rustfs/backlog#1261 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * feat(rio): add MD5 as an additional checksum (x-amz-checksum-md5) (S6) Wire MD5 into ChecksumType as an additional (flexible) checksum, distinct from the legacy Content-MD5 / ETag path: header x-amz-checksum-md5, 16-byte digest, COMPOSITE-only, md-5 hasher. Pinned to the official empty-input MD5 vector. Thanks to the single-source-of-truth wiring from S2, every dispatch site (GetObject/HeadObject/PutObject echo, multipart complete_part_checksum and the COMPOSITE assembly) picks MD5 up automatically via base()/key()/the catch-all arm — no handler changes needed. Tests are extended to cover MD5 across them. Coordination with #4513: that PR made the OUTBOUND rustfs-checksums client reject "md5" so it could never silently fall back to CRC32. This change is on the server-side rio path and never falls back — it implements MD5 correctly rather than substituting another algorithm — so the #4513 intent is preserved, and the outbound client keeps rejecting md5 (S7). Refs rustfs/backlog#1258 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * perf(rio): drop per-request to_uppercase alloc in checksum parsing (S11) from_string_with_obj_type ran alg.to_uppercase() on every checksummed request, allocating a String just to compare against a fixed set of algorithm names. Replace it with eq_ignore_ascii_case, which is allocation-free and, for the ASCII algorithm names involved, exactly equivalent. A test locks that case-insensitivity, the CRC64NVME full-object assumption, composite-only FULL_OBJECT rejection, and unknown/empty handling are all unchanged. The other S11 notes are intentionally not acted on: the Phase-0 header scan is N/A (we chose full support over rejection, so there is no reject guard), and parallelizing the serialized hash passes is deferred pending a measured need. Refs rustfs/backlog#1263 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * refactor(checksums): collapse 5 duplicated response-checksum loops into one Review of the accumulated commits found the same "iterate decrypted checksums, match five typed algorithms, drop the rest" loop copy-pasted across five response paths (GetObject, HeadObject, GetObjectAttributes object-level and part-level, CompleteMultipartUpload). That was patch-on-patch duplication. Collapse it into a single source of truth: - rustfs-rio gains ChecksumType::is_s3s_typed() — the one place that defines the five-typed vs additional-algorithm split. - object_usecase gains ResponseChecksums + classify_response_checksums(), which performs the typed/extra split once. All five call sites now destructure its result; additional_checksum_echo_pairs() also uses is_s3s_typed() instead of a hand-rolled five-way comparison. Behaviour is unchanged (GetObjectAttributes still cannot surface the additional algorithms — an s3s XML-body limitation, now documented in one spot). One pass over the map; extra pairs pushed only when a new-algorithm checksum is present. Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * test(checksums): unit tests for classifier/echo helpers + fix unused import Add direct unit tests for the refactored single-source-of-truth helpers: - rio ChecksumType::is_s3s_typed() — exhaustive typed-vs-additional split, and that flags (FULL_OBJECT/MULTIPART) on a base type don't change classification. - object_usecase classify_response_checksums() — typed fields vs `extra` headers, the checksum-type marker, and empty input. - additional_checksum_echo_pairs() — echo pair only for additional algorithms, none for the five typed ones, none for None. - inject_additional_checksum_headers() — writes all pairs; empty is a no-op. Also drop the now-unused AMZ_CHECKSUM_TYPE import in multipart_usecase.rs left by the classifier refactor (would fail the -D warnings gate). Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * style(rio): fix typo flagged by CI (mis-decoding -> decoding a wrong length) The Typos CI check flagged "mis-decoding" (it reads "mis" as a word). Reword the S8 forward-compat comment; no code change. Refs rustfs/backlog#1260 Co-Authored-By: heihutu <heihutu@gmail.com> * test(e2e): integration test for XXHash/SHA-512/MD5 additional checksums (S10) Permanent verify-on-write integration test in the e2e suite for the AWS 2026-04 additional algorithms. aws_sdk_s3 has no typed builder for these, so the x-amz-checksum-<algo> header is injected via mutate_request (value from rustfs-rio, byte-for-byte identical to awscrt). Uses a client with automatic checksum calculation disabled (request_checksum_calculation=WhenRequired) so the injected header is the only checksum on the wire. For each of XXHash3/64/128, SHA-512 and MD5: a correct value is accepted and the object stored intact; a mismatched value is rejected with BadDigest and nothing is stored. Verified passing locally (1 passed) alongside a boto3+awscrt round-trip that additionally confirms the HEAD/GET header echo (14/14). Refs rustfs/backlog#1262 rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> * style(get): allow too_many_arguments on finalize_get_object_response The classifier refactor added an extra_checksum_headers parameter, pushing finalize_get_object_response to 8 args and tripping clippy::too_many_arguments under CI's `-D warnings`. Add the same #[allow] the sibling GET helpers already carry; no behavior change. Refs rustfs/backlog#1252 Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
6886dca7d1 |
feat(ecstore): make GET codec-streaming a single rollout switch (backlog#1183) (#4752)
backlog#1183, staged rollout step. Simplify enabling the zero-duplex GET
codec-streaming fast path to a single switch — RUSTFS_GET_CODEC_STREAMING_ROLLOUT
(default "off") — now that body/header parity is proven (parity e2e net + bench
A/B on backlog#1183).
- Add a clean production rollout token "on" (aliases "full"/"production"); the
legacy "internal"/"benchmark" tokens remain accepted.
- Flip DEFAULT_RUSTFS_GET_CODEC_STREAMING_ENABLE and the two ..._COMPAT_CONFIRMED
defaults to true. They are retained as emergency kill-switches (set any to
false to force the fast path off) but no longer gate enablement — the rollout
switch does.
No production behavior change: with no env set the rollout switch defaults to
"off", so GET stays on the legacy duplex path exactly as before. Flipping the
hard default to on is deferred to a follow-up after a production soak.
Note (intentional semantics change): with the rollout switch opted in
("on"/"internal"/"benchmark"), codec streaming now activates without also
setting the two ..._COMPAT_CONFIRMED vars — compatibility is confirmed, so those
confirmations are baked in.
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
houseme
|
c2362bca14 |
perf(ecstore): slice in-memory shards instead of copying them twice (#4687)
* perf(ecstore): slice in-memory shards instead of copying them twice (backlog#1159)
The GET path reads a shard out of the page cache into a `Bytes`, then
`open_disk_reader` erased it behind `Box<dyn AsyncRead>` by wrapping it in
a `Cursor`. Downstream, `BitrotReader` could only get it back by copying:
once out of the `Cursor` into its scratch buffer, and once from there into
the caller's buffer. CPU profiling of a cached 1 MiB GET (device reads = 0)
attributed 8.23% of the whole server to `Cursor::poll_read` alone — a copy
of data that was already sitting in memory.
Keep the source concrete instead of erasing it. `ShardReader` is an enum of
`InMemory(Cursor<Bytes>)` and `Stream(Box<dyn AsyncRead ...>)`, and the new
`ShardSource::try_take_block(n)` lets an in-memory source hand over the
`[hash][data]` block as a slice. `read_appending` uses it to verify the hash
on the slice and `extend_from_slice` the shard straight into the caller's
buffer: one copy instead of two.
`try_take_block` defaults to `None`, so a streaming source keeps the old
path byte for byte, along with its short-read and EOF semantics. A source
that cannot serve `n` bytes declines rather than truncating, so a partial
block still becomes UnexpectedEof rather than a short shard. The hash is
still checked before anything is appended, so a corrupt shard never reaches
the caller's buffer on either path. The deferred parity reader opens its
source lazily and stays on the streaming path; parity is only read when a
data shard fails.
Tests gate equivalence and non-vacuity:
* `try_take_block` fires for `Cursor<Bytes>`, advances the position exactly
as a read of the same length would, declines when fewer than `n` bytes
remain, and returns `None` for a non-`Bytes` source — without this the
equivalence test below would silently compare one path against itself;
* both paths return identical bytes for the same shard;
* a corrupt shard fails on the fast path too, appending nothing.
Verified: clippy --tests -D warnings clean; `erasure::` 215 passed, 0 failed;
`set_disk::core::io_primitives` 49 and `io_support::` 22 pass.
Stacked on #4681 (`read_appending`), which this builds on.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(ecstore): implement ShardSource for Cursor<&[u8]> used by the erasure bench
`crates/ecstore/benches/erasure_benchmark.rs` builds
`BitrotReader<Cursor<&[u8]>>`, which the new `ShardSource` bound on
`ParallelReader`/`decode` does not accept. `cargo clippy --tests` does not
compile bench targets, so this only surfaced in CI's `--all-targets` run.
A borrowed slice carries no `Bytes` to hand out, so it takes the default
`try_take_block` and keeps the old streaming copy path — no behavior change.
Verified with the same target set CI uses:
`cargo clippy -p rustfs-ecstore --all-targets -- -D warnings` clean.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
|
||
|
GatewayJ
|
a269f8df05 | fix(ecstore): require write quorum for metadata early stop (#4300) | ||
|
Zhengchao An
|
91a5c87132 | refactor(startup): thread explicit InstanceContext through the storage startup path (#4611) | ||
|
Zhengchao An
|
ce6bc30b26 | test(ecstore): harden validation gate and EC coverage tests (#4590) | ||
|
Zhengchao An
|
ed81d2f6b8 |
test(ecstore): complete EC validation coverage gate
* test(ecstore): complete EC validation coverage gate * test(ecstore): stabilize validation suite after rebase * test(ecstore): fix rio-v2 clippy lint |
||
|
houseme
|
f262fcfce0 |
perf(hotpath): add fine-grained PUT-path stage guards (HP-14) (#4541)
Close the ~10ms instrumentation residual on the PUT success path that the existing coarse writer_setup/encode/rename stage metrics do not attribute. Adds `hp_guard!` measurement scopes to the previously uninstrumented sub-stages called out in backlog#935 item 2: - SetDisks::acquire_read_lock / acquire_write_lock (namespace lock acquire) - S3::put_object_prelookup (pre-write get_object_info lookup) - MultiWriter::shutdown (bitrot writer flush/close) - SetDisks::commit_rename_data_dir (old data-dir reclaim) - S3Access::put_object (S3 authorization segment) Instrumentation only: `hp_guard!` expands to nothing without the `hotpath` feature, so this is a pure-observation change with zero behavior impact and zero cost in default builds. The pre-lookup site wraps only the lookup call in a scoped block so the guard measures that slice exactly while preserving the existing match and control flow. Verified: cargo check -p rustfs-ecstore (default and --features hotpath) and cargo check -p rustfs (default and --features hotpath) all pass. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
15808254d3 |
fix(ecstore): correct codec-streaming byte accounting and partNumber routing (#4535)
Two correctness defects on the opt-in codec-streaming GET path. ECA-02 (#943): ErasureDecodeReader only decremented `remaining` for the main fill buffer. Under the default DualInFlight policy each fill also produces a queued stripe that is delivered to the client via `prefetched_bufs.pop_front()` without touching `remaining`, so any object larger than one erasure block finished with `remaining > 0` and the GET terminated with LessData despite delivering all bytes. The inflated `remaining` was also fed back into the fill worker, which used it to trim the final stripe and to decide whether to read past EOF. Account for the queued-stripe bytes when they enter the prefetch queue; queued buffers come only from `Ok(true)` decodes so they are non-empty and bounded by `remaining - main_buf.len()`, ruling out underflow. ECA-04 (#945): the codec-streaming gate did not inspect `opts.part_number`. A partNumber GET carries `range == None`, so it was not classified as a Range request and reached the full-object codec-streaming reader, which drops the storage offset/length returned by GetObjectReader::new. A partNumber >= 2 request would then stream the whole object. Mirror the direct-memory part_number fallback and route any partNumber request back to the legacy duplex path, which applies the offset/length correctly. Regression tests: DualInFlight read_to_end on a multi-block object and on a non-block-aligned object; SingleInFlight vs DualInFlight byte-identical output; gate fallback on partNumber requests. Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
3531abb34a |
feat(heal): disk-walk UNION enumeration to heal sub-quorum versions (backlog#920) (#4527)
B5 switched heal enumeration to list_object_versions, which only reflects the read-quorum metadata view: a version present on fewer than read-quorum disks was never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw + mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and feeds each to the existing per-version heal_object. - filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the cross-disk version union at one tested seam. - ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1, dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator + HealWalkVersion. - ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta physically probes part files via check_parts; when >= data_blocks data shards survive (meta lost but data recoverable) it regenerates xl.meta from a surviving FileInfo with the correct per-disk shard index instead of dangling-deleting. Genuine torn writes (< data_blocks) keep the current behavior — no resurrection. - heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor, idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else the unchanged B5 path; anti-loop guard aborts on (empty && truncated). Closes rustfs/backlog#920 |
||
|
Zhengchao An
|
91dec123d9 |
refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type (#4413)
* refactor(ecstore): add per-instance InstanceContext, migrate erasure setup type Phase 5 of the global-singleton consolidation (backlog#939): begin moving runtime identity state out of process globals so multiple ECStore instances can coexist in one process. Isolation is carried by the object graph (ECStore -> Sets -> SetDisks holding an Arc<InstanceContext>), not a task-local, which does not propagate across the many internal tokio::spawn boundaries in the data/background paths. This first slice migrates the erasure setup type -- previously three independent process-global bools -- into a single per-instance RwLock<SetupType> that derives is_erasure / is_dist_erasure / is_erasure_sd, removing a triple source of truth that could drift out of sync. - New runtime::instance module: InstanceContext + process bootstrap context. - The legacy free-function facade (is_erasure/update_erasure_type/...) keeps its signatures and forwards to the current instance's context, falling back to the bootstrap context before a store is published. - ECStore gains a pub(crate) ctx field and setup_is_* accessors; its constructors adopt the bootstrap context (never mint a fresh one) so startup writes and post-construction reads share one cell -- single-instance behavior is byte-for-byte unchanged. Tests: erasure predicate derivation vs the legacy behavior, object-graph carrier isolation across two ECStore instances, and bootstrap adoption. Refs: backlog#939 (Phase 5, Slice 1), backlog#653 (item 8) * refactor(ecstore): thread InstanceContext down the object graph (Phase 5 Slice 2) (#4415) * refactor(ecstore): source the namespace lock manager per-instance (#4417) refactor(ecstore): source the namespace lock manager per-instance (Phase 5 Slice 3) Phase 5 Slice 3 (backlog#939): give each instance its own lock namespace by sourcing SetDisks' lock manager from the instance context instead of the process singleton. This removes the false cross-instance mutual exclusion (and attendant ABBA risk) that a shared GlobalLockManager would cause once multiple instances coexist. - InstanceContext gains a `lock_manager: Arc<GlobalLockManager>`. `new()` mints a fresh manager (independent per-instance); `bootstrap_ctx()` aliases the process singleton via get_global_lock_manager(), so a single-instance deployment keeps exactly one shared namespace. - SetDisks::new sources `local_lock_manager` from `ctx.lock_manager()` (the ctx it already adopts), not `runtime_sources::global_lock_manager()`. Single instance: same Arc as before, so behavior is unchanged. - Remove the now-unused `runtime_sources::global_lock_manager()` wrapper. Tests: bootstrap lock manager aliases the process singleton; two fresh contexts own distinct managers; a SetDisks' lock manager is the one from its context and aliases the global singleton in a single-instance build. Verification: cargo test -p rustfs-ecstore (10 Phase 5 + set_disk locking regressions green), cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass). Refs: backlog#939 (Phase 5, Slice 3). Stacked on #4415 (Slice 2). |
||
|
Zhengchao An
|
1e6207c08e |
fix(lock): fence write commit on lock loss (#4406)
fix(lock): fence write commit on lock loss (backlog#899 Phase 2) Phase 0+1 (#4388) made object write locks refreshable and marks the guard lost when the heartbeat can no longer refresh a quorum, but does not act on it. Under a partition a long write's lock can expire on an unreachable node and be reclaimed, letting a third party re-acquire it; the original writer keeps going and both commit -- a double write. Expose the loss signal through NamespaceLockGuard::is_lock_lost() and ObjectLockDiagGuard::is_lock_lost(), and fence the commit in put_object and complete_multipart_upload: immediately before rename_data (the atomic commit point), abort with a retryable NamespaceLockQuorumUnavailable (503) if the lock was lost. In multipart the check precedes cleanup_multipart_path so a lost lock leaves the upload intact and retryable. A write that already reached rename_data Ok is durable and never aborted. The loss criterion is unchanged (reacts to Phase 1's signal). Heal and the long-GET read side are deferred follow-ups. |
||
|
houseme
|
a413729b16 | perf(delete): gate and parallelize DeleteObjects per-object stat fanout (#4398) | ||
|
Zhengchao An
|
afc7f1d6f9 |
fix(ecstore): make post-commit old data dir cleanup best-effort (#4386)
* fix(ecstore): make post-commit old data dir cleanup best-effort (backlog#898) A write is authoritatively committed once rename_data returns Ok (the new version is durable on >= write_quorum disks and immediately readable). The subsequent reclamation of the now-dereferenced old object/<data_dir> is pure space reclamation, yet commit_rename_data_dir propagated a below-quorum GC failure via `?` into ErasureWriteQuorum -> 503, producing a false-negative ACK for an already-persisted write. This is a deliberate divergence from MinIO (erasure-object.go:1577), which couples the two; the divergence is justified by durability semantics, not parity. Changes: - commit_rename_data_dir now returns a structured OldDataDirCleanup receipt and never returns Err. Adds an old==committed-dir anti-misdelete guard and a committed_data_dir parameter. Classification is extracted into pure functions (classify_old_data_dir_cleanup / map_cleanup_join_result / is_cleanup_not_found) so it is unit-testable. Task panic/cancel is mapped to a non-ignored DiskError::other (never DiskNotFound), and not-found is normalized to reclaimed. - object.rs / multipart.rs consume the receipt instead of `?`. The result reverts to Ok, so the invalidate_get_object_metadata_cache self-heal and the capacity/compression accounting that a `?` early-return previously skipped now run on the cleanup-failure path too. - On residue, report_old_data_dir_cleanup emits leak metrics and enqueues an object heal over the existing heal channel (disk-health signal replacing the 503). heal_object -> reclaim_orphan_data_dirs already reclaims unreferenced local data dirs, closing the loop end to end. - Adds rustfs_old_data_dir_* counters (attempted/reclaimed/leaked/below_quorum) as the operator-visible backstop for leaked residue. - Adds a test-only (#[cfg(test)]) delete fault-injection seam; in production it inlines to a no-op None and has no behavioral effect. Tests: pure-function A/C group + join-error mapping + actions decision; A5/A5b real-disk guard/reclaim integration; end-to-end overwrite returning 200 while old-data-dir cleanup fails. #864 rollback guard test remains green. * fix(ecstore): resolve merge conflicts with origin/main in io_primitives.rs --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: houseme <housemecn@gmail.com> |
||
|
Zhengchao An
|
65953bfdb3 | fix(ecstore): reduce rename data version signatures (#4383) | ||
|
houseme
|
58114f49f2 |
perf(ecstore): k-way heap merge for ListObjects, drop clone-to-parse (#4347)
* perf(ecstore): replace linear merge scan with k-way heap and drop clone-to-parse (backlog#874 backlog#875) merge_entry_channels advanced the k-way merge with a linear scan over all channel heads (O(entries x channels)) and allocated two fresh Strings per pairwise comparison via path::clean. Every step also cloned MetaCacheEntry values, including entry.clone().xl_meta() clone-to-parse calls. - Introduce MergeHead with a cached cleaned name (allocated only when the raw name is not already clean) and drive the merge with a BinaryHeap of boxed heads: O(log channels) per entry, allocation-free comparisons. - Move entries through the merge instead of cloning; the winner is sent without an intermediate copy. - Remove the dead merge_file_meta_versions block: it only ran for prefix-dir groups whose entries have empty metadata, so xl_meta() always failed; cross-drive version merging happens in the resolve path. - Keep legacy same-name semantics (dir groups collapse, objects shadow prefix dirs, later object candidate wins) and add regression tests for interleaved ordering, dir/object precedence, uncleaned-name grouping, and prefix-dir collapse. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): honor ascending versions_sort in ListObjects walk (#4348) * fix(ecstore): honor ascending versions_sort in walk and document ordering invariant (backlog#876) The walk loop carried a bare `//TODO: SORT` inside the `WalkVersionsSortOrder::Ascending` branch, so the requested ascending order was silently ignored and versions streamed newest-first (the raw FileMeta order). WalkOptions defaults to Ascending, so every default walker -- notably replication resync, which replays versions and needs oldest-first to preserve the version-stack order -- received the exact opposite of the contract. FileMeta maintains versions newest-first (sort_by_mod_time is descending) and into_file_info_versions preserves that order, so ascending emission is the exact reverse of file_info_versions output. Reverse in place when ascending is requested and add a regression test locking the newest-first invariant plus the reversal contract. Key-ordering audit result (no gap found): per-disk walkers emit sorted streams, merge_entry_channels performs an ordered k-way merge, and gather_results only filters by marker/limit, so ListObjects key order is guaranteed upstream and needs no post-sort. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): enable GET metadata early-stop by default (#4349) * perf(ecstore): enable GET metadata early-stop by default with env opt-out (backlog#872) The metadata early-stop fanout (read_all_fileinfo_early_stop) has been implemented and instrumented for a while but stayed behind an opt-in flag, so default GETs always waited for every disk to answer the metadata read even after quorum agreement was reached. Flip RUSTFS_GET_METADATA_EARLY_STOP_ENABLE to default-on. The gate stays conservative: should_allow_metadata_early_stop only admits metadata-only reads (read_data=false) without version_id, healing, or free-version requirements, everything else falls back to the full-wait fanout, and setting the env var to false restores the old behavior entirely. The version-aware gate (RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE) remains opt-in because versioned reads carry a higher stale-selection risk profile. Also replace the stale "optimize concurrency" TODO in get_object_fileinfo with a pointer to the early-stop implementation and add regression tests for the new default plus the explicit opt-out path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): lazily construct codec streaming multipart readers (#4350) * perf(ecstore): lazily construct codec streaming multipart part readers (backlog#871) get_object_decode_reader_with_fileinfo opened shard readers for every part of a multipart object before returning the streaming reader, so TTFB paid for parts x disks file opens up front and an early client disconnect wasted the setup work for every unread part. Replace the eager loop with LazyMultipartCodecStreamingReader: the first part is still built eagerly so the dominant fallback conditions (missing shards / read quorum) are detected before any byte is streamed and the whole request can fall back to the legacy duplex path exactly as before. Each subsequent part is built on demand -- when the previous part hits EOF -- via a spawned task handle owned by the reader; dropping the reader aborts an in-flight build so disconnects stop all further IO. If a later part hits a fallback condition mid-stream (a shard vanished after the request started), the reader surfaces an explicit read error with a pipeline-failure metric instead of silently degrading; the client's retry then detects the condition on the eager first-part setup and takes the legacy path cleanly. Adds unit tests for in-order streaming across lazy boundaries, deferred construction (no build when the client stops within part 1), and the mid-stream fallback error path. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): prefetch next multipart part reader setup during decode (#4351) * perf(ecstore): prefetch next multipart part reader setup during decode (backlog#870) get_object_with_fileinfo processed multipart parts strictly serially: the next part's bitrot reader setup (file opens + read-quorum wait across all disks) only started after the current part finished decoding, so large multipart reads paid full setup latency between every part. Overlap the two stages with a depth-one pipeline: right after the current part's readers are obtained, the next part's setup is spawned (shared inputs behind Arc) and joined when the loop reaches that part. The shared setup_multipart_part_readers helper keeps stage-duration metrics semantics identical for both paths; a failed or stale prefetch falls back to the synchronous setup, and the PrefetchedReaderSetup guard aborts the in-flight task on error returns, early breaks, or caller drop so disconnects stop background disk IO. Gate: RUSTFS_GET_MULTIPART_READER_SETUP_PREFETCH (default on, env opt-out). Adds a three-part end-to-end read test covering the prefetch hit path and cross-part content ordering. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): move FileInfo through GET shuffle instead of cloning (#4352) perf(ecstore): move FileInfo entries through the GET shuffle instead of cloning (backlog#873) shuffle_disks_and_parts_metadata_by_index deep-cloned every valid FileInfo (parts, erasure info, metadata map) once per disk on each GET. Add an ownership-taking variant that runs the same by-index consistency check as a read-only first pass and then moves entries into their shuffled slots with mem::take, and switch get_object_with_fileinfo to it -- that call site already owned the parts metadata vector. Disk handles are Arc clones and stay cheap. Scope notes from the backlog#873 audit: - get_object_fileinfo's disks.clone() stays: DiskStore is Arc<Disk>, so the clone is per-slot refcounting and correctly avoids holding the RwLock read guard across the metadata fanout awaits. - get_object_decode_reader_with_fileinfo keeps the borrowing shuffle: its caller must retain files/disks for the legacy fallback path, so an owned variant would just shift the same clone upstream. - The metadata-cache hit path still clones parts_metadata; sharing the cached entry via Arc changes the read-path return types and is left as a follow-up. Equivalence tests cover both the by-index placement and the mod-time fallback against the borrowing variant. Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> * fix(ecstore): gate merge emission on cleaned key and clear clippy redundant_clone Address review + CI findings on the ListObjects/GET optimization PR: - merge_entry_channels gated emission on the raw entry name while the heap orders by the cleaned key, so entries whose cleaned order and raw byte order disagree (e.g. redundant slashes) could be dropped. Gate on the same cleaned sort key the heap uses; add a regression test (`a//c` after `a/b`). - Drop three redundant `.clone()` calls in test code flagged by clippy::redundant_clone (owned-shuffle equivalence tests and the walk ascending-versions contract test) that failed the CI clippy gate. - Document the known mid-stream fallback limitation of the opt-in multipart codec streaming reader (default off) and mark the in-place per-part legacy degradation as a follow-up. Co-Authored-By: heihutu <heihutu@gmail.com> * fix(ecstore): force full metadata fanout for object tagging writes (backlog#872) put_object_tags reads the object fileinfo with read_data=false and then writes the updated tags to the online-disk set that read returned. With metadata early-stop enabled by default, that read now returns as soon as read quorum is reached, so the online-disk set is only a read-quorum subset. Writing tags to that subset fails write quorum -> ErasureWriteQuorum -> S3 SlowDown, which is exactly the s3-tests tagging failures (PutObjectTagging/DeleteObjectTagging, reached max retries). Thread a caller-controlled `allow_early_stop` gate through read_all_fileinfo_observed/_inner and add get_object_fileinfo_gated; put_object_tags calls it with allow_early_stop=false so the metadata read does the full quorum fanout and returns the complete online-disk set as the write target. Pure-read callers (GET/HEAD/tag read) keep the early-stop fast path unchanged. Extract metadata_early_stop_permitted() as the single gate and add a unit test locking the invariant: caller opt-out (and observe=false, and data reads) never early-stop even with the env flags on. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
c9b976ad46 |
feat(ecstore): reclaim orphaned data dirs during heal (#4356)
* feat(ecstore): reclaim orphaned data dirs during heal (backlog #3231/#3191) Pre-#3510, an unversioned overwrite leaked one UUID-named data dir per PUT. The write path now cleans up going forward, but pre-existing strays stay on disk forever: heal's dangling logic only removes whole objects whose data is missing, never surplus data dirs of an otherwise-healthy object. That leaked space is what eventually made ListObjects scan tens of GB and time out. Add SetDisks::reclaim_orphan_data_dirs, a fail-closed, quorum-safe sweep that mirrors purge_orphan_dir_object: - referenced set is the UNION of get_data_dirs() across every online replica's xl.meta, so a dir named by any replica is kept; - if a disk holds the object dir but its xl.meta is missing or unparseable the object is treated as degraded and nothing is removed; - only UUID-named subdirectories are ever considered. Wire it into heal_object's healthy tail (under the object write lock) as a best-effort step so the background heal scanner and admin heal recover the leaked space automatically; a reclaim failure never fails the heal. Covered by four unit tests: removal of an unreferenced dir, no-op when all dirs are referenced, fail-closed abort on missing metadata, and cross-replica union preservation. Co-Authored-By: heihutu <heihutu@gmail.com> * docs(ecstore): fix typo unparseable -> unparsable Satisfies the repository typos CI check. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
31c6859965 |
chore: converge stale TODOs and apply safe fills (backlog#646) (#4322)
Second TODO-convergence round over the current tree (backlog#646). All line numbers in the old inventory had gone stale after the set_disk / diagnostics / cluster refactors, so this re-scans and reduces the marker count from 144 to 99. STALE removals (comment describes already-implemented behavior, or dead commented-out blocks) across ecstore (set_disk ops/core, store, cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and rustfs auth/object_usecase. No behavior change. Safe fills, each verified: - filemeta: replication_info_equals now also compares replication_state_internal (function currently has no callers; adds a regression test). - bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify and the now-unused `sum` on LocalDisk::bitrot_verify, removing a Bytes::copy_from_slice allocation. Streaming verify uses the file's embedded per-shard hash, never the passed sum. - signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the non_upper_case_globals allow. - admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a JSON body (would panic); rewire to serde_json::from_slice to match the production decode path, add assertions, un-ignore. Verified: cargo fmt; cargo check on touched crates; tests pass (filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts pass. |
||
|
Zhengchao An
|
3e4c15da5d | fix(object-lock): prevent locked version deletes (#4297) | ||
|
Zhengchao An
|
a8d8e56478 | refactor(ecstore): relocate NamespaceLocking + finalize God-Object split (backlog#822) (#4291) |