Commit Graph

7595 Commits

Author SHA1 Message Date
Chris fca83c72e4 ci: move compile-free and prebuilt-binary lanes to hosted runners (#8679)
* ci: move compile-free and prebuilt-binary lanes to hosted runners

Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.

* docs(ci): document runner billing and placement rules

Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
2026-10-19 08:55:29 +09:00
Chris 22740876b3 docs(architecture): record scanner usage authority and PUT-tail decisions (#8678)
Supersede the 2026-09-03 decision: hard quota admission is owned by the
durable reservation ledger, and scanner data usage is best-effort reporting
input. Record that a pending ordinary PUT commit is a write-barrier event
rather than a usage-publication veto, and split the scanner plan identity
into a coverage identity and a reuse identity.

Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904.
2026-10-19 08:33:28 +09:00
overtrue e0f0628bc1 feat(storage): seal artifact writes across one disk inventory 2026-10-19 08:15:20 +09:00
Chris ab7001a714 docs: unify agent instructions across tools (#8675) 2026-10-19 07:52:51 +09:00
Hauser b2770312f2 feat(policy): support object creation operation conditions (#8671)
* feat(policy): support object creation operation conditions

Derive the condition from the resolved S3 operation and keep multipart staging separate from object publication. Reject request and claim spoofing and cover policy evaluation and HTTP upload paths.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(protocols): preserve multipart staging authorization

Keep the trusted protocol operation until the shared policy context is built, so FTPS and SFTP honor object creation conditions on multipart staging. Reuse the server-only marker and test real backend decisions for IAM and bucket policies.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): cover ObjectCreationOperation in STS session policies

Run AWS's conditional-write example 1 as an inline AssumeRole session
policy: object publication needs a request header, while multipart
staging is exempt through s3:ObjectCreationOperation.

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: oc7o <oc7o@sloow.de>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-19 07:17:42 +09:00
Chris e0e18908d5 fix(heal): place checkpoint serialization impls before tests (#8663)
* fix(heal): move impls before test module to satisfy clippy items_after_test_module lint

Clippy in Rust 1.99 flags items defined after #[cfg(test)] mod tests.
Move Serialize and Deserialize impls for CheckpointJson before the
test module in crates/heal/src/heal/resume/checkpoint/integrity.rs.

* fix(heal): restore checkpoint integrity file newline

Add the missing final newline so cargo fmt --all --check passes.
The checkpoint serializer implementations and tests remain unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 07:08:49 +09:00
Ramakrishna Chilaka 34e883e16a perf(filemeta): avoid clones and intermediate allocations in xl.meta codec (#8661)
* perf(filemeta): avoid clones and intermediate allocations in xl.meta codec

- Slice version headers and metadata payloads directly from the xl.meta buffer in FileMeta::unmarshal_msg instead of allocating temporary header/metadata buffers per version.
- Decode FileInfo directly from borrowed xl.meta slices in get_file_info without constructing an intermediate FileMeta, Vec<FileMetaShallowVersion>, or InlineData copy.
- Validate and search InlineData keys on borrowed slices without per-key Vec/String allocations.
- Borrow FileMetaVersion in FileMetaVersionHeader::from and stream object/delete-marker signatures directly into Xxh64Writer instead of cloning full version structs and marshaling temporary buffers.
- Encode FileMetaVersionHeader into a stack buffer in FileMeta::marshal_msg and pre-size output buffers.
- Decode msgpack struct field names onto a 16-byte stack buffer via read_small_str and unwrap PrependByteReader after reading submap headers.

* test(filemeta): pin xl.meta codec compatibility

Pin pre-optimization signatures for historical RustFS and MinIO fixtures,
and cover direct reads across selectors and decode options. Preserve null
inline keys, free-version selection, empty histories, and allocation
boundaries for large version bodies.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 07:01:11 +09:00
0x70 3a5742c5fd feat(policy): support s3:if-none-match and s3:if-match condition keys (#8615)
* feat(policy): support s3:if-none-match and s3:if-match condition keys

Bucket, IAM, and STS session policies that enforce conditional writes
with s3:if-none-match or s3:if-match were rejected with "invalid key
name". Accept both keys and populate them from the If-None-Match and
If-Match request headers. A blank header is left out of the condition
values because the write ignores it, so it cannot satisfy a Null check
for a write that runs unconditionally.

* fix(policy): classify conditional write keys as request-derived

s3:if-match and s3:if-none-match are filled from request headers, but
is_server_derived() reported them as server-derived because their names
lack the x-amz- prefix. Add an explicit arm so header input is never
refused for these keys if they join COMMON_KEYS.

* fix(s3): reject non-wildcard If-None-Match on writes

S3 accepts If-None-Match on PutObject, CopyObject and
CompleteMultipartUpload only as "*" and answers other values with 501
NotImplemented. RustFS compared such a value as an ETag, so a write with
If-None-Match: "<other etag>" overwrote the key while satisfying a
policy that requires s3:if-none-match.

Reject ETag values in the three handlers before any I/O. The store-level
check now treats any non-blank If-None-Match as create-only, so a caller
that skips the handler check cannot weaken the precondition. Drop the
CompleteMultipartUpload ETag pre-check that the rejection supersedes.

* test: tighten conditional write policy coverage

Pin trimming and the non-UTF-8 case in the condition values, and drop
the auth test that re-ran policy evaluation already covered by the
policy crate. Run the three policy e2e scenarios on one server, assert
that ETag If-None-Match cannot overwrite under a Null: false policy and
that CompleteMultipartUpload with If-None-Match: * returns 412. Share
the AssumeRole response parser through common.rs.

* test(ecstore): isolate manual delete recovery from workers

Stop fixture background services before seeding recovery journals so the
explicit recovery pass owns the decisions its assertions inspect.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): enforce request-derived conditional write guards

Validate raw If-None-Match headers as well as DTO fields so POST form
uploads cannot skip rejection of unsupported ETag preconditions. Derive
conditional policy keys after identity claims and clear colliding claims
when the actual request has no usable precondition.

Align multipart and pinned Ceph regressions with wildcard-only writes,
retaining If-Match, object-content, version and delete-marker assertions.
Document that 501 is the implementation's unsupported-value response.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): eliminate conditional write request copies

Borrow request headers during policy condition collection and exclude
conditional keys at the identity-claim boundary instead of inserting and
removing them. Preserve existing aliases and multivalue header behavior.

Move the CompleteMultipartUpload input out of its request, compare strong
ETags by reference, and remove the ETag If-None-Match branch made unreachable
by the write guard. Keep both form/HTTP input checks and storage commit
preconditions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): align ignored conditional PUT and COPY cases

Use If-None-Match: * for the existing-object 412 regressions and assert 501
for matching and nonmatching ETag values. Cover both header-only callers
and parsed DTO-plus-header requests in the serial ILM fixtures.

Check object bytes and live version counts after each rejected operation.
Keep the existing test names and ignored-test lane selection unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 07:00:21 +09:00
overtrue 6de3034e36 fix(storage): reject ordinary deletes from managed artifact buckets
Reject single, batch, and exact-version deletes before staging mutations.
Transfer real lifecycle ownership through detached deletion and reuse the
cleaner's existing owner to avoid a queued-writer deadlock.

Correct the physical-tail cancellation observer and cover disconnected PUT
publication plus managed delete refusal with four-node tests.
2026-10-19 06:57:35 +09:00
InitCool e2583ddcd6 feat(oidc): make the Console session duration configurable (#8568)
* feat(oidc): make the Console session duration configurable

The OIDC callback hard-coded the Console credential lifetime at 3600
seconds, so a Console session always ended after about an hour.
RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour,
is the Console HTTP request timeout and cannot extend a session.

Resolve the lifetime from the process-level
RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to
complete_authorization_code(). Unset, unparseable, zero, and negative
values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so
existing deployments keep the previous behaviour and a misconfigured
variable can never issue an already expired credential. Values above
CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that
maximum with a warning, which also caps how far an operator can stretch a
credential by accident.

Keep the Console logout token for that same lifetime. OidcStateStore
pinned the logout cache to a hard-coded hour, so a session configured to
last longer than that dropped its logout token while the credential was
still valid: RP-initiated logout could no longer resolve, and the browser
was sent back to the Console login page with the IdP session still
active. The logout cache now derives its TTL from the same
console_session_duration_secs(), which applies the same clamp, and the
callback reads its lifetime from that one function instead of a local
copy, so the credential lifetime and the logout-token retention cannot
drift apart. Add tests covering the default hour, an extended session,
the maximum, and out-of-range values.

The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that
list is suffix-scanned per provider, and this setting applies to every
provider. It has no identity_openid KVS equivalent and needs a restart,
like the other process-level settings in the same document.

Document the setting in the login flow, the process-level table, and the
troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a
request timeout rather than a session lifetime.

* test(ecstore): retain offline marker fixtures across reconnect

Keep simulated drive handles attached and reject I/O until explicit rejoin.
Attempt reconnect in the current-marker case so the former CI race is
deterministic while retaining physical cleanup and historical-data checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(oidc): align session diagnostics and lifetime regression tests

Use structured IAM diagnostics for clamping and move configuration tests
to their owning crate. Cover duration boundaries, logout retention and
signed credential expiration through the federation binding.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 06:52:05 +09:00
Henry Guo 317191ce00 fix(scanner): scale backup deadline to checkpoint save cost (#8659) 2026-10-19 06:21:33 +09:00
overtrue ae259e127d feat(storage): seal distributed artifact repair writes 2026-10-19 05:49:40 +09:00
overtrue 8233e3fe4c feat(storage): enable distributed artifact PUT sealing 2026-10-19 02:38:26 +09:00
Chris f3c4a790dd fix(listing): accept literal prefixes with repeated slashes (#8605)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-19 01:27:51 +08:00
overtrue f88b7b6ba5 fix(storage): bind volume creation to the receiving instance 2026-10-19 01:52:02 +09:00
overtrue 0307c9a7e2 fix(storage): retain admission through HTTP stream writes 2026-10-19 01:44:38 +09:00
overtrue 6a8d0aba58 fix(storage): retain admission through deletion cleanup 2026-10-19 01:06:40 +09:00
overtrue ee6c2afe43 fix(storage): serialize multipart publication and rollback 2026-10-19 00:09:40 +09:00
overtrue 9ca8bc3ca6 fix(storage): record failed rename admission metrics 2026-10-18 23:21:59 +09:00
overtrue 3678703a98 fix(storage): retain dual-volume admission through file rename 2026-10-18 23:11:43 +09:00
overtrue fe798c13e7 fix(rpc): admit grouped version deletion before classification 2026-10-18 22:35:09 +09:00
overtrue 24b77349aa feat(storage): admit legacy bulk version deletion 2026-10-18 22:22:18 +09:00
overtrue 8c55396bfe fix(rpc): admit staged single-object version deletion 2026-10-18 22:04:43 +09:00
overtrue 5be8badb20 fix(storage): retain admission across staged version deletion 2026-10-18 21:48:17 +09:00
overtrue 46daaadbb3 fix(rpc): admit legacy single-object delete rollback 2026-10-18 21:40:48 +09:00
overtrue f3b6c4df6b fix(storage): retain admission through single-object delete rollback 2026-10-18 21:38:26 +09:00
overtrue 7183849227 fix(storage): admit ordinary version deletion before classification 2026-10-18 18:41:13 +08:00
Hauser 6681dd12e3 fix(scanner): expose mixed sweep checkpoint transitions (#8657)
* fix(scanner): expose mixed sweep checkpoint transitions

Correlate checkpoint admission, incomplete sweeps, and successful terminal
Partial persistence by bucket/source and cycle/leader epoch. Share the
traversal-position predicate and local/remote save diagnostic without
changing certification, acknowledgement, or persistence contracts.

Extend real-disk cache regressions across cycles and leader changes and
verify that failed terminal saves retain an older periodic frontier until
a successful terminal save permits fresh certification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): report the aggregate incomplete sweep gate

Expose the existing unresolved_objects decision in the incomplete-sweep
event. Nested cache-entry failures can keep a sweep partial while both
the root counter and the capped diagnostic failure map are empty.

Extend the real-disk regression with diagnostics disabled and distinguish
this case from mixed-only coverage without adding another cache walk.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 16:50:42 +08:00
Henry Guo 3b0ee83466 fix(scanner): pace checkpoints by snapshot construction cost (#8656) 2026-10-18 13:36:58 +08:00
Henry Guo 02c4f4c97f fix(ecstore): clean completed PUT staging on pre-commit cancellation (#8654)
fix(ecstore): clean staged PUT data on pre-commit cancellation
2026-10-18 13:36:33 +08:00
Hauser c28e02a722 fix(ecstore): preserve inline heal rollback staging lifetime (#8650)
* fix(ecstore): preserve inline heal rollback staging lifetime

Retain the TMP source prefix in the existing publication lease until the owned receiver finishes rollback. Serialize recursive cleanup and stale GC with the same key, and avoid decoding an unused old version during inline healing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reuse staging lease during parent cleanup

Reuse the existing staging lease when pruning its parent directory, including already-missing prefixes, and retain it in the physical rmdir executor after cancellation. Match the inline-heal fsync test hook to the canonical group path on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 12:58:57 +08:00
Henry Guo 1df96624df feat(table-catalog): support conditional maintenance worker retries (#8562)
* feat(table-catalog): support conditional maintenance worker retries

* fix(table-catalog): refresh conditional worker claim time

* style(table-catalog): format conditional worker expiry test

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-18 10:03:58 +08:00
Henry Guo ee534fa19b test(table-catalog): validate current backing contracts (#8559)
* test(table-catalog): qualify current backing contracts

* fix(table-catalog): verify smoke cleanup after object rename

* fix(ci): include the baseline writer test in full E2E selection

* fix(ci): reserve test capacity for disk-backed lock probes

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-18 10:03:30 +08:00
Chris 668bc7a4ba test(utils): cover mixed-case metadata alias conflicts (#8651) 2026-10-18 10:02:47 +08:00
overtrue 942ce79c4e fix(storage): admit ordinary undo before artifact classification 2026-10-18 06:08:50 +08:00
overtrue fec1262f1a fix(storage): admit metadata updates before bucket classification 2026-10-18 05:39:45 +08:00
overtrue 5eb46cb29d fix(rpc): admit both volumes before legacy rename classification 2026-10-18 05:30:33 +08:00
overtrue 2cd904d0e7 feat(storage): prepare legacy renames with dual-volume ownership 2026-10-18 05:18:24 +08:00
overtrue 61eec6da18 fix(storage): reject contended rollback backup publication 2026-10-18 04:53:45 +08:00
overtrue 5b8cd2a5dc test(storage): initialize real lifecycle owners in fixtures 2026-10-18 04:51:11 +08:00
overtrue d7824049b2 fix(storage): bind metadata RPCs to prepared volume leases 2026-10-18 04:32:19 +08:00
Hauser e29a76f391 fix(heal): verify checkpoint integrity before applying defaults (#8649)
* fix(heal): verify checkpoint integrity before applying defaults

Validate the persisted JSON shape so intact signed legacy checkpoints can
reach the existing safe schema rewind. Preserve deployed field ordering and
read historical sorted-key digests without changing the checkpoint schema.
Reuse the signing value when saving and retain strict duplicate-field parsing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): preserve checkpoint writer rollback compatibility

Keep ordered checkpoint parsing local rather than enabling preserve_order
throughout the server. Sort writer keys recursively to retain the production
base reader's digest format while accepting both historical read orders.

Cover real writer canonicalization, nested scalar ordering and depth limits.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 04:03:16 +08:00
overtrue d46e4c54a6 fix(storage): bind legacy file RPCs to volume ownership 2026-10-18 04:00:46 +08:00
overtrue 3ca0b5f35e fix(storage): retain volume locks through physical mutations 2026-10-18 03:26:41 +08:00
overtrue 00b5ddc0ad feat(storage): persist artifact receiver write fences 2026-10-18 03:01:03 +08:00
Chris 3ddd307304 chore(release): align installation references for 1.0.3 (#8646)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-18 02:58:32 +08:00
Ramakrishna Chilaka 9946c67b53 perf(filemeta): avoid clones and redundant map scans in into_fileinfo (#8647)
FileMetaShallowVersion::into_fileinfo and to_fileinfo_with_part_checksums previously decoded an owned FileMetaVersion and then converted it via shared reference (&self), cloning meta_user, meta_sys, part_etags, and part_indices before dropping the temporary FileMetaVersion and scanning the metadata map up to 10 times for optional internal suffixes. Additionally, metadata_compat helpers (get_str, contains_key_str, remove_str, get_consistent_value) allocated two temporary heap Strings via both_keys(suffix) on every absent-key fallback.

- Replace both_keys(suffix) in case-insensitive fallback scans with allocation-free prefix+suffix slice matching.
- Add consuming into_fileinfo_with_part_checksums on FileMetaVersion and MetaObject to move decoded part vectors and metadata maps in place.
- Fold transition-version metadata formatting and optional suffix presence detection into the single meta_sys pass, and skip target_delete_marker_versions in get_internal_replication_state when no delete-marker version keys are present.
2026-10-18 02:57:21 +08:00
Hauser 286e4cf30b perf(ecstore): keep inline heal rollback backups in staging (#8648)
* perf(ecstore): keep inline heal rollback backups in staging

Keep inline repair rollback local to the disk commit and avoid publishing
a live backup that needs a second cleanup operation. Preserve legacy peer
and non-inline backup reclamation, with production-mode rollback coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Co-Authored-By: Ramakrishna Chilaka <rkchilaka378@gmail.com>

* test(ecstore): route inline heal imports through test facade

Keep ECStore integration-test API access behind storage_api.rs to satisfy
the architecture migration boundary without changing test behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Ramakrishna Chilaka <rkchilaka378@gmail.com>
2026-10-18 01:51:52 +08:00
Hauser 453be3dd84 fix(scanner): attribute incomplete census and lease waits (#8643)
* fix(scanner): attribute incomplete census and lease waits

Report bucket sweep blockers and correlate partial remote work and rejected
roots with their set and cycle. Keep distinct accepted bucket names in the
result collector so completion counts and missing-bucket attribution share
one source of truth, including during cancellation drain.

Distinguish worker completion from coverage, retain peer/purpose context in
lease failures without changing retry suffixes, and avoid claiming that an
unverified leader fence changed or a waiting scan was complete.

Document the census stages and preserve publication, dirty acknowledgement,
remote failure classification and checkpoint safety rules.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): move set caches through the result collector

Give the sole result collector exclusive ownership of the aggregate cache
and return it together with the accepted bucket names after channel drain.
Move that cache into complete persistence or incomplete updates instead of
sharing a mutex and cloning the terminal cache.

Return each accepted result's owned name from cache application so coverage
tracking also avoids an extra name allocation. Preserve worker joins,
cancellation drain, publication metadata and all certification fences.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 01:05:14 +08:00
唐小鸭 222beaba00 fix(scanner): certify dirty usage per bucket and complete cycles with pending buckets (#8542)
* fix(scanner): treat replication status writebacks as status-class hits

A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).

The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.

No persisted format, protocol, admin API or configuration changes.

This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.

Refs #8375, #8494, rustfs/backlog#2816.

* fix(scanner): refresh replication statistics during maintenance

Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): avoid expanding the legacy S3 DTO surface

Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): attribute activity guardrail vetoes per field

The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.

* feat(data-usage): carry pending-bucket certification in DataUsageInfo

Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.

* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets

A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.

* feat(scanner): accept unconverged complete publications as the cache baseline

complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.

Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.

* feat(admin): expose scanner guardrail vetoes in usage freshness

Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.

* docs(scanner): record per-bucket certification in the design and changelog

Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.

* refactor(scanner): drop the unreachable failed-bucket input from the pending set

A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.

* test(scanner): name the activity field mutation table type

* test(scanner): fill the post-walk gate in merged replication fixtures

The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.

* ci(nextest): run the admin target-repair fixtures on the ecstore base stack

repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (65f30bd5c) passes at 2032 KiB and overflows at 2016 KiB,
this branch passes at 2040 KiB and overflows at 2032 KiB; on both trees
the overflow is in the test's delete_bucket step on the real store, and
the futures on that path are small (the peak is nested poll depth, not
one large future). The module was given this stack in #7272 and removed
in #7280 to check that boxing (#7277) sufficed; with the default now
exhausted on main itself, give the fixtures the same 4 MiB base stack as
the other real-storage rustfs fixtures, in both profiles.

* test(scanner): fill the post-walk gate in the walkers-per-disk fixture

The merge from main brought the walkers_per_disk cycle fixture (#8539),
whose ScannerCycleRequest initializer predates the test-only
post_walk_gate field this branch adds; the scanner test target did not
compile. Fill the default so the merged head builds.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 00:25:02 +08:00