mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-26 22:07:53 +00:00
e2583ddcd62be68913552adea492d82016d264bf
909 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 InitCool
|
e2583ddcd6 |
feat(oidc): make the Console session duration configurable (#8568)
* feat(oidc): make the Console session duration configurable The OIDC callback hard-coded the Console credential lifetime at 3600 seconds, so a Console session always ended after about an hour. RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour, is the Console HTTP request timeout and cannot extend a session. Resolve the lifetime from the process-level RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to complete_authorization_code(). Unset, unparseable, zero, and negative values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so existing deployments keep the previous behaviour and a misconfigured variable can never issue an already expired credential. Values above CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that maximum with a warning, which also caps how far an operator can stretch a credential by accident. Keep the Console logout token for that same lifetime. OidcStateStore pinned the logout cache to a hard-coded hour, so a session configured to last longer than that dropped its logout token while the credential was still valid: RP-initiated logout could no longer resolve, and the browser was sent back to the Console login page with the IdP session still active. The logout cache now derives its TTL from the same console_session_duration_secs(), which applies the same clamp, and the callback reads its lifetime from that one function instead of a local copy, so the credential lifetime and the logout-token retention cannot drift apart. Add tests covering the default hour, an extended session, the maximum, and out-of-range values. The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that list is suffix-scanned per provider, and this setting applies to every provider. It has no identity_openid KVS equivalent and needs a restart, like the other process-level settings in the same document. Document the setting in the login flow, the process-level table, and the troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a request timeout rather than a session lifetime. * test(ecstore): retain offline marker fixtures across reconnect Keep simulated drive handles attached and reject I/O until explicit rejoin. Attempt reconnect in the current-marker case so the former CI race is deterministic while retaining physical cleanup and historical-data checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(oidc): align session diagnostics and lifetime regression tests Use structured IAM diagnostics for clamping and move configuration tests to their owning crate. Cover duration boundaries, logout retention and signed credential expiration through the federation binding. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
6681dd12e3 |
fix(scanner): expose mixed sweep checkpoint transitions (#8657)
* fix(scanner): expose mixed sweep checkpoint transitions Correlate checkpoint admission, incomplete sweeps, and successful terminal Partial persistence by bucket/source and cycle/leader epoch. Share the traversal-position predicate and local/remote save diagnostic without changing certification, acknowledgement, or persistence contracts. Extend real-disk cache regressions across cycles and leader changes and verify that failed terminal saves retain an older periodic frontier until a successful terminal save permits fresh certification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): report the aggregate incomplete sweep gate Expose the existing unresolved_objects decision in the incomplete-sweep event. Nested cache-entry failures can keep a sweep partial while both the root counter and the capped diagnostic failure map are empty. Extend the real-disk regression with diagnostics disabled and distinguish this case from mixed-only coverage without adding another cache walk. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Henry Guo
|
1df96624df |
feat(table-catalog): support conditional maintenance worker retries (#8562)
* feat(table-catalog): support conditional maintenance worker retries * fix(table-catalog): refresh conditional worker claim time * style(table-catalog): format conditional worker expiry test --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Henry Guo
|
ee534fa19b |
test(table-catalog): validate current backing contracts (#8559)
* test(table-catalog): qualify current backing contracts * fix(table-catalog): verify smoke cleanup after object rename * fix(ci): include the baseline writer test in full E2E selection * fix(ci): reserve test capacity for disk-backed lock probes --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
453be3dd84 |
fix(scanner): attribute incomplete census and lease waits (#8643)
* fix(scanner): attribute incomplete census and lease waits Report bucket sweep blockers and correlate partial remote work and rejected roots with their set and cycle. Keep distinct accepted bucket names in the result collector so completion counts and missing-bucket attribution share one source of truth, including during cancellation drain. Distinguish worker completion from coverage, retain peer/purpose context in lease failures without changing retry suffixes, and avoid claiming that an unverified leader fence changed or a waiting scan was complete. Document the census stages and preserve publication, dirty acknowledgement, remote failure classification and checkpoint safety rules. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(scanner): move set caches through the result collector Give the sole result collector exclusive ownership of the aggregate cache and return it together with the accepted bucket names after channel drain. Move that cache into complete persistence or incomplete updates instead of sharing a mutex and cloning the terminal cache. Return each accepted result's owned name from cache application so coverage tracking also avoids an extra name allocation. Preserve worker joins, cancellation drain, publication metadata and all certification fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
唐小鸭
|
222beaba00 |
fix(scanner): certify dirty usage per bucket and complete cycles with pending buckets (#8542)
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): attribute activity guardrail vetoes per field
The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.
* feat(data-usage): carry pending-bucket certification in DataUsageInfo
Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.
* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets
A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.
* feat(scanner): accept unconverged complete publications as the cache baseline
complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.
Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.
* feat(admin): expose scanner guardrail vetoes in usage freshness
Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.
* docs(scanner): record per-bucket certification in the design and changelog
Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.
* refactor(scanner): drop the unreachable failed-bucket input from the pending set
A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.
* test(scanner): name the activity field mutation table type
* test(scanner): fill the post-walk gate in merged replication fixtures
The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.
* ci(nextest): run the admin target-repair fixtures on the ecstore base stack
repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (
|
||
|
Hauser
|
f64cb816db |
fix(scanner): preserve checkpoint progress across object failures (#8642)
Separate completed traversal from persisted object health debt so leader handoffs reuse healthy subtrees and retry failures without losing progress. Fence old readers with a new receipt digest domain and preserve exact accounting under diagnostic eviction, compaction and punctuation ordering. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
b1f3ebb184 |
fix(ecstore): converge partial pool identity writes (#8636)
* fix(ecstore): converge partial pool identity writes * fix(ecstore): bound pool metadata CAS future state |
||
|
唐小鸭
|
353e67bbf3 |
fix(ecstore): resolve Windows drive paths in URL endpoints (#8629)
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only form multi-node and pool-expansion layouts accept) produced the disk path `/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path` put it back, and `get_file_path` only removed it for `file` endpoints. Windows resolves `/C:/data1` as an invalid `C:` component under the current drive, so disk validation and `resolve_local_disk_root` failed. - `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged. - UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected with an explicit error instead of becoming a file URL whose host is the share server and whose path drops the share name. - A drive root (`http://host:9000/D:/`) is accepted in URL form, matching the path-style `D:\` endpoint that was already accepted. - The "distinct physical disks" error explains on Windows that directories on one volume share a device identity, alongside the existing RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint. - The Windows filesystem lane now runs on layout changes and runs the layout::endpoint:: tests; new Windows tests carry the windows_ prefix. - Document Windows endpoint forms, the same-volume check, and the SNSD expansion limit in cluster-lifecycle-operations.md. |
||
|
唐小鸭
|
78ea655d87 |
fix(startup): drive startup on an explicitly sized thread (#8630)
`run_process` called `block_on(async_main())` on the native main thread, so the whole startup chain ran on whatever stack the platform hands the main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE default on Windows. A Windows debug build overflowed before it listened. `RUST_MIN_STACK` does not reach the native main thread. Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack, keeping the budget Linux and macOS already had and making it independent of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised on the main thread so the exit status is unchanged. Measured on macOS arm64 by lowering the main-thread stack with `ulimit -s` before this change: debug startup overflows at 1792 KiB and reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is ready at 768 KiB. After it, debug starts with a 256 KiB main thread. Add a Windows `Startup Smoke` job that builds the debug server, runs `--version`, and waits for /health/ready on a single volume. |
||
|
Henry Guo
|
749a1686aa |
feat(table-catalog): plan maintenance on durable backing (#8567)
* feat(table-catalog): plan maintenance on durable backing * test(table-catalog): bootstrap strong planning handler fixture |
||
|
唐小鸭
|
741de6f81c |
fix(replication): locate encrypted replicas without comparing ETags (#8602)
Since 1.0.2 an encrypted object's ETag is a keyed fingerprint, not the plaintext MD5. A target that computes its own ETag (AWS S3, Wasabi) never reports it, so every "find the existing replica by exact ETag" step missed encrypted objects on targets that mint their own version ids: - a heal / MRF re-drive after a lost PUT response wrote a second version; - an existing-object resync re-PUT a replica its ledger already named (the ledger-located HEAD was confirmed by ETag only), and on the AWS 400 shape the ledger was not consulted at all; - resync verification counted such replicas as failed; - a ledger-less version purge reported the replica as already gone. Encrypted sources now recognise their replica by shape: the byte count the replication PUT sent (plaintext for managed SSE, ciphertext for SSE-C) and a target write time no earlier than the source version. An ETag match still wins (RustFS/MinIO targets). The ledger is consulted first on both the 404 and the AWS 400 shape. Without a ledger, only a unique shape match among the unclaimed ListObjectVersions candidates is adopted and recorded; several shape matches mean re-PUT for convergence and a refused purge. The AWS current-version fallback stays ETag-only, and a listing the target refuses on the 400/403 shape keeps that fallback (re-PUT) rather than stalling. An unversioned target's `null` version is never matched by shape: it holds whichever generation was written last. Plaintext sources keep exact ETag identity. Builds on the ETag-computing fake target mode and the ComputesOwnEtags matrix row from rustfs#8583. Adds an SSE-S3 variant of the mint-own-ids re-drive test (red before this change: the heal left two target versions), and both variants now also assert the purge. |
||
|
cxymds
|
12ae143bee | fix(quota): keep writes available with a degraded fleet (#8604) | ||
|
cxymds
|
628663f468 |
fix(heal): recover replacements after pool removal (#8600)
* fix(heal): recover replacements after pool removal * test(e2e): gate upgrade probes on writable baseline nodes * test(e2e): register baseline readiness regression in smoke digest |
||
|
唐小鸭
|
d457378c02 |
feat(scanner): decouple bucket walkers from disk count (#8539)
* fix(scanner): treat replication status writebacks as status-class hits A replication status writeback (persist_replication_state_if_current in Update mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path: it advanced the dirty-usage generation, woke the scanner, reached the durable dirty journal and entered the activity snapshot. Any replicated bucket therefore made every scanner cycle Superseded, the authoritative usage baseline was never certified and usage freshness stayed stale for as long as replication ran (#8375). The writeback is now status class: record_status_class_mutation records only the process-local producer coverage bit and nothing else. There is no rescan queue for it, because a deployment with an active replication rule already walks every bucket on every cycle (ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes replication_stats. The routing lives in a named, unit-tested function in rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data and stays on the namespace path. No persisted format, protocol, admin API or configuration changes. This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md, which this change adds together with its index entry. Refs #8375, #8494, rustfs/backlog#2816. * fix(scanner): refresh replication statistics during maintenance Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): avoid expanding the legacy S3 DTO surface Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(scanner): keep raw enumeration bookkeeping linear per entry The page writer inserted into its building page with a push-and-sort on every entry and committed each page through a clone. Insert in sorted position and move the committed page into the index instead. Add a regression test that records 20,000 entries in one prefix and pins the cost to one hash per entry and well under a second, so the quadratic re-normalization removed by #8114 (rustfs#8113) cannot return once pacing builds on this walker. * feat(scanner): decouple bucket walkers from disk count Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK, 1 through 16, default 1). Each local coordinator disk of a set spawns that many bucket walkers; remote disks keep one. The walker count is what max_concurrent_disk_scans caps and the disk count stays the floor. The walkers of one disk share its scanner admission through LocalDiskAdmissionShare: the first walker to start a bucket takes the per-disk permit and the last to finish releases it, so a peer's remote request still sees one scan session per physical disk. Without the share the second walker was rejected as busy and retired. The disk_concurrency_budget log line reports walkers_per_disk, walkers and the reason for the effective limit. The default stays 1 because no local disk reports rotational media today; the design keeps a higher flash default as the intent. * docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime controls table. Add it to the Helm chart (values, configmap rendering, README table, template test), the configuration crate's environment reference with operator guidance, and a short section in the operations manual on when to raise it and how to confirm it took effect. --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
8c8c74cbc3 |
fix(ecstore): unify bounded delete recovery across upgrades (#8594)
* fix(ecstore): complete delete recovery across upgrades and restarts Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): expect typed quorum error after delete rollback Assert the exact InsufficientWriteQuorum bucket and object instead of the removed Io wrapper, retaining the rollback and cleanup ownership checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
唐小鸭
|
b08331335a |
test(replication): cover SSE-S3 replicas on ETag-computing targets (#8583)
The outbound target matrix only modeled targets that adopt the forwarded source ETag, so the post-PUT replica ETag audit was never exercised against a target that computes its own ETag (AWS S3 and most compatible stores), and no matrix shape was encrypted. That gap let the SSE-S3 "replica etag mismatch" regression fixed in #8573 ship unnoticed. Add an ignore_source_etags mode to the fake S3 target, a ComputesOwnEtags matrix row that uses it, and an SseS3 object shape (the source runs with the local KMS). The shape asserts its source ETag is not the plaintext MD5 so the cell cannot pass vacuously. Against a build without #8573 the new row fails exactly computes-own-etags/SseS3 with "replica etag mismatch"; with it the whole matrix passes. Also correct the #8573 docs and comment that called the replication PUT checksum the integrity check for encrypted sources: managed-SSE sources forward no checksum, so their replicas have no post-PUT content audit. |
||
|
Chris
|
7a32ca2adb |
fix(replication): skip the replica ETag audit for encrypted sources (#8573)
New encrypted writes store a keyed HMAC fingerprint as the object ETag (docs/operations/encrypted-etags.md). Replication forwards that ETag on x-rustfs-source-etag, but a target that computes its own ETag (AWS S3 and most compatible stores) answers the plaintext PUT with the MD5 of what it received. Both values are 32 hex characters, so verify_single_part_replica treated the fingerprint as a comparable MD5 and failed every single-part SSE-S3 / SSE-KMS replica with "replica etag mismatch"; MRF and resync then retried the object forever. RustFS and MinIO targets adopt the source ETag and were unaffected. Exempt sources whose stored ETag cannot describe the wire bytes: any managed-SSE or SSE-C source, decided by the existing replication source-encryption classifier. The checksum header on the replication PUT remains the integrity check for those objects; plaintext sources keep the audit unchanged. Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797 Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Ramakrishna Chilaka
|
606d8a3a8a | fix(scanner): clear transient cycle recovery on follower lock contention (#8566) | ||
|
Hauser
|
8a96fedfdd |
fix(ecstore): retain pool metadata repair ownership (#8552)
* fix(ecstore): retain system metadata repair ownership Return system metadata convergence failures to the fenced writer after committed cleanup instead of submitting an unexecutable MRF handoff. Drain system rename tails before acknowledging and preserve ordinary incarnation-bound object repair. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): separate PUT commit and completion outcomes Keep committed metadata writes on their retained cleanup owner when the writer must handle repair. Propagate the completion error at the API boundary without a duplicate temporary-data deletion or worker wait. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): scope repair completion to pool metadata owners Keep ordinary system metadata quorum-compatible and explicitly select owner-observed repair completion for fenced pool metadata CAS writes. Seed format-heal metadata before injecting the missing-format fault. Preserve degraded bucket recreation and test all completion modes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
ce00d8f7e5 |
feat(s3): support conditional bulk deletes (#8535)
* feat(s3): support conditional bulk deletes * fix(s3): report null marker identity in bulk delete responses * fix(s3): isolate conditional bulk delete actions and tier guards * fix(tier): preserve backend identity validation diagnostics --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
8b75145a29 |
fix(s3): preserve catalog lock failures at the error boundary (#8550)
Retain typed retryable catalog lock failures and route S3 publication errors through the canonical storage conversion for bounded metrics, public-message redaction, and existing retry advice. Borrow CAS results for metrics instead of cloning opaque error sources. Cover authorization under a real publication writer, HTTP serialization, source preservation, recovery, and established catalog error categories. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
唐小鸭
|
c9f88cff66 |
fix(scanner): treat replication status writebacks as status-class hints (#8509)
* fix(scanner): treat replication status writebacks as status-class hits A replication status writeback (persist_replication_state_if_current in Update mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path: it advanced the dirty-usage generation, woke the scanner, reached the durable dirty journal and entered the activity snapshot. Any replicated bucket therefore made every scanner cycle Superseded, the authoritative usage baseline was never certified and usage freshness stayed stale for as long as replication ran (#8375). The writeback is now status class: record_status_class_mutation records only the process-local producer coverage bit and nothing else. There is no rescan queue for it, because a deployment with an active replication rule already walks every bucket on every cycle (ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes replication_stats. The routing lives in a named, unit-tested function in rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data and stays on the namespace path. No persisted format, protocol, admin API or configuration changes. This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md, which this change adds together with its index entry. Refs #8375, #8494, rustfs/backlog#2816. * fix(scanner): refresh replication statistics during maintenance Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): avoid expanding the legacy S3 DTO surface Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): seed the ordinary-timeout peer fixture under the default drive deadline The one-second ENV_DRIVE_MAX_TIMEOUT_DURATION in target_ordinary_timeout_keeps_its_physical_namespace_owner wrapped the whole test, so the fixture's write_metadata and the final read_version were bounded by it too. On a loaded CI runner the seed took longer than a second and the test failed with "seed real metadata: Timeout" before the paused rename under test ran. Only the rename now runs under the one-second deadline; the fixture and the tail read use the default. --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
08ac1ec218 |
fix(s3): advise retry after typed lock timeouts (#8546)
Add a one-second Retry-After hint at the canonical S3 error conversion only for ServiceUnavailable responses retaining a typed lock timeout. Keep existing error codes, source chains, and lock budgets unchanged. Cover actual HTTP/XML serialization, nested causes, private-resource redaction, and unrelated disk/quorum/text failures. Document the response contract and the limits of log-count and five-second timeout attribution. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
388d1f17e6 |
fix(heal): safely recover missing bucket directories by default (#8517)
* fix(heal): safely recover missing bucket directories by default * fix(heal): allow bound admin recovery before namespace checks |
||
|
Henry Guo
|
22c093e66b | feat(table-catalog): add opt-in periodic maintenance queueing (#8537) | ||
|
Henry Guo
|
fa0ac13e9b |
feat(table-catalog): support durable-strong table maintenance (#8523)
* feat(table-catalog): enable durable maintenance control * fix(table-catalog): revalidate unlocked worker planning * fix(table-catalog): preserve maintenance state across rename * fix(table-catalog): fence maintenance reads across rename * fix(table-catalog): bound scheduler history read fencing * test(table-catalog): inject loss using the held client lease * refactor(table-catalog): remove unused metadata log wrapper |
||
|
Hauser
|
b80590f1a7 |
perf(heal): index replacement history reconciliation (#8527)
refactor(heal): index replacement generation reconciliation Reconcile owned records and their definitive sources in one pass. Group claims by target slot and use a shared-anchor fast path while retaining pairwise ambiguity checks. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
025ed4c6d1 |
fix(heal): preserve retained replacement histories (#8525)
Keep definitive local histories while rejecting ambiguous overlapping peer records. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
adf17bbfba | fix(heal): reject incomplete administrator bucket discovery (#8512) | ||
|
GatewayJ
|
fb5c002a30 | refactor(iam): encapsulate federated authorization models (#8514) | ||
|
Hauser
|
9ca0e9b717 |
fix(obs): isolate bucket usage dashboards by cluster (#8510)
Keep same-name buckets in independent clusters separate while deduplicating replicated usage observations. Document the existing scanner-backed exporter and cover its gauge contract and cluster-scoped queries. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
34803e8165 |
perf(ecstore): isolate cleanup and preserve metadata authority (#8289)
* chore: pin s3s to upstream git revision Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4. Allow the official s3s git source in cargo-deny so source checks continue to pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): instrument put object hot path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): split cleanup worker wait metrics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * upgrade version * chore(metrics): add PUT post-store stage probes Add lightweight PUT post-store stage probes for issue 2615 validation. * fix cargo deny check warn * upgrade version * perf(ecstore): bound and isolate cleanup execution (#8151) * perf(ecstore): bound and isolate cleanup execution * fix(ecstore): preserve cleanup cancellation and lease release * feat(ecstore): default isolated cleanup to CPUs 1-2 * test(ecstore): decouple budget tests from CPU affinity * upgrade version * perf(ecstore): reuse pinned directory handles for object I/O * feat(observability): add sampled local GetObject stage summaries Add an opt-in GetObject stage aggregation path that works without the OTLP metrics export. When RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED is set, io-metrics keeps per-(path, stage, object class, size bucket) sampled counts and duration totals in a bounded in-process map, and a startup task flushes them periodically to /var/log/rustfs/get-stage-summary.log, tunable via the sample rate and interval env keys. * build(deps): bump workspace dependency releases Patch releases: async-compression 0.4.49, tokio-rustls 0.26.6, xxhash-rust 0.8.19, suppaftp 12.1.1, russh-sftp 3.0.1. Minor release: hotpath 0.27.0. * fix(ecstore): fence distributed metadata cache mutations Add authenticated cluster-wide mutation fences for the opt-in DistErasure GetObject metadata cache. Guard recursive prefix and batch deletes, fail closed on configuration mismatch or uncertain quorum, and cover replay, restart, peer loss, and mixed-configuration behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT observers on improve/base (#8039) * perf(object): instrument storage hot paths Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT stage observation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> * build(deps): bump workspace dependency releases * ci: allowlist rustix OFlags WRONLY in typos config * refactor(storage): route ecstore symbols through storage_api boundaries * perf(ecstore): cache quorum-selected early-stop metadata (#8308) Publish eligible latest-version GET metadata after a validated read quorum when early-stop cancels the remaining fanout. Keep metadata errors, selection quorum, online read quorum, and mutation-generation checks as publication gates. Add slow-peer repeat-hit and concurrent mutation-fence regression tests. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: merge latest main into improve/base (#8332) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): borrow metadata cache lookup key (#8333) Avoid request-scoped Arc<str> allocations on eligible cache lookups while preserving full key identity. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: sync improve/base with latest main (#8334) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish metrics by process identity (#8343) Attach process.pid to the OpenTelemetry resource and promote it for Prometheus so counter series from restarted processes remain distinct. Preserve the stable RustFS node and service instance attributes. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync origin/main into improve/base (#8345) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(storage): retain typed owners across borrowed writes (#8329) * fix(storage): retain typed owners across borrowed writes * fix(storage): preserve Copy write completion policy * refactor(storage): remove redundant write owner references * fix(ecstore): hide rollback backup residue in prefix listings * test(connect): isolate short HTTP fixtures in nextest * fix(protocols): enforce S3 policies for file protocol operations (#8262) * test(ecstore): enforce main core regression gates (#8338) * test(ecstore): enforce main core regression gates * test(ecstore): pin external shards and completion regressions * fix(test): restore authenticated cluster fault scenarios (#8340) * fix(test): restore authenticated cluster fault scenarios * fix(test): use the E2E storage API boundary * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * fix(ecstore): own listing workers through cancellation (#8342) Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route commit sync through guarded helper Use the Linux-capable guarded directory sync entry point for both commit sync sites. This keeps group commit and file-sync admission behavior while ensuring the already-open commit directory remains the sync target. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): expose rename directory guard in production The commit path now acquires the directory guard in production builds, not only tests, so destination metadata reads and durable sync share the guarded context. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> * fix(ecstore): remove unsafe metadata cache admission paths Keep distributed GET metadata on quorum reads and quota admission on authoritative storage. Preserve local cache invalidation, remove the experimental RPC surface, and gate diagnostic work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): cover cold degraded mutation admission Check first PUT and DELETE with one peer absent and no SDK retries, then verify that losing write quorum still rejects writes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain guarded Linux destination metadata reads Use the existing commit directory guard when reading old xl.meta on Linux. Cover symlinked destination metadata for inline and non-inline renames without changing the split commit structure. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(test): remove retired cache RPC import Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * merge: sync latest main into improve/base (#8349) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync main auth and scanner updates (#8352) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) * perf(scanner): reduce scan and checkpoint peak allocations (#8346) * perf(scanner): reduce scan and checkpoint peak allocations * test(scanner): isolate partial commit recovery scenarios * fix(auth): return ExpiredToken for retained STS credentials (#8344) * fix(auth): return ExpiredToken for retained STS credentials Validate session identity independently of credential expiry in both IAM rejection and claims validation, and preserve typed S3 errors at the caller. Keep token binding, disabled-account precedence, and service-account behavior. Add exact expiry, IAM, and HTTP/XML regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): synchronize the metadata slowtail hedge regression Park the slow metadata task at a test-only fault checkpoint and release healthy reads after the fault is active. Assert the spare response and cancelled-task drain without imposing a 300 ms disk/scheduler deadline. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ilm): synchronize copy compensation transition stages Observe automatic compensation reaching the mock tier PUT checkpoint before waiting for the final metadata commit. Assert forced enqueue failure, automatic backfill, one completed tier upload, and payload integrity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(ecstore): explain cleanup scheduler conflict Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(metrics): separate GET request and timeout outcomes (#8354) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * docs: refresh security advisory lessons (#8350) * fix(heal): bound durable MRF rescheduling (#8351) Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(rustfs): route recovery errors through S3 gateway Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(io-metrics): use is_multiple_of for sampling Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): satisfy strict Clippy pagination checks Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning Copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * add ignore .mimosa * fix(zip): adapt archive readers to zip 9 APIs Upgrade zip and its required dependencies alongside fallible archive-name handling in log ingestion, report bundles, bucket metadata import, and archive tests. Preserve existing skip and error propagation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reuse quorum metadata for body GETs Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): align storage test stacks and peer reset classification Apply the existing bounded ECStore stack setup to the affected cross-crate storage fixtures. Validate conditional-only metadata publication separately from full body-layout cache reuse. Classify typed h2 shutdown resets as unreachable while retaining application, authentication, cancellation, and deadline distinctions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve omitted-disk GET recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore: integrate published workspace dependency updates Preserve the concurrent dependency refresh and retain the typed h2 reset classification and bounded storage-test stack setup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): preserve snapshot size budget after fallback move * fix(test): use one barrier for compensation COPY Let the forced-enqueue helper own the PUT barrier and compensation assertions. Waiting on an outer barrier overwritten by the helper always timed out after the upload had completed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(ecstore): apply current lint fixes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update sha2 to 0.11.1 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Hauser
|
e7e6556f83 |
fix(scanner): bound superseded waits for regular maintenance (#8502)
Preserve ordinary lifecycle and replication cadence while retaining usage-only convergence backoff, explicit cycle floors and publication fencing. Cover repeated snapshots and the real convergence waiting path. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
a57a3ae51d |
fix(scanner): reuse observations and expose complete usage (#8498)
* fix(scanner): reuse observations and expose complete usage Allow superseded retries to reach the existing validated cache planner while bootstrap remains pending. Serve complete observations in server info without changing quota authority or dirty acknowledgements. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): route fixture setup through storage boundary Keep the single-disk scanner fixture on the existing storage_api boundary instead of importing the ECStore API facade directly. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
0ef133bbb0 |
fix: refresh peer metadata without blocking bucket mutations (#8486)
* fix: refresh peer bucket metadata after creation * fix(s3): release bucket mutation slots before peer reloads * test(ecstore): distinguish delete lock waits from disk latency |
||
|
Hauser
|
dd90bb7488 |
fix(s3): short-circuit conditional reads at metadata quorum (#8459)
* fix(s3): evaluate conditional reads before opening object bodies Reuse locked metadata snapshots for conditional GETs and unify HEAD preconditions and cache headers. Preserve body-cache hooks and validate encryption metadata and key authorization before returning bodyless errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(s3): unify conditional reads at metadata quorum Apply scoped metadata conditions to the existing reader pipeline instead of maintaining parallel prepared-read branches. Preserve quorum checks, early-stop, coalescing, and cache-hook behavior on condition misses. Track metadata-only termination through selection and reject body access or a changed decision. Keep concrete reader futures off request stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): strengthen conditional read acceptance evidence Assert zero shard, decode, cache-fill, and remote-body calls on matching conditions with live positive controls. Exercise GET and HEAD through s3s and Hyper with conditional-header and precedence matrices. Add reproducible paired application-read benchmarks with isolated metadata controls and strict drift, artifact, and resource gates. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(s3): remove obsolete conditional comparison helper Keep ETag comparison tests on the canonical read-condition evaluator, remove unused date-trait imports, and name the HTTP case type. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): validate full conditional etag fields Reject adjacent tags and malformed trailing list members before accepting a match. Keep the existing request-scoped metadata read and authorization order, while removing redundant validation wrappers and unused benchmark overrides. Replace simulated phase coverage with real 304/412 cache-fill probes and extend HTTP/1.1 and HTTP/2 ingress regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): cache terminal conditional metadata safely Preserve quorum-validated conditional metadata when early termination cancels the remaining fanout. Restrict those entries to terminal GET readers; body and full-layout consumers still obtain their own metadata. Serialize cache publication per key so conditional entries cannot replace complete layouts or renew their TTL. Retain generation fencing and the final selected-metadata predicate check before any body access. Cover real four-disk hedging, nonterminal body and range reads, consumer isolation, generation races, atomic publication and actual cache expiry. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(log-analyzer): align heal writer failure rule with current logs Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
GatewayJ
|
0018bbf5d7 |
ci: guard OIDC architecture boundaries (#8447)
* ci: guard OIDC architecture boundaries * fix(ci): guard OIDC session ownership * fix(ci): ignore OIDC authorization destructuring * fix(ci): accept typed authorization destructuring --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
4166b9503f |
fix(ecstore): recover interrupted delete transactions (#8468)
* fix(ecstore): recover crash-interrupted delete retries Retain set-quorum cleanup proof and restore metadata before dropping rollback markers. Reclaim eligible exact-key residue during explicit delete retries, with bucket identity, version, retention and namespace fencing guards. Preserve prefix children and snapshot ownership through batched physical cleanup, and cover interrupted transactions and failure boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route recovery test imports through contracts Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve missing-key deletes at quorum * feat(ecstore): add certified set-wide delete recovery Atomically prepare deletion journals and exact targets, publish durable set receipts, and recover through one fenced disk/RPC core used by startup and scoped administrator healing. Preserve returning generations, ordinary missing-key quorum semantics, and legacy rollback wire compatibility. Retain namespace and volume ownership through late physical operations, order related transactions causally, and keep proof until full convergence. Remove the superseded local-only recovery path and add storage, protocol, RPC, durability and isolated Docker regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim completed multipart delete sidecars * fix(ecstore): recover multipart sidecars with verified proofs Recognize native multipart sidecars at the shared recovery boundary and restore them without treating metadata or proof files as shard payloads. Validate required proof generations and bind every present live/staged shard copy to its metadata-bound Merkle proof before restoring metadata. Cover protected native MPU rollback, split source layouts, missing or corrupt proofs, and payload corruption with recomputed bitrot checksums. Add real multipart Docker regressions in both integrity modes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): settle superseded committed version deletes Allow a validated committed decision to finish its own data cleanup after a later decision removes current metadata. Prepared copies need an inspected same-transaction receipt matching immutable local intent before promotion, and retain their local journal without importing peer hashes. Reuse the authority matcher for dry-run and settlement. Cover chained explicit-version deletions, partial receipt publication, mismatch rejection and no-resurrection, while retaining the no-decision absence protection. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): clarify delete recovery case label Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): restore recovery CI guard coverage Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): preserve owned OIDC validation errors Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): unify recovery authority and snapshot work Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve delete recovery invariants --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
1266d4c270 |
feat(tls): reload and publish external certificate generations (#8467)
* feat(tls): reload external certificate snapshots coherently Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(tls): publish server certificates through a fenced admin API Add opt-in server-leaf publication with dedicated IAM actions, actual TLS transport checks, immutable generations, durable captured-token CAS and idempotent receipts. Preserve independent trust and client identity sources and distinguish visible publication state from the full serving chain. Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test, and eight live local API scenarios including least privilege and restart. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route certificate handler types through S3 facade Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(admin): include TLS certificate routes in route matrix Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): reconcile startup trust before skipping reloads Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): reuse pinned generations and prepared listener state Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): preserve startup trust source requirements --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Chris
|
ba2047b184 | ci: reuse workspace test builds for serial ILM checks (#8465) | ||
|
GatewayJ
|
da42266bfb | refactor(iam): organize OIDC runtime modules (#8445) | ||
|
Chris
|
367f13f3f9 | ci: reduce repeated PR runner work (#8463) | ||
|
Hauser
|
9ecb00d388 |
fix(scanner): avoid placeholder retention and resume checkpoints (#8460)
* fix(scanner): bound old-cache lookups and recover checkpoint saves Keep missing child lookups read-only and preserve the checkpoint receiver after ordinary persistence errors. Reload on stale CAS or indeterminate revision refresh, and report confirmed periodic saves separately. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): remove redundant checkpoint error conversion Return the existing StorageError directly so strict Clippy passes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(scanner): clarify checkpoint persistence guarantees Distinguish a confirmed periodic save from validated restart coverage. Keep adoption and subtree-skip requirements explicit. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): preserve resumable periodic coverage Materialize active ancestor accounting in bounded periodic snapshots and seal completed coverage independently of raw enumeration state. Preserve durable coverage during early resumed enumeration and exclude unfinished out-of-order heal work from periodic and cancellation receipts. Reuse ancestor frame slots to avoid growing child-set copies on descent, retain compacted accumulator safety, and report checkpoint adoption resets. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): reject covered active ancestors in checkpoints Check every suspended ancestor alongside the current folder before certifying completed coverage. An unfinished covered ancestor must not become a resumable subtree merely because its descendant sorts later. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): retain an incomplete root for missing namespaces Create the empty bucket root explicitly at the missing-namespace boundary. Preserve nonempty durable caches and keep namespace results unpublished. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
b77bd26874 | Merge commit from fork | ||
|
Hauser
|
8474feaa26 |
fix: reconcile bucket generations and observe lock failures (#8421)
* fix: reconcile bucket generations and observe lock failures Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix: fence bucket recovery through cancellation and quota reuse Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): route tests through storage API boundary Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): use typed lock errors in recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
d386f7741d |
fix(lifecycle): coordinate tier free-version recovery scans (#8387)
* fix(lifecycle): coordinate tier free-version recovery scans * fix(lifecycle): remove obsolete recovery enqueue entry point * test(upgrade): wait for writers after each rolling replacement |
||
|
cppudge
|
99ae115731 |
fix(ecstore): make deletes cancellation-safe and reclaim their residue (#8378)
* fix(ecstore): keep object deletes running when the caller is dropped DeleteObject and DeleteObjects commit on every disk and then run a separate cleanup pass, both inside the request future. When the client disconnects, hyper drops that future and the delete stops between the two passes: xl.meta is gone, but the rollback backup and the marked data dir stay on disk, hidden from listings and never reclaimed (#6898). Run the ECStore delete entry points on a detached task tracked by the instance context. The task owns the object locks, both disk passes and any rollback, so dropping the caller only stops the wait. Server shutdown waits up to 30s for detached deletes after the HTTP drain. Two smaller leaks of the same kind: a cancelled data-dir delete no longer leaves its snapshot-lease entry marked as deleting forever, and DiskStore::delete_versions releases its waiting slot through a guard. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): finish committed delete residue on repeated deletes Residue that an older build stranded (#6898) has no xl.meta, so deleting the key again reports it missing and leaves everything in place. On a disk, a delete-data.<T> marker in one of the object's data dirs proves that transaction T removed the metadata there and committed: the data it marked can no longer be read, healed or rolled back on that disk. When delete_versions or delete_version finds no xl.meta, remove what the cleanup pass of every transaction proven that way would have removed, under the object's metadata lease. The rollback backup goes before the marked data dirs, so an attempt cut short still leaves the marker for the next one. Reserved markers, rollback backups without a committed marker and directories that also hold xl.meta or other keys are left alone. The disk result stays FileNotFound; the extra cost is one directory read per missing key inside the disk call that already runs, with no new RPC, and the read stops at the first child that is not a data dir. Finished transactions are counted in rustfs_disk_delete_residue_finished_total instead of logged per object. A single DeleteObject in an unversioned bucket answers a missing key from the metadata read quorum without reaching the disks, so existing residue is reclaimed by DeleteObjects or by the empty-listing purge. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): purge the rollback backup of a committed delete on listing After an empty listing, the orphan purge (#7955, #8261) already removes data dirs that carry a committed delete-data.<T> marker, but it treated the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data and blocked the whole object directory and its ancestors. Residue left by an abandoned delete (#6898) therefore survived the listing, and DeleteBucket kept failing with BucketNotEmpty. Treat a UUID dir that holds only xl.meta.bkp as purgeable when a purgeable data dir of the same object on the same disk carries the committed marker of that transaction. A backup without such a marker still blocks as before, since it may be needed to roll the delete back. The other purge gates are unchanged: every disk of the set must be online, and the backoff applies. xl.meta or unmarked data on any disk blocks that directory and its ancestors on every disk. Like a marked data dir before, a backup below such a directory is still removed from a disk that holds no xl.meta there and proves the transaction with its own marker. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): move caller-held delete locks into the detached delete A force (recursive) delete takes the bucket lifecycle write lock in the S3 handler and passes only its fence down, so the store does not take that lock again. Since the delete runs on a detached task, dropping the request released the lock while the delete kept running: a PUT under the same prefix could succeed and then be removed by the old delete. Lifecycle expiry had the same shape with its table-bucket publication read lock, of which only the lock-lost signal reaches the delete. Add ECStore::delete_object_with_tier_delete_journal_and_guards. It moves the caller's guards into the detached task and releases them after the delete and its listing bookkeeping. The force-delete handler and lifecycle expiry hand their guards over; DeleteObjects holds none. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(startup): keep unclean-shutdown markers while deletes still run Shutdown waits up to 30s for detached deletes, but it cleared the unclean-shutdown markers even when that wait timed out. A delete then cut short by the runtime teardown went unnoticed: the next startup skipped the erasure-set heal that the marker triggers. Clear the markers only when the heal handoff completed and every detached mutation finished, and log that they are kept otherwise. Wait after the FTP, FTPS, SFTP and WebDAV servers have stopped as well, since their deletes take the same detached path. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * test(ecstore): simulate tier-delete crashes below the detached delete The chunk crash-boundary tests model a crash by aborting the task that runs delete_object_with_tier_delete_journal while it is paused at a TierDeleteChunkTestBarrier. That entry point now runs the delete on a detached task, so aborting the caller no longer stops it: the delete resumed on the old store once the barrier was dropped, and building the restarted store on the same disks tripped the tier-delete journal proof assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources failed in CI). Run the delete through handle_delete_object_with_journal, the body below the detached task, so the abort stops it at the barrier as before. tier_delete_chunk_missing_child_with_journals_fails_closed used the same pattern; its resumed delete only failed fast because the tier rejects removes, so it gets the same change. Signed-off-by: Petr Slavyagin <disident47@gmail.com> --------- Signed-off-by: Petr Slavyagin <disident47@gmail.com> |
||
|
cxymds
|
855586255f |
feat(auth): allow custom secret keys beyond 40 bytes (#8419)
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints. |
||
|
Hauser
|
bf47a919ac |
feat(quota): add sharded durable reservation protocol (#8402)
* feat(quota): add sharded durable reservation protocol Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): preserve typed task error context Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): initialize all durable reservation shards Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): persist bootstrap shard eligibility Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(quota): reuse allocator bootstrap state Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): widen inventory startup wait Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(quota): reserve protocol v3 for unknown fixtures Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): reload allocator after concurrent credit refills --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |