mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-27 14:27:10 +00:00
1.0.2
888 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris
|
2b243d8659 |
fix(scanner): clear transient cycle recovery on follower lock contention (1.0.2 backport) (#8596)
fix(scanner): clear transient cycle recovery on follower lock contention (#8566)
(cherry picked from commit
|
||
|
Chris
|
f0f2b0cf48 |
fix(replication): skip the replica ETag audit for encrypted sources (1.0.2 backport) (#8580)
Backport of #8573 (main commit |
||
|
Hauser
|
08ac1ec218 |
fix(s3): advise retry after typed lock timeouts (#8546)
Add a one-second Retry-After hint at the canonical S3 error conversion only for ServiceUnavailable responses retaining a typed lock timeout. Keep existing error codes, source chains, and lock budgets unchanged. Cover actual HTTP/XML serialization, nested causes, private-resource redaction, and unrelated disk/quorum/text failures. Document the response contract and the limits of log-count and five-second timeout attribution. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
388d1f17e6 |
fix(heal): safely recover missing bucket directories by default (#8517)
* fix(heal): safely recover missing bucket directories by default * fix(heal): allow bound admin recovery before namespace checks |
||
|
Henry Guo
|
22c093e66b | feat(table-catalog): add opt-in periodic maintenance queueing (#8537) | ||
|
Henry Guo
|
fa0ac13e9b |
feat(table-catalog): support durable-strong table maintenance (#8523)
* feat(table-catalog): enable durable maintenance control * fix(table-catalog): revalidate unlocked worker planning * fix(table-catalog): preserve maintenance state across rename * fix(table-catalog): fence maintenance reads across rename * fix(table-catalog): bound scheduler history read fencing * test(table-catalog): inject loss using the held client lease * refactor(table-catalog): remove unused metadata log wrapper |
||
|
Hauser
|
b80590f1a7 |
perf(heal): index replacement history reconciliation (#8527)
refactor(heal): index replacement generation reconciliation Reconcile owned records and their definitive sources in one pass. Group claims by target slot and use a shared-anchor fast path while retaining pairwise ambiguity checks. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
025ed4c6d1 |
fix(heal): preserve retained replacement histories (#8525)
Keep definitive local histories while rejecting ambiguous overlapping peer records. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
adf17bbfba | fix(heal): reject incomplete administrator bucket discovery (#8512) | ||
|
GatewayJ
|
fb5c002a30 | refactor(iam): encapsulate federated authorization models (#8514) | ||
|
Hauser
|
9ca0e9b717 |
fix(obs): isolate bucket usage dashboards by cluster (#8510)
Keep same-name buckets in independent clusters separate while deduplicating replicated usage observations. Document the existing scanner-backed exporter and cover its gauge contract and cluster-scoped queries. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
34803e8165 |
perf(ecstore): isolate cleanup and preserve metadata authority (#8289)
* chore: pin s3s to upstream git revision Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4. Allow the official s3s git source in cargo-deny so source checks continue to pass. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): instrument put object hot path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): split cleanup worker wait metrics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * upgrade version * chore(metrics): add PUT post-store stage probes Add lightweight PUT post-store stage probes for issue 2615 validation. * fix cargo deny check warn * upgrade version * perf(ecstore): bound and isolate cleanup execution (#8151) * perf(ecstore): bound and isolate cleanup execution * fix(ecstore): preserve cleanup cancellation and lease release * feat(ecstore): default isolated cleanup to CPUs 1-2 * test(ecstore): decouple budget tests from CPU affinity * upgrade version * perf(ecstore): reuse pinned directory handles for object I/O * feat(observability): add sampled local GetObject stage summaries Add an opt-in GetObject stage aggregation path that works without the OTLP metrics export. When RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED is set, io-metrics keeps per-(path, stage, object class, size bucket) sampled counts and duration totals in a bounded in-process map, and a startup task flushes them periodically to /var/log/rustfs/get-stage-summary.log, tunable via the sample rate and interval env keys. * build(deps): bump workspace dependency releases Patch releases: async-compression 0.4.49, tokio-rustls 0.26.6, xxhash-rust 0.8.19, suppaftp 12.1.1, russh-sftp 3.0.1. Minor release: hotpath 0.27.0. * fix(ecstore): fence distributed metadata cache mutations Add authenticated cluster-wide mutation fences for the opt-in DistErasure GetObject metadata cache. Guard recursive prefix and batch deletes, fail closed on configuration mismatch or uncertain quorum, and cover replay, restart, peer loss, and mixed-configuration behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT observers on improve/base (#8039) * perf(object): instrument storage hot paths Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(metrics): consolidate PUT stage observation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> * build(deps): bump workspace dependency releases * ci: allowlist rustix OFlags WRONLY in typos config * refactor(storage): route ecstore symbols through storage_api boundaries * perf(ecstore): cache quorum-selected early-stop metadata (#8308) Publish eligible latest-version GET metadata after a validated read quorum when early-stop cancels the remaining fanout. Keep metadata errors, selection quorum, online read quorum, and mutation-generation checks as publication gates. Add slow-peer repeat-hit and concurrent mutation-fence regression tests. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: merge latest main into improve/base (#8332) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): borrow metadata cache lookup key (#8333) Avoid request-scoped Arc<str> allocations on eligible cache lookups while preserving full key identity. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * chore: sync improve/base with latest main (#8334) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish metrics by process identity (#8343) Attach process.pid to the OpenTelemetry resource and promote it for Prometheus so counter series from restarted processes remain distinct. Preserve the stable RustFS node and service instance attributes. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync origin/main into improve/base (#8345) * refactor(storage): bind rename rollback to its commit context (#8324) * refactor(storage): separate metadata scheduling from quorum decisions (#8327) * test(storage): verify rejected rollback inspection on real disks (#8331) * test(storage): make commit fault checkpoints independent (#8326) * test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup * fix(listing): count logical prefixes before pagination (#8328) * fix(listing): count logical prefixes before pagination * test(listing): normalize the fixed corpus seed * fix(scanner): isolate and coalesce dirty usage journals (#8325) * fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(storage): retain typed owners across borrowed writes (#8329) * fix(storage): retain typed owners across borrowed writes * fix(storage): preserve Copy write completion policy * refactor(storage): remove redundant write owner references * fix(ecstore): hide rollback backup residue in prefix listings * test(connect): isolate short HTTP fixtures in nextest * fix(protocols): enforce S3 policies for file protocol operations (#8262) * test(ecstore): enforce main core regression gates (#8338) * test(ecstore): enforce main core regression gates * test(ecstore): pin external shards and completion regressions * fix(test): restore authenticated cluster fault scenarios (#8340) * fix(test): restore authenticated cluster fault scenarios * fix(test): use the E2E storage API boundary * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * ci: coerce the pool timeout input to a number timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to the string '240' and the runner rejected the template: 'An error occurred when attempting to determine the step timeout ... Unexpected value 240' (chain run 36505625402 job at 02:25:52, standalone run 36393453114 at 07:46:37). Both pool runs since #8155 merged have therefore executed without the intended step budget. Wrap the expression in fromJSON and update the policy test that pinned the broken literal. * fix(ecstore): own listing workers through cancellation (#8342) Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route commit sync through guarded helper Use the Linux-capable guarded directory sync entry point for both commit sync sites. This keeps group commit and file-sync admission behavior while ensuring the already-open commit directory remains the sync target. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): expose rename directory guard in production The commit path now acquires the directory guard in production builds, not only tests, so destination metadata reads and durable sync share the guarded context. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> * fix(ecstore): remove unsafe metadata cache admission paths Keep distributed GET metadata on quorum reads and quota admission on authoritative storage. Preserve local cache invalidation, remove the experimental RPC surface, and gate diagnostic work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): cover cold degraded mutation admission Check first PUT and DELETE with one peer absent and no SDK retries, then verify that losing write quorum still rejects writes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retain guarded Linux destination metadata reads Use the existing commit directory guard when reading old xl.meta on Linux. Cover symlinked destination metadata for inline and non-inline renames without changing the split commit structure. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(test): remove retired cache RPC import Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * merge: sync latest main into improve/base (#8349) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * merge: sync main auth and scanner updates (#8352) * test: allow cleanup fixture CI headroom (#8336) * fix(s3): encode multipart listing response keys (#8347) * perf(scanner): reduce scan and checkpoint peak allocations (#8346) * perf(scanner): reduce scan and checkpoint peak allocations * test(scanner): isolate partial commit recovery scenarios * fix(auth): return ExpiredToken for retained STS credentials (#8344) * fix(auth): return ExpiredToken for retained STS credentials Validate session identity independently of credential expiry in both IAM rejection and claims validation, and preserve typed S3 errors at the caller. Keep token binding, disabled-account precedence, and service-account behavior. Add exact expiry, IAM, and HTTP/XML regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): synchronize the metadata slowtail hedge regression Park the slow metadata task at a test-only fault checkpoint and release healthy reads after the fault is active. Assert the spare response and cancelled-task drain without imposing a 300 ms disk/scheduler deadline. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ilm): synchronize copy compensation transition stages Observe automatic compensation reaching the mock tier PUT checkpoint before waiting for the final metadata commit. Assert forced enqueue failure, automatic backfill, one completed tier upload, and payload integrity. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reject overlapping cleanup schedulers Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(ecstore): explain cleanup scheduler conflict Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(metrics): separate GET request and timeout outcomes (#8354) Co-authored-by: zhi22915 <qiuzgang@gmail.com> * docs: refresh security advisory lessons (#8350) * fix(heal): bound durable MRF rescheduling (#8351) Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(rustfs): route recovery errors through S3 gateway Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(io-metrics): use is_multiple_of for sampling Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): satisfy strict Clippy pagination checks Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): avoid cloning Copy purge identity Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * add ignore .mimosa * fix(zip): adapt archive readers to zip 9 APIs Upgrade zip and its required dependencies alongside fallible archive-name handling in log ingestion, report bundles, bucket metadata import, and archive tests. Preserve existing skip and error propagation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reuse quorum metadata for body GETs Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ci): align storage test stacks and peer reset classification Apply the existing bounded ECStore stack setup to the affected cross-crate storage fixtures. Validate conditional-only metadata publication separately from full body-layout cache reuse. Classify typed h2 shutdown resets as unreachable while retaining application, authentication, cancellation, and deadline distinctions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update workspace dependencies Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve omitted-disk GET recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore: integrate published workspace dependency updates Preserve the concurrent dependency refresh and retain the typed h2 reset classification and bounded storage-test stack setup. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): preserve snapshot size budget after fallback move * fix(test): use one barrier for compensation COPY Let the forced-enqueue helper own the PUT barrier and compensation assertions. Waiting on an outer barrier overwritten by the helper always timed out after the upload had completed. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(ecstore): apply current lint fixes Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * chore(deps): update sha2 to 0.11.1 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: hector <42570491+majinghe@users.noreply.github.com> Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Hauser
|
e7e6556f83 |
fix(scanner): bound superseded waits for regular maintenance (#8502)
Preserve ordinary lifecycle and replication cadence while retaining usage-only convergence backoff, explicit cycle floors and publication fencing. Cover repeated snapshots and the real convergence waiting path. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
a57a3ae51d |
fix(scanner): reuse observations and expose complete usage (#8498)
* fix(scanner): reuse observations and expose complete usage Allow superseded retries to reach the existing validated cache planner while bootstrap remains pending. Serve complete observations in server info without changing quota authority or dirty acknowledgements. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): route fixture setup through storage boundary Keep the single-disk scanner fixture on the existing storage_api boundary instead of importing the ECStore API facade directly. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
0ef133bbb0 |
fix: refresh peer metadata without blocking bucket mutations (#8486)
* fix: refresh peer bucket metadata after creation * fix(s3): release bucket mutation slots before peer reloads * test(ecstore): distinguish delete lock waits from disk latency |
||
|
Hauser
|
dd90bb7488 |
fix(s3): short-circuit conditional reads at metadata quorum (#8459)
* fix(s3): evaluate conditional reads before opening object bodies Reuse locked metadata snapshots for conditional GETs and unify HEAD preconditions and cache headers. Preserve body-cache hooks and validate encryption metadata and key authorization before returning bodyless errors. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(s3): unify conditional reads at metadata quorum Apply scoped metadata conditions to the existing reader pipeline instead of maintaining parallel prepared-read branches. Preserve quorum checks, early-stop, coalescing, and cache-hook behavior on condition misses. Track metadata-only termination through selection and reject body access or a changed decision. Keep concrete reader futures off request stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(s3): strengthen conditional read acceptance evidence Assert zero shard, decode, cache-fill, and remote-body calls on matching conditions with live positive controls. Exercise GET and HEAD through s3s and Hyper with conditional-header and precedence matrices. Add reproducible paired application-read benchmarks with isolated metadata controls and strict drift, artifact, and resource gates. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(s3): remove obsolete conditional comparison helper Keep ETag comparison tests on the canonical read-condition evaluator, remove unused date-trait imports, and name the HTTP case type. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): validate full conditional etag fields Reject adjacent tags and malformed trailing list members before accepting a match. Keep the existing request-scoped metadata read and authorization order, while removing redundant validation wrappers and unused benchmark overrides. Replace simulated phase coverage with real 304/412 cache-fill probes and extend HTTP/1.1 and HTTP/2 ingress regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): cache terminal conditional metadata safely Preserve quorum-validated conditional metadata when early termination cancels the remaining fanout. Restrict those entries to terminal GET readers; body and full-layout consumers still obtain their own metadata. Serialize cache publication per key so conditional entries cannot replace complete layouts or renew their TTL. Retain generation fencing and the final selected-metadata predicate check before any body access. Cover real four-disk hedging, nonterminal body and range reads, consumer isolation, generation races, atomic publication and actual cache expiry. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(log-analyzer): align heal writer failure rule with current logs Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
GatewayJ
|
0018bbf5d7 |
ci: guard OIDC architecture boundaries (#8447)
* ci: guard OIDC architecture boundaries * fix(ci): guard OIDC session ownership * fix(ci): ignore OIDC authorization destructuring * fix(ci): accept typed authorization destructuring --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
4166b9503f |
fix(ecstore): recover interrupted delete transactions (#8468)
* fix(ecstore): recover crash-interrupted delete retries Retain set-quorum cleanup proof and restore metadata before dropping rollback markers. Reclaim eligible exact-key residue during explicit delete retries, with bucket identity, version, retention and namespace fencing guards. Preserve prefix children and snapshot ownership through batched physical cleanup, and cover interrupted transactions and failure boundaries. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): route recovery test imports through contracts Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve missing-key deletes at quorum * feat(ecstore): add certified set-wide delete recovery Atomically prepare deletion journals and exact targets, publish durable set receipts, and recover through one fenced disk/RPC core used by startup and scoped administrator healing. Preserve returning generations, ordinary missing-key quorum semantics, and legacy rollback wire compatibility. Retain namespace and volume ownership through late physical operations, order related transactions causally, and keep proof until full convergence. Remove the superseded local-only recovery path and add storage, protocol, RPC, durability and isolated Docker regression coverage. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim completed multipart delete sidecars * fix(ecstore): recover multipart sidecars with verified proofs Recognize native multipart sidecars at the shared recovery boundary and restore them without treating metadata or proof files as shard payloads. Validate required proof generations and bind every present live/staged shard copy to its metadata-bound Merkle proof before restoring metadata. Cover protected native MPU rollback, split source layouts, missing or corrupt proofs, and payload corruption with recomputed bitrot checksums. Add real multipart Docker regressions in both integrity modes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): settle superseded committed version deletes Allow a validated committed decision to finish its own data cleanup after a later decision removes current metadata. Prepared copies need an inspected same-transaction receipt matching immutable local intent before promotion, and retain their local journal without importing peer hashes. Reuse the authority matcher for dry-run and settlement. Cover chained explicit-version deletions, partial receipt publication, mismatch rejection and no-resurrection, while retaining the no-decision absence protection. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): clarify delete recovery case label Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): restore recovery CI guard coverage Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): preserve owned OIDC validation errors Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): unify recovery authority and snapshot work Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve delete recovery invariants --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
1266d4c270 |
feat(tls): reload and publish external certificate generations (#8467)
* feat(tls): reload external certificate snapshots coherently Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * feat(tls): publish server certificates through a fenced admin API Add opt-in server-leaf publication with dedicated IAM actions, actual TLS transport checks, immutable generations, durable captured-token CAS and idempotent receipts. Preserve independent trust and client identity sources and distinguish visible publication state from the full serving chain. Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test, and eight live local API scenarios including least privilege and restart. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(admin): route certificate handler types through S3 facade Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(admin): include TLS certificate routes in route matrix Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): reconcile startup trust before skipping reloads Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(tls): reuse pinned generations and prepared listener state Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(tls): preserve startup trust source requirements --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Chris
|
ba2047b184 | ci: reuse workspace test builds for serial ILM checks (#8465) | ||
|
GatewayJ
|
da42266bfb | refactor(iam): organize OIDC runtime modules (#8445) | ||
|
Chris
|
367f13f3f9 | ci: reduce repeated PR runner work (#8463) | ||
|
Hauser
|
9ecb00d388 |
fix(scanner): avoid placeholder retention and resume checkpoints (#8460)
* fix(scanner): bound old-cache lookups and recover checkpoint saves Keep missing child lookups read-only and preserve the checkpoint receiver after ordinary persistence errors. Reload on stale CAS or indeterminate revision refresh, and report confirmed periodic saves separately. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): remove redundant checkpoint error conversion Return the existing StorageError directly so strict Clippy passes. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs(scanner): clarify checkpoint persistence guarantees Distinguish a confirmed periodic save from validated restart coverage. Keep adoption and subtree-skip requirements explicit. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): preserve resumable periodic coverage Materialize active ancestor accounting in bounded periodic snapshots and seal completed coverage independently of raw enumeration state. Preserve durable coverage during early resumed enumeration and exclude unfinished out-of-order heal work from periodic and cancellation receipts. Reuse ancestor frame slots to avoid growing child-set copies on descent, retain compacted accumulator safety, and report checkpoint adoption resets. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): reject covered active ancestors in checkpoints Check every suspended ancestor alongside the current folder before certifying completed coverage. An unfinished covered ancestor must not become a resumable subtree merely because its descendant sorts later. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): retain an incomplete root for missing namespaces Create the empty bucket root explicitly at the missing-namespace boundary. Preserve nonempty durable caches and keep namespace results unpublished. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
b77bd26874 | Merge commit from fork | ||
|
Hauser
|
8474feaa26 |
fix: reconcile bucket generations and observe lock failures (#8421)
* fix: reconcile bucket generations and observe lock failures Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix: fence bucket recovery through cancellation and quota reuse Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): route tests through storage API boundary Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): use typed lock errors in recovery Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
d386f7741d |
fix(lifecycle): coordinate tier free-version recovery scans (#8387)
* fix(lifecycle): coordinate tier free-version recovery scans * fix(lifecycle): remove obsolete recovery enqueue entry point * test(upgrade): wait for writers after each rolling replacement |
||
|
cppudge
|
99ae115731 |
fix(ecstore): make deletes cancellation-safe and reclaim their residue (#8378)
* fix(ecstore): keep object deletes running when the caller is dropped DeleteObject and DeleteObjects commit on every disk and then run a separate cleanup pass, both inside the request future. When the client disconnects, hyper drops that future and the delete stops between the two passes: xl.meta is gone, but the rollback backup and the marked data dir stay on disk, hidden from listings and never reclaimed (#6898). Run the ECStore delete entry points on a detached task tracked by the instance context. The task owns the object locks, both disk passes and any rollback, so dropping the caller only stops the wait. Server shutdown waits up to 30s for detached deletes after the HTTP drain. Two smaller leaks of the same kind: a cancelled data-dir delete no longer leaves its snapshot-lease entry marked as deleting forever, and DiskStore::delete_versions releases its waiting slot through a guard. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): finish committed delete residue on repeated deletes Residue that an older build stranded (#6898) has no xl.meta, so deleting the key again reports it missing and leaves everything in place. On a disk, a delete-data.<T> marker in one of the object's data dirs proves that transaction T removed the metadata there and committed: the data it marked can no longer be read, healed or rolled back on that disk. When delete_versions or delete_version finds no xl.meta, remove what the cleanup pass of every transaction proven that way would have removed, under the object's metadata lease. The rollback backup goes before the marked data dirs, so an attempt cut short still leaves the marker for the next one. Reserved markers, rollback backups without a committed marker and directories that also hold xl.meta or other keys are left alone. The disk result stays FileNotFound; the extra cost is one directory read per missing key inside the disk call that already runs, with no new RPC, and the read stops at the first child that is not a data dir. Finished transactions are counted in rustfs_disk_delete_residue_finished_total instead of logged per object. A single DeleteObject in an unversioned bucket answers a missing key from the metadata read quorum without reaching the disks, so existing residue is reclaimed by DeleteObjects or by the empty-listing purge. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): purge the rollback backup of a committed delete on listing After an empty listing, the orphan purge (#7955, #8261) already removes data dirs that carry a committed delete-data.<T> marker, but it treated the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data and blocked the whole object directory and its ancestors. Residue left by an abandoned delete (#6898) therefore survived the listing, and DeleteBucket kept failing with BucketNotEmpty. Treat a UUID dir that holds only xl.meta.bkp as purgeable when a purgeable data dir of the same object on the same disk carries the committed marker of that transaction. A backup without such a marker still blocks as before, since it may be needed to roll the delete back. The other purge gates are unchanged: every disk of the set must be online, and the backoff applies. xl.meta or unmarked data on any disk blocks that directory and its ancestors on every disk. Like a marked data dir before, a backup below such a directory is still removed from a disk that holds no xl.meta there and proves the transaction with its own marker. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): move caller-held delete locks into the detached delete A force (recursive) delete takes the bucket lifecycle write lock in the S3 handler and passes only its fence down, so the store does not take that lock again. Since the delete runs on a detached task, dropping the request released the lock while the delete kept running: a PUT under the same prefix could succeed and then be removed by the old delete. Lifecycle expiry had the same shape with its table-bucket publication read lock, of which only the lock-lost signal reaches the delete. Add ECStore::delete_object_with_tier_delete_journal_and_guards. It moves the caller's guards into the detached task and releases them after the delete and its listing bookkeeping. The force-delete handler and lifecycle expiry hand their guards over; DeleteObjects holds none. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(startup): keep unclean-shutdown markers while deletes still run Shutdown waits up to 30s for detached deletes, but it cleared the unclean-shutdown markers even when that wait timed out. A delete then cut short by the runtime teardown went unnoticed: the next startup skipped the erasure-set heal that the marker triggers. Clear the markers only when the heal handoff completed and every detached mutation finished, and log that they are kept otherwise. Wait after the FTP, FTPS, SFTP and WebDAV servers have stopped as well, since their deletes take the same detached path. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * test(ecstore): simulate tier-delete crashes below the detached delete The chunk crash-boundary tests model a crash by aborting the task that runs delete_object_with_tier_delete_journal while it is paused at a TierDeleteChunkTestBarrier. That entry point now runs the delete on a detached task, so aborting the caller no longer stops it: the delete resumed on the old store once the barrier was dropped, and building the restarted store on the same disks tripped the tier-delete journal proof assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources failed in CI). Run the delete through handle_delete_object_with_journal, the body below the detached task, so the abort stops it at the barrier as before. tier_delete_chunk_missing_child_with_journals_fails_closed used the same pattern; its resumed delete only failed fast because the tier rejects removes, so it gets the same change. Signed-off-by: Petr Slavyagin <disident47@gmail.com> --------- Signed-off-by: Petr Slavyagin <disident47@gmail.com> |
||
|
cxymds
|
855586255f |
feat(auth): allow custom secret keys beyond 40 bytes (#8419)
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints. |
||
|
Hauser
|
bf47a919ac |
feat(quota): add sharded durable reservation protocol (#8402)
* feat(quota): add sharded durable reservation protocol Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): preserve typed task error context Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): initialize all durable reservation shards Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): persist bootstrap shard eligibility Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(quota): reuse allocator bootstrap state Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): widen inventory startup wait Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(quota): reserve protocol v3 for unknown fixtures Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(quota): reload allocator after concurrent credit refills --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Chris
|
681c95f359 |
ci: add daily main sync for the gateway integration branch (#8401)
Merge main into gateway/integration every weekday with a --no-ff merge commit; abort a conflicting merge and open a sync/<source>-<date> pull request listing the conflicting files instead. Run ci.yml on pull requests that target gateway/integration so the branch ruleset's required checks can report, and document both in the CI gate matrix. Refs rustfs/backlog#2736, rustfs/backlog#2734 |
||
|
Hauser
|
32d98ff8b2 |
feat(scanner): admit background disk I/O (#8389)
* feat(scanner): admit background disk I/O Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(startup): place tests after module items Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Lindroos Hsu
|
db2abcf0a6 |
feat(http3): add opt-in QUIC transport (#7903)
* build(http3): add optional s3s-http3 dependency Signed-off-by: tommady <tommady@users.noreply.github.com> * feat(server): add opt-in HTTP/3 transport Signed-off-by: tommady <tommady@users.noreply.github.com> * test(server): cover HTTP/3 TLS configuration * feat(server): add opt-in HTTP/3 support Serve S3 requests over QUIC with the existing middleware stack. Signed-off-by: tommady <tommady@users.noreply.github.com> * fix(http3): cancel pending body reads without panicking * test(http3): keep transport independent of s3s --------- Signed-off-by: tommady <tommady@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Henry Guo
|
3b92708011 |
feat(table-catalog): add durable catalog backups (#8266)
* feat(table-catalog): add durable catalog backups * fix(table-catalog): preserve migration lock ordering * fix(table-catalog): preserve backup recovery fences * fix(table-catalog): order backup publication locks * fix(table-catalog): preserve recovery publication fences * test(table-catalog): use the storage facade for recovery fencing * test(ecstore): repair catalog publication fixtures * fix(table-catalog): address catalog backup clippy failures * test(table-catalog): adapt generation races to publication fencing --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
GatewayJ
|
477f768f7e |
feat(table-catalog): support disabling empty catalogs (#8376)
* feat(table-catalog): support disabling empty catalogs * fix(admin): use the storage error domain module * style(admin): format the publication fence error * fix(table-catalog): use shared API error constructors |
||
|
Chris
|
14adc5ed3f |
test(ecstore): cover real metadata cache and late identity reads (#8371)
* test(ecstore): cover real metadata cache and late identity reads * test(e2e): bound max-keys fixture upload concurrency |
||
|
Chris
|
073b4d838e | test(ecstore): verify multi-generation authority model (#8339) | ||
|
Hauser
|
003570b61e |
fix(heal): bound durable MRF rescheduling (#8351)
Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
4783adff36 |
test(ecstore): enforce main core regression gates (#8338)
* test(ecstore): enforce main core regression gates * test(ecstore): pin external shards and completion regressions |
||
|
GatewayJ
|
68ad1a1d7b | fix(protocols): enforce S3 policies for file protocol operations (#8262) | ||
|
Hauser
|
5e1bd498ce |
fix(scanner): isolate and coalesce dirty usage journals (#8325)
* fix(scanner): isolate and coalesce dirty usage journals Bind dirty replay records to a stable node owner so peers cannot overwrite or delete another node's pending journal. Replace the unbounded command queue with bounded coalesced state and revision-aware background saves. Keep legacy records isolated and enforce replay byte limits with conservative whole-bucket or unverified fallback. Add publication-stage diagnostics and storage regressions for ownership, clearing, retry, and restart behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): collapse confirmed journal clear condition Use a let-chain to retain the generation fence before removing a confirmed bucket and updating its scope byte accounting. Satisfy collapsible_if without changing journal clearing behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
a760d567b6 |
test(storage): make commit fault checkpoints independent (#8326)
* test(storage): make commit fault checkpoints independent * test(storage): simplify commit checkpoint lookup |
||
|
Chris
|
d087416ff9 | refactor(storage): separate metadata scheduling from quorum decisions (#8327) | ||
|
Chris
|
309128f034 | refactor(storage): bind rename rollback to its commit context (#8324) | ||
|
Nils Melchert
|
7a0f8561b6 |
fix(oidc): ignore groups without a matching policy at login (#8164)
* fix(oidc): ignore groups without a matching policy at login OIDC login failed with "OIDC policy mapping did not resolve to current policies" whenever any mapped group lacked a policy of the same name. Directory-backed providers such as Active Directory always emit groups like `DOMAIN\Domain Users` that can never be mapped, so group-based authorization was impossible there. Keep only policy names that resolve to an existing policy and are valid in session claims, and reject the login only when none remain. The signed `policy` claim still lists only resolved names, so request-time evaluation and site replication receivers keep their invariant. Refs #8163 * fix(oidc): only ignore unmapped groups-claim values Limit the relaxed resolution to policy names derived solely from the groups claim. Names from ROLE_POLICY, a dedicated CLAIM_NAME claim, or an explicit IAM policy mapping must still all resolve. Reject the login when a mapped name refers to an existing policy whose name is not allowed in session claims: dropping it could remove an explicit Deny and broaden access. Claim-unsafe names are only checked against the in-memory policy cache and never passed to storage-backed policy loading. Refs #8163 * docs(oidc): document ignorable roles claim values and cover wiring Roles claim values are merged into the canonical groups, so values without a matching policy are ignored like groups-claim values. This covers built-in provider roles such as Keycloak's `offline_access`. Document that in the field and method docs and in the provider requirements, and pin it with a test. Assert that the OIDC authorization carries the group claim policies so a regression in the wiring cannot silently disable the relaxation. Refs #8163 * fix(oidc): ignore group policies only after confirmed absence merge_policies drops names whose load fails, so a storage or decode error for an uncached group policy was indistinguishable from a missing one. The group name was then ignored and the session signed without it, which could remove an existing Deny that request-time checks cannot restore. Add IamSys::policy_exists, which consults the cache and then storage and reports false only for NoSuchPolicy while returning every other error. The OIDC binding now ignores a group-derived name only after a confirmed absence and rejects the login when a lookup fails. Refs #8163 --------- Co-authored-by: cxymds <cxymds@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
Chris
|
e77c76272a | docs(storage): verify object generation authority contract (#8322) | ||
|
Chris
|
c0e63ee543 | docs: trim redundant project skills (#8323) | ||
|
GatewayJ
|
753e8aa657 |
perf(ecstore): reduce multipart and object I/O overhead (#8095)
* perf(ecstore): share multipart cleanup paths across disks * perf(ecstore): avoid encode copies and cached descriptor duplication * test(ecstore): cover default ingest selection and overrides * test(ecstore): cover cached positioned reads in both modes --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
Chris
|
641c4b3493 |
chore(release): merge 1.0.1 back into main and refresh installation references (#8314)
* fix(ci): include pagination regression in full E2E selection * fix(deps): replace yanked yoke-derive release * fix(scanner): expose pause backlog replica diagnostics (#8258) * fix(scanner): expose pause backlog replica diagnostics Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(connect): stabilize runtime profile lease cancellation Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(connect): tolerate delayed schedule startup in CI Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retry quota reads during usage warmup Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): avoid meta-bucket incarnation self-deadlock Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(test): use persisted incarnation in heal fixture Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): reconcile stale counters after lifecycle expiration (#8108) * fix(usage): reconcile stale counters after lifecycle expiration * fix(usage): account lifecycle expiry during continuous writes * test(usage): run lifecycle usage scenarios on one scanner store * test(usage): use a Windows-representable pre-mutation offset * fix(usage): harden expiry accounting recovery and quota checks Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(usage): recover stale persisted counts through the scanner Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test. Refs rustfs/backlog#2689 Refs rustfs/backlog#2691 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): bound delimiter pagination fixture concurrency The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics. Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(upgrade): establish a persisted previous-release baseline Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions. Refs rustfs/backlog#2689 Refs rustfs/backlog#2384 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): bound cancellation metadata persistence waits Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions. Refs rustfs/backlog#2697 Refs rustfs/backlog#2689 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Chris <anzhengchao@gmail.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain startup and shutdown failure diagnostics * fix(test): supply CPU workload for sampler regression * fix(ci): locate security chain scripts in the workspace * fix(usage): recover historical counters with generation fencing (#8273) * fix(usage): recover historical counters during continued writes Use newer converged scanner snapshots to reconcile stale absolute usage baselines while preserving concurrent mutation and expiry receipt fences. Cover durable publication, admin and quota reads, and legacy generations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(usage): fence snapshots and move preserved cache entries Apply the cached scanner generation floor before every reconciliation path and retain it even when an older snapshot happens to match core counts. Move preserved usage entries instead of cloning their histogram maps under the cache lock, retaining expiry receipt identity and cancellation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * test(scanner): verify checkpoint takeover and repair dispatch (#8275) * test(scanner): cover checkpoint handoff and repair dispatch Drive runtime budget expiry, partial-cycle persistence, leadership claims, and stale checkpoint rejection between real disk-backed fixture scans. Verify that a metadata repair beyond the first bounded prefix is saved in the scanner ledger and dispatched by the MRF consumer. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * ci: isolate scanner fixtures and refresh full e2e membership Reserve nextest capacity for the real-disk scanner publication and MRF admission fixtures. Bind both platform membership checks to the reviewed pagination deadline test added on main. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(scanner): consolidate checkpoint fixture lifecycle Keep one durable control store across timeout and leadership transitions, and inject generation advancement into the shared checkpoint scenario. Check the actual saved metadata path so late-write rejection also proves that existing checkpoint bytes remain intact. Centralize MRF fixture isolation and reuse nextest process isolation when the startup environment already satisfies the test contract. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(obs): distinguish allocator counters from live memory (#8274) * fix(obs): distinguish allocator counters from live memory Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(obs): parse allocator statistics from one node Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): isolate late parity recovery from metadata hedges Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix(usage): combine identical snapshot retention branches * fix(test): await HTTP sender readiness in Top RPC fixture * [release/1.0.1] Gate multipart copy through write admission (#8284) Gate multipart copy through write admission Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * Gate multipart copy through write admission (#8283) Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission. Co-authored-by: zhi22915 <qiuzgang@gmail.com> * fix: add UploadPart OOM validation guardrails (#8287) * docs(release): validate candidates on release branch * fix(scanner): validate checkpoints against global cycle fence (#8278) ## Related Issues Related to rustfs/backlog#2701. ## Summary of Changes Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission. ## Verification Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance. ## Impact Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired. ## Additional Notes Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior. * fix(ci): restore E2E membership and pagination timeouts (#8281) * ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279) ## Related Issues Follow-up to #8229. ## Summary of Changes Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence. ## Verification The exact PR head |
||
|
Henry Guo
|
5efa6254eb |
feat(table-catalog): expose durable strong diagnostics (#8232)
* feat(table-catalog): expose durable strong diagnostics * fix(table-catalog): stabilize snapshot diagnostics and metadata scans Reuse one recovery observation for exports and diagnostics in both backing modes. Follow the persisted metadata directory for renamed and registered tables, and bound diagnostics under continuous snapshot changes. Remove redundant recovery work and fix the production dead-code and clone lint failures without relaxing checks. Add deterministic race, retry, and metadata reachability coverage and clarify read-only candidate semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(e2e): bound pagination fixture upload concurrency Keep the 1200-object delimiter boundary fixture while preparing it with at most 16 concurrent PUTs. Serial preparation exhausted the existing 120-second smoke deadline before the listing request was issued. Report fixture preparation time and verify the exact common prefixes, empty contents, KeyCount, and absent continuation token without relaxing the timeout, durability settings, or dataset size. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: GatewayJ <835269233@qq.com> |
||
|
唐小鸭
|
5c99417073 |
fix(sse): guard copy-source SSE-C keys over TLS and reject bucket-keyed KMS context (#8296)
* fix(sse): count copy-source SSE-C headers in the TLS transport guard
The SSE-C transport guard only looked at the object's own customer-key
headers. A CopyObject or UploadPartCopy that read an SSE-C source into a
non-SSE-C destination carried the source key only in the
x-amz-copy-source-server-side-encryption-customer-* headers, so it was
accepted on a plaintext transport with RUSTFS_SSE_C_REQUIRE_TLS=true and
was missing from rustfs_ssec_plaintext_requests_total.
Treat the copy-source triple as SSE-C headers too.
* fix(sse): reject a KMS context key that replaces the location entry
Managed SSE wraps each data key under an encryption context that carries
{bucket: bucket/key}, added with or_insert, while only the client context
is persisted and the entry is rebuilt on read. A client context entry
keyed by the bucket name therefore replaced the location entry on write
and on every read, so the data key was no longer tied to the object's
location.
Reject such a context with 400 InvalidArgument at the single managed-SSE
write entry, before the KMS is called. The read side is unchanged, so
objects stored with such an entry stay readable, and other keys,
including other bucket names, are still accepted.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
|