Compare commits

..

147 Commits

Author SHA1 Message Date
Chris 802ac8a1bd test(heal): avoid needless collect in legacy root recovery test (1.0.2 backport) (#8551)
test(heal): avoid needless collect in legacy root recovery test
2026-10-14 08:22:12 -07:00
Chris 9408b90f13 chore(release): prepare 1.0.2 (#8549)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-14 22:12:55 +08:00
Hauser 08ac1ec218 fix(s3): advise retry after typed lock timeouts (#8546)
Add a one-second Retry-After hint at the canonical S3 error conversion
only for ServiceUnavailable responses retaining a typed lock timeout.
Keep existing error codes, source chains, and lock budgets unchanged.

Cover actual HTTP/XML serialization, nested causes, private-resource
redaction, and unrelated disk/quorum/text failures. Document the response
contract and the limits of log-count and five-second timeout attribution.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 20:52:14 +08:00
cxymds 388d1f17e6 fix(heal): safely recover missing bucket directories by default (#8517)
* fix(heal): safely recover missing bucket directories by default

* fix(heal): allow bound admin recovery before namespace checks
2026-10-14 03:57:09 -07:00
Henry Guo 22c093e66b feat(table-catalog): add opt-in periodic maintenance queueing (#8537) 2026-10-14 03:55:36 -07:00
Chris 24ab34a5ee fix(ci): restore main lint and test reliability (#8543)
* fix(ci): remove needless returns from Linux disk helpers

* fix(test): verify encrypted purge replicas by version body

* fix(test): expire namespace timeout after fixture preparation

* fix(ci): remove remaining needless collection and borrows

* fix(test): wait for active admin heal at interruption checkpoint
2026-10-14 18:28:10 +08:00
Hauser 9068d83400 fix(heal): unblock automatic replacement recovery (#8533)
* fix(heal): unblock automatic replacement recovery

Repair bucket publication directories before adopting survivor delete proofs.
Keep explicit empty-volume removal after proof settlement and preserve the
admitted bucket incarnation for administrator replay.

Discover unfinished intents across every local set and resume the durable
owner with its exact target scope before admitting new sibling replacements.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): route scanner fixtures through the storage boundary

Expose the existing disk-layout bootstrap symbols as test-only aliases in
heal::storage_api and use that boundary from scanner recovery regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(heal): index replacement discovery within each scan

Keep a single endpoint-keyed disk inventory with first-snapshot precedence
and deterministic discovery order. Index local target lookups and move
resume state into generation candidates instead of cloning it.

Retain fresh durable reads and the existing conflict and mutation fences.
Cover duplicate disk handles, endpoint uniqueness and discovery ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): retain committed journals in recovery fixtures

Build one certified survivor delete decision for both replacement and
empty-volume removal tests. Clear target contents without moving its root
inode so Linux disk handles observe the same empty target. Drain PUT tails
and assert the proof remains before the real Heal entry.

Expose the existing recovery state through the ECStore disk API for typed
fixture checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): use server stack for replacement regression

The replacement regression polls the composed Heal future on libtest's
smaller calling-thread stack and aborts before reporting an assertion.
Run it on a bounded 8 MiB test thread and Tokio workers, matching the
debug server's stack budget. Keep all storage and object assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 17:04:36 +08:00
Hauser 20fc897036 chore(deps): refresh crates and use Pyroscope 2.1.2 (#8540)
* chore(deps): refresh crates and use Pyroscope 2.1.2

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh lockfile

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 16:13:57 +08:00
Hauser a86117590c fix(ci): prepare pinned Compose for Connect acceptance (#8531)
* fix(ci): prepare pinned Compose for Connect acceptance

Install the verified Compose CLI through one local action shared by all 11 connected acceptance entry points. Publish only checksum-verified binaries and keep daemon, provenance and timeout gates unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): use daemon-visible Connect temporary inputs

Match the existing official Connect acceptance TMPDIR contract in all 11 real E2E invocations so daemon bind mounts can read generated inputs. Preserve key modes, read-only mounts and cleanup checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 10:42:35 +08:00
Hauser ceb53fef85 fix(ci): select Python 3.14 for Connect acceptance (#8530)
The shared connected preflight uses uuid.uuid7, which is unavailable in older runner Python environments. Select Python 3.14 with the existing pinned setup action in all eleven acceptance callers.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 09:02:04 +08:00
Nikita Bakun 4e0ff037ec fix(replication): skip version-addressed marker pre-check on targets that mint their own version ids (#8492)
* fix(replication): skip version-addressed marker pre-check on targets that mint their own version ids

* fix(replication): keep bodiless 403 fail-closed and verify resynced markers on targets that mint their own ids

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-13 14:00:54 -07:00
Henry Guo fa0ac13e9b feat(table-catalog): support durable-strong table maintenance (#8523)
* feat(table-catalog): enable durable maintenance control

* fix(table-catalog): revalidate unlocked worker planning

* fix(table-catalog): preserve maintenance state across rename

* fix(table-catalog): fence maintenance reads across rename

* fix(table-catalog): bound scheduler history read fencing

* test(table-catalog): inject loss using the held client lease

* refactor(table-catalog): remove unused metadata log wrapper
2026-10-14 03:54:16 +08:00
Hauser c7848df9a0 fix(build): enforce GNU ABI floor and canonical target metadata (#8524)
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata

gnu release binaries were built natively on Ubuntu 24.04, embedding
GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised
glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062).
The two linux-gnu legs now build through cargo-zigbuild pinned to the
2.31 baseline (same-arch; zig acts as the linker), with post-build
guards asserting both the embedded version metadata and the highest
referenced GLIBC symbol version, keeping the package declarations
honest.

Version metadata itself now comes from cargo's TARGET cfg instead of
shadow-rs' host stamping, so builds whose host differs from their
target (cross compilation, musl targets on a glibc-hosted toolchain)
report the platform they were compiled for (#8048 follow-up).

The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in
the setup action now that it is load-bearing for releases, the build
job gets an explicit matrix-independent name so the connect-*
acceptance workflows keep matching it when matrix fields change, and
the preview-release workflow contract is updated for the TARGET_ARG
build commands.

Fixes #8062
Follow-up on #8048

* fix(build): enforce GNU ABI floor and canonical target metadata

Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and
build GNU releases against the declared glibc 2.31 floor. Validate both
release ELF binaries using their loader version requirements.

Generate target metadata once through shadow-rs and check the actual
release version output without adding a parallel metadata facade.

Refs: rustfs/backlog#2824

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(build): reuse GNU release profile across binaries

Switching the CLI back to the default strip profile restarted 419
compilation units after the server build. Keep the profiling build
profile for both binaries, then strip only the CLI artifact.

Retain serial links, server profiling symbols, and post-strip ABI gates.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(build): limit release artifacts to the rustfs server

Build only rustfs in the release workflow and remove the CLI build,
strip, and packaging paths. Preserve GNU profiling symbols and catalog
sidecars, and align strict acceptance archive readers with the new set.

Exercise all six build targets, Linux package checks, and all four
strict acceptance extractors using their actual workflow scripts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 12:00:40 -07:00
Hauser 4502a1f9b0 fix(ci): repair profiling and compact transition test gates (#8526)
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344

Vendor the published Pyroscope package with an object 0.40 dependency and
lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add
bounded decoder regressions, and run the patched package in audit CI.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>

* fix(deps): correct vendored pyroscope documentation typos

Correct copied comments and changelog spelling for the repository typo
check, and describe these comment-only changes in the vendor provenance.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>

* fix(ecstore): refresh compact transition proof per test case

Install the short-lived homogeneous fleet proof immediately before each
kill-point transaction. Earlier cancellation and recovery work must not
age the next case's writer capability fixture.
Report an early transition result instead of waiting for an unreachable
kill point. Preserve cancellation, remote-owner and payload assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 11:59:59 -07:00
Hauser 93d51d172f fix(server): cancel request watchdogs with response futures (#8528)
Tie watchdog cancellation to response future ownership before the first
poll, and prefer cancellation when the watchdog deadline is also ready.
Preserve completion logging and genuine slow-request warnings.

Add paused-clock regression coverage for dropped, completed, panicking,
and still-pending request futures.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 11:19:22 -07:00
Hauser b80590f1a7 perf(heal): index replacement history reconciliation (#8527)
refactor(heal): index replacement generation reconciliation

Reconcile owned records and their definitive sources in one pass. Group claims by target slot and use a shared-anchor fast path while retaining pairwise ambiguity checks.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 11:06:21 -07:00
Hauser 025ed4c6d1 fix(heal): preserve retained replacement histories (#8525)
Keep definitive local histories while rejecting ambiguous overlapping peer records.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 00:36:33 +08:00
hector 3b428bffef fix(heal): merge disjoint replacement generations in cluster status (#8521) 2026-10-13 08:18:58 -07:00
Hauser da52303f2a fix(server): restore socket peer fallback for client rate limits (#8519)
Read the optional socket peer extension supplied by external ingress so
API and Console client limits remain effective without trusted proxies.
Keep validated client precedence and the existing missing-peer policy.

Exercise the full external stack across HTTP request versions, direct
IPv4/IPv6 peers, forged headers, and distinct trusted proxy clients.
Align API and Console fixtures with the production extension contract.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 20:32:27 +08:00
Chris a79c6072f1 fix: suppress clippy too_many_arguments on rename_data_commit_from_observations (#8515)
fix(ecstore): suppress clippy too_many_arguments on rename_data_commit_from_observations

The function takes 9 slice parameters for the renamedata commit phase.
Annotate with #[allow(clippy::too_many_arguments)] to match the existing
codebase pattern for internal multi-slice commit helpers.
2026-10-13 20:31:23 +08:00
Hauser ab512768d5 fix(ecstore): group rename data commit observations (#8516)
Group disk observations into a borrowed struct to keep the commit helper
within Clippy's argument limit without changing quorum snapshot logic.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 20:05:36 +08:00
cxymds adf17bbfba fix(heal): reject incomplete administrator bucket discovery (#8512) 2026-10-13 04:12:28 -07:00
GatewayJ fb5c002a30 refactor(iam): encapsulate federated authorization models (#8514) 2026-10-13 03:59:29 -07:00
Hauser 8a94c64934 fix(audit): preserve request identity and verified client address (#8513)
Capture the authenticated request access key when building the audit entry
and preserve it through completion instead of replacing it with root
credentials. Resolve remotehost from validated client context or the
socket peer, and ignore untrusted forwarding headers.

Cover request identities, completion outcomes, client address precedence,
and audit serialization without credential secrets.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 18:56:44 +08:00
Hauser 9ca0e9b717 fix(obs): isolate bucket usage dashboards by cluster (#8510)
Keep same-name buckets in independent clusters separate while deduplicating
replicated usage observations. Document the existing scanner-backed exporter
and cover its gauge contract and cluster-scoped queries.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 03:00:44 -07:00
Henry Guo bd83bd3f6b fix(table-catalog): allow commits after object-backed rename (#8508) 2026-10-13 17:12:00 +08:00
Hauser 34803e8165 perf(ecstore): isolate cleanup and preserve metadata authority (#8289)
* chore: pin s3s to upstream git revision

Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4.

Allow the official s3s git source in cargo-deny so source checks continue to pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): instrument put object hot path

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): split cleanup worker wait metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* upgrade version

* chore(metrics): add PUT post-store stage probes

Add lightweight PUT post-store stage probes for issue 2615 validation.

* fix cargo deny check warn

* upgrade version

* perf(ecstore): bound and isolate cleanup execution (#8151)

* perf(ecstore): bound and isolate cleanup execution

* fix(ecstore): preserve cleanup cancellation and lease release

* feat(ecstore): default isolated cleanup to CPUs 1-2

* test(ecstore): decouple budget tests from CPU affinity

* upgrade version

* perf(ecstore): reuse pinned directory handles for object I/O

* feat(observability): add sampled local GetObject stage summaries

Add an opt-in GetObject stage aggregation path that works without the
OTLP metrics export. When RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED is
set, io-metrics keeps per-(path, stage, object class, size bucket)
sampled counts and duration totals in a bounded in-process map, and a
startup task flushes them periodically to
/var/log/rustfs/get-stage-summary.log, tunable via the sample rate and
interval env keys.

* build(deps): bump workspace dependency releases

Patch releases: async-compression 0.4.49, tokio-rustls 0.26.6,
xxhash-rust 0.8.19, suppaftp 12.1.1, russh-sftp 3.0.1. Minor release:
hotpath 0.27.0.

* fix(ecstore): fence distributed metadata cache mutations

Add authenticated cluster-wide mutation fences for the opt-in DistErasure GetObject metadata cache. Guard recursive prefix and batch deletes, fail closed on configuration mismatch or uncertain quorum, and cover replay, restart, peer loss, and mixed-configuration behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(metrics): consolidate PUT observers on improve/base (#8039)

* perf(object): instrument storage hot paths

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(metrics): consolidate PUT stage observation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>

* build(deps): bump workspace dependency releases

* ci: allowlist rustix OFlags WRONLY in typos config

* refactor(storage): route ecstore symbols through storage_api boundaries

* perf(ecstore): cache quorum-selected early-stop metadata (#8308)

Publish eligible latest-version GET metadata after a validated read quorum
when early-stop cancels the remaining fanout. Keep metadata errors, selection
quorum, online read quorum, and mutation-generation checks as publication gates.
Add slow-peer repeat-hit and concurrent mutation-fence regression tests.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore: merge latest main into improve/base (#8332)

* refactor(storage): bind rename rollback to its commit context (#8324)

* refactor(storage): separate metadata scheduling from quorum decisions (#8327)

* test(storage): verify rejected rollback inspection on real disks (#8331)

* test(storage): make commit fault checkpoints independent (#8326)

* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup

* fix(listing): count logical prefixes before pagination (#8328)

* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): borrow metadata cache lookup key (#8333)

Avoid request-scoped Arc<str> allocations on eligible cache lookups while preserving full key identity.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore: sync improve/base with latest main (#8334)

* refactor(storage): bind rename rollback to its commit context (#8324)

* refactor(storage): separate metadata scheduling from quorum decisions (#8327)

* test(storage): verify rejected rollback inspection on real disks (#8331)

* test(storage): make commit fault checkpoints independent (#8326)

* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup

* fix(listing): count logical prefixes before pagination (#8328)

* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed

* fix(scanner): isolate and coalesce dirty usage journals (#8325)

* fix(scanner): isolate and coalesce dirty usage journals

Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): collapse confirmed journal clear condition

Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(obs): distinguish metrics by process identity (#8343)

Attach process.pid to the OpenTelemetry resource and promote it for Prometheus
so counter series from restarted processes remain distinct. Preserve the stable
RustFS node and service instance attributes.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* merge: sync origin/main into improve/base (#8345)

* refactor(storage): bind rename rollback to its commit context (#8324)

* refactor(storage): separate metadata scheduling from quorum decisions (#8327)

* test(storage): verify rejected rollback inspection on real disks (#8331)

* test(storage): make commit fault checkpoints independent (#8326)

* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup

* fix(listing): count logical prefixes before pagination (#8328)

* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed

* fix(scanner): isolate and coalesce dirty usage journals (#8325)

* fix(scanner): isolate and coalesce dirty usage journals

Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): collapse confirmed journal clear condition

Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(storage): retain typed owners across borrowed writes (#8329)

* fix(storage): retain typed owners across borrowed writes

* fix(storage): preserve Copy write completion policy

* refactor(storage): remove redundant write owner references

* fix(ecstore): hide rollback backup residue in prefix listings

* test(connect): isolate short HTTP fixtures in nextest

* fix(protocols): enforce S3 policies for file protocol operations (#8262)

* test(ecstore): enforce main core regression gates (#8338)

* test(ecstore): enforce main core regression gates

* test(ecstore): pin external shards and completion regressions

* fix(test): restore authenticated cluster fault scenarios (#8340)

* fix(test): restore authenticated cluster fault scenarios

* fix(test): use the E2E storage API boundary

* ci: coerce the pool timeout input to a number

timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.

* ci: coerce the pool timeout input to a number

timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.

* fix(ecstore): own listing workers through cancellation (#8342)

Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route commit sync through guarded helper

Use the Linux-capable guarded directory sync entry point for both commit sync sites. This keeps group commit and file-sync admission behavior while ensuring the already-open commit directory remains the sync target.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): expose rename directory guard in production

The commit path now acquires the directory guard in production builds, not only tests, so destination metadata reads and durable sync share the guarded context.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>

* fix(ecstore): remove unsafe metadata cache admission paths

Keep distributed GET metadata on quorum reads and quota admission on authoritative storage. Preserve local cache invalidation, remove the experimental RPC surface, and gate diagnostic work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): cover cold degraded mutation admission

Check first PUT and DELETE with one peer absent and no SDK retries, then verify that losing write quorum still rejects writes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): retain guarded Linux destination metadata reads

Use the existing commit directory guard when reading old xl.meta on Linux. Cover symlinked destination metadata for inline and non-inline renames without changing the split commit structure.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(test): remove retired cache RPC import

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* merge: sync latest main into improve/base (#8349)

* test: allow cleanup fixture CI headroom (#8336)

* fix(s3): encode multipart listing response keys (#8347)

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* merge: sync main auth and scanner updates (#8352)

* test: allow cleanup fixture CI headroom (#8336)

* fix(s3): encode multipart listing response keys (#8347)

* perf(scanner): reduce scan and checkpoint peak allocations (#8346)

* perf(scanner): reduce scan and checkpoint peak allocations

* test(scanner): isolate partial commit recovery scenarios

* fix(auth): return ExpiredToken for retained STS credentials (#8344)

* fix(auth): return ExpiredToken for retained STS credentials

Validate session identity independently of credential expiry in both IAM
rejection and claims validation, and preserve typed S3 errors at the caller.
Keep token binding, disabled-account precedence, and service-account behavior.
Add exact expiry, IAM, and HTTP/XML regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): synchronize the metadata slowtail hedge regression

Park the slow metadata task at a test-only fault checkpoint and release
healthy reads after the fault is active. Assert the spare response and
cancelled-task drain without imposing a 300 ms disk/scheduler deadline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ilm): synchronize copy compensation transition stages

Observe automatic compensation reaching the mock tier PUT checkpoint before
waiting for the final metadata commit. Assert forced enqueue failure,
automatic backfill, one completed tier upload, and payload integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reject overlapping cleanup schedulers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reject overlapping cleanup schedulers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(ecstore): explain cleanup scheduler conflict

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(metrics): separate GET request and timeout outcomes (#8354)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* docs: refresh security advisory lessons (#8350)

* fix(heal): bound durable MRF rescheduling (#8351)

Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(rustfs): route recovery errors through S3 gateway

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update workspace dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(io-metrics): use is_multiple_of for sampling

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): satisfy strict Clippy pagination checks

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid cloning Copy purge identity

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* add ignore .mimosa

* fix(zip): adapt archive readers to zip 9 APIs

Upgrade zip and its required dependencies alongside fallible archive-name handling in log ingestion, report bundles, bucket metadata import, and archive tests. Preserve existing skip and error propagation behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reuse quorum metadata for body GETs

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): align storage test stacks and peer reset classification

Apply the existing bounded ECStore stack setup to the affected cross-crate storage fixtures. Validate conditional-only metadata publication separately from full body-layout cache reuse. Classify typed h2 shutdown resets as unreachable while retaining application, authentication, cancellation, and deadline distinctions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update workspace dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve omitted-disk GET recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore: integrate published workspace dependency updates

Preserve the concurrent dependency refresh and retain the typed h2 reset classification and bounded storage-test stack setup.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): preserve snapshot size budget after fallback move

* fix(test): use one barrier for compensation COPY

Let the forced-enqueue helper own the PUT barrier and compensation assertions. Waiting on an outer barrier overwritten by the helper always timed out after the upload had completed.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ecstore): apply current lint fixes

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update sha2 to 0.11.1

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-12 22:07:50 -07:00
cxymds 9af2f7c04e fix(ilm): make missing lifecycle expiry targets idempotent (#8504)
fix(ilm): treat missing expiry targets as success
2026-10-12 20:05:21 -07:00
cxymds 9d6f48c325 fix(s3): match AWS duplicate bucket responses (#8503) 2026-10-12 19:25:12 -07:00
Hauser e7e6556f83 fix(scanner): bound superseded waits for regular maintenance (#8502)
Preserve ordinary lifecycle and replication cadence while retaining
usage-only convergence backoff, explicit cycle floors and publication
fencing. Cover repeated snapshots and the real convergence waiting path.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 07:04:27 +08:00
Hauser 0a6aeda191 test(e2e): bound scanner witnesses by the reported schedule (#8501)
* test(e2e): bound scanner witnesses by the reported schedule

Require a fresh completed observed walk and account once for planned
superseded waiting, preserving the active-cycle execution budget.
Cover invalid schedules and early wakeup, and refresh exact platform
selection digests without removing existing tests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): refresh nightly scanner budget membership

Record the three scanner budget regressions in both platform nightly
selection digests, preserving every previously selected test.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 13:06:42 -07:00
Hauser 0e0f656ae1 test: isolate license and rebalance cancellation fixtures (#8500)
Preserve canonical claim encoding for the signature tamper test and pause the admin cancellation fixture at its existing run fence instead of durable deletion.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 10:59:46 -07:00
Hauser a57a3ae51d fix(scanner): reuse observations and expose complete usage (#8498)
* fix(scanner): reuse observations and expose complete usage

Allow superseded retries to reach the existing validated cache planner while
bootstrap remains pending. Serve complete observations in server info without
changing quota authority or dirty acknowledgements.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): route fixture setup through storage boundary

Keep the single-disk scanner fixture on the existing storage_api boundary
instead of importing the ECStore API facade directly.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 09:50:59 -07:00
GatewayJ 5d8e143505 fix(admin): add OIDC policy log fields (#8496) 2026-10-12 08:07:46 -07:00
Hauser d6fe5c30ce test(e2e): borrow URLs in internode RPC helpers (#8497)
Pass existing string slices directly to the client constructor to satisfy strict Clippy without temporary allocations.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 23:02:02 +08:00
Chris ac6cf8a50d fix(test): discover data dirs by listing instead of info.data_dir (#8495)
The conditional_get_304_uses_metadata_without_decoding_damaged_data test
constructed the data directory path from info.data_dir.to_string(), but
the on-disk directory name may not match the UUID string form. Switch to
iterating subdirectories of the object path (matching the
delete_object_part_shards helper) so the test is robust regardless of
the data_dir naming convention.

Verified: cargo nextest run -p rustfs -- conditional_get_304 → PASS
2026-10-12 23:01:37 +08:00
Hauser a7188a16b4 test(storage): make recovery and upgrade checks deterministic (#8491)
* test(upgrade): separate rc.5 data and candidate query checks

Verify historical fixture bytes before upgrading without requiring the
pinned rc.5 reader to satisfy the candidate's cross-part range contract.
Reuse the complete-body oracle and retain all upgraded range, partNumber,
attributes and replication assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): replace timing races with recovery evidence

Prove no_lock batch progress with the existing cleanup barrier while the
caller retains its guard. Check namespace counters and every drive's
metadata absence rather than using a one-second disk I/O deadline.

Keep durable ILM's manual recovery as the sole tier journal consumer and
verify exact migrated input before retaining the original recovery stats,
receipts and remote-version assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 21:17:58 +08:00
Chris 0195732a1f test(ecstore): cover delete cleanup crashes and retries (#8471)
fix(ecstore): preserve retry evidence during delete cleanup

Bind normal cleanup to the coordinator transaction after quorum succeeds, and remove the rollback backup before marked data. Preserve data and markers if backup cleanup fails.

Add deterministic LocalDisk crash and retry regressions plus real four-disk single and batch coordinator coverage. This prevents one post-quorum cleanup window; it does not resolve ambiguous inline or historical residue in #8464.

Co-authored-by: Zhengchao An <overtrue.us@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 21:02:03 +08:00
cxymds 0ef133bbb0 fix: refresh peer metadata without blocking bucket mutations (#8486)
* fix: refresh peer bucket metadata after creation

* fix(s3): release bucket mutation slots before peer reloads

* test(ecstore): distinguish delete lock waits from disk latency
2026-10-12 05:03:16 -07:00
GatewayJ 5228463a1c fix(oidc): recover providers after startup discovery failures (#8475)
* fix(oidc): recover providers after startup discovery failures

* fix(oidc): report recovery failures with throttled diagnostics
2026-10-12 03:40:07 -07:00
Hauser dd90bb7488 fix(s3): short-circuit conditional reads at metadata quorum (#8459)
* fix(s3): evaluate conditional reads before opening object bodies

Reuse locked metadata snapshots for conditional GETs and unify HEAD
preconditions and cache headers. Preserve body-cache hooks and validate
encryption metadata and key authorization before returning bodyless errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): unify conditional reads at metadata quorum

Apply scoped metadata conditions to the existing reader pipeline instead
of maintaining parallel prepared-read branches. Preserve quorum checks,
early-stop, coalescing, and cache-hook behavior on condition misses.

Track metadata-only termination through selection and reject body access
or a changed decision. Keep concrete reader futures off request stacks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): strengthen conditional read acceptance evidence

Assert zero shard, decode, cache-fill, and remote-body calls on matching
conditions with live positive controls. Exercise GET and HEAD through
s3s and Hyper with conditional-header and precedence matrices.

Add reproducible paired application-read benchmarks with isolated
metadata controls and strict drift, artifact, and resource gates.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): remove obsolete conditional comparison helper

Keep ETag comparison tests on the canonical read-condition evaluator,
remove unused date-trait imports, and name the HTTP case type.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): validate full conditional etag fields

Reject adjacent tags and malformed trailing list members before accepting a
match. Keep the existing request-scoped metadata read and authorization order,
while removing redundant validation wrappers and unused benchmark overrides.
Replace simulated phase coverage with real 304/412 cache-fill probes and extend
HTTP/1.1 and HTTP/2 ingress regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): cache terminal conditional metadata safely

Preserve quorum-validated conditional metadata when early termination
cancels the remaining fanout. Restrict those entries to terminal GET
readers; body and full-layout consumers still obtain their own metadata.

Serialize cache publication per key so conditional entries cannot replace
complete layouts or renew their TTL. Retain generation fencing and the
final selected-metadata predicate check before any body access.

Cover real four-disk hedging, nonterminal body and range reads, consumer
isolation, generation races, atomic publication and actual cache expiry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(log-analyzer): align heal writer failure rule with current logs

Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 16:59:21 +08:00
GatewayJ 0018bbf5d7 ci: guard OIDC architecture boundaries (#8447)
* ci: guard OIDC architecture boundaries

* fix(ci): guard OIDC session ownership

* fix(ci): ignore OIDC authorization destructuring

* fix(ci): accept typed authorization destructuring

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-12 16:13:36 +08:00
Hauser 13ee76a4a9 fix(ecstore): validate recovery evidence before mutation (#8487)
Reject causal cycles before source repair and share conservative backup
classification between discovery and inspection. Fence public raw deletion
and derive exact-key removals from successful settlement without an extra
metadata read. Preserve the existing physical phases and strict source checks.

Make outage and renewal fixtures deterministic without weakening their
convergence assertions, and complete strict TLS/RPC lint cleanup.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 01:06:01 -07:00
cppudge 3dc09c9397 test(ecstore): deflake bucket sentinel and dispatch rollback tests (#8488)
* test(ecstore): bound sentinel tests by the lock wait, not recreation IO

Both bucket incarnation sentinel tests gave the bucket recreation that
follows the paused delete (force delete_bucket, make_bucket, put_object
on four fsync'ing disks) 10 s in total. The delete releases the sentinel
milliseconds after the barrier, but on a runner starved for IO the
recreation alone can exceed 10 s:
delete_objects_holds_bucket_incarnation_sentinel_through_commit failed
that way on main in run 37760321449 (job 113259251979).

That bound is not what catches a kept sentinel: delete_bucket waits for
it on its lifecycle write lock, which fails after the lock acquire
timeout, and the recreation's expect reports the lock error. Keep that
check and give the recreation 120 s only as a hang guard.

The recreation starts that lock wait before the barrier is released, so
the wait also has to cover the delete's own commit, and the 5 s default
leaves a starved runner little room for it. Run both tests with
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30 and bound the delete, which was
not bounded, by the same 30 s. A kept sentinel still fails the tests,
after 30 s instead of 5 s.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* test(ecstore): keep background recovery out of dispatch rollback tests

dispatch_manifest_detached_workers_are_store_bounded_and_shutdown_cancellable
seeds six aborting dispatch manifests, installs a pause hook and expects
its own recovery passes to park four detached workers on it. Its first
store also ran the background tier-delete journal recovery loop, which
runs a pass every TIER_DELETE_JOURNAL_RECOVERY_INTERVAL (60 s). When the
setup took longer than that, a background pass rolled the manifests
back before the hook was installed, no worker paused, and the 30 s wait
failed. This happened on main on a runner starved for IO, in run
37760321449 (job 113259251979).

Build the first store with background recovery suppressed, as the
neighbouring dispatch tests do. The restarted store keeps it: the second
half of the test checks that it picks up the retained manifests.

dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member
and dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence
have the same exposure: each seeds an aborting manifest on a store with
background recovery and only then installs its slow-delete hook. They
have not failed in CI, but with the setup stretched past 60 s they fail
the same way. Suppress background recovery for them too.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

---------

Signed-off-by: Petr Slavyagin <disident47@gmail.com>
2026-10-12 01:02:33 -07:00
Hauser 71c0efa97d fix(scanner): account remote progress and preserve dispatch limits (#8485)
* fix(scanner): account remote progress and preserve dispatch limits

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): isolate remote progress by cycle

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 00:29:10 -07:00
Hauser 4166b9503f fix(ecstore): recover interrupted delete transactions (#8468)
* fix(ecstore): recover crash-interrupted delete retries

Retain set-quorum cleanup proof and restore metadata before dropping rollback
markers. Reclaim eligible exact-key residue during explicit delete retries,
with bucket identity, version, retention and namespace fencing guards.

Preserve prefix children and snapshot ownership through batched physical
cleanup, and cover interrupted transactions and failure boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route recovery test imports through contracts

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve missing-key deletes at quorum

* feat(ecstore): add certified set-wide delete recovery

Atomically prepare deletion journals and exact targets, publish durable set
receipts, and recover through one fenced disk/RPC core used by startup and
scoped administrator healing. Preserve returning generations, ordinary
missing-key quorum semantics, and legacy rollback wire compatibility.

Retain namespace and volume ownership through late physical operations,
order related transactions causally, and keep proof until full convergence.
Remove the superseded local-only recovery path and add storage, protocol,
RPC, durability and isolated Docker regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reclaim completed multipart delete sidecars

* fix(ecstore): recover multipart sidecars with verified proofs

Recognize native multipart sidecars at the shared recovery boundary and
restore them without treating metadata or proof files as shard payloads.
Validate required proof generations and bind every present live/staged
shard copy to its metadata-bound Merkle proof before restoring metadata.

Cover protected native MPU rollback, split source layouts, missing or
corrupt proofs, and payload corruption with recomputed bitrot checksums.
Add real multipart Docker regressions in both integrity modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): settle superseded committed version deletes

Allow a validated committed decision to finish its own data cleanup after
a later decision removes current metadata. Prepared copies need an
inspected same-transaction receipt matching immutable local intent before
promotion, and retain their local journal without importing peer hashes.

Reuse the authority matcher for dry-run and settlement. Cover chained
explicit-version deletions, partial receipt publication, mismatch rejection
and no-resurrection, while retaining the no-decision absence protection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): clarify delete recovery case label

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore recovery CI guard coverage

Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): preserve owned OIDC validation errors

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify recovery authority and snapshot work

Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve delete recovery invariants

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-12 13:47:37 +08:00
cxymds 201b5f62d2 fix(s3): normalize PUT read quorum failures as unavailable (#8484) 2026-10-11 21:55:23 -07:00
Chris 902c97ce05 fix: auto-repair breaking changes from acd26afd8 (#8483) 2026-10-11 20:09:55 -07:00
dependabot[bot] 7ad4fa439e build(deps): bump the dependencies group across 1 directory with 3 updates (#8481)
Bumps the dependencies group with 3 updates in the / directory: [datafusion](https://github.com/apache/datafusion), rust-embed and [hotpath](https://github.com/pawurb/hotpath-rs).


Updates `datafusion` from 55.1.0 to 55.2.0
- [Changelog](https://github.com/apache/datafusion/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apache/datafusion/compare/55.1.0...55.2.0)

Updates `rust-embed` from 8.12.0 to 8.13.0

Updates `hotpath` from 0.28.5 to 0.28.6
- [Release notes](https://github.com/pawurb/hotpath-rs/releases)
- [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.28.5...v0.28.6)

---
updated-dependencies:
- dependency-name: datafusion
  dependency-version: 55.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.28.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rust-embed
  dependency-version: 8.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 19:09:19 -07:00
Hauser 1266d4c270 feat(tls): reload and publish external certificate generations (#8467)
* feat(tls): reload external certificate snapshots coherently

Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(tls): publish server certificates through a fenced admin API

Add opt-in server-leaf publication with dedicated IAM actions, actual TLS
transport checks, immutable generations, durable captured-token CAS and
idempotent receipts. Preserve independent trust and client identity sources
and distinguish visible publication state from the full serving chain.

Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test,
and eight live local API scenarios including least privilege and restart.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): route certificate handler types through S3 facade

Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(admin): include TLS certificate routes in route matrix

Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tls): reconcile startup trust before skipping reloads

Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(tls): reuse pinned generations and prepared listener state

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tls): preserve startup trust source requirements

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-12 08:53:11 +08:00
Anton acd26afd8b fix(scanner): skip child prefixes removed before they are opened (#8479)
A child prefix can be deleted after its parent listed it and before scan_folder opens it. read_dir then returned NotFound, which propagated through the compacted subtree and dropped the whole top-level child (child_scan_failed), so the cycle never produced a durable usage snapshot under continuous deletes.

Treat NotFound on open like a directory that vanished during iteration for non-root folders. A missing bucket root still reports NotFound so namespace removal handling is unchanged.

Refs #8478

Co-authored-by: Anton <5527364+basic-system@users.noreply.github.com>
2026-10-11 15:25:14 -07:00
Rafael Kansy 5924a55931 fix(obs): honor RUSTFS_OBS_USE_STDOUT with per-signal endpoints (#8476)
use_stdout was forced to true whenever the root RUSTFS_OBS_ENDPOINT was
empty, so a deployment that sets only a per-signal endpoint such as
RUSTFS_OBS_TRACE_ENDPOINT could not turn the stdout exporters off: every
exported span was also printed to stdout as a multi-line text block.

Count a non-empty trace, metric or log endpoint as a configured endpoint.
With no endpoint at all the stdout default is still forced on. Add a test
covering each per-signal endpoint and an empty value.
2026-10-11 13:00:24 -07:00
GatewayJ 2d2679a710 refactor(oidc): isolate config updates and replication view (#8446) 2026-10-11 08:53:29 -07:00
Chris 257c9f11ae docs: refresh security advisory lessons (#8472) 2026-10-11 22:43:28 +08:00
Hauser 1324e296f8 test(scanner): bound expansion fixture preparation I/O (#8473)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-11 22:43:08 +08:00
Hauser 38263ba896 fix(log-analyzer): match structured heal writer failures (#8469)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-11 20:46:44 +08:00
Chris ba2047b184 ci: reuse workspace test builds for serial ILM checks (#8465) 2026-10-11 13:55:48 +08:00
GatewayJ da42266bfb refactor(iam): organize OIDC runtime modules (#8445) 2026-10-11 13:27:46 +08:00
Chris 367f13f3f9 ci: reduce repeated PR runner work (#8463) 2026-10-10 22:19:25 -07:00
Hauser 0263d5fcb9 fix(storage): preserve heal failures and timeout responses (#8462)
* fix(storage): preserve heal failures and timeout responses

Stop repairs with no writable target before opening source readers. Consolidate target failure aggregation, preserve typed causes, and distinguish all-ignored diagnostics from actual successful nil entries. Map typed disk timeouts to retryable S3 responses without changing quorum decisions.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(heal): unify staging failure cleanup

Route data-stage failures through one local result boundary after temporary
readers and writers are released. Preserve typed report failures, outer
reconstruction errors, partial repair, and the existing commit fences.

Exercise actual source loss after second-part writer admission using the
existing test pause registry. Assert all staged parts are removed, committed
metadata stays unchanged, and restoring the source allows a full-byte retry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-11 12:41:13 +08:00
Chris 93a64fdca3 fix(dev): restore VS Code debug configurations (#8450) 2026-10-11 12:34:41 +08:00
Hauser 9ecb00d388 fix(scanner): avoid placeholder retention and resume checkpoints (#8460)
* fix(scanner): bound old-cache lookups and recover checkpoint saves

Keep missing child lookups read-only and preserve the checkpoint receiver
after ordinary persistence errors. Reload on stale CAS or indeterminate
revision refresh, and report confirmed periodic saves separately.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): remove redundant checkpoint error conversion

Return the existing StorageError directly so strict Clippy passes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(scanner): clarify checkpoint persistence guarantees

Distinguish a confirmed periodic save from validated restart coverage.
Keep adoption and subtree-skip requirements explicit.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): preserve resumable periodic coverage

Materialize active ancestor accounting in bounded periodic snapshots and
seal completed coverage independently of raw enumeration state. Preserve
durable coverage during early resumed enumeration and exclude unfinished
out-of-order heal work from periodic and cancellation receipts.

Reuse ancestor frame slots to avoid growing child-set copies on descent,
retain compacted accumulator safety, and report checkpoint adoption resets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): reject covered active ancestors in checkpoints

Check every suspended ancestor alongside the current folder before
certifying completed coverage. An unfinished covered ancestor must not
become a resumable subtree merely because its descendant sorts later.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): retain an incomplete root for missing namespaces

Create the empty bucket root explicitly at the missing-namespace boundary.
Preserve nonempty durable caches and keep namespace results unpublished.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-10 20:54:12 -07:00
cxymds b8889fccc9 fix(ecstore): allow cold-start recovery at EC read quorum (#8458)
* fix(ecstore): load existing formats at EC read quorum

* fix(ecstore): complete startup at EC read quorum
2026-10-11 11:50:24 +08:00
GatewayJ a424d10fa0 refactor(iam): construct OIDC runtime directly (#8436)
* refactor(iam): remove legacy OIDC federation paths

* refactor(iam): construct OIDC runtime directly
2026-10-10 04:01:40 -07:00
Chris 072b57b375 fix: auto-repair breaking changes from 446479791 (#8451)
* fix: repair s3s footprint ratchet violation, clippy warnings, and stack overflow

- Replace s3_error! macro with S3Error::with_message in ecfs.rs to keep
  s3s footprint ratchet at baseline (1586) after retention validation refactor
- Remove redundant clone on Copy type in list_objects.rs (clippy::redundant_clone)
- Remove clone_on_copy calls in access.rs test (clippy::clone_on_copy)
- Use manual thread with 8MB stack for heal_concurrent_delete_test to fix
  stack overflow on macOS (consistent with sibling tests in same file)

* test: align CI assertions with version authorization rules
2026-10-09 17:01:48 -07:00
GatewayJ 446479791e refactor(iam): remove legacy OIDC federation paths (#8433)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-09 07:11:50 -07:00
Chris 99af59a914 Merge commit from fork 2026-10-09 21:43:53 +08:00
Chris b77bd26874 Merge commit from fork 2026-10-09 21:43:08 +08:00
Chris 956c1cf027 Merge commit from fork 2026-10-09 21:40:59 +08:00
Chris 8b9d44110e Merge commit from fork 2026-10-09 21:40:08 +08:00
Chris 44ce8d114f Merge commit from fork 2026-10-09 21:39:18 +08:00
Chris 769b0d7578 Merge commit from fork 2026-10-09 21:28:19 +08:00
Chris 03e775947d Merge commit from fork 2026-10-09 21:18:27 +08:00
Hauser 25e73d3bd6 fix(zip): statically link liblzma on macOS (#8434)
* fix(zip): statically link liblzma on macOS

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(zip): pin macOS liblzma minimum to 0.4.8

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): route bucket recovery errors through facade

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): group version pagination markers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid cloning copy purge identity

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-09 16:25:24 +08:00
Hauser a464b45367 fix(rustfs): keep recovery handler within S3 gateway (#8429)
* fix(rustfs): route recovery errors through S3 gateway

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): satisfy strict Clippy pagination checks

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid cloning Copy purge identity

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: restore one-second drive request budgets

* fix(ecstore): resume pending lock bucket creation

Allow retries to reuse an uncommitted Object Lock bucket intent when the same incarnation remains unretired and all physical volumes are absent. Isolate the recovery entrypoint test from the shared production leader lock.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: provision stack for deep catalog test futures

Apply the existing 4 MiB test-thread stack setup to table-catalog handler/store tests and the relocated-pool GET regression in both local and CI nextest profiles. These cases overflowed libtest's 2 MiB default stack on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-09 14:08:12 +08:00
overtrue 5972a09cc3 test: restore one-second drive request budgets 2026-10-08 23:55:57 +08:00
Chris 8a5e730bc9 fix(storage): warn on stalled local disk operations (#8395) 2026-10-08 21:50:31 +08:00
Hauser 1df874b724 fix(obs): pass RustFS build version to telemetry (#8425)
fix(obs): align telemetry version with build identity

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 21:27:24 +08:00
eye-gu 68dcad8387 fix(listing): continue versioned delimiter listings from raw key (#8424)
* fix(listing): continue versioned delimiter listings from raw key

Versioned listings with a delimiter could end a page with
IsTruncated=false while keys remained past the folded scan window.
Continue from the last raw scanned key; a folded prefix cannot
advance forward_past on versioned walks.

Fixes #8423

* fix(listing): filter resumed version prefixes

Apply the ListObjectVersions CommonPrefixes key-marker boundary before
page truncation and resume at the last raw key represented by the final
returned prefix.

Refs rustfs/backlog#2791

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 21:26:57 +08:00
Hauser 8474feaa26 fix: reconcile bucket generations and observe lock failures (#8421)
* fix: reconcile bucket generations and observe lock failures

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: fence bucket recovery through cancellation and quota reuse

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): route tests through storage API boundary

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): use typed lock errors in recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 21:26:12 +08:00
cxymds d386f7741d fix(lifecycle): coordinate tier free-version recovery scans (#8387)
* fix(lifecycle): coordinate tier free-version recovery scans

* fix(lifecycle): remove obsolete recovery enqueue entry point

* test(upgrade): wait for writers after each rolling replacement
2026-10-08 21:24:34 +08:00
GatewayJ 5b4432620e refactor(iam): separate OIDC authentication mapping (#8416) 2026-10-08 21:21:22 +08:00
Chris 1a450bf228 ci(e2e): require fresh distributed JUnit reports (#8384)
* ci(e2e): require fresh distributed JUnit reports

* test(e2e): emit JUnit from successful workflow mocks
2026-10-08 06:21:19 -07:00
Chris 371241afcb test(connect): restore subscriber readiness deadlines (#8285) 2026-10-08 06:21:11 -07:00
Chris 61b31e7b45 test(connect): reject unknown health flags (#8142) 2026-10-08 06:21:03 -07:00
Chris 0d1d1487f3 test(ecstore): cover mixed metadata arrival orders (#8385) 2026-10-08 06:05:23 -07:00
Chris 3824779520 test(ecstore): cover late GET object generation changes (#8383) 2026-10-08 06:05:03 -07:00
cxymds a9670ebb4d fix(ci): preserve heal quorum retries and synchronize background tests (#8422) 2026-10-08 05:03:33 -07:00
cppudge 99ae115731 fix(ecstore): make deletes cancellation-safe and reclaim their residue (#8378)
* fix(ecstore): keep object deletes running when the caller is dropped

DeleteObject and DeleteObjects commit on every disk and then run a
separate cleanup pass, both inside the request future. When the client
disconnects, hyper drops that future and the delete stops between the
two passes: xl.meta is gone, but the rollback backup and the marked data
dir stay on disk, hidden from listings and never reclaimed (#6898).

Run the ECStore delete entry points on a detached task tracked by the
instance context. The task owns the object locks, both disk passes and
any rollback, so dropping the caller only stops the wait. Server
shutdown waits up to 30s for detached deletes after the HTTP drain.

Two smaller leaks of the same kind: a cancelled data-dir delete no
longer leaves its snapshot-lease entry marked as deleting forever, and
DiskStore::delete_versions releases its waiting slot through a guard.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): finish committed delete residue on repeated deletes

Residue that an older build stranded (#6898) has no xl.meta, so deleting
the key again reports it missing and leaves everything in place. On a
disk, a delete-data.<T> marker in one of the object's data dirs proves
that transaction T removed the metadata there and committed: the data it
marked can no longer be read, healed or rolled back on that disk.

When delete_versions or delete_version finds no xl.meta, remove what the
cleanup pass of every transaction proven that way would have removed,
under the object's metadata lease. The rollback backup goes before the
marked data dirs, so an attempt cut short still leaves the marker for
the next one. Reserved markers, rollback backups without a committed
marker and directories that also hold xl.meta or other keys are left
alone. The disk result stays FileNotFound; the extra cost is one
directory read per missing key inside the disk call that already runs,
with no new RPC, and the read stops at the first child that is not a
data dir. Finished transactions are counted in
rustfs_disk_delete_residue_finished_total instead of logged per object.

A single DeleteObject in an unversioned bucket answers a missing key
from the metadata read quorum without reaching the disks, so existing
residue is reclaimed by DeleteObjects or by the empty-listing purge.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): purge the rollback backup of a committed delete on listing

After an empty listing, the orphan purge (#7955, #8261) already removes
data dirs that carry a committed delete-data.<T> marker, but it treated
the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data
and blocked the whole object directory and its ancestors. Residue left
by an abandoned delete (#6898) therefore survived the listing, and
DeleteBucket kept failing with BucketNotEmpty.

Treat a UUID dir that holds only xl.meta.bkp as purgeable when a
purgeable data dir of the same object on the same disk carries the
committed marker of that transaction. A backup without such a marker
still blocks as before, since it may be needed to roll the delete back.
The other purge gates are unchanged: every disk of the set must be
online, and the backoff applies. xl.meta or unmarked data on any disk
blocks that directory and its ancestors on every disk. Like a marked
data dir before, a backup below such a directory is still removed from
a disk that holds no xl.meta there and proves the transaction with its
own marker.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): move caller-held delete locks into the detached delete

A force (recursive) delete takes the bucket lifecycle write lock in the
S3 handler and passes only its fence down, so the store does not take
that lock again. Since the delete runs on a detached task, dropping the
request released the lock while the delete kept running: a PUT under
the same prefix could succeed and then be removed by the old delete.
Lifecycle expiry had the same shape with its table-bucket publication
read lock, of which only the lock-lost signal reaches the delete.

Add ECStore::delete_object_with_tier_delete_journal_and_guards. It
moves the caller's guards into the detached task and releases them
after the delete and its listing bookkeeping. The force-delete handler
and lifecycle expiry hand their guards over; DeleteObjects holds none.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(startup): keep unclean-shutdown markers while deletes still run

Shutdown waits up to 30s for detached deletes, but it cleared the
unclean-shutdown markers even when that wait timed out. A delete then
cut short by the runtime teardown went unnoticed: the next startup
skipped the erasure-set heal that the marker triggers.

Clear the markers only when the heal handoff completed and every
detached mutation finished, and log that they are kept otherwise. Wait
after the FTP, FTPS, SFTP and WebDAV servers have stopped as well,
since their deletes take the same detached path.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* test(ecstore): simulate tier-delete crashes below the detached delete

The chunk crash-boundary tests model a crash by aborting the task that
runs delete_object_with_tier_delete_journal while it is paused at a
TierDeleteChunkTestBarrier. That entry point now runs the delete on a
detached task, so aborting the caller no longer stops it: the delete
resumed on the old store once the barrier was dropped, and building the
restarted store on the same disks tripped the tier-delete journal proof
assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources
failed in CI).

Run the delete through handle_delete_object_with_journal, the body
below the detached task, so the abort stops it at the barrier as before.
tier_delete_chunk_missing_child_with_journals_fails_closed used the same
pattern; its resumed delete only failed fast because the tier rejects
removes, so it gets the same change.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

---------

Signed-off-by: Petr Slavyagin <disident47@gmail.com>
2026-10-08 02:58:14 -07:00
cxymds 545e794945 fix(admin): resolve event destination subscription target types (#8386) 2026-10-08 02:56:49 -07:00
cxymds 855586255f feat(auth): allow custom secret keys beyond 40 bytes (#8419)
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints.
2026-10-08 17:37:15 +08:00
Hauser bf47a919ac feat(quota): add sharded durable reservation protocol (#8402)
* feat(quota): add sharded durable reservation protocol

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): preserve typed task error context

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): initialize all durable reservation shards

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): persist bootstrap shard eligibility

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(quota): reuse allocator bootstrap state

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): widen inventory startup wait

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(quota): reserve protocol v3 for unknown fixtures

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): reload allocator after concurrent credit refills

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-08 16:58:32 +08:00
GatewayJ bdd23da034 refactor(iam): separate federated provider views (#8415) 2026-10-08 16:53:17 +08:00
Valentin Belyakov 89a277c3d9 feat: support object lock remaining retention days (#8407) 2026-10-08 16:52:27 +08:00
Henry Guo 3e87004032 fix(table-catalog): reuse held entry locks in warehouse scans (#8418) 2026-10-08 01:01:56 -07:00
Henry Guo 76386572f6 fix(table-catalog): drop migrated tables after rename (#8417) 2026-10-08 00:07:24 -07:00
Hauser 34917c7947 fix: resume stalled GET bodies and classify transient storage failures (#8414)
* fix: resume stalled GET bodies

Retry a stalled source through the existing identity-fenced resume budget so transient stalls do not truncate committed responses.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: preserve data usage read quorum errors

Apply the existing system-volume error mapping to snapshot body reads so a metadata-volume outage remains a retryable quorum failure.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: handle transient heal lock failures

Align erasure-set heal classification with the shared lock taxonomy and avoid duplicate error logging for retried scanner journal writes.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-07 23:11:36 -07:00
GatewayJ f0b2080e0c refactor(iam): add verified federated identity model (#8413) 2026-10-07 23:10:34 -07:00
Hauser 9c6b0e182b fix: stabilize scanner leases and connect diagnostics (#8394)
* fix(scanner): scope observational publication leases

Separate ordinary namespace commit churn from data-movement fences and retry transient lease acquisition conflicts. Bind observational leases to the observed usage object while preserving strict authoritative publication checks.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): expect namespace admission conflict

Assert the retryable namespace-specific result when a commit finishes while strict lease admission awaits its snapshot.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(connect): add headroom to real performance probes

Give real filesystem and object integration probes enough bounded time on slow CI runners, and keep cleanup independent from measurement latency. Use a genuinely unwritable Linux fixture when tests run as root.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-07 19:04:34 -07:00
Chris ab9af5523c test: record multipart target response diagnostics (#8382)
* test(e2e): capture mixed multipart failure snapshots

* test: record multipart target response diagnostics
2026-10-07 18:15:00 -07:00
Hauser b18c5afc8d feat(targets): support Kafka SASL without TLS (#8398)
* feat(targets): support Kafka SASL without TLS

Propagate the explicit Kafka transport mode through notify and audit targets while retaining TLS as the default.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update RustFS Kafka crates to 1.4.0

Use the published Kafka clients after the SASL plaintext transport support release.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-07 16:57:09 -07:00
Hauser df988cfd86 fix(obs): export bucket gauges from live snapshots (#8399)
* fix(obs): export bucket gauges from live snapshots

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(obs): collapse snapshot update condition

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 02:12:48 +08:00
Henry Guo 04bc5dffd6 fix(table-catalog): preserve credential vending after rename (#8400) 2026-10-08 02:10:56 +08:00
Chris 681c95f359 ci: add daily main sync for the gateway integration branch (#8401)
Merge main into gateway/integration every weekday with a --no-ff merge
commit; abort a conflicting merge and open a sync/<source>-<date> pull
request listing the conflicting files instead. Run ci.yml on pull requests
that target gateway/integration so the branch ruleset's required checks can
report, and document both in the CI gate matrix.

Refs rustfs/backlog#2736, rustfs/backlog#2734
2026-10-08 02:08:57 +08:00
Hauser 32d98ff8b2 feat(scanner): admit background disk I/O (#8389)
* feat(scanner): admit background disk I/O

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(startup): place tests after module items

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 01:01:55 +08:00
cxymds 6b1554003e fix(heal): defer replacement retries until dangling grace expires (#8388) 2026-10-07 07:04:12 -07:00
GatewayJ 19978b2cb6 test(iam): freeze OIDC federation boundaries (#8393) 2026-10-07 20:05:10 +08:00
Lindroos Hsu db2abcf0a6 feat(http3): add opt-in QUIC transport (#7903)
* build(http3): add optional s3s-http3 dependency

Signed-off-by: tommady <tommady@users.noreply.github.com>

* feat(server): add opt-in HTTP/3 transport

Signed-off-by: tommady <tommady@users.noreply.github.com>

* test(server): cover HTTP/3 TLS configuration

* feat(server): add opt-in HTTP/3 support

Serve S3 requests over QUIC with the existing middleware stack.

Signed-off-by: tommady <tommady@users.noreply.github.com>

* fix(http3): cancel pending body reads without panicking

* test(http3): keep transport independent of s3s

---------

Signed-off-by: tommady <tommady@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-07 03:59:21 -07:00
Hauser 14eb28dcd6 chore(deps): refresh workspace dependencies and hotpath profiling (#8380)
* chore(deps): update workspace dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: allow hotpath label marker attributes

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: refine hotpath profiling attribution

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: guard object hotpath attribution

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* Add ProfileReasonCode to CPU profile capture

Signed-off-by: Hauser <housemecn@gmail.com>

* fix: restore CPU profile test import

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-07 03:02:39 -07:00
ormandj 2f0143ef15 fix(ecstore): skip warning for absent .minio.sys volume (#8373)
* fix(ecstore): skip warning for absent .minio.sys volume

Startup MinIO migrations read from .minio.sys and treat its absence as expected, but resolve_read_all_result logged a disk_local_read_version_fallback warning for every such read. Keep the access check and the VolumeNotFound return, and skip only the warning for the migrating metadata volume.

Refs #8372

* test(ecstore): cover .minio.sys descendants and lookalike volumes

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-07 01:02:47 -07:00
Henry Guo 3b92708011 feat(table-catalog): add durable catalog backups (#8266)
* feat(table-catalog): add durable catalog backups

* fix(table-catalog): preserve migration lock ordering

* fix(table-catalog): preserve backup recovery fences

* fix(table-catalog): order backup publication locks

* fix(table-catalog): preserve recovery publication fences

* test(table-catalog): use the storage facade for recovery fencing

* test(ecstore): repair catalog publication fixtures

* fix(table-catalog): address catalog backup clippy failures

* test(table-catalog): adapt generation races to publication fencing

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-06 23:53:53 -07:00
Chris 8fea8ef600 test(e2e): bound mixed-version multipart acknowledgement (#8381) 2026-10-07 14:08:47 +08:00
GatewayJ 477f768f7e feat(table-catalog): support disabling empty catalogs (#8376)
* feat(table-catalog): support disabling empty catalogs

* fix(admin): use the storage error domain module

* style(admin): format the publication fence error

* fix(table-catalog): use shared API error constructors
2026-10-07 14:07:56 +08:00
shurkus 47ed6717de fix(tier): drop reserved internal headers from transition PUT (#8363)
* fix(tier): drop reserved internal headers from transition PUT

* test(tier): transition PUT against a target rejecting reserved headers

The fake S3 target gains a mode that answers 400 InvalidArgument to any
PutObject or UploadPart carrying an x-minio-internal-* request header,
the rule MinIO and the Storj gateway enforce. RT-14 transitions an
inline and a non-inline object through a real server to that target and
asserts the stored bytes, both transition identity pairs and read-back.

The e2e-full linux selection digest is refreshed for the new test.

---------

Co-authored-by: Vector <vector@wcomp.io>
Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-07 14:06:14 +08:00
Chris 480312e5ad fix(ecstore): refresh parsed bucket configs on every load (#8379) 2026-10-06 22:59:14 -07:00
yihong 1c5bbc0c47 ci: do not run some actions on fork (#8318)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2026-10-06 22:02:55 -07:00
Chris 14adc5ed3f test(ecstore): cover real metadata cache and late identity reads (#8371)
* test(ecstore): cover real metadata cache and late identity reads

* test(e2e): bound max-keys fixture upload concurrency
2026-10-06 22:02:48 -07:00
dependabot[bot] 714706328b build(deps): bump rustls from 0.23.42 to 0.23.45 in /fuzz in the cargo group across 1 directory (#8370)
build(deps): bump rustls in /fuzz in the cargo group across 1 directory

Bumps the cargo group with 1 update in the /fuzz directory: [rustls](https://github.com/rustls/rustls).


Updates `rustls` from 0.23.42 to 0.23.45
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.42...v/0.23.45)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.45
  dependency-type: indirect
  dependency-group: cargo
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-06 20:05:36 -07:00
Chris cd0b895ad7 fix(protocols): share the server context with protocol storage (#8367)
* fix(protocols): bind storage authorization to the startup server context

* test(protocols): publish the context through the runtime boundary
2026-10-06 19:58:58 -07:00
Chris 073b4d838e test(ecstore): verify multi-generation authority model (#8339) 2026-10-06 19:58:13 -07:00
Hauser 95268a3b9e fix(obs): avoid reactor panic in OTLP periodic exports (#8368)
fix(obs): use blocking clients for OTLP exporters

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-06 23:31:56 +08:00
Chris f7c4ac11b2 fix(heal): retry typed RPC transport failures after peer restart (#8365)
* fix(heal): retry typed RPC transport failures after peer restart

* test(heal): keep original errors in GOAWAY clone coverage
2026-10-06 08:11:19 -07:00
Sven Gottwald 9d9e92bab8 fix(compose): Refactor volume paths and add service_completed_success… (#8355)
fix(compose): Refactor volume paths and add service_completed_successfully

The volume paths for the volume-permission-helper service have been updated and the command has been adjusted to reflect the new paths.

Ensure that the volume-permission-helper service runs successfully before starting the rustfs service.

Signed-off-by: Sven Gottwald <2502366+svengo@users.noreply.github.com>
2026-10-06 22:42:15 +08:00
overtrue 0ad0a6d8de fix(heal): increase stack size for absence receipt replay test
The  test used
 with default stack size, but
the 16-disk deep heal path overflows the default macOS thread stack
(~8MB). Switch to an explicit  with 32MB stack
and manual tokio runtime, matching the pattern used by sibling tests
in the same file.

Verified: test passes in isolation after the change.
2026-10-05 04:10:55 -07:00
GatewayJ 051a54f129 fix(table-catalog): verify Iceberg metadata path signatures (#8291)
* fix(table-catalog): verify Iceberg metadata path signatures

* fix(deps): allow pinned s3s Git source
2026-10-05 02:03:41 -07:00
dependabot[bot] c71e4f72ea build(deps): bump the dependencies group with 4 updates (#8359)
Bumps the dependencies group with 4 updates: [aws-sdk-kms](https://github.com/awslabs/aws-sdk-rust), [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust), [aws-sdk-sts](https://github.com/awslabs/aws-sdk-rust) and [hotpath](https://github.com/pawurb/hotpath-rs).


Updates `aws-sdk-kms` from 1.122.0 to 1.123.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-sdk-s3` from 1.151.0 to 1.152.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-sdk-sts` from 1.118.0 to 1.119.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `hotpath` from 0.28.0 to 0.28.2
- [Release notes](https://github.com/pawurb/hotpath-rs/releases)
- [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.28.0...v0.28.2)

---
updated-dependencies:
- dependency-name: aws-sdk-kms
  dependency-version: 1.123.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-sdk-s3
  dependency-version: 1.152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-sdk-sts
  dependency-version: 1.119.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.28.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-05 10:38:53 +08:00
Hauser 003570b61e fix(heal): bound durable MRF rescheduling (#8351)
Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 09:04:11 -07:00
Chris 0c3efda7bd docs: refresh security advisory lessons (#8350) 2026-10-04 22:48:56 +08:00
Hauser 2c7e7d195c fix(auth): return ExpiredToken for retained STS credentials (#8344)
* fix(auth): return ExpiredToken for retained STS credentials

Validate session identity independently of credential expiry in both IAM
rejection and claims validation, and preserve typed S3 errors at the caller.
Keep token binding, disabled-account precedence, and service-account behavior.
Add exact expiry, IAM, and HTTP/XML regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): synchronize the metadata slowtail hedge regression

Park the slow metadata task at a test-only fault checkpoint and release
healthy reads after the fault is active. Assert the spare response and
cancelled-task drain without imposing a 300 ms disk/scheduler deadline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ilm): synchronize copy compensation transition stages

Observe automatic compensation reaching the mock tier PUT checkpoint before
waiting for the final metadata commit. Assert forced enqueue failure,
automatic backfill, one completed tier upload, and payload integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 06:16:40 -07:00
Hauser 0764785754 perf(scanner): reduce scan and checkpoint peak allocations (#8346)
* perf(scanner): reduce scan and checkpoint peak allocations

* test(scanner): isolate partial commit recovery scenarios
2026-10-04 06:04:19 -07:00
Chris 182b2bfe98 fix(s3): encode multipart listing response keys (#8347) 2026-10-04 20:24:26 +08:00
cxymds 00a549504c test: allow cleanup fixture CI headroom (#8336) 2026-10-04 20:20:05 +08:00
Hauser 83fa2611f4 fix(ecstore): own listing workers through cancellation (#8342)
Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 02:04:01 -07:00
hector e22414fe68 ci: coerce the pool timeout input to a number
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
2026-10-04 01:45:12 -07:00
hector 45f97bccf9 ci: coerce the pool timeout input to a number
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
2026-10-04 01:45:12 -07:00
Chris 046fb21d86 fix(test): restore authenticated cluster fault scenarios (#8340)
* fix(test): restore authenticated cluster fault scenarios

* fix(test): use the E2E storage API boundary
2026-10-04 01:03:59 -07:00
Chris 4783adff36 test(ecstore): enforce main core regression gates (#8338)
* test(ecstore): enforce main core regression gates

* test(ecstore): pin external shards and completion regressions
2026-10-04 01:03:42 -07:00
GatewayJ 68ad1a1d7b fix(protocols): enforce S3 policies for file protocol operations (#8262) 2026-10-03 21:40:56 -07:00
overtrue 7c289001ed test(connect): isolate short HTTP fixtures in nextest 2026-10-03 21:40:18 -07:00
overtrue 7b1ffbfdc1 fix(ecstore): hide rollback backup residue in prefix listings 2026-10-03 21:40:18 -07:00
Chris 562e03670e fix(storage): retain typed owners across borrowed writes (#8329)
* fix(storage): retain typed owners across borrowed writes

* fix(storage): preserve Copy write completion policy

* refactor(storage): remove redundant write owner references
2026-10-03 20:02:39 -07:00
Hauser 5e1bd498ce fix(scanner): isolate and coalesce dirty usage journals (#8325)
* fix(scanner): isolate and coalesce dirty usage journals

Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): collapse confirmed journal clear condition

Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 03:02:22 +08:00
Chris 5aca750a6e fix(listing): count logical prefixes before pagination (#8328)
* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed
2026-10-04 01:24:39 +08:00
Chris a760d567b6 test(storage): make commit fault checkpoints independent (#8326)
* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup
2026-10-04 01:19:28 +08:00
Chris 47d208325e test(storage): verify rejected rollback inspection on real disks (#8331) 2026-10-04 01:13:39 +08:00
Chris d087416ff9 refactor(storage): separate metadata scheduling from quorum decisions (#8327) 2026-10-04 01:04:37 +08:00
Chris 309128f034 refactor(storage): bind rename rollback to its commit context (#8324) 2026-10-03 23:55:33 +08:00
536 changed files with 82947 additions and 14217 deletions
@@ -45,11 +45,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
### S3 object actions, copy, multipart, and upload policy validation
- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
@@ -60,12 +63,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
### Filesystem paths and object key traversal
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
### Secrets, defaults, and cryptographic misuse
@@ -106,6 +111,8 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### Object Lock and retention invariants
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
### Serde deserialization and input validation
@@ -135,16 +142,17 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
## Minimum Regression Test Expectations
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=98dff0961c6eb21d6128bfd2d4a6100a624ddf1fb7db05dba360a5d3d0a310c2
sha256-linux=9ff0da7406de9b89c0e848ff29f2238b30ba9da89c7e9df97c246ce86631350f
sha256-darwin=1520cd3027c3a6924afc6a204eddd5475bc30f0109d4c2847456a66177263244
sha256-linux=26155b7f1c5bf6bf596ff24a4136e6767ea4ee6461c32db7fbe1400791da2c28
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17
sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392
+85
View File
@@ -61,6 +61,11 @@
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
},
{
"invariant": "rollback-closed-inspection-channel",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_closed_heal_receiver_preserves_recovery_material"
},
{
"invariant": "rollback-cancellation",
"suite": "rustfs-ecstore",
@@ -86,6 +91,11 @@
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
},
{
"invariant": "seeded-put-tail-pressure",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::seeded_put_completion_pressure_drains_every_real_tail"
},
{
"invariant": "borrowed-lock-tail",
"suite": "rustfs-ecstore",
@@ -151,6 +161,11 @@
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
},
{
"invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
},
{
"invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
"suite": "rustfs-ecstore",
@@ -166,6 +181,21 @@
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
},
{
"invariant": "metadata-cache-real-get-and-range-preserve-body-across-read-modes",
"suite": "rustfs-ecstore",
"name": "set_disk::read::metadata_cache_tests::metadata_cache_real_get_and_range_preserve_body_across_read_modes"
},
{
"invariant": "late-metadata-refresh-rejects-changed-generation-and-mapped-shards",
"suite": "rustfs-ecstore",
"name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
},
{
"invariant": "late-full-get-rejects-changed-object-generation",
"suite": "rustfs-ecstore",
"name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
},
{
"invariant": "list-objects-shared-corpus-static-namespace",
"suite": "rustfs-ecstore",
@@ -260,6 +290,46 @@
"invariant": "logical-prefix-limit",
"suite": "rustfs-ecstore",
"name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
},
{
"invariant": "quota-same-name-owner-and-disk-authority",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority"
},
{
"invariant": "quota-snapshot-scope-and-owner",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard"
},
{
"invariant": "quota-snapshot-lifecycle-and-incarnation",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation"
},
{
"invariant": "marker-purge-receipt-reopen",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::marker_purge_receipt_tests::quorum_marker_purge_receipt_survives_reopen_with_stale_member"
},
{
"invariant": "external-minio-shards",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_minio_shards_decode_and_reconstruct_plaintext"
},
{
"invariant": "external-legacy-shards",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_legacy_shards_decode_and_reconstruct_plaintext"
},
{
"invariant": "external-shard-bitrot",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_erasure_shards_reject_bitrot_corruption"
},
{
"invariant": "external-shard-quorum-codec",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_erasure_shards_reject_insufficient_quorum_and_wrong_codec"
}
],
"fixtures": [
@@ -302,6 +372,21 @@
"path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
"sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
"source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/minio/manifest.json",
"sha256": "c398f8dcb76903f6cf8eee07931c76592895ced201ff9d59ed4ff9b0ead2b13c",
"source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/erasure-shards/README.md"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/legacy/manifest.json",
"sha256": "dce397d5b2f42748eb5091b4355bb9d133a50cbba003626fafac28095dbe540b",
"source": "RustFS 1.0.0-alpha.37; crates/ecstore/tests/fixtures/erasure-shards/README.md"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/plaintext.bin",
"sha256": "b5a83332327961a31b65a8731c64559fd04ff8ac9f6124022201dbbdba5ffa01",
"source": "Independent fixed SHA256 counter-stream oracle; crates/ecstore/tests/fixtures/erasure-shards/README.md"
}
]
}
+3
View File
@@ -52,6 +52,9 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
+24 -3
View File
@@ -88,10 +88,21 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
[[profile.default.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
[[profile.default.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -170,10 +181,11 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
# Registration, inventory delivery, and real drive/object probes include
# durable filesystem IO in short deadlines. Reserve capacity so sibling test
# filesystem or HTTP IO in short deadlines. Reserve capacity so sibling test
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
@@ -332,10 +344,19 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
[[profile.ci.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -428,7 +449,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
@@ -1748,7 +1748,7 @@
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
"expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
@@ -1848,7 +1848,7 @@
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
"expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
+1
View File
@@ -61,6 +61,7 @@ scrape_configs:
otlp:
promote_resource_attributes:
- service.instance.id
- process.pid
- service.name
- service.namespace
- cloud.availability_zone
@@ -0,0 +1,48 @@
evaluation_interval: 1m
tests:
- name: bucket usage keeps clusters separate and deduplicates reporting nodes
interval: 1m
input_series:
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
values: '4096 1024'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
values: '4096 1024'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
values: '65536 65536'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
values: '0 0'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
values: '3 1'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
values: '3 1'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
values: '70 70'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
values: '0 0'
promql_expr_test:
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1024
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1024
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
value: 65536
- expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
eval_time: 1m
exp_samples: []
+4
View File
@@ -98,6 +98,10 @@ runs:
shell: bash
run: make script-tests
- name: Check object generation protocol model
shell: bash
run: python3 ./scripts/test_object_generation_protocol_model.py
- name: Check test wiring
shell: bash
run: python3 ./scripts/check_test_wiring.py
@@ -0,0 +1,39 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Set up Docker Compose
description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
runs:
using: composite
steps:
- name: Install verified Docker Compose CLI plugin
shell: bash
run: |
set -euo pipefail
[[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
mkdir -p "$plugin_dir"
compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
trap 'rm -f "$compose_download" "$compose_staged"' EXIT
curl --fail --location --retry 3 \
https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
--output "$compose_download"
printf '%s %s\n' \
dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
"$compose_download" | sha256sum --check --strict
install -m 0755 "$compose_download" "$compose_staged"
mv -f "$compose_staged" "$plugin_dir/docker-compose"
[[ $(docker compose version --short) == 2.40.3 ]]
+15 -1
View File
@@ -36,6 +36,16 @@ inputs:
description: "Install cross-compilation tools"
required: false
default: "false"
zig-version:
description: >-
Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
baseline semantics cannot drift with an untested zig release.
required: false
default: "0.17.0"
cargo-zigbuild-version:
description: "cargo-zigbuild version installed for cross builds."
required: false
default: "0.23.4"
target:
description: "Target architecture to add"
required: false
@@ -99,10 +109,14 @@ runs:
- name: Install Zig
if: inputs.install-cross-tools == 'true'
uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
with:
version: ${{ inputs.zig-version }}
- name: Install cargo-zigbuild
if: inputs.install-cross-tools == 'true'
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
with:
tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
@@ -126,5 +140,5 @@ runs:
# normalises, so cargo fingerprints stay valid.
cache-all-crates: false
cache-on-failure: true
shared-key: ${{ inputs.cache-shared-key }}
shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
save-if: ${{ inputs.cache-save-if }}
+3 -3
View File
@@ -88,7 +88,7 @@ jobs:
# database lookup with a second ignore list to maintain.
cargo-deny:
name: Cargo Deny
if: github.event_name != 'pull_request' || github.event.action != 'closed'
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: sm-standard-2
timeout-minutes: 20
steps:
@@ -131,7 +131,7 @@ jobs:
workflow-pin-report:
name: Workflow Pin Report
if: github.event_name != 'pull_request' || github.event.action != 'closed'
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: sm-standard-2
timeout-minutes: 5
steps:
@@ -203,7 +203,7 @@ jobs:
# workflow produces, and until now it was only visible to whoever happened to
# open the Actions tab. Same ci-8 mechanism coverage.yml and
# e2e-replication-nightly.yml already use.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+67 -39
View File
@@ -213,12 +213,12 @@ jobs:
fi
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
if [[ "${selected}" == "all" ]]; then
@@ -246,7 +246,8 @@ jobs:
echo "Selected platforms: ${selected}"
build-rustfs:
name: Build RustFS
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
@@ -273,7 +274,7 @@ jobs:
with:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -410,35 +411,64 @@ jobs:
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
BINARY_NAMES=(rustfs)
if [[ "${{ matrix.platform }}" == "linux" ]]; then
BINARY_NAMES+=(rustfs-cli)
fi
FEATURE_ARGS=()
if [[ -n "${{ matrix.features }}" ]]; then
FEATURE_ARGS=(--features "${{ matrix.features }}")
fi
# Build one binary at a time so release LTO links cannot exhaust a
# self-hosted runner by running concurrently.
for binary in "${BINARY_NAMES[@]}"; do
PROFILE_ARGS=()
if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
# All cross targets in the matrix are Linux; zigbuild handles them.
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
else
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
fi
done
# gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
# cargo-zigbuild) so release binaries link no symbols newer than it
# and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
TARGET_ARG="$BUILD_TARGET"
if [[ -n "${{ matrix.glibc }}" ]]; then
TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
fi
PROFILE_ARGS=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
else
cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
fi
# All release targets use same-architecture runners, including GNU
# builds that use Zig only to select the glibc ABI floor.
bin="target/${{ matrix.target }}/release/rustfs"
if [[ -f "${bin}.exe" ]]; then
bin="${bin}.exe"
fi
version_output=$("$bin" --version)
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \
! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
echo "release binary reports an unexpected compile target" >&2
exit 1
fi
if [[ -n "${{ matrix.glibc }}" ]]; then
python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
"target/${{ matrix.target }}/release/rustfs"
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: matrix.platform == 'macos'
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
echo "macOS release binary must not depend on a dynamic liblzma library" >&2
exit 1
fi
"$binary" --version
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
@@ -543,9 +573,6 @@ jobs:
fi
BINARY_FILES=("$BINARY_NAME")
if [[ "${{ matrix.platform }}" == "linux" ]]; then
BINARY_FILES+=("rustfs-cli")
fi
if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
@@ -660,7 +687,7 @@ jobs:
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux binaries
- name: Verify packaged Linux binary
if: matrix.platform == 'linux'
shell: bash
run: |
@@ -668,7 +695,7 @@ jobs:
package_file="${{ steps.package.outputs.package_file }}"
archive_members="$(unzip -Z1 "$package_file" | sort)"
expected_members=$'rustfs\nrustfs-cli'
expected_members='rustfs'
target="${{ matrix.target }}"
features="${{ matrix.features }}"
if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
@@ -680,13 +707,14 @@ jobs:
exit 1
fi
# Every Linux target builds natively, so executing the binaries here
# musl targets build natively; the gnu legs are same-arch
# cargo-zigbuild builds pinned to the release glibc baseline. Either
# way the binaries run on their runner, so executing them here
# proves each artifact matches its runner architecture.
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs" --version
"$cli_dir/rustfs-cli" --help >/dev/null
binary_dir="$(mktemp -d)"
trap 'rm -rf "$binary_dir"' EXIT
unzip -q "$package_file" rustfs -d "$binary_dir"
"$binary_dir/rustfs" --version
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
+1 -1
View File
@@ -99,7 +99,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
+56 -86
View File
@@ -36,7 +36,9 @@ on:
- "flake.lock"
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
# gateway/integration carries main's ruleset (rustfs/backlog#2734), so its
# pull requests must produce the same required checks.
branches: [ main, gateway/integration ]
merge_group:
types: [ checks_requested ]
schedule:
@@ -166,13 +168,10 @@ jobs:
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/test-and-lint/run-metadata.txt
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
# Keep full Clippy validation on main, merge groups, and scheduled/manual
# runs. PRs retain the complete default test suite and compile-free checks.
- name: Run clippy lints
if: github.event_name != 'pull_request'
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -281,6 +280,49 @@ jobs:
- name: Verify rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
# The workspace already compiled these ignored tests. Keep the same Cargo
# package selection to reuse their binaries, then run them serially in
# separate processes because they share fixed ports and global state.
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
if: needs.classify-changes.outputs.mode == 'full'
env:
# Match the preceding workspace build and reuse its feature graph.
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run --profile default -j1 --run-ignored ignored-only \
--all --exclude e2e_test \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always() && needs.classify-changes.outputs.mode == 'full'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/ilm-integration
retention-days: 3
# This gate builds into a fresh target directory to isolate the E2E root.
# Give its cold build a separate budget from workspace tests and migration proofs.
offline-enrollment-e2e:
@@ -312,83 +354,11 @@ jobs:
- name: Check offline enrollment E2E root boundary
run: ./scripts/check_offline_enrollment_e2e.sh
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
# fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
# marked #[ignore] to keep them out of the default parallel `cargo nextest run
# --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
# `-j1` so no two of them share global state or race for a port at once.
# serial_test's #[serial] does NOT serialize across nextest's process-per-test
# boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
# See rustfs/backlog#1148 (ilm-1) and #1155.
test-ilm-integration-serial:
name: ILM Integration (serial)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
env:
# Match the measured Test and Lint link budget. The default exposed
# all 14 pod CPUs and a cold cache spent the full 80m compiling
# without starting one ILM test (main run 32982910990).
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/ilm-integration
target/nextest/ci/junit.xml
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
# Alternate feature configurations run on main, merge groups, and scheduled
# or manual validation; the default workspace remains the PR gate.
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -504,12 +474,12 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
strategy:
# Preserve both independent results when one protocol fails, so diagnosis
# Preserve independent results when one protocol fails, so diagnosis
# and a focused fix do not require rebuilding an interrupted sibling.
fail-fast: false
matrix:
@@ -518,6 +488,8 @@ jobs:
flags: "--features swift"
- name: sftp
flags: "--features sftp"
- name: http3
flags: "--features http3,swift"
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -1260,7 +1232,6 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- connect-short-credential-boundary
- test-and-lint-protocols
@@ -1294,7 +1265,6 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- test-and-lint-protocols
- build-rustfs-debug-binary
@@ -57,6 +57,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -94,7 +102,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
@@ -137,7 +145,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
@@ -162,7 +170,7 @@ jobs:
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule
@@ -57,6 +57,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -127,7 +135,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -172,7 +180,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
@@ -199,7 +207,7 @@ jobs:
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=health-service
@@ -73,6 +73,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive
@@ -73,6 +73,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=performance-network
@@ -77,6 +77,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -118,7 +126,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.jobs[] | select(.name == $name)] as $matches
@@ -173,7 +181,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
@@ -57,6 +57,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -127,7 +135,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -172,7 +180,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
@@ -198,7 +206,7 @@ jobs:
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory
@@ -81,6 +81,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -148,7 +156,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -201,7 +209,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads
@@ -73,6 +73,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-api
@@ -68,6 +68,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -138,7 +146,7 @@ jobs:
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
@@ -183,7 +191,7 @@ jobs:
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
@@ -213,7 +221,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-disk
@@ -73,6 +73,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-locks
@@ -73,6 +73,14 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc
+4 -1
View File
@@ -78,6 +78,7 @@ concurrency:
jobs:
distributed:
if: github.repository == 'rustfs/rustfs'
name: Distributed 4-node 4-disk e2e
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
@@ -167,11 +168,13 @@ jobs:
FILTER: ${{ inputs.filter }}
run: |
set -euo pipefail
rm -f target/nextest/e2e-distributed/junit.xml
if [ -n "${FILTER}" ]; then
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
else
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
fi
test -s target/nextest/e2e-distributed/junit.xml
- name: Upload distributed e2e diagnostics
if: always()
@@ -200,7 +203,7 @@ jobs:
alert-on-failure:
name: Alert on scheduled failure
needs: [distributed]
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -53,6 +53,7 @@ concurrency:
jobs:
repl-nightly:
if: github.repository == 'rustfs/rustfs'
name: Replication e2e (nightly)
# dual-node tests spawn two full rustfs servers each; the extra headroom
# over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
@@ -119,6 +120,7 @@ jobs:
if-no-files-found: ignore
cluster-nightly:
if: github.repository == 'rustfs/rustfs'
name: Cluster fault e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
@@ -168,6 +170,7 @@ jobs:
if-no-files-found: warn
protocols-nightly:
if: github.repository == 'rustfs/rustfs'
name: Protocol e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
@@ -229,7 +232,7 @@ jobs:
# Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
# manual workflow_dispatch runs stay quiet so a debugging run never files a
# spurious alert.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+37 -41
View File
@@ -96,38 +96,8 @@ jobs:
overwrite: true
upgrade:
name: ${{ matrix.name }}
name: Upgrade compatibility
needs: build
strategy:
fail-fast: false
matrix:
include:
# The two `_from_rc2_` tests keep their names: they assert
# release-independent object contracts and pass unchanged against the
# newer pinned source, so renaming them would only churn history and
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
runs-on: sm-standard-2
timeout-minutes: 60
env:
@@ -142,8 +112,7 @@ jobs:
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-tests
# One main-branch consumer warms the shared test-harness dependencies.
cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
@@ -170,22 +139,49 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
- name: Run upgrade compatibility test
# Reuse one compiled harness, while keeping each case in its own process.
# Continue after a failure to retain the former matrix's complete verdict.
- name: Run upgrade compatibility tests
shell: bash
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
export no_proxy="$NO_PROXY"
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
cases=(
direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
)
status=0
for entry in "${cases[@]}"; do
artifact="${entry%%:*}"
test="${entry#*:}"
export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
echo "::group::$test"
if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
echo "PASS: $test"
else
echo "::error::Upgrade compatibility failed: $test"
status=1
fi
echo "::endgroup::"
done
exit "$status"
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
name: upgrade-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
if-no-files-found: ignore
retention-days: 14
@@ -193,7 +189,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
name: upgrade-server-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
if-no-files-found: warn
retention-days: 14
@@ -11,6 +11,7 @@ concurrency:
cancel-in-progress: false
jobs:
collect:
if: github.repository == 'rustfs/rustfs'
runs-on: smoke-testing
timeout-minutes: 15
steps:
+6 -4
View File
@@ -72,9 +72,11 @@ jobs:
fuzz-build:
name: Build Fuzz Harness
if: >
(github.event_name == 'pull_request' && github.event.action != 'closed') ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
github.repository == 'rustfs/rustfs' && (
(github.event_name == 'pull_request' && github.event.action != 'closed') ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
)
runs-on: sm-standard-4
timeout-minutes: 60
env:
@@ -239,7 +241,7 @@ jobs:
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
# ci-8); PR and manual dispatch failures are already watched by a human.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -0,0 +1,197 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Gateway Integration Sync
# Merges `main` into the long-lived `gateway/integration` branch every weekday
# so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736)
# never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit
# pushed straight to the branch. A conflicting merge is aborted and surfaced as
# a pull request from `sync/<source>-<date>`, whose head is the un-merged
# source commit, so the conflict is resolved the same day, in review, with no
# other change riding along. The run itself stays green in that case: the pull
# request is the signal. Any other failure (push rejected, pull request not
# created) fails the run and, on the schedule, files the shared failure issue.
#
# Resolving a conflict pull request: merge `origin/gateway/integration` into
# the sync branch, resolve, push, get the review. Land it with a fast-forward
# push of the resolved merge commit to `gateway/integration`, not with the
# squash button: a squash drops the merge ancestry and the same hunks conflict
# again on the next run.
#
# Write scope: only `gateway/integration` and `sync/*` are ever pushed, and
# both push targets below are literal refs. The pushes use a one-hour
# installation token of the organisation's own GitHub App (contents: write)
# because that app is the only non-admin bypass actor of the
# `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor,
# so it cannot push through the required reviews and checks. The conflict pull
# request is opened with FLAKE_UPDATE_TOKEN, the PAT that already opens the
# weekly flake.lock pull requests, because this repository keeps "Allow GitHub
# Actions to create and approve pull requests" switched off. GITHUB_TOKEN is
# therefore read-only here.
#
# `schedule` fires only from the default branch, so the cron is live once this
# file exists on `main`. `workflow_dispatch` runs from any ref that carries the
# file and always acts on `gateway/integration`.
on:
schedule:
- cron: "0 2 * * 1-5" # Weekdays 02:00 UTC
workflow_dispatch:
inputs:
source_ref:
description: >-
Branch to merge into gateway/integration. The schedule always merges
main; only the conflict acceptance test (rustfs/backlog#2736) sets
another branch.
required: false
default: main
type: string
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
env:
TARGET_BRANCH: gateway/integration
jobs:
sync:
name: Merge main into gateway/integration
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Create the branch push token
id: push-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
with:
app-id: ${{ vars.CLA_BOT_APP_ID }}
private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: rustfs
permission-contents: write
- name: Checkout gateway/integration
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push
# credential — the merge commit and the sync branch are pushed with it
# below. It is a one-hour installation token scoped to contents.
ref: gateway/integration
fetch-depth: 0
token: ${{ steps.push-token.outputs.token }}
- name: Configure the bot identity
run: |
git config user.name "rustfs-cla-assistant[bot]"
git config user.email "273121464+rustfs-cla-assistant[bot]@users.noreply.github.com"
- name: Merge the source branch
id: merge
env:
SOURCE_REF: ${{ inputs.source_ref || 'main' }}
run: |
set -euo pipefail
if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \
|| [[ "$SOURCE_REF" == *..* ]] \
|| [[ "$SOURCE_REF" == "$TARGET_BRANCH" ]]; then
echo "::error::refusing source_ref '${SOURCE_REF}'"
exit 1
fi
git fetch origin "refs/heads/${SOURCE_REF}:refs/remotes/origin/${SOURCE_REF}"
source_sha=$(git rev-parse "origin/${SOURCE_REF}")
target_sha=$(git rev-parse HEAD)
{
echo "source_ref=${SOURCE_REF}"
echo "source_sha=${source_sha}"
echo "target_sha=${target_sha}"
} >> "$GITHUB_OUTPUT"
if git merge --no-ff --no-edit "origin/${SOURCE_REF}"; then
if [[ "$(git rev-parse HEAD)" == "$target_sha" ]]; then
echo "result=up-to-date" >> "$GITHUB_OUTPUT"
echo "\`${TARGET_BRANCH}\` already contains \`${SOURCE_REF}\` (${source_sha}); nothing to push." >> "$GITHUB_STEP_SUMMARY"
else
echo "result=merged" >> "$GITHUB_OUTPUT"
fi
exit 0
fi
git diff --name-only --diff-filter=U > "$RUNNER_TEMP/conflicts.txt"
git merge --abort
if [[ ! -s "$RUNNER_TEMP/conflicts.txt" ]]; then
echo "::error::git merge failed without reporting conflicting paths; this is not a conflict"
exit 1
fi
echo "result=conflict" >> "$GITHUB_OUTPUT"
- name: Push the merge commit
if: steps.merge.outputs.result == 'merged'
env:
SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
run: |
set -euo pipefail
git push origin HEAD:refs/heads/gateway/integration
echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY"
- name: Open the conflict pull request
if: steps.merge.outputs.result == 'conflict'
env:
GH_TOKEN: ${{ secrets.FLAKE_UPDATE_TOKEN }}
SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
TARGET_SHA: ${{ steps.merge.outputs.target_sha }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
date_utc=$(date -u +%Y-%m-%d)
sync_branch="sync/${SOURCE_REF//\//-}-${date_utc}"
git push origin "refs/remotes/origin/${SOURCE_REF}:refs/heads/${sync_branch}"
{
echo "The scheduled merge of \`${SOURCE_REF}\` into \`${TARGET_BRANCH}\` stopped on conflicts and was aborted; \`${TARGET_BRANCH}\` is unchanged."
echo
echo "This branch is the un-merged \`${SOURCE_REF}\` head. Merge \`origin/${TARGET_BRANCH}\` into it, resolve the files below, push, and get the review. Land the resolved merge commit with a fast-forward push to \`${TARGET_BRANCH}\`; a squash merge drops the merge ancestry and the same hunks conflict again on the next run. Do not include any change that is not conflict resolution."
echo
echo "- Run: ${RUN_URL}"
echo "- Source: \`${SOURCE_REF}\` at ${SOURCE_SHA}"
echo "- Target: \`${TARGET_BRANCH}\` at ${TARGET_SHA}"
echo
echo "## Conflicting files"
echo
echo "Output of \`git diff --name-only --diff-filter=U\` captured before \`git merge --abort\`:"
echo
sed 's/^/- /' "$RUNNER_TEMP/conflicts.txt"
} > "$RUNNER_TEMP/pr-body.md"
title="sync: merge ${SOURCE_REF} ${date_utc} (conflicts)"
existing=$(gh pr list --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --state open --json number --jq '.[0].number // empty')
if [[ -n "$existing" ]]; then
gh pr edit "$existing" --repo "$GITHUB_REPOSITORY" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md"
pr_url=$(gh pr view "$existing" --repo "$GITHUB_REPOSITORY" --json url --jq .url)
else
pr_url=$(gh pr create --repo "$GITHUB_REPOSITORY" --base "$TARGET_BRANCH" --head "$sync_branch" --title "$title" --body-file "$RUNNER_TEMP/pr-body.md")
fi
echo "Conflicts in \`${SOURCE_REF}\` -> \`${TARGET_BRANCH}\`; pull request: ${pr_url}" >> "$GITHUB_STEP_SUMMARY"
alert-on-failure:
name: Alert on scheduled failure
needs: [sync]
# Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that
# could not push or could not open the conflict pull request must not stay
# visible only in the Actions tab.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
+18 -4
View File
@@ -43,6 +43,7 @@ env:
jobs:
resolve-source:
if: github.repository == 'rustfs/rustfs'
name: Resolve nightly source
runs-on: sm-standard-2
timeout-minutes: 10
@@ -79,13 +80,24 @@ jobs:
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: build-x86_64-unknown-linux-gnu
cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
cache-save-if: 'false'
install-cross-tools: 'true'
target: x86_64-unknown-linux-gnu
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Build RustFS
run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
- name: Build DEB package
id: deb
@@ -496,8 +508,10 @@ jobs:
name: Alert on scheduled failure
needs: [build, kms-vault-lane, kms-vault-ha-failover]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+1
View File
@@ -32,6 +32,7 @@ concurrency:
jobs:
update-flake:
if: github.repository == 'rustfs/rustfs'
name: Update flake.lock
runs-on: sm-standard-2
timeout-minutes: 90
+4 -2
View File
@@ -20,8 +20,9 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc.rs"
- "crates/iam/src/oidc/**"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -31,8 +32,9 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc.rs"
- "crates/iam/src/oidc/**"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
+5 -2
View File
@@ -49,6 +49,7 @@ env:
jobs:
warp-ab:
if: github.repository == 'rustfs/rustfs'
name: Warp A/B budget gate
runs-on: sm-standard-2
# A normal nightly restores the last successful binary and builds only the
@@ -364,8 +365,10 @@ jobs:
# composite action already reports cancelled/timed-out jobs in the issue
# body.
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+2 -1
View File
@@ -42,6 +42,7 @@ concurrency:
jobs:
check-ephemerality:
if: github.repository == 'rustfs/rustfs'
name: Check runner ephemerality
runs-on: sm-standard-2
timeout-minutes: 15
@@ -65,7 +66,7 @@ jobs:
# Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
# scheduled runs file a tracking issue, manual dispatch stays quiet so
# debugging never produces a spurious alert.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+4 -20
View File
@@ -266,7 +266,10 @@ jobs:
# the previous 45m step cap killed four consecutive runs mid-rebalance
# with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
# in ~21m) plus one decommission pass; override per dispatch.
timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
# fromJSON coerces the string dispatch input to the number that
# timeout-minutes requires; the bare || form yielded the string '240'
# and the runner rejected the template (Unexpected value '240').
timeout-minutes: ${{ fromJSON(inputs.pool_timeout_minutes || '240') }}
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
@@ -583,25 +586,6 @@ jobs:
echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
failed=1
fi
# A runner-level kill (step timeout) truncates the suite without a
# FAIL verdict or an [ERROR] line, so require every expected
# [POOL-STEP] marker to be present with PASS - missing markers mean
# the run was truncated, not green. This also closes the gap for
# standalone dispatches, where the chain driver's evidence
# validation does not run.
last_step=6
if [ "${{ inputs.pools || '3' }}" = "3" ]; then
last_step=8
if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
last_step=9
fi
fi
for n in $(seq 1 "${last_step}"); do
grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
failed=1
}
done
[ "${failed}" -eq 0 ] || exit 1
- name: Upload functional report to dashboard
@@ -28,6 +28,7 @@ concurrency:
jobs:
check-freshness:
if: github.repository == 'rustfs/rustfs'
name: Check scheduled validation freshness
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
@@ -39,6 +39,7 @@ jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.event == 'schedule' &&
github.event.workflow_run.conclusion != 'success' &&
github.event.workflow_run.conclusion != 'failure'
+1
View File
@@ -26,6 +26,7 @@ on:
jobs:
stale:
if: github.repository == 'rustfs/rustfs'
runs-on: sm-standard-2
timeout-minutes: 30
steps:
+1
View File
@@ -14,6 +14,7 @@ concurrency:
jobs:
update:
if: github.repository == 'rustfs/rustfs'
runs-on: sm-standard-2
timeout-minutes: 10
steps:
+5 -2
View File
@@ -40,6 +40,7 @@ env:
jobs:
targets-live:
if: github.repository == 'rustfs/rustfs'
name: PostgreSQL, MySQL, AMQP, and NATS
runs-on: sm-standard-2
timeout-minutes: 90
@@ -175,8 +176,10 @@ jobs:
name: Alert on scheduled failure
needs: [targets-live]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
+1
View File
@@ -90,3 +90,4 @@ worktrees/*
# insta scratch files; the accepted .snap files ARE the assertions and are committed
*.snap.new
*.pending-snap
.mimosa
+21 -27
View File
@@ -7,7 +7,9 @@
"name": "Debug(only) executable 'rustfs'",
"env": {
"RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
"RUSTFS_SKIP_BACKGROUND_TASK": "on"
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
// Local test volumes share one physical disk.
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -44,6 +46,7 @@
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -117,7 +120,8 @@
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
//"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -135,41 +139,30 @@
{
"type": "lldb",
"request": "launch",
"name": "Debug test_lifecycle_transition_basic",
"name": "Debug test_transition_and_restore_flows",
"cargo": {
"args": [
"test",
"--no-run",
"-p",
"rustfs-scanner",
"--test",
"lifecycle_integration_test",
"serial_tests::test_lifecycle_transition_basic",
"-j",
"1"
]
],
"filter": {
"name": "lifecycle_integration_test",
"kind": "test"
}
},
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/test",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
"args": [
"--skip",
"test_lifecycle_expiry_basic",
"--skip",
"test_lifecycle_expiry_deletemarker",
//"--skip",
//"test_lifecycle_transition_basic",
],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"sourceLanguages": [
"rust"
"serial_tests::test_transition_and_restore_flows",
"--exact",
"--ignored",
"--test-threads=1"
],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/rustfs with sse kms",
@@ -285,7 +278,7 @@
"RUSTFS_COMPRESSION_ENABLED": "true",
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
@@ -307,6 +300,7 @@
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_REGION": "us-east-1",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
@@ -315,7 +309,7 @@
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
Generated
+521 -462
View File
File diff suppressed because it is too large Load Diff
+97 -89
View File
@@ -73,8 +73,8 @@ resolver = "3"
edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.98.1"
version = "1.0.1"
rust-version = "1.99.0"
version = "1.0.2"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -91,56 +91,56 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.1" }
rustfs-heal = { path = "crates/heal", version = "1.0.1" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.1" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.1" }
rustfs-audit = { path = "crates/audit", version = "1.0.1" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.1" }
rustfs-common = { path = "crates/common", version = "1.0.1" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.1" }
rustfs-config = { path = "./crates/config", version = "1.0.1" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.1" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.1" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.1" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.1" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.1" }
rustfs-iam = { path = "crates/iam", version = "1.0.1" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.1" }
rustfs-license = { path = "crates/license", version = "1.0.1" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.1" }
rustfs-kms = { path = "crates/kms", version = "1.0.1" }
rustfs-lock = { path = "crates/lock", version = "1.0.1" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.1" }
rustfs-notify = { path = "crates/notify", version = "1.0.1" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.1" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.1" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.1" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.1", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.1" }
rustfs-obs = { path = "crates/obs", version = "1.0.1" }
rustfs-policy = { path = "crates/policy", version = "1.0.1" }
rustfs-protos = { path = "crates/protos", version = "1.0.1" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.1" }
rustfs-replication = { path = "crates/replication", version = "1.0.1" }
rustfs-rio = { path = "crates/rio", version = "1.0.1" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.1" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.1" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.1" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.1" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.1" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.1" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.1" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.1" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.1" }
rustfs-signer = { path = "crates/signer", version = "1.0.1" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.1" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.1" }
rustfs-targets = { path = "crates/targets", version = "1.0.1" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.1" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.1" }
rustfs-utils = { path = "crates/utils", version = "1.0.1" }
rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
# Async Runtime and Networking
async-channel = "2.5.0"
@@ -156,8 +156,9 @@ futures-lite = "2.6.1"
futures-util = "0.3.34"
pollster = "1.0.1"
pulsar = { default-features = false, version = "6.9.0" }
lapin = { default-features = false, version = "4.12.0" }
hyper = { version = "1.11.1" }
lapin = { default-features = false, version = "4.12.2" }
hyper = { version = "1.12.0" }
h2 = { version = "0.4.20" }
hyper-rustls = { default-features = false, version = "0.27.10" }
hyper-util = { version = "0.1.21" }
http = "1.5.0"
@@ -166,18 +167,25 @@ http-body-util = "0.1.5"
memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.3.1" }
rustfs-kafka-async = { version = "1.4.0" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.53.1" }
tokio = { version = "1.53.3" }
tokio-rustls = { default-features = false, version = "0.26.6" }
tokio-stream = { version = "0.1.19" }
tokio-test = "0.4.6"
tokio-util = { version = "0.7.19" }
tokio-util = { version = "0.7.20" }
tonic = { version = "0.14.6" }
tonic-prost = { version = "0.14.6" }
tonic-prost-build = { version = "0.14.6" }
tower = { version = "0.5.3" }
tower-http = { version = "0.7.1" }
h3 = "0.0.8"
h3-quinn = "0.0.10"
quinn = { version = "0.11.12", default-features = false, features = [
"futures-io",
"runtime-tokio",
"rustls-aws-lc-rs",
] }
# Serialization and Data Formats
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
@@ -193,7 +201,7 @@ rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.229" }
serde_ignored = { version = "0.1" }
serde_with = { version = "3", default-features = false, features = ["macros", "std"] }
serde_json = { version = "1.0.151" }
serde_json = { version = "1.0.154" }
serde_urlencoded = "0.7.1"
# Cryptography and Security
@@ -217,15 +225,15 @@ rustls-native-certs = "0.8"
rustls-pki-types = "1.15.1"
x509-parser = "0.18.1"
sha1 = "0.11.0"
sha2 = "0.11.0"
sha2 = "0.11.1"
subtle = "2.6"
zeroize = { version = "1.9.0" }
zeroize = { version = "1.9.1" }
proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.37" }
jiff = { version = "0.2.38" }
time = { version = "0.3.55" }
# Database
@@ -244,9 +252,9 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.12.0" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.122.0" }
aws-sdk-s3 = { default-features = false, version = "1.151.0" }
aws-sdk-sts = { default-features = false, version = "1.118.0" }
aws-sdk-kms = { default-features = false, version = "1.123.0" }
aws-sdk-s3 = { default-features = false, version = "1.152.0" }
aws-sdk-sts = { default-features = false, version = "1.119.0" }
aws-smithy-async = { version = "1.3.0" }
aws-smithy-http-client = { default-features = false, version = "1.5.0" }
aws-smithy-runtime-api = { version = "1.19.0" }
@@ -261,7 +269,7 @@ crossbeam-queue = "0.3.14"
crossbeam-channel = "0.5.17"
crossbeam-deque = "0.8.8"
crossbeam-utils = "0.8.23"
datafusion = { default-features = false, version = "55.1.0" }
datafusion = { default-features = false, version = "55.2.0" }
derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.14"
@@ -279,7 +287,7 @@ highway = { version = "1.3.0" }
hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.1"
libc = "0.2.189"
libc = "0.2.190"
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
log = "0.4"
@@ -299,7 +307,7 @@ percent-encoding = "2.3.2"
# Server-side QR rendering for TOTP enrollment, so neither the console nor the
# CLI needs its own QR encoder. No default features: the image/render backends
# pull in an image stack this only needs SVG and text output from.
qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
qrcode-rs = { version = "2.2.0", default-features = false, features = ["std", "svg"] }
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.3" }
@@ -312,17 +320,17 @@ rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.7.1" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.5" }
rust-embed = { version = "8.12.0" }
rust-embed = { version = "8.13.0" }
rustc-hash = { version = "2.1.3" }
s3s = { version = "0.17.0", features = ["minio"] }
s3s-sigv4 = "0.17.0"
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
smallvec = { version = "1.16.2" }
smallvec = { version = "1.16.3" }
compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.3.0" }
starshard = { version = "2.4.0" }
strum = { version = "0.28.0" }
sysinfo = "0.39.6"
temp-env = "0.3.6"
@@ -333,12 +341,12 @@ tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.34" }
tracing-opentelemetry = { version = "0.34.0" }
tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.26.1" }
uuid = { version = "1.28.0" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
@@ -346,28 +354,28 @@ winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.19" }
zip = "8.6.0"
zstd = "0.14.0"
zip = "9.0.3"
zstd = "0.14.2"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.2", default-features = false }
opentelemetry = { version = "0.33.0" }
opentelemetry-appender-tracing = { version = "0.33.0" }
opentelemetry-otlp = { version = "0.33.0" }
opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.33.0" }
opentelemetry-semantic-conventions = { version = "0.33.0" }
opentelemetry-stdout = { version = "0.33.0" }
pyroscope = { version = "2.1.1" }
dial9 = { version = "0.5.4", default-features = false }
opentelemetry = { version = "0.33.1" }
opentelemetry-appender-tracing = { version = "0.33.1" }
opentelemetry-otlp = { version = "0.33.1" }
opentelemetry-proto = { version = "0.33.1", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.33.1" }
opentelemetry-semantic-conventions = { version = "0.33.1" }
opentelemetry-stdout = { version = "0.33.1" }
pyroscope = { version = "2.1.2" }
# FTP and SFTP
libunftp = { version = "0.23.1" }
unftp-core = "0.1.0"
suppaftp = { version = "12.1.1" }
suppaftp = { version = "12.2.0" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.63.3" }
russh = { version = "0.64.1" }
russh-sftp = "3.0.1"
# WebDAV
@@ -376,9 +384,9 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.5.6" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.28.0", default-features = false }
hotpath = { version = "0.28.6", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
insta = { version = "1.49" }
[workspace.metadata.cargo-shear]
ignored = ["hotpath", "rustfs"]
+2
View File
@@ -51,6 +51,8 @@ unparseable = "unparseable"
# Disaster-recovery objectives: recovery time and recovery point.
RTO = "RTO"
rto = "rto"
# rustix OFlags variant (POSIX O_WRONLY equivalent).
WRONLY = "WRONLY"
[files]
extend-exclude = []
+1
View File
@@ -34,6 +34,7 @@ hotpath-alloc = ["hotpath", "hotpath/hotpath-alloc"]
hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu"]
[dependencies]
arc-swap.workspace = true
hotpath.workspace = true
tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
tonic = { workspace = true, features = ["gzip", "deflate"] }
+131 -5
View File
@@ -12,9 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use arc_swap::ArcSwap;
use std::collections::HashMap;
use std::sync::LazyLock;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, LazyLock};
use tokio::sync::RwLock;
use tonic::transport::Channel;
@@ -27,6 +28,42 @@ pub static GLOBAL_ROOT_CERT: LazyLock<RwLock<Option<Vec<u8>>>> = LazyLock::new(|
pub static GLOBAL_MTLS_IDENTITY: LazyLock<RwLock<Option<MtlsIdentityPem>>> = LazyLock::new(|| RwLock::new(None));
pub static GLOBAL_OUTBOUND_TLS_GENERATION: LazyLock<AtomicU64> = LazyLock::new(|| AtomicU64::new(0));
#[derive(Clone, Default)]
pub struct OutboundTlsSnapshot {
pub generation: u64,
pub root_ca_pem: Option<Vec<u8>>,
pub mtls_identity: Option<MtlsIdentityPem>,
}
static GLOBAL_OUTBOUND_TLS_STATE: LazyLock<ArcSwap<OutboundTlsSnapshot>> =
LazyLock::new(|| ArcSwap::from_pointee(OutboundTlsSnapshot::default()));
pub fn get_global_outbound_tls_snapshot() -> Arc<OutboundTlsSnapshot> {
GLOBAL_OUTBOUND_TLS_STATE.load_full()
}
/// Publish identity, trust and generation together; the individual globals are legacy views.
pub async fn publish_global_outbound_tls_snapshot(snapshot: OutboundTlsSnapshot) -> u64 {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
if current.generation != 0
&& current.root_ca_pem == snapshot.root_ca_pem
&& current.mtls_identity == snapshot.mtls_identity
{
return Arc::clone(current);
}
Arc::new(OutboundTlsSnapshot {
generation: snapshot.generation.max(current.generation.saturating_add(1)),
root_ca_pem: snapshot.root_ca_pem.clone(),
mtls_identity: snapshot.mtls_identity.clone(),
})
});
let accepted = get_global_outbound_tls_snapshot();
*GLOBAL_ROOT_CERT.write().await = accepted.root_ca_pem.clone();
*GLOBAL_MTLS_IDENTITY.write().await = accepted.mtls_identity.clone();
GLOBAL_OUTBOUND_TLS_GENERATION.store(accepted.generation, Ordering::Relaxed);
accepted.generation
}
/// Log level to use when reporting cached gRPC connection eviction.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum ConnectionEvictionLogLevel {
@@ -85,17 +122,27 @@ pub async fn get_global_addr() -> String {
/// # Arguments
/// * `cert` - A vector of bytes representing the PEM-encoded root CA certificate.
pub async fn set_global_root_cert(cert: Vec<u8>) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.root_ca_pem = Some(cert.clone());
Arc::new(next)
});
*GLOBAL_ROOT_CERT.write().await = Some(cert);
}
/// Clear the global root CA certificate for outbound gRPC clients.
pub async fn clear_global_root_cert() {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.root_ca_pem = None;
Arc::new(next)
});
*GLOBAL_ROOT_CERT.write().await = None;
}
/// Get the global root CA certificate for outbound gRPC clients.
pub async fn get_global_root_cert() -> Option<Vec<u8>> {
GLOBAL_ROOT_CERT.read().await.clone()
get_global_outbound_tls_snapshot().root_ca_pem.clone()
}
/// Set the global mTLS identity (cert+key PEM) for outbound gRPC clients.
@@ -105,22 +152,32 @@ pub async fn get_global_root_cert() -> Option<Vec<u8>> {
/// # Arguments
/// * `identity` - An optional MtlsIdentityPem struct containing the cert and key PEM.
pub async fn set_global_mtls_identity(identity: Option<MtlsIdentityPem>) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.mtls_identity = identity.clone();
Arc::new(next)
});
*GLOBAL_MTLS_IDENTITY.write().await = identity;
}
/// Get the global mTLS identity for outbound gRPC clients.
pub async fn get_global_mtls_identity() -> Option<MtlsIdentityPem> {
GLOBAL_MTLS_IDENTITY.read().await.clone()
get_global_outbound_tls_snapshot().mtls_identity.clone()
}
/// Set the global outbound TLS generation.
pub fn set_global_outbound_tls_generation(generation: u64) {
GLOBAL_OUTBOUND_TLS_STATE.rcu(|current| {
let mut next = (**current).clone();
next.generation = generation;
Arc::new(next)
});
GLOBAL_OUTBOUND_TLS_GENERATION.store(generation, Ordering::Relaxed);
}
/// Get the global outbound TLS generation.
pub fn get_global_outbound_tls_generation() -> u64 {
GLOBAL_OUTBOUND_TLS_GENERATION.load(Ordering::Relaxed)
get_global_outbound_tls_snapshot().generation
}
/// Evict a stale/dead connection from the global connection cache.
@@ -194,8 +251,77 @@ pub async fn clear_all_connections() {
///
/// When present, gRPC clients will present this identity to servers requesting/requiring mTLS.
/// When absent, clients proceed with standard server-authenticated TLS.
#[derive(Clone, Debug)]
#[derive(Clone, PartialEq, Eq)]
pub struct MtlsIdentityPem {
pub cert_pem: Vec<u8>,
pub key_pem: Vec<u8>,
}
impl std::fmt::Debug for MtlsIdentityPem {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("MtlsIdentityPem")
.field("cert_bytes", &self.cert_pem.len())
.field("key_pem", &"[REDACTED]")
.finish()
}
}
#[cfg(test)]
mod tls_snapshot_tests {
use super::*;
#[tokio::test]
async fn outbound_snapshot_publication_is_coherent_and_skips_unchanged_material() {
let start = get_global_outbound_tls_generation().saturating_add(1);
let snapshot = OutboundTlsSnapshot {
generation: start,
root_ca_pem: Some(vec![1]),
mtls_identity: Some(MtlsIdentityPem {
cert_pem: vec![1],
key_pem: vec![1],
}),
};
let first = publish_global_outbound_tls_snapshot(snapshot.clone()).await;
assert_eq!(
publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
generation: first.saturating_add(1),
..snapshot
})
.await,
first
);
let writer = tokio::spawn(async move {
for tag in 2_u8..=255 {
publish_global_outbound_tls_snapshot(OutboundTlsSnapshot {
generation: start.saturating_add(u64::from(tag)),
root_ca_pem: Some(vec![tag]),
mtls_identity: Some(MtlsIdentityPem {
cert_pem: vec![tag],
key_pem: vec![tag],
}),
})
.await;
tokio::task::yield_now().await;
}
});
while !writer.is_finished() {
let state = get_global_outbound_tls_snapshot();
let identity = state.mtls_identity.as_ref().expect("identity in complete snapshot");
assert_eq!(state.root_ca_pem.as_ref().expect("roots in complete snapshot"), &identity.cert_pem);
assert_eq!(identity.cert_pem, identity.key_pem);
tokio::task::yield_now().await;
}
writer.await.expect("writer completes");
}
#[test]
fn mtls_identity_debug_redacts_private_key() {
let identity = MtlsIdentityPem {
cert_pem: b"public certificate".to_vec(),
key_pem: b"PRIVATE-KEY-MARKER".to_vec(),
};
let debug = format!("{identity:?}");
assert!(!debug.contains("PRIVATE-KEY-MARKER"));
assert!(debug.contains("[REDACTED]"));
}
}
+3 -13
View File
@@ -18,10 +18,8 @@ use const_str::concat;
/// Default value: RustFS
/// Environment variable: RUSTFS_APP_NAME
pub const APP_NAME: &str = "RustFS";
/// Application version
/// Default value: 1.0.0
/// Environment variable: RUSTFS_VERSION
pub const VERSION: &str = "1.0.0";
/// Cargo package version for this workspace member.
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
/// Default configuration logger level
/// Default value: error
@@ -39,12 +37,6 @@ pub const SAMPLE_RATIO: f64 = 1.0;
/// Default value: 30
pub const METER_INTERVAL: u64 = 30;
/// Default configuration service version
/// Default value: 1.0.0
/// Environment variable: RUSTFS_OBS_SERVICE_VERSION
/// Uses the same value as VERSION constant
pub const SERVICE_VERSION: &str = "1.0.0";
/// Default configuration environment
/// Default value: production
pub const ENVIRONMENT: &str = "production";
@@ -410,7 +402,6 @@ mod tests {
APP_NAME,
VERSION,
DEFAULT_LOG_LEVEL,
SERVICE_VERSION,
ENVIRONMENT,
RUSTFS_TLS_KEY,
RUSTFS_TLS_CERT,
@@ -446,8 +437,7 @@ mod tests {
fn test_configuration_consistency() {
// Test configuration consistency
// Version consistency
assert_eq!(VERSION, SERVICE_VERSION, "Application version should match service version");
assert_eq!(VERSION, env!("CARGO_PKG_VERSION"));
// Port conflict check
let ports = [DEFAULT_PORT, DEFAULT_CONSOLE_PORT];
+6
View File
@@ -207,3 +207,9 @@ pub const ENV_TLS_RELOAD_INTERVAL: &str = "RUSTFS_TLS_RELOAD_INTERVAL";
/// Default interval for TLS certificate reload check
pub const DEFAULT_TLS_RELOAD_INTERVAL: u64 = 30;
/// Environment variable to enable experimental HTTP/3 support.
pub const ENV_HTTP3_ENABLE: &str = "RUSTFS_HTTP3_ENABLE";
/// HTTP/3 is disabled unless explicitly enabled.
pub const DEFAULT_HTTP3_ENABLE: bool = false;
+14
View File
@@ -51,6 +51,11 @@ pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_
/// PUT records multiple timers and histograms when attribution is active.
pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
/// Enables sampled in-process GetObject stage summaries without OTLP histograms.
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE";
pub const ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS: &str = "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -159,6 +164,15 @@ mod tests {
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
assert_eq!(ENV_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED, "RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED");
assert_eq!(
ENV_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE,
"RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_SAMPLE_RATE"
);
assert_eq!(
ENV_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS,
"RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_INTERVAL_SECS"
);
// Test log cleanup related env keys
assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
@@ -33,6 +33,10 @@
//! tracked separately.
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use crate::fault_proxy::FaultMode;
use aws_sdk_s3::Client;
use std::time::Duration;
use tokio::time::{Instant, timeout};
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
@@ -144,7 +148,99 @@ async fn cluster_volume_fault_proxy_pass_smoke() -> TestResult {
cluster.start().await?;
cluster.create_test_bucket(BUCKET).await?;
let payload = vec![0x6Du8; 256 * 1024];
put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await
put_get_roundtrip(&cluster, "volume-proxy/object", &payload).await?;
// Node 1 owns only two disks; a successful three-vote write must
// authenticate at least one disk mutation through node 0's proxy.
let client = cluster.create_s3_client(1)?;
let client = Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
);
let forwarded_before = proxy.forwarded_bytes();
client
.put_object()
.bucket(BUCKET)
.key("volume-proxy/authenticated")
.body(bytes::Bytes::copy_from_slice(&payload).into())
.send()
.await?;
assert!(
proxy.forwarded_bytes() > forwarded_before,
"successful peer-disk publication must traverse the proxy"
);
let dropped_before = proxy.dropped_bytes();
proxy.set_mode(FaultMode::Blackhole);
assert_eq!(proxy.mode(), FaultMode::Blackhole);
let error = timeout(
Duration::from_secs(30),
client
.put_object()
.bucket(BUCKET)
.key("volume-proxy/rejected")
.body(bytes::Bytes::copy_from_slice(&payload).into())
.send(),
)
.await?
.expect_err("two local disks cannot commit a write while the peer is blackholed");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
assert!(
proxy.dropped_bytes() > dropped_before,
"the injected blackhole must discard actual peer traffic"
);
proxy.set_mode(FaultMode::Pass);
assert_eq!(proxy.mode(), FaultMode::Pass);
let http = reqwest::Client::builder()
.no_proxy()
.timeout(Duration::from_secs(3))
.build()?;
let deadline = Instant::now() + Duration::from_secs(30);
loop {
let response = http.get(format!("{}/health/ready", cluster.nodes[1].url)).send().await?;
let status = response.status();
let body: serde_json::Value = response.json().await?;
// Reachable peers can still have Returning disk handles that reject
// writes. Wait for this node's live writer inventory to be Online.
if status.as_u16() == 200
&& body["details"]["storage"]["source"] == "local_runtime"
&& body["details"]["storage"]["writeQuorum"] == true
&& body["details"]["storage"]["unavailableDrives"]
.as_array()
.is_some_and(Vec::is_empty)
&& body["details"]["lock"]["ready"] == true
{
break;
}
assert!(Instant::now() < deadline, "peer write quorum did not recover: HTTP {status}, {body}");
tokio::time::sleep(Duration::from_millis(200)).await;
}
client
.put_object()
.bucket(BUCKET)
.key("volume-proxy/recovered")
.body(bytes::Bytes::copy_from_slice(&payload).into())
.send()
.await?;
for key in ["volume-proxy/object", "volume-proxy/authenticated", "volume-proxy/recovered"] {
let got = client.get_object().bucket(BUCKET).key(key).send().await?;
let body = got.body.collect().await?.into_bytes();
assert_eq!(body.as_ref(), payload, "proxy recovery must preserve the full object body for {key}");
}
let rejected = client
.head_object()
.bucket(BUCKET)
.key("volume-proxy/rejected")
.send()
.await
.expect_err("a rejected proxied write must never become visible after recovery");
assert_eq!(rejected.raw_response().map(|response| response.status().as_u16()), Some(404));
Ok(())
}
.await;
+29 -2
View File
@@ -35,7 +35,7 @@ use s3s::Body;
use serde_json;
use std::fs as stdfs;
use std::io::ErrorKind;
use std::net::SocketAddr;
use std::net::{Ipv6Addr, SocketAddr};
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Stdio};
use std::sync::Once;
@@ -1442,7 +1442,11 @@ impl RustFSTestClusterEnvironment {
return Err(format!("a volume proxy is already configured for node {node_idx}").into());
}
let target = self.nodes[node_idx].address.parse::<SocketAddr>()?;
let proxy = crate::fault_proxy::FaultProxy::start(target).await?;
// Endpoint locality and RPC audience both use the advertised authority.
// Keep the node's port while the second loopback IP separates the proxy
// listener from the direct IPv4 server and S3 client address.
let listen = SocketAddr::from((Ipv6Addr::LOCALHOST, target.port()));
let proxy = crate::fault_proxy::FaultProxy::start_on(listen, target).await?;
self.volume_proxy_addresses[node_idx] = Some(proxy.local_addr());
Ok(proxy)
}
@@ -1538,6 +1542,15 @@ impl RustFSTestClusterEnvironment {
self.start_node_from_binary(node_idx, &binary_path).await
}
/// Start a process and wait only for TCP, allowing tests to bring up a
/// quorum one node at a time before the S3 service can become ready.
pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let binary_path = rustfs_binary_path();
let volumes_arg = self.build_volumes_arg();
self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
}
/// Start one stopped cluster node with a specific RustFS binary while
/// preserving the cluster's volume layout and that node's data directory.
pub async fn start_node_from_binary(
@@ -2467,6 +2480,20 @@ mod tests {
assert!(volumes.contains(&proxied), "volumes must use the proxy address: {volumes}");
assert!(!volumes.contains(&direct), "volumes must not retain the direct address: {volumes}");
assert_eq!(proxy.local_addr().port(), proxy.target_addr().port());
assert_ne!(proxy.local_addr().ip(), proxy.target_addr().ip());
assert_eq!(env.nodes[0].address, direct, "S3 clients must retain the direct address");
let mut endpoint =
crate::storage_api::Endpoint::try_from(volumes.as_str()).expect("proxied volume should be a valid endpoint");
endpoint
.update_is_local(proxy.target_addr().port())
.expect("proxied endpoint locality should resolve");
assert!(endpoint.is_local, "the proxied node must keep its local disk and first-disk authority");
assert_eq!(
endpoint.host_port(),
proxied,
"the local RPC audience must be the advertised proxy authority"
);
proxy.shutdown().await;
}
@@ -31,6 +31,7 @@ use super::harness::{
use crate::common::{
AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
};
use crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use std::path::{Path, PathBuf};
@@ -400,6 +401,19 @@ async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> Tes
Ok(())
}
async fn replace_node_and_wait_for_write_readiness(dist: &mut DistCluster, seed: &UpgradeSeed, node_idx: usize) -> TestResult {
dist.replace_node_with_current_binary(node_idx).await?;
// Node readiness proves reads can succeed; a rolling writer must also
// recover before replacing the next peer or checking upgrade contracts.
wait_for_upgrade_write_readiness(
&dist.clients()?,
&seed.history_bucket,
&format!("replaced-node-{node_idx}"),
CREDENTIAL_TIMEOUT,
)
.await
}
#[tokio::test]
async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> TestResult {
init_logging();
@@ -410,15 +424,15 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
let seed = seed_history_and_iam(&dist).await?;
dist.replace_node_with_current_binary(0).await?;
replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
assert_history_and_iam(&dist, &seed, "one-current-node").await?;
for node_idx in [1, 2] {
dist.replace_node_with_current_binary(node_idx).await?;
replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
}
assert_history_and_iam(&dist, &seed, "one-previous-node").await?;
dist.replace_node_with_current_binary(3).await?;
replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
Ok(())
}
+379 -14
View File
@@ -50,8 +50,8 @@ use std::collections::{BTreeMap, HashMap, VecDeque};
use std::error::Error;
use std::io;
use std::net::SocketAddr;
use std::sync::{Arc, Mutex, MutexGuard};
use std::time::{Duration, SystemTime};
use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use tokio::net::TcpListener;
use tokio::sync::{OwnedSemaphorePermit, Semaphore, watch};
use tokio::task::JoinHandle;
@@ -363,6 +363,14 @@ pub struct RequestRecord {
pub upload_id: Option<String>,
pub part_number: Option<i32>,
pub content_length: Option<u64>,
pub source_version_id: Option<String>,
pub source_mtime: Option<String>,
pub source_etag: Option<String>,
pub source_replication_request: bool,
pub journaled_at_unix_millis: Option<u64>,
/// Computed by the target, before response stall/disconnect faults. This
/// does not prove that any response bytes reached the client.
pub prepared_response: Option<PreparedResponse>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub proxy_headers: ProxyHeaderSnapshot,
@@ -381,6 +389,20 @@ pub struct RequestRecord {
pub fault: Option<FaultAction>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PreparedResponse {
pub status: u16,
pub version_id: Option<String>,
pub etag: Option<String>,
pub last_modified: Option<String>,
pub content_length: Option<u64>,
pub prepared_at_unix_millis: Option<u64>,
pub elapsed: Duration,
}
#[derive(Clone, Default)]
struct RequestSequence(Arc<OnceLock<u64>>);
/// Integrity and framing headers of an upload. A remote target's acceptance
/// rules key off exactly these (rustfs#6853: `aws-chunked` framing stored
/// verbatim; rustfs#7082: Object Lock PUTs need `Content-MD5` or
@@ -430,6 +452,10 @@ impl TransportSnapshot {
#[derive(Debug, Clone, Default, PartialEq, Eq)]
struct JournaledHeaders {
content_length: Option<u64>,
source_version_id: Option<String>,
source_mtime: Option<String>,
source_etag: Option<String>,
source_replication_request: bool,
replication_timestamps: ReplicationTimestampHeaders,
proxy_headers: ProxyHeaderSnapshot,
transport: TransportSnapshot,
@@ -444,6 +470,10 @@ impl JournaledHeaders {
.get(CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok()),
source_version_id: header_value(headers, &SOURCE_VERSION_ID_HEADERS).map(bounded_journal_value),
source_mtime: header_value(headers, &SOURCE_MTIME_HEADERS).map(bounded_journal_value),
source_etag: header_value(headers, &SOURCE_ETAG_HEADERS).map(bounded_journal_value),
source_replication_request: header_value(headers, &SOURCE_REPLICATION_REQUEST_HEADERS).as_deref() == Some("true"),
replication_timestamps: ReplicationTimestampHeaders::from_headers(headers),
proxy_headers: ProxyHeaderSnapshot::from_headers(headers),
transport: TransportSnapshot::from_headers(headers),
@@ -511,6 +541,10 @@ struct StoreState {
/// version id the target never had answers 404 `NoSuchVersion` instead of
/// the idempotent 204 RustFS/MinIO give.
reject_unknown_version_deletes: bool,
/// Models MinIO and its derivatives (Storj): `X-Minio-Internal-` is a reserved
/// metadata prefix, so a PutObject or UploadPart carrying any such request
/// header is refused with `InvalidArgument` (rustfs#8362).
reject_reserved_internal_headers: bool,
limits: StoreLimits,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
@@ -738,6 +772,7 @@ impl FakeS3Target {
let connection_limit = Arc::new(Semaphore::new(MAX_CONNECTIONS));
let (active_connections_tx, active_connections) = watch::channel(0usize);
let task_connection_limit = Arc::clone(&connection_limit);
let task_control = Arc::clone(&control);
let task = tokio::spawn(async move {
let mut connections = tokio::task::JoinSet::new();
@@ -751,6 +786,7 @@ impl FakeS3Target {
};
active_connections_tx.send_modify(|count| *count += 1);
let service = service.clone();
let control = Arc::clone(&task_control);
let active_connections = active_connections_tx.clone();
let mut connection_gate = connection_gate.clone();
connections.spawn(async move {
@@ -764,7 +800,7 @@ impl FakeS3Target {
{
return;
}
let handler = service_fn(move |request| handle_request(request, service.clone()));
let handler = service_fn(move |request| handle_request(request, service.clone(), Arc::clone(&control)));
let mut connection = http1::Builder::new();
connection
.keep_alive(false)
@@ -975,6 +1011,43 @@ impl FakeS3Target {
lock(&self.backend.store).reject_unknown_version_deletes = enabled;
}
/// Refuse every PutObject / UploadPart that carries an `x-minio-internal-*`
/// request header, the way MinIO and Storj do.
pub fn reject_reserved_internal_headers(&self, enabled: bool) {
lock(&self.backend.store).reject_reserved_internal_headers = enabled;
}
/// Keys of every stored, non-delete object in the bucket, sorted.
pub fn stored_keys(&self, bucket: &str) -> Vec<String> {
let state = lock(&self.backend.store);
let mut keys: Vec<String> = state
.buckets
.get(bucket)
.map(|bucket| {
bucket
.objects
.iter()
.filter(|(_, versions)| versions.last().is_some_and(|version| !version.delete_marker))
.map(|(key, _)| key.clone())
.collect()
})
.unwrap_or_default();
keys.sort();
keys
}
/// Body and user metadata (names as stored, without the `x-amz-meta-` prefix)
/// of the latest non-delete version of an object.
pub fn stored_object(&self, bucket: &str, key: &str) -> Option<(Bytes, HashMap<String, String>)> {
lock(&self.backend.store)
.buckets
.get(bucket)
.and_then(|bucket| bucket.objects.get(key))
.and_then(|versions| versions.last())
.filter(|version| !version.delete_marker)
.map(|version| (version.body.clone(), version.metadata.clone().unwrap_or_default()))
}
pub fn require_checksum_for_object_lock(&self, enabled: bool) {
lock(&self.backend.store).require_checksum_for_object_lock = enabled;
}
@@ -1161,7 +1234,10 @@ impl S3Access for FaultAccess {
let operation = operation_from_s3_name(context.s3_op().name());
let journaled = JournaledHeaders::from_headers(context.headers());
let aws_chunked = journaled.transport.aws_chunked;
let fault = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
let (sequence, fault) = record_request(&self.control, operation, context.method().clone(), parsed, journaled);
if let Some(slot) = context.extensions_mut().get::<RequestSequence>() {
let _ = slot.0.set(sequence);
}
if let Some(status) = fault.as_ref().and_then(|fault| scripted_status(&fault.action)) {
return Err(scripted_status_error(status));
}
@@ -1174,6 +1250,15 @@ impl S3Access for FaultAccess {
"this target does not decode aws-chunked request bodies; send a plain signed payload with an exact Content-Length"
));
}
if matches!(operation, Operation::PutObject | Operation::UploadPart)
&& lock(&self.store).reject_reserved_internal_headers
&& context
.headers()
.keys()
.any(|name| name.as_str().starts_with("x-minio-internal-"))
{
return Err(s3s::s3_error!(InvalidArgument, "Your metadata headers are not supported."));
}
let prebody_permit = if operation == Operation::CompleteMultipartUpload {
Some(
timeout(
@@ -1228,7 +1313,7 @@ fn record_request(
method: Method,
parsed: ParsedRequest,
headers: JournaledHeaders,
) -> Option<RequestFault> {
) -> (u64, Option<RequestFault>) {
let mut state = lock(control);
let action = parsed
.key
@@ -1251,6 +1336,12 @@ fn record_request(
upload_id: parsed.upload_id.map(bounded_journal_value),
part_number: parsed.part_number,
content_length: headers.content_length,
source_version_id: headers.source_version_id,
source_mtime: headers.source_mtime,
source_etag: headers.source_etag,
source_replication_request: headers.source_replication_request,
journaled_at_unix_millis: unix_millis(),
prepared_response: None,
consumed_bytes: None,
replication_timestamps: headers.replication_timestamps,
proxy_headers: headers.proxy_headers,
@@ -1261,7 +1352,14 @@ fn record_request(
continuation_token: parsed.continuation_token.map(bounded_journal_value),
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
(sequence, action.map(|action| RequestFault { sequence, action }))
}
fn unix_millis() -> Option<u64> {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.ok()
.and_then(|elapsed| u64::try_from(elapsed.as_millis()).ok())
}
fn bounded_journal_value(mut value: String) -> String {
@@ -1269,14 +1367,31 @@ fn bounded_journal_value(mut value: String) -> String {
while !value.is_char_boundary(end) {
end -= 1;
}
value.truncate(end);
value
if value.capacity() > MAX_RETAINED_IDENTIFIER_BYTES {
value[..end].to_owned()
} else {
value.truncate(end);
value
}
}
async fn handle_request(request: Request<Incoming>, service: S3Service) -> Result<Response<Body>, BoxError> {
let mut response = timeout(MAX_REQUEST_DURATION, call_s3(service, request.map(Body::from)))
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
async fn handle_request(
mut request: Request<Incoming>,
service: S3Service,
control: Arc<Mutex<ControlState>>,
) -> Result<Response<Body>, BoxError> {
let started = Instant::now();
let sequence = RequestSequence::default();
request.extensions_mut().insert(sequence.clone());
let mut response = timeout(MAX_REQUEST_DURATION, async {
let result = call_s3(service, request.map(Body::from)).await;
if let (Ok(response), Some(sequence)) = (&result, sequence.0.get()) {
update_prepared_response(&control, *sequence, response, started.elapsed());
}
result
})
.await
.map_err(|_| io::Error::new(io::ErrorKind::TimedOut, "fake target request exceeded 65 seconds"))??;
if response.headers_mut().remove(DISCONNECT_HEADER).is_some() {
return Err(io::Error::new(io::ErrorKind::ConnectionAborted, "fake target scripted disconnect").into());
}
@@ -1531,6 +1646,25 @@ fn update_consumed(control: &Mutex<ControlState>, sequence: u64, consumed: usize
}
}
fn update_prepared_response(control: &Mutex<ControlState>, sequence: u64, response: &Response<Body>, elapsed: Duration) {
let prepared = PreparedResponse {
status: response.status().as_u16(),
version_id: header_value(response.headers(), &["x-amz-version-id"]).map(bounded_journal_value),
etag: header_value(response.headers(), &[ETAG.as_str()]).map(bounded_journal_value),
last_modified: header_value(response.headers(), &[LAST_MODIFIED.as_str()]).map(bounded_journal_value),
content_length: response
.headers()
.get(CONTENT_LENGTH)
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok()),
prepared_at_unix_millis: unix_millis(),
elapsed,
};
if let Some(record) = lock(control).requests.iter_mut().find(|record| record.sequence == sequence) {
record.prepared_response = Some(prepared);
}
}
async fn apply_non_body_fault(fault: Option<&RequestFault>, control: &Mutex<ControlState>) -> S3Result<()> {
match fault.map(|fault| &fault.action) {
Some(action @ (FaultAction::Status(_) | FaultAction::ResponseStatus(_))) => {
@@ -3230,10 +3364,38 @@ mod tests {
let retry = client.clone();
requests.spawn(async move { get_bytes(&retry, bucket, "held", None).await });
timeout(Duration::from_secs(2), gate.entered.wait_for(|count| *count == 2)).await??;
let held_retry = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("held"))
.expect("the held retry must be journaled");
assert_eq!(held_retry.prepared_response, None);
assert_eq!(
timeout(Duration::from_secs(2), get_bytes(&client, bucket, "unrelated", None)).await??,
Bytes::from_static(b"payload")
);
let unrelated = target
.requests()
.into_iter()
.find(|record| record.operation == Operation::GetObject && record.key.as_deref() == Some("unrelated"))
.expect("the unrelated GET must be journaled");
let unrelated_response = unrelated.prepared_response.expect("the unrelated GET prepared a response");
assert_eq!(unrelated_response.status, 200);
assert_eq!(
unrelated_response.version_id.as_deref(),
Some(target.stored_versions(bucket, "unrelated")[0].0.as_str())
);
assert!(held_retry.sequence < unrelated.sequence);
assert_eq!(
target
.requests()
.into_iter()
.find(|record| record.sequence == held_retry.sequence)
.expect("the held retry remains journaled")
.prepared_response,
None
);
assert!(requests.try_join_next().is_none(), "the retry must remain behind the gate");
drop(gate);
assert_eq!(
@@ -3242,6 +3404,18 @@ mod tests {
.expect("retried GET task")??,
Bytes::from_static(b"payload")
);
let held_response = target
.requests()
.into_iter()
.find(|record| record.sequence == held_retry.sequence)
.and_then(|record| record.prepared_response)
.expect("the released retry prepared its own response");
assert_eq!(held_response.status, 200);
assert_eq!(
held_response.version_id.as_deref(),
Some(target.stored_versions(bucket, "held")[0].0.as_str())
);
assert_ne!(held_response.version_id, unrelated_response.version_id);
assert_eq!(get_bytes(&client, bucket, "held", None).await?, Bytes::from_static(b"payload"));
assert_eq!(target.count_requests(Operation::GetObject, "held"), 3);
Ok(())
@@ -3586,6 +3760,23 @@ mod tests {
.await?;
assert_eq!(first_head.content_length(), Some(5));
assert_eq!(first_head.version_id(), first.version_id.as_deref());
let journaled_head = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::HeadObject)
.expect("the version HEAD must be journaled");
assert_eq!(journaled_head.version_id, first.version_id);
assert!(journaled_head.journaled_at_unix_millis.is_some());
let prepared_head = journaled_head
.prepared_response
.expect("the version HEAD prepared a response");
assert_eq!(prepared_head.status, 200);
assert_eq!(prepared_head.version_id.as_deref(), first_head.version_id());
assert_eq!(prepared_head.etag.as_deref(), first_head.e_tag());
assert_eq!(prepared_head.content_length, Some(5));
assert!(prepared_head.last_modified.is_some());
assert!(prepared_head.prepared_at_unix_millis.is_some());
assert_eq!(
get_bytes(&client, "target-bucket", "nested/key", None).await?,
Bytes::from_static(b"second")
@@ -3611,6 +3802,20 @@ mod tests {
.await
.expect_err("deleted version must not remain readable");
assert_sdk_error!(deleted, 404, "NoSuchVersion");
let deleted_record = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::GetObject)
.expect("the deleted version GET must be journaled");
assert_eq!(deleted_record.version_id, first.version_id);
assert_eq!(
deleted_record
.prepared_response
.expect("the error response was prepared")
.status,
404
);
let marker = client
.delete_object()
.bucket("target-bucket")
@@ -4042,7 +4247,21 @@ mod tests {
.await
.expect_err("wrong part ETag must fail without consuming the upload");
assert_sdk_error!(invalid_complete, 400, "InvalidPart");
client
let rejected_complete = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::CompleteMultipartUpload)
.expect("the rejected completion must be journaled");
assert_eq!(rejected_complete.upload_id.as_deref(), Some(upload_id.as_str()));
assert_eq!(
rejected_complete
.prepared_response
.expect("the rejection was prepared")
.status,
400
);
let successful_complete = client
.complete_multipart_upload()
.bucket("target-bucket")
.key("multipart")
@@ -4050,6 +4269,18 @@ mod tests {
.multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(parts)).build())
.send()
.await?;
let completed_record = target
.requests()
.into_iter()
.rev()
.find(|record| record.operation == Operation::CompleteMultipartUpload)
.expect("the successful completion must be journaled");
assert_eq!(completed_record.upload_id.as_deref(), Some(upload_id.as_str()));
let prepared_complete = completed_record
.prepared_response
.expect("the completion response was prepared");
assert_eq!(prepared_complete.status, 200);
assert_eq!(prepared_complete.version_id.as_deref(), successful_complete.version_id());
let completed = get_bytes(&client, "target-bucket", "multipart", None).await?;
assert_eq!(completed.len(), MIN_MULTIPART_PART_BYTES + 5);
assert_eq!(&completed[..MIN_MULTIPART_PART_BYTES], first_body.as_ref());
@@ -4430,6 +4661,32 @@ mod tests {
.send()
.await?;
assert_eq!(completed.version_id(), Some(completed_version.as_str()));
let receipt_records = target.requests();
let receipt_create = receipt_records
.iter()
.find(|record| {
record.operation == Operation::CreateMultipartUpload && record.key.as_deref() == Some("multipart-receipt-time")
})
.expect("the source version must be retained on MPU creation");
assert_eq!(receipt_create.source_version_id.as_deref(), Some(completed_version.as_str()));
let receipt_complete = receipt_records
.iter()
.find(|record| {
record.operation == Operation::CompleteMultipartUpload
&& record.upload_id.as_deref() == Some(receipt_upload.as_str())
})
.expect("the source mtime must be retained on completion");
assert_eq!(receipt_complete.source_mtime.as_deref(), Some("1970-01-01T00:00:00Z"));
assert!(receipt_complete.source_replication_request);
assert_eq!(
receipt_complete
.prepared_response
.as_ref()
.expect("the receipt-time response was prepared")
.version_id
.as_deref(),
completed.version_id()
);
assert_eq!(
get_bytes(&client, "target-bucket", "multipart-receipt-time", None).await?,
Bytes::from_static(b"completed"),
@@ -4648,6 +4905,23 @@ mod tests {
.expect_err("disconnected request must not store an object");
assert_sdk_error!(disconnected, 404, "NotFound");
target.inject_for_key(Operation::PutObject, "lost-response", FaultAction::DisconnectAfterResponse, 1);
client
.put_object()
.bucket("target-bucket")
.key("lost-response")
.body(ByteStream::from_static(b"stored"))
.send()
.await
.expect_err("a prepared response does not prove client delivery");
let lost_response_head = client
.head_object()
.bucket("target-bucket")
.key("lost-response")
.send()
.await?;
assert_eq!(lost_response_head.content_length(), Some(6));
assert_eq!(
parse_request(&Method::PUT, &"/target-bucket/key?uploadId=upload&partNumber=bad".parse::<Uri>()?).operation,
Operation::Unknown
@@ -4661,6 +4935,28 @@ mod tests {
.count(),
4
);
assert!(
requests
.iter()
.filter(|request| request.fault == Some(FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE)))
.all(|request| {
request
.prepared_response
.as_ref()
.is_some_and(|response| response.status == 503)
})
);
let lost_response = requests
.iter()
.find(|request| request.operation == Operation::PutObject && request.key.as_deref() == Some("lost-response"))
.expect("the disconnected response must be journaled");
assert_eq!(lost_response.fault, Some(FaultAction::DisconnectAfterResponse));
let prepared_lost_response = lost_response
.prepared_response
.as_ref()
.expect("the target prepared the successful response");
assert_eq!(prepared_lost_response.status, 200);
assert_eq!(prepared_lost_response.version_id.as_deref(), lost_response_head.version_id());
assert!(
requests
.iter()
@@ -4851,8 +5147,44 @@ mod tests {
assert_eq!(records.len(), MAX_REQUEST_RECORDS);
assert_eq!(records.front().map(|record| record.sequence), Some(2));
assert_eq!(records.back().map(|record| record.sequence), Some((MAX_REQUEST_RECORDS + 1) as u64));
let mut oversized_allocation = String::with_capacity(MAX_RETAINED_IDENTIFIER_BYTES * 16);
oversized_allocation.push_str("short");
let short_value = bounded_journal_value(oversized_allocation);
assert_eq!(short_value, "short");
assert!(short_value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
let mut response = Response::new(Body::empty());
for name in ["x-amz-version-id", "etag", "last-modified"] {
response.headers_mut().insert(
name,
HeaderValue::from_str(&"r".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
);
}
response.headers_mut().insert(CONTENT_LENGTH, HeaderValue::from_static("17"));
response
.headers_mut()
.insert("authorization", HeaderValue::from_static("unrecorded-response-authorization"));
response.headers_mut().insert(
"x-amz-server-side-encryption-customer-key",
HeaderValue::from_static("unrecorded-response-customer-key"),
);
update_prepared_response(&control, 1, &response, Duration::ZERO);
update_prepared_response(&control, u64::MAX, &response, Duration::ZERO);
assert_eq!(lock(&control).requests, records, "misses and evicted records must not be resurrected");
let bounded_control = Mutex::new(ControlState::default());
let utf8_boundary = format!("{}é", "a".repeat(MAX_RETAINED_IDENTIFIER_BYTES - 1));
let mut source_headers = HeaderMap::new();
for name in ["x-rustfs-source-version-id", "x-minio-source-mtime", "x-rustfs-source-etag"] {
source_headers.insert(
name,
HeaderValue::from_str(&"s".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)).expect("header"),
);
}
source_headers.insert("x-minio-source-replication-request", HeaderValue::from_static("true"));
source_headers.insert("authorization", HeaderValue::from_static("unrecorded-request-authorization"));
source_headers.insert(
"x-amz-server-side-encryption-customer-key",
HeaderValue::from_static("unrecorded-request-customer-key"),
);
record_request(
&bounded_control,
Operation::GetObject,
@@ -4865,8 +5197,9 @@ mod tests {
upload_id: Some("u".repeat(MAX_RETAINED_IDENTIFIER_BYTES + 1)),
..Default::default()
},
JournaledHeaders::default(),
JournaledHeaders::from_headers(&source_headers),
);
update_prepared_response(&bounded_control, 1, &response, Duration::from_millis(7));
{
let bounded_records = lock(&bounded_control);
let bounded = &bounded_records.requests[0];
@@ -4874,7 +5207,39 @@ mod tests {
assert_eq!(bounded.key.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES - 1));
assert_eq!(bounded.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.upload_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.source_version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.source_mtime.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(bounded.source_etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
for value in [&bounded.source_version_id, &bounded.source_mtime, &bounded.source_etag]
.into_iter()
.flatten()
{
assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
}
assert!(bounded.source_replication_request);
let prepared = bounded
.prepared_response
.as_ref()
.expect("the retained record receives its response");
assert_eq!(prepared.version_id.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(prepared.etag.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
assert_eq!(prepared.last_modified.as_ref().map(String::len), Some(MAX_RETAINED_IDENTIFIER_BYTES));
for value in [&prepared.version_id, &prepared.etag, &prepared.last_modified]
.into_iter()
.flatten()
{
assert!(value.capacity() <= MAX_RETAINED_IDENTIFIER_BYTES);
}
assert_eq!(prepared.content_length, Some(17));
assert_eq!(prepared.elapsed, Duration::from_millis(7));
assert!(
!format!("{bounded:?}").contains("unrecorded-"),
"authorization and customer keys must not enter the journal"
);
}
lock(&bounded_control).requests.clear();
update_prepared_response(&bounded_control, 1, &response, Duration::ZERO);
assert!(lock(&bounded_control).requests.is_empty(), "a drained record must not be resurrected");
let target = FakeS3Target::start().await.expect("target");
target.create_bucket("b".repeat(63));
let empty_bucket = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
+49 -6
View File
@@ -61,6 +61,8 @@
use std::io;
use std::net::{Ipv4Addr, SocketAddr};
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
@@ -108,19 +110,33 @@ pub struct FaultProxy {
mode_tx: watch::Sender<FaultMode>,
shutdown_tx: watch::Sender<bool>,
accept_task: JoinHandle<()>,
traffic: Arc<ProxyTraffic>,
}
#[derive(Default)]
struct ProxyTraffic {
forwarded_bytes: AtomicUsize,
dropped_bytes: AtomicUsize,
}
impl FaultProxy {
/// Bind a listener on `127.0.0.1:0` and start forwarding accepted
/// connections to `target`. Starts in [`FaultMode::Pass`].
pub async fn start(target: SocketAddr) -> io::Result<Self> {
let listener = TcpListener::bind((Ipv4Addr::LOCALHOST, 0)).await?;
Self::start_on(SocketAddr::from((Ipv4Addr::LOCALHOST, 0)), target).await
}
/// Bind a specific address so advertised volume endpoints can retain the
/// server's port and authenticated RPC identity on another loopback IP.
pub async fn start_on(listen: SocketAddr, target: SocketAddr) -> io::Result<Self> {
let listener = TcpListener::bind(listen).await?;
let listen_addr = listener.local_addr()?;
let (mode_tx, mode_rx) = watch::channel(FaultMode::Pass);
let (shutdown_tx, shutdown_rx) = watch::channel(false);
let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx));
let traffic = Arc::new(ProxyTraffic::default());
let accept_task = tokio::spawn(accept_loop(listener, target, mode_rx, shutdown_rx, traffic.clone()));
Ok(Self {
listen_addr,
@@ -128,9 +144,20 @@ impl FaultProxy {
mode_tx,
shutdown_tx,
accept_task,
traffic,
})
}
/// Bytes actually forwarded in either direction, excluding dropped traffic.
pub fn forwarded_bytes(&self) -> usize {
self.traffic.forwarded_bytes.load(Ordering::Relaxed)
}
/// Bytes actually discarded by a blackhole or one-way partition.
pub fn dropped_bytes(&self) -> usize {
self.traffic.dropped_bytes.load(Ordering::Relaxed)
}
/// Local address the proxy is listening on. Point clients here instead of
/// the real target to route their traffic through the proxy.
pub fn local_addr(&self) -> SocketAddr {
@@ -171,6 +198,7 @@ async fn accept_loop(
target: SocketAddr,
mode_rx: watch::Receiver<FaultMode>,
mut shutdown_rx: watch::Receiver<bool>,
traffic: Arc<ProxyTraffic>,
) {
loop {
tokio::select! {
@@ -188,6 +216,7 @@ async fn accept_loop(
target,
mode_rx.clone(),
shutdown_rx.clone(),
traffic.clone(),
));
}
// A transient accept error should not tear the proxy down.
@@ -205,6 +234,7 @@ async fn handle_connection(
target: SocketAddr,
mode_rx: watch::Receiver<FaultMode>,
shutdown_rx: watch::Receiver<bool>,
traffic: Arc<ProxyTraffic>,
) {
let server = match TcpStream::connect(target).await {
Ok(s) => s,
@@ -221,8 +251,9 @@ async fn handle_connection(
server_wr,
mode_rx.clone(),
shutdown_rx.clone(),
traffic.clone(),
));
let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx));
let down = tokio::spawn(pump(Direction::ServerToClient, server_rd, client_wr, mode_rx, shutdown_rx, traffic));
let _ = up.await;
let _ = down.await;
@@ -237,6 +268,7 @@ async fn pump<R, W>(
mut to: W,
mut mode_rx: watch::Receiver<FaultMode>,
mut shutdown_rx: watch::Receiver<bool>,
traffic: Arc<ProxyTraffic>,
) where
R: AsyncReadExt + Unpin,
W: AsyncWriteExt + Unpin,
@@ -257,15 +289,22 @@ async fn pump<R, W>(
let mode = *mode_rx.borrow_and_update();
match mode {
// Whole connection is a black hole: drain and drop.
FaultMode::Blackhole => continue,
FaultMode::Blackhole => {
traffic.dropped_bytes.fetch_add(n, Ordering::Relaxed);
continue;
}
// This direction is partitioned off: drop its bytes; the peer keeps
// reading nothing while the other direction may still flow.
FaultMode::Partition(blocked) if blocked == dir => continue,
FaultMode::Partition(blocked) if blocked == dir => {
traffic.dropped_bytes.fetch_add(n, Ordering::Relaxed);
continue;
}
FaultMode::Latency(delay) => {
tokio::time::sleep(delay).await;
if to.write_all(&buf[..n]).await.is_err() {
break;
}
traffic.forwarded_bytes.fetch_add(n, Ordering::Relaxed);
if to.flush().await.is_err() {
break;
}
@@ -274,6 +313,7 @@ async fn pump<R, W>(
if to.write_all(&buf[..n]).await.is_err() {
break;
}
traffic.forwarded_bytes.fetch_add(n, Ordering::Relaxed);
if to.flush().await.is_err() {
break;
}
@@ -285,7 +325,6 @@ async fn pump<R, W>(
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
use std::time::Instant;
use tokio::sync::Mutex;
@@ -375,6 +414,8 @@ mod tests {
assert_eq!(got, payload, "pass mode must forward bytes unchanged");
assert_eq!(echo.received().await, payload, "server must have received the bytes");
assert_eq!(proxy.forwarded_bytes(), payload.len() * 2);
assert_eq!(proxy.dropped_bytes(), 0);
proxy.shutdown().await;
}
@@ -414,6 +455,8 @@ mod tests {
assert!(got.is_empty(), "blackhole mode must not return any bytes, got {got:?}");
assert!(echo.received().await.is_empty(), "blackhole must not forward to the server");
assert_eq!(proxy.forwarded_bytes(), 0);
assert_eq!(proxy.dropped_bytes(), b"into-the-void".len());
// Proxy is still alive and steerable after a blackholed connection.
proxy.set_mode(FaultMode::Pass);
@@ -1191,11 +1191,59 @@ mod tests {
.into())
}
fn scanner_cycle_completion_budget(
cycle_end: u64,
interval: Option<u64>,
now: u64,
active: bool,
) -> Result<Duration, &'static str> {
if cycle_end > now {
return Err("scanner cycle timestamp is in the future");
}
if active {
return Ok(Duration::from_secs(60));
}
let interval = interval
.filter(|interval| *interval != 0)
.ok_or("scanner schedule has no positive interval")?;
let next_cycle = cycle_end
.checked_add(interval)
.ok_or("scanner schedule timestamp overflows")?;
let budget = next_cycle
.saturating_sub(now)
.checked_add(60)
.ok_or("scanner completion budget overflows")?;
Ok(Duration::from_secs(budget))
}
#[test]
fn scanner_cycle_completion_budget_accounts_for_captured_retry_schedule() {
assert_eq!(
scanner_cycle_completion_budget(1_791_815_390, Some(256), 1_791_815_447, false),
Ok(Duration::from_secs(259)),
);
assert_eq!(scanner_cycle_completion_budget(100, Some(1), 110, false), Ok(Duration::from_secs(60)));
}
#[test]
fn scanner_cycle_completion_budget_does_not_grant_planned_sleep_to_active_cycle() {
assert_eq!(scanner_cycle_completion_budget(100, Some(256), 110, true), Ok(Duration::from_secs(60)));
assert_eq!(scanner_cycle_completion_budget(0, None, 110, true), Ok(Duration::from_secs(60)));
}
#[test]
fn scanner_cycle_completion_budget_rejects_invalid_schedule() {
for (end, interval, now) in [(101, 1, 100), (100, 0, 100), (u64::MAX, 1, u64::MAX), (1, u64::MAX - 1, 1)] {
assert!(scanner_cycle_completion_budget(end, Some(interval), now, false).is_err());
}
}
async fn wait_for_scanner_cycle_after(
cluster: &RustFSTestClusterEnvironment,
previous_cycle_end: u64,
) -> Result<u64, Box<dyn Error + Send + Sync>> {
let deadline = Instant::now() + Duration::from_secs(60);
let mut deadline = Instant::now() + Duration::from_secs(60);
let mut schedule_accounted = false;
loop {
let mut latest_cycle_end = 0;
let mut versions_observed = false;
@@ -1237,14 +1285,36 @@ mod tests {
let versions_scanned = metrics["versions_scanned"]
.as_u64()
.ok_or("scanner status is missing its version-coverage counter")?;
latest_cycle_end = latest_cycle_end.max(cycle_end);
let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
if cycle_end > now {
return Err("scanner cycle timestamp is in the future".into());
}
let schedule = &status["cycle_schedule"];
if !schedule_accounted && schedule["execution_role"].as_str() == Some("leader") {
let active = metrics["current_cycle_active"]
.as_bool()
.ok_or("scanner status is missing its active-cycle flag")?;
if active || (cycle_end != 0 && schedule["effective_interval_available"].as_bool() == Some(true)) {
let interval = schedule["effective_interval_seconds"].as_u64();
let budget = scanner_cycle_completion_budget(cycle_end, interval, now, active)?;
deadline = Instant::now().checked_add(budget).ok_or("scanner deadline overflows")?;
schedule_accounted = true;
}
}
// Superseded walks complete all sets while withholding quota authority.
// Deferred and partial attempts do not certify completed scanning.
if matches!(metrics["last_cycle_result"].as_str(), Some("success" | "superseded")) {
latest_cycle_end = latest_cycle_end.max(cycle_end);
}
versions_observed |= versions_scanned > 0;
observations.push(format!(
"node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}",
"node{node_index}: end={cycle_end}, versions={versions_scanned}, cycle={}, active={}, leader={}, result={}, partial_reason={}, schedule={}",
metrics["current_cycle"],
metrics["current_cycle_active"],
metrics["leader_lock_state"],
metrics["last_cycle_result"],
metrics["last_cycle_partial_reason"],
schedule,
));
}
// The coordinator records cycle completion, but remote workers
@@ -1555,7 +1625,8 @@ mod tests {
}
if background_enabled {
wait_for_scanner_cycle_after(&cluster, 0).await?;
let baseline_cycle_floor = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH)?.as_secs();
wait_for_scanner_cycle_after(&cluster, baseline_cycle_floor).await?;
}
cluster.stop_node(1)?;
@@ -1898,11 +1969,16 @@ mod tests {
let status_body = signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key).await?;
let status: serde_json::Value = serde_json::from_str(&status_body)
.map_err(|err| format!("pre-interrupt background heal status is not JSON ({err}): {status_body}"))?;
if background_enabled || only_admin_heal_is_active(&status) {
let admin_active = background_rejoin_heal_evidence
|| (status["state"].as_str() == Some("active")
&& status["healOperations"]["activeBySource"]["admin"].as_u64() == Some(1));
if admin_active && (background_enabled || only_admin_heal_is_active(&status)) {
break status;
}
if Instant::now() >= partial_deadline {
return Err(format!("non-admin heal did not drain before the interruption deadline: {status}").into());
return Err(
format!("heal tasks did not reach the required interruption state before the deadline: {status}").into(),
);
}
sleep(Duration::from_millis(50)).await;
};
@@ -2562,4 +2638,157 @@ mod tests {
};
assert!(!outage_candidate_targets_replacement(&insufficient_peer_shards, 12, replacement_set_slot));
}
async fn two_node_heal_status(
cluster: &RustFSTestClusterEnvironment,
bucket: Option<&str>,
recreate: bool,
scan_mode: u8,
) -> Result<serde_json::Value, Box<dyn Error + Send + Sync>> {
let url = format!("{}/rustfs/admin/v3/heal/{}", cluster.nodes[0].url, bucket.unwrap_or_default());
let options = serde_json::json!({
"recursive": true, "dryRun": false, "remove": false,
"recreate": recreate, "scanMode": scan_mode, "updateParity": false, "nolock": false,
});
let started =
start_root_heal_when_control_ready(&url, &options.to_string(), &cluster.access_key, &cluster.secret_key).await?;
let started: serde_json::Value = serde_json::from_str(&started)?;
let token = started["clientToken"]
.as_str()
.ok_or_else(|| format!("missing heal token: {started}"))?;
let status_url = format!("{url}?clientToken={token}");
let deadline = Instant::now() + Duration::from_secs(240);
loop {
let body = timeout(
Duration::from_secs(20),
signed_admin_post(&status_url, None, &cluster.access_key, &cluster.secret_key),
)
.await??;
let status: serde_json::Value = serde_json::from_str(&body)?;
match status["summary"].as_str() {
Some("finished" | "stopped") => return Ok(status),
Some("running") if Instant::now() < deadline => sleep(Duration::from_millis(250)).await,
_ => return Err(format!("heal did not reach a terminal state: {status}").into()),
}
}
}
async fn verify_two_node_directory_recovery(
remove_bucket: bool,
scan_mode: u8,
bucket_scoped: bool,
recreate: bool,
) -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut cluster = RustFSTestClusterEnvironment::new(2).await?;
// Isolate explicit administrator recovery from scanner, MRF, and disk healing.
for key in [
"RUSTFS_HEAL_AUTO_HEAL_ENABLE",
"RUSTFS_HEAL_MRF_ENABLE",
"RUSTFS_SCANNER_ENABLED",
"RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE",
] {
cluster.set_env(key, "false");
}
cluster.start().await?;
let source = cluster.create_s3_client(0)?;
let target = cluster.create_s3_client(1)?;
let bucket = "two-node-directory-recovery";
source.create_bucket().bucket(bucket).send().await?;
let objects = [
("empty", Vec::new()),
("nested/inline", vec![0x3c; 1024]),
("nested/external", (0_u8..=255).cycle().take(1024 * 1024).collect::<Vec<_>>()),
];
for (key, bytes) in &objects {
source
.put_object()
.bucket(bucket)
.key(*key)
.body(ByteStream::from(bytes.clone()))
.send()
.await?;
}
let target_bucket = Path::new(&cluster.nodes[1].data_dir).join(bucket);
let original = objects
.iter()
.map(|(key, _)| census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None))
.collect::<Result<Vec<_>, _>>()?;
for census in &original {
assert!(census.has_xl_meta, "fixture must have a target metadata copy: {census:?}");
}
assert!(
!original[2].present_part_fingerprints.is_empty(),
"large fixture must exercise external parts"
);
let removed = Path::new(&cluster.temp_dir).join("removed");
tokio::fs::create_dir(&removed).await?;
if remove_bucket {
tokio::fs::rename(&target_bucket, removed.join(bucket)).await?;
} else {
for (index, (key, _)) in objects.iter().enumerate() {
tokio::fs::rename(target_bucket.join(key), removed.join(index.to_string())).await?;
}
}
let status = two_node_heal_status(&cluster, bucket_scoped.then_some(bucket), recreate, scan_mode).await?;
assert_eq!(status["summary"], "finished", "administrator recovery should complete: {status}");
assert_eq!(status["outcome"]["execution"]["state"], "completed");
assert_eq!(status["outcome"]["coverage"], "complete");
for ((key, _), before) in objects.iter().zip(&original) {
let after = census_object_version_on_disk(Path::new(&cluster.nodes[1].data_dir), bucket, key, None)?;
assert!(after.has_xl_meta, "metadata must be restored before GET: {key}");
assert_eq!(
after.present_part_fingerprints, before.present_part_fingerprints,
"external shard bytes must match: {key}"
);
assert_eq!(
after.inline_data_fingerprint, before.inline_data_fingerprint,
"inline shard bytes must match: {key}"
);
}
cluster.stop_node(0)?;
for (key, expected) in &objects {
let actual = timeout(Duration::from_secs(30), async {
target
.get_object()
.bucket(bucket)
.key(*key)
.send()
.await?
.body
.collect()
.await
.map(|body| body.into_bytes())
.map_err(Box::<dyn Error + Send + Sync>::from)
})
.await??;
assert_eq!(actual.len(), expected.len(), "target-only read length: {key}");
assert_eq!(sha256_hex(&actual), sha256_hex(expected), "target-only content: {key}");
}
Ok(())
}
#[tokio::test]
async fn test_two_node_bucket_discovery_and_explicit_recreate() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(true, 2, true, true).await
}
#[tokio::test]
async fn test_two_node_missing_bucket_default_cluster_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(true, 2, false, false).await
}
#[tokio::test]
async fn test_two_node_missing_bucket_default_scoped_recovery() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(true, 2, true, false).await
}
#[tokio::test]
async fn test_two_node_object_directory_recovery_normal() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(false, 1, false, false).await
}
#[tokio::test]
async fn test_two_node_object_directory_recovery_deep() -> Result<(), Box<dyn Error + Send + Sync>> {
verify_two_node_directory_recovery(false, 2, false, false).await
}
}
@@ -264,7 +264,7 @@ async fn call_make_volume_response(
request: MakeVolumeRequest,
headers: HeaderMap,
) -> Result<Response<MakeVolumeResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(&url.to_string())
let mut client = node_service_time_out_client_no_auth(url)
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(request);
@@ -273,7 +273,7 @@ async fn call_make_volume_response(
}
async fn call_ping_response(url: &str, headers: HeaderMap) -> Result<Response<PingResponse>, Status> {
let mut client = node_service_time_out_client_no_auth(&url.to_string())
let mut client = node_service_time_out_client_no_auth(url)
.await
.map_err(|err| Status::unavailable(format!("cannot reach the node service: {err}")))?;
let mut rpc_request = Request::new(PingRequest {
@@ -562,17 +562,23 @@ mod tests {
create_bucket(&client, bucket).await.expect("Failed to create bucket");
let object_count = 1002;
for i in 0..object_count {
let key = format!("object{:04}.txt", i);
client
.put_object()
.bucket(bucket)
.key(&key)
.body(ByteStream::from_static(b"test content"))
.send()
.await
.expect("Failed to put object");
}
// Finish every durable PUT before listing, with the same bounded fanout as the delimiter fixtures.
stream::iter(0..object_count)
.for_each_concurrent(16, |i| {
let client = &client;
async move {
let key = format!("object{i:04}.txt");
client
.put_object()
.bucket(bucket)
.key(&key)
.body(ByteStream::from_static(b"test content"))
.send()
.await
.unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
}
})
.await;
eprintln!("Seeded {object_count} objects in {bucket}; starting ListObjectsV2 pagination");
let deadline = Instant::now() + PAGINATION_TIMEOUT;
+441 -24
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::RustFSTestClusterEnvironment;
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use bytes::Bytes;
@@ -185,21 +185,113 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
return Err(format!(
"read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
)
.into());
}
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}
}
#[tokio::test]
async fn test_cold_degraded_object_mutations_require_only_storage_quorum() -> TestResult {
crate::common::init_logging();
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUSTFS_GET_OBJECT_METADATA_CACHE_DISTRIBUTED_ENABLE", "false");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = "cold-degraded-mutations";
clients[0].create_bucket().bucket(bucket).send().await?;
// No object mutation may warm a process-wide configuration probe before
// the missing peer is introduced. Three shards still satisfy EC:2 writes.
cluster.stop_node(3)?;
let admission_deadline = Instant::now() + Duration::from_secs(30);
for client in &clients[..3] {
loop {
match client
.get_object()
.bucket(bucket)
.key("never-created-readiness-probe")
.send()
.await
{
Err(error) if error.as_service_error().and_then(|error| error.meta().code()) == Some("NoSuchKey") => break,
Err(error)
if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
&& Instant::now() < admission_deadline =>
{
tokio::time::sleep(Duration::from_millis(100)).await;
}
Err(error) => return Err(error.into()),
Ok(_) => panic!("a cold readiness probe must not create an object"),
}
}
}
let body = b"first write while one peer is offline";
clients[0]
.put_object()
.bucket(bucket)
.key("object")
.body(Bytes::from_static(body).into())
.send()
.await?;
for client in &clients[..3] {
assert_quorum_object_body(client, bucket, "object", body).await?;
}
// This coordinator has not previously committed an object mutation either.
clients[1].delete_object().bucket(bucket).key("object").send().await?;
let error = clients[2]
.get_object()
.bucket(bucket)
.key("object")
.send()
.await
.expect_err("acknowledged deletion must be visible at another entrypoint");
assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchKey"));
cluster.stop_node(2)?;
let error = clients[0]
.put_object()
.bucket(bucket)
.key("below-quorum")
.body(Bytes::from_static(body).into())
.send()
.await
.expect_err("removing the cache gate must not weaken storage write quorum");
assert_eq!(error.as_service_error().and_then(|error| error.meta().code()), Some("ServiceUnavailable"));
cluster.stop();
Ok(())
}
#[tokio::test]
async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
// Wait for every seed fanout before removing any physical shard.
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -300,17 +392,28 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
}
cluster.stop_node(read_quorum - 1)?;
let lock_read_quorum = node_count - node_count / 2;
// EC:2 on six nodes still has three shared lock votes after losing
// its fourth data shard. Preserve the distinct data-quorum error.
let expected_code = if read_quorum - 1 < lock_read_quorum {
"ServiceUnavailable"
} else {
"SlowDownRead"
};
for client in clients.iter().take(read_quorum - 1) {
match client.get_object().bucket(&bucket).key("below-quorum").send().await {
Ok(response) => assert!(
response.body.collect().await.is_err(),
"fewer than {read_quorum} valid fragments must not reconstruct an uncached object"
),
Err(error) => assert_eq!(
error.as_service_error().and_then(|error| error.meta().code()),
Some("ServiceUnavailable"),
"a quorum loss must not be mistaken for a missing object"
),
Err(error) => {
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_eq!(
error.as_service_error().and_then(|error| error.meta().code()),
Some(expected_code),
"a quorum loss must not be mistaken for a missing object"
);
}
}
}
@@ -327,6 +430,277 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
Ok(())
}
#[tokio::test]
async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "warn");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = format!("cold-read-quorum-{node_count}-{parity}");
clients[0].create_bucket().bucket(&bucket).send().await?;
let small = b"cold startup follows the stored EC layout".to_vec();
let large = vec![73; 256 * 1024];
for (key, body) in [
("warm-small", &small),
("warm-large", &large),
("cold-large", &large),
("below-quorum", &large),
] {
clients[0]
.put_object()
.bucket(&bucket)
.key(key)
.body(Bytes::copy_from_slice(body).into())
.send()
.await?;
for node in &cluster.nodes {
let census =
crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
}
}
for node in 0..node_count {
cluster.stop_node(node)?;
}
// Rebuild every process's in-memory state below read quorum, then
// cross the EC boundary. A format bootstrap must not admit S3 reads
// merely because half the peers are online (e.g. six disks, EC:2).
for node in 0..read_quorum - 1 {
cluster.start_node_process(node).await?;
}
let error = clients[0]
.get_object()
.bucket(&bucket)
.key("below-quorum")
.send()
.await
.expect_err("cold startup below the configured EC read quorum must stay unavailable");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(read_quorum - 1).await?;
wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
for client in clients.iter().take(read_quorum) {
assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
let write = client
.put_object()
.bucket(&bucket)
.key("cold-quorum-write")
.body(Bytes::copy_from_slice(&small).into())
.send()
.await;
if read_quorum >= write_quorum {
write?;
assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
} else {
let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
}
for node in read_quorum..node_count {
cluster.start_node(node).await?;
}
for client in &clients {
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
}
}
Ok(())
}
#[tokio::test]
async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
crate::common::init_logging();
let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
// Explicit URLs are consumed in order. Place one drive from each node in
// every set, matching a four-host, six-drive-per-host deployment.
let volumes = (0..6)
.flat_map(|drive| {
cluster
.nodes
.iter()
.map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
})
.collect::<Vec<_>>()
.join(" ");
cluster.set_env("RUSTFS_VOLUMES", volumes);
cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "error");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = "cold-start-read-quorum";
clients[0].create_bucket().bucket(bucket).send().await?;
let mut fixtures = Vec::new();
let mut covered = [[false; 2]; 6];
for index in 0..128 {
let key = match index {
0 => "warm-small".to_string(),
1 => "warm-large".to_string(),
_ => format!("cold-{index}"),
};
let size_class = index % 2;
let size = if size_class == 0 { 71 } else { 256 * 1024 };
let body = (0..size)
.map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
.collect::<Vec<_>>();
clients[0]
.put_object()
.bucket(bucket)
.key(&key)
.body(Bytes::copy_from_slice(&body).into())
.send()
.await?;
for node in &cluster.nodes {
let mut complete_shards = 0;
for (drive, path) in node.data_dirs.iter().enumerate() {
let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
if census.has_xl_meta {
assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
covered[drive][size_class] = true;
complete_shards += 1;
}
}
assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
}
fixtures.push((key, body));
if covered.iter().flatten().all(|covered| *covered) {
break;
}
}
assert!(
covered.iter().flatten().all(|covered| *covered),
"fixtures must cover all six sets with inline and large objects"
);
for first in 0..4 {
for second in first + 1..4 {
for node in 0..4 {
cluster.stop_node(node)?;
}
cluster.start_node_process(first).await?;
let error = clients[first]
.get_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("one node cannot satisfy read quorum after a cold start");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(second).await?;
wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
for node in [first, second] {
for (key, body) in &fixtures {
assert_quorum_object_body(&clients[node], bucket, key, body).await?;
}
let range = clients[node]
.get_object()
.bucket(bucket)
.key("warm-large")
.range("bytes=17-1023")
.send()
.await?;
assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
for (key, _) in &fixtures {
assert!(
listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
"listing omitted {key}"
);
}
let error = clients[node]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
.expect_err("two nodes must not authorize a write");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
let error = clients[first]
.delete_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("two nodes must not authorize a delete");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
cluster.start_node(remaining[0]).await?;
let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
loop {
match clients[first]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
{
Ok(_) => break,
Err(error)
if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
&& tokio::time::Instant::now() < deadline =>
{
tokio::time::sleep(Duration::from_millis(200)).await
}
Err(error) => return Err(error.into()),
}
}
for node in [first, second, remaining[0]] {
assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
}
cluster.start_node(remaining[1]).await?;
for client in &clients {
for (key, body) in &fixtures {
assert_quorum_object_body(client, bucket, key, body).await?;
}
assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
}
}
}
Ok(())
}
async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
client
.put_object()
@@ -425,6 +799,60 @@ async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum()
Ok(())
}
fn unavailable_readiness_drive_indices(payload: &serde_json::Value) -> Vec<usize> {
let mut indices = payload["details"]["storage"]["unavailableDrives"]
.as_array()
.expect("node readiness must report unavailable drives")
.iter()
.map(|drive| {
usize::try_from(drive["diskIndex"].as_u64().expect("drive index must be unsigned"))
.expect("drive index must fit usize")
})
.collect::<Vec<_>>();
indices.sort_unstable();
indices
}
fn assert_node_readiness_contract(payload: &serde_json::Value, survivors: usize) {
let read_ready = survivors >= 2;
let write_ready = survivors >= 3;
let storage = &payload["details"]["storage"];
assert_eq!(payload["ready"], read_ready);
assert_eq!(storage["ready"], read_ready);
assert_eq!(storage["readQuorum"], read_ready);
assert_eq!(storage["writeQuorum"], write_ready);
assert_eq!(storage["readinessScope"], "read_quorum");
assert_eq!(storage["source"], "local_runtime");
assert_eq!(storage["status"], if read_ready { "connected" } else { "disconnected" });
assert_eq!(payload["details"]["poolMetadata"]["ready"], true);
assert_eq!(payload["details"]["iam"]["ready"], true);
assert_eq!(payload["details"]["lock"]["ready"], read_ready);
assert_eq!(unavailable_readiness_drive_indices(payload), (survivors..4).collect::<Vec<_>>());
for drive in storage["unavailableDrives"].as_array().expect("unavailable drives") {
assert_eq!(drive["poolIndex"], 0);
assert_eq!(drive["setIndex"], 0);
let state = drive["runtimeState"].as_str().expect("runtime state must be reported");
let host_online = drive["hostOnline"].as_bool().expect("host reachability must be reported");
assert!(
matches!(state, "online" | "suspect" | "offline" | "returning"),
"unknown runtime state: {state}"
);
assert!(
state != "online" || !host_online,
"a reachable online drive cannot be excluded from quorum"
);
}
if !read_ready {
assert!(
payload["degradedReasons"]
.as_array()
.expect("degraded reasons")
.iter()
.any(|reason| reason == "storage_and_lock_unavailable")
);
}
}
async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvironment) -> TestResult {
let clients: Vec<_> = cluster
.create_all_clients()?
@@ -483,27 +911,14 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
&& payload["details"]["poolMetadata"]["ready"] == true
&& payload["details"]["iam"]["ready"] == true
&& payload["details"]["lock"]["ready"] == read_quorum
&& unavailable_readiness_drive_indices(&payload) == (survivors..4).collect::<Vec<_>>()
{
break payload;
}
assert!(Instant::now() < deadline, "node {idx}, survivors={survivors}: HTTP {status}, {payload}");
tokio::time::sleep(Duration::from_millis(200)).await;
};
assert_eq!(payload["details"]["storage"]["readinessScope"], "read_quorum");
assert_eq!(payload["details"]["storage"]["source"], "local_runtime");
assert_eq!(
payload["details"]["storage"]["status"],
if node_ready { "connected" } else { "disconnected" }
);
if !node_ready {
assert!(
payload["degradedReasons"]
.as_array()
.expect("degraded reasons")
.iter()
.any(|reason| reason == "storage_and_lock_unavailable")
);
}
assert_node_readiness_contract(&payload, survivors);
for path in ["/health/ready", "/minio/health/ready"] {
let head = http.head(format!("{url}{path}")).send().await?;
@@ -512,7 +927,9 @@ async fn assert_node_readiness_tracks_quorum(cluster: &mut RustFSTestClusterEnvi
let response = http.get(format!("{url}{path}")).send().await?;
assert_eq!(response.status().as_u16(), node_status);
let body: serde_json::Value = response.json().await?;
assert_eq!(body["details"]["storage"], payload["details"]["storage"]);
// Each response samples live diagnostics; suspect -> offline
// must not change the quorum contract or excluded topology.
assert_node_readiness_contract(&body, survivors);
assert_eq!(body["details"]["poolMetadata"], payload["details"]["poolMetadata"]);
}
let live = http.get(format!("{url}/health/live")).send().await?;
@@ -465,6 +465,13 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn recover_delete_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionRequest>,
) -> Result<Response<rustfs_protos::proto_gen::node_service::RecoverDeleteTransactionResponse>, Status> {
Err(Status::unimplemented("delete recovery is not supported by the lock fixture"))
}
async fn settle_part_transaction(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::SettlePartTransactionRequest>,
@@ -3945,7 +3945,15 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
(sse_key, sse_put.version_id().ok_or("SSE PUT omitted version ID")?.to_string()),
(large_key, large_put.version_id().ok_or("large PUT omitted version ID")?.to_string()),
];
assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
// Managed-SSE listings may omit ETags. Check the replicated version's body
// before deleting it rather than comparing keyless listing fingerprints.
for (key, version_id) in &purged {
let source_body = get_version_body(&source_client, &source_bucket, key, version_id).await?;
let expected_sha256: [u8; 32] = Sha256::digest(&source_body).into();
wait_for_replicated_sha256(&target_client, &target_bucket, key, expected_sha256).await?;
let target_body = get_version_body(&target_client, &target_bucket, key, version_id).await?;
assert_eq!(target_body, source_body, "replicated version body differs for {key}/{version_id}");
}
for (key, version_id) in &purged {
source_client
@@ -3956,10 +3964,6 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
.send()
.await?;
}
assert_replication_converged(&source_client, &source_bucket, &target_client, &target_bucket).await?;
let target_state = list_replication_state(&target_client, &target_bucket).await?;
assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
// The purge state is applied on the source asynchronously after the target
// acknowledges the delete; only then does the retained version go away and
// the bucket become deletable. A listing that is empty while DeleteBucket
@@ -3984,6 +3988,9 @@ async fn test_bucket_replication_version_purge_of_non_inline_object_releases_sou
}
}
let target_state = list_replication_state(&target_client, &target_bucket).await?;
assert!(target_state.is_empty(), "target retained an explicitly purged version: {target_state:?}");
Ok(())
}
+1 -1
View File
@@ -15,7 +15,7 @@
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::bucket::bucket_target_sys::BucketTargetSys;
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::disk::{RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
pub(crate) use rustfs_ecstore::api::disk::{Endpoint, RUSTFS_META_BUCKET, VolumeInfo, WalkDirOptions};
pub(crate) use rustfs_ecstore::api::rpc::{AuthenticatedChannel, TonicInterceptor, node_service_time_out_client_no_auth};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::{
+1 -1
View File
@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.version_id(version)
.send()
.await?;
opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
client.list_objects_v2().bucket(source).send().await?;
opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
client.list_object_versions().bucket(source).send().await?;
@@ -165,4 +165,210 @@ mod tests {
info!("RT-13c PASS: scanner/config persists across restart");
Ok(())
}
/// RT-14: a transition PUT must not carry reserved `x-minio-internal-*` headers.
///
/// Regression pattern: transition forwarded the object's internal metadata as raw
/// request headers, so MinIO-derived remotes (Storj) refused every transition with
/// `400 InvalidArgument` while `tier add` still succeeded (rustfs#8362). The fake
/// target refuses any PutObject carrying such a header, like MinIO and Storj do.
#[tokio::test]
async fn test_transition_put_omits_reserved_internal_headers() -> TestResult {
use crate::fake_s3_target::{BucketMode, FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, Operation};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, LifecycleRuleFilter, Transition,
TransitionStorageClass,
};
use std::time::{Duration, Instant};
init_logging();
info!("RT-14: transition PUT omits reserved internal headers");
const TIER: &str = "RT14COLD";
const TARGET_BUCKET: &str = "rt14-cold";
const SOURCE_BUCKET: &str = "rt14-hot";
let target = FakeS3Target::start().await.expect("start fake target");
target.create_bucket_with_mode(TARGET_BUCKET, BucketMode::Unversioned);
target.reject_reserved_internal_headers(true);
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_CONSOLE_ENABLE", "false"),
("RUSTFS_TIER_RUSTFS_ALLOW_LOOPBACK_ENDPOINT", "true"),
],
)
.await
.expect("start RustFS");
let tier_body = serde_json::json!({
"type": "rustfs",
"rustfs": {
"name": TIER,
"endpoint": target.endpoint(),
"accessKey": FAKE_ACCESS_KEY,
"secretKey": FAKE_SECRET_KEY,
"bucket": TARGET_BUCKET,
"prefix": "cold/",
"region": "us-east-1",
"storageClass": ""
}
})
.to_string();
let deadline = Instant::now() + Duration::from_secs(30);
loop {
match admin_ok(&env, http::Method::PUT, "/rustfs/admin/v3/tier", Some(tier_body.clone())).await {
Ok(_) => break,
Err(err) if Instant::now() < deadline => {
info!(" add tier not ready yet: {err}");
tokio::time::sleep(Duration::from_millis(500)).await;
}
Err(err) => panic!("RT-14 FAIL: add tier: {err}"),
}
}
let client = env.create_s3_client();
client
.create_bucket()
.bucket(SOURCE_BUCKET)
.send()
.await
.expect("create bucket");
let objects: Vec<(&str, Vec<u8>)> = vec![
("small.bin", (0..1024u32).map(|i| (i % 251) as u8).collect()),
("medium.bin", (0..200_000u32).map(|i| (i.wrapping_mul(31) % 253) as u8).collect()),
];
for (key, body) in &objects {
client
.put_object()
.bucket(SOURCE_BUCKET)
.key(*key)
.body(ByteStream::from(body.clone()))
.send()
.await
.expect("put source object");
}
let rule = LifecycleRule::builder()
.id("to-cold")
.status(ExpirationStatus::Enabled)
.filter(LifecycleRuleFilter::builder().prefix("").build())
.transitions(
Transition::builder()
.days(0)
.storage_class(TransitionStorageClass::from(TIER))
.build(),
)
.build()
.expect("build rule");
client
.put_bucket_lifecycle_configuration()
.bucket(SOURCE_BUCKET)
.lifecycle_configuration(
BucketLifecycleConfiguration::builder()
.rules(rule)
.build()
.expect("build config"),
)
.send()
.await
.expect("put lifecycle");
admin_ok(
&env,
http::Method::POST,
&format!("/rustfs/admin/v3/ilm/transition/run?bucket={SOURCE_BUCKET}&prefix=&tier={TIER}&dryRun=false"),
None,
)
.await
.expect("run transition");
let deadline = Instant::now() + Duration::from_secs(60);
while target.stored_keys(TARGET_BUCKET).len() < objects.len() {
assert!(
Instant::now() < deadline,
"RT-14 FAIL: transition did not reach the target; stored keys: {:?}",
target.stored_keys(TARGET_BUCKET)
);
tokio::time::sleep(Duration::from_millis(500)).await;
}
let keys = target.stored_keys(TARGET_BUCKET);
assert_eq!(keys.len(), objects.len(), "RT-14 FAIL: unexpected remote objects: {keys:?}");
let mut stored_bodies = Vec::new();
for key in &keys {
let (body, metadata) = target.stored_object(TARGET_BUCKET, key).expect("remote object");
for name in [
"x-rustfs-internal-transition-transaction-id",
"x-minio-internal-transition-transaction-id",
"x-rustfs-internal-transition-tier-destination-id",
"x-minio-internal-transition-tier-destination-id",
] {
assert!(
metadata.contains_key(name),
"RT-14 FAIL: remote object {key} lost transition identity {name}: {:?}",
metadata.keys().collect::<Vec<_>>()
);
}
stored_bodies.push(body.to_vec());
}
let mut expected_bodies: Vec<Vec<u8>> = objects.iter().map(|(_, body)| body.clone()).collect();
expected_bodies.sort();
stored_bodies.sort();
assert_eq!(stored_bodies, expected_bodies, "RT-14 FAIL: remote bytes differ from the source objects");
let transition_puts = target
.requests()
.iter()
.filter(|record| {
record.operation == Operation::PutObject
&& record
.key
.as_deref()
.is_some_and(|key| key.contains("transition-transactions"))
})
.count();
assert_eq!(
transition_puts,
objects.len(),
"RT-14 FAIL: transition PUTs were retried or refused by the target"
);
for (key, body) in &objects {
let head = client
.head_object()
.bucket(SOURCE_BUCKET)
.key(*key)
.send()
.await
.expect("head transitioned");
assert_eq!(
head.storage_class().map(|class| class.as_str()),
Some(TIER),
"RT-14 FAIL: {key} is not reported as transitioned"
);
let got = client
.get_object()
.bucket(SOURCE_BUCKET)
.key(*key)
.send()
.await
.expect("get transitioned")
.body
.collect()
.await
.expect("read body")
.into_bytes();
assert_eq!(got.as_ref(), body.as_slice(), "RT-14 FAIL: read-back of {key} differs");
}
target.shutdown().await;
info!("RT-14 PASS: transition PUT omits reserved internal headers");
Ok(())
}
}
+371 -33
View File
@@ -33,6 +33,7 @@ use aws_sdk_s3::types::{
};
use http::{Method, StatusCode};
use serde_json::Value;
use std::future::Future;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
@@ -57,6 +58,10 @@ const UPGRADE_READINESS_BODY: &[u8] = b"upgrade write readiness";
const PREVIOUS_RELEASE_SEED_KEY: &str = ".upgrade-readiness/previous-seed/node-0";
const MULTIPART_WORKERS: usize = 16;
const MULTIPART_UPLOADS_PER_WORKER: usize = 16;
// Bound each 64 KiB upload, including completion acknowledgement. A peer
// that accepts the request but never responds must fail the compatibility
// check with the writer and object, rather than consume the whole CI job.
const MIXED_MULTIPART_UPLOAD_TIMEOUT: Duration = Duration::from_secs(30);
// Peers keep a restarted node's drive in Suspect/Returning for roughly
// probe_interval (2s) x success_threshold (3) after it comes back; 30s
// comfortably covers that window plus CI scheduling jitter.
@@ -335,14 +340,21 @@ fn configure_cluster_logs(cluster: &mut RustFSTestClusterEnvironment) -> TestRes
async fn write_multipart_load(clients: &[Client], phase: &str) -> Result<Vec<String>, Box<dyn std::error::Error + Send + Sync>> {
let mut tasks = JoinSet::new();
for worker in 0..MULTIPART_WORKERS {
let client = clients[worker % clients.len()].clone();
let node = worker % clients.len();
let client = clients[node].clone();
let phase = phase.to_string();
tasks.spawn(async move {
let mut keys = Vec::with_capacity(MULTIPART_UPLOADS_PER_WORKER);
for upload in 0..MULTIPART_UPLOADS_PER_WORKER {
let key = format!("{phase}/multipart/{worker:02}/{upload:02}");
let part = vec![u8::try_from(worker)?; 64 * 1024];
write_multipart(&client, MIXED_BUCKET, &key, &[part]).await?;
mixed_multipart_before(
Instant::now() + MIXED_MULTIPART_UPLOAD_TIMEOUT,
node,
&key,
write_multipart(&client, MIXED_BUCKET, &key, &[part]),
)
.await?;
keys.push(key);
}
Ok::<_, Box<dyn std::error::Error + Send + Sync>>(keys)
@@ -356,6 +368,172 @@ async fn write_multipart_load(clients: &[Client], phase: &str) -> Result<Vec<Str
Ok(keys)
}
async fn mixed_multipart_before(
deadline: Instant,
node: usize,
key: &str,
upload: impl Future<Output = TestResult>,
) -> TestResult {
tokio::time::timeout_at(deadline, upload)
.await
.map_err(|_| MultipartAcknowledgementDeadline {
node,
key: key.to_owned(),
})?
}
#[derive(Debug)]
struct MultipartAcknowledgementDeadline {
node: usize,
key: String,
}
impl std::fmt::Display for MultipartAcknowledgementDeadline {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
formatter,
"node {}: multipart upload {MIXED_BUCKET}/{} acknowledgement deadline exceeded",
self.node, self.key
)
}
}
impl std::error::Error for MultipartAcknowledgementDeadline {}
// Decode only the public lock snapshot fields. Unknown fields, including
// credential/configuration payloads, never enter the failure artifact.
#[derive(serde::Deserialize, serde::Serialize)]
struct MixedLockSnapshot {
total: Option<u64>,
truncated: Option<bool>,
capability_note: Option<String>,
locks: Vec<MixedLockEntry>,
}
#[derive(serde::Deserialize, serde::Serialize)]
struct MixedLockEntry {
resource: Option<String>,
bucket: Option<String>,
object: Option<String>,
version: Option<String>,
owner: Option<String>,
#[serde(rename = "type")]
lock_type: Option<String>,
priority: Option<String>,
since: Option<String>,
elapsed_secs: Option<u64>,
ttl_secs: Option<u64>,
}
async fn read_mixed_lock_snapshot(response: &mut reqwest::Response) -> Result<MixedLockSnapshot, &'static str> {
const MAX_BODY_BYTES: usize = 1024 * 1024;
let mut body = Vec::new();
while let Some(chunk) = response.chunk().await.map_err(|_| "body_read_error")? {
if chunk.len() > MAX_BODY_BYTES - body.len() {
return Err("body_limit_exceeded");
}
body.extend_from_slice(&chunk);
}
serde_json::from_slice(&body).map_err(|_| "invalid_lock_snapshot")
}
async fn probe_mixed_multipart_failure(
cluster: &RustFSTestClusterEnvironment,
node: usize,
path: &str,
head: bool,
deadline: Instant,
) -> Value {
let started = Instant::now();
let method = if head { Method::HEAD } else { Method::GET };
let mut observation = serde_json::json!({
"node": node, "method": method.as_str(), "path": path,
"status": null, "observation": "unknown", "timeout": false,
});
// The shared absolute deadline bounds both headers and a slow/infinite
// response body; a send-only timeout would leave the cluster alive forever.
let result = tokio::time::timeout_at(deadline, async {
let mut response = signed_request(
method,
&format!("{}{path}", cluster.nodes[node].url.trim_end_matches('/')),
&cluster.access_key,
&cluster.secret_key,
None,
None,
)
.await
.map_err(|_| "request_error")?;
let status = response.status();
observation["status"] = serde_json::json!(status.as_u16());
if head {
observation["observation"] = serde_json::json!(match status {
StatusCode::OK => "visible",
StatusCode::NOT_FOUND => "not_visible",
_ => "unknown",
});
for (field, header) in [
("content_length", "content-length"),
("etag", "etag"),
("version", "x-amz-version-id"),
] {
observation[field] = serde_json::json!(response.headers().get(header).and_then(|value| value.to_str().ok()));
}
} else if status == StatusCode::OK {
observation["snapshot"] =
serde_json::to_value(read_mixed_lock_snapshot(&mut response).await?).map_err(|_| "snapshot_encode_error")?;
observation["observation"] = serde_json::json!("captured");
}
Ok::<_, &'static str>(())
})
.await;
match result {
Ok(Ok(())) => {}
Ok(Err(reason)) => observation["error"] = serde_json::json!(reason),
Err(_) => observation["timeout"] = serde_json::json!(true),
}
observation["elapsed_secs"] = serde_json::json!(started.elapsed().as_secs_f64());
observation
}
async fn observe_mixed_multipart_failure(
cluster: &RustFSTestClusterEnvironment,
phase: &str,
error: BoxError,
log_dir: Option<&Path>,
) -> TestResult {
if let (Some(log_dir), Some(failure)) = (log_dir, error.downcast_ref::<MultipartAcknowledgementDeadline>()) {
let started = Instant::now();
let deadline = started + Duration::from_secs(5);
// Reserve the last second for serialization and bounded artifact I/O.
// All probes run directly in this future; no diagnostic task is spawned.
let network_deadline = started + Duration::from_secs(4);
let path = format!("/{MIXED_BUCKET}/{}", failure.key);
let _ = tokio::time::timeout_at(deadline, async {
let (heads, locks) = tokio::join!(
futures::future::join_all((0..cluster.nodes.len()).map(|node| probe_mixed_multipart_failure(
cluster,
node,
&path,
true,
network_deadline
))),
probe_mixed_multipart_failure(cluster, 0, "/rustfs/admin/v3/top/locks?count=1000", false, network_deadline),
);
let snapshot = serde_json::json!({
"phase": phase, "bucket": MIXED_BUCKET, "key": failure.key,
"writer_node": failure.node, "heads": heads, "top_locks": locks,
});
let body = serde_json::to_vec_pretty(&snapshot).map_err(|_| ())?;
tokio::fs::write(log_dir.join("mixed-multipart-failure.json"), body)
.await
.map_err(|_| ())
})
.await;
}
// Visibility and diagnostic failures cannot acknowledge the original MPU.
Err(error)
}
/// Assert that `client` eventually lists exactly `expected` objects under
/// `{phase}/`, polling until [`LISTING_CONVERGENCE_TIMEOUT`].
///
@@ -400,7 +578,7 @@ async fn exercise_mixed_cluster(
previous_node: usize,
) -> TestResult {
let clients = cluster.create_all_clients()?;
wait_for_upgrade_write_readiness(&clients, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, phase, LISTING_CONVERGENCE_TIMEOUT).await?;
let current_client = &clients[current_node];
let previous_client = &clients[previous_node];
@@ -426,7 +604,13 @@ async fn exercise_mixed_cluster(
.await?;
assert_eq!(read_object(current_client, MIXED_BUCKET, &previous_key, None).await?.1, previous_body);
let multipart_keys = write_multipart_load(&clients, phase).await?;
let multipart_keys = match write_multipart_load(&clients, phase).await {
Ok(keys) => keys,
Err(error) => {
let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR");
return observe_mixed_multipart_failure(cluster, phase, error, log_dir.as_deref().map(Path::new)).await;
}
};
let expected_count = multipart_keys.len() + 2;
for (label, client) in [("current", current_client), ("previous", previous_client)] {
assert_eq!(
@@ -462,7 +646,12 @@ fn upgrade_probe_client(client: &Client) -> Client {
)
}
async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budget: Duration) -> TestResult {
pub(crate) async fn wait_for_upgrade_write_readiness(
clients: &[Client],
bucket: &str,
phase: &str,
budget: Duration,
) -> TestResult {
// ListBuckets can succeed before peers recover a restarted disk. Cluster
// health also accepts Returning disks whose write health is still FAULTY.
// Probe every writer outside the asserted phase prefix; compatibility
@@ -482,7 +671,7 @@ async fn wait_for_upgrade_write_readiness(clients: &[Client], phase: &str, budge
deadline,
client
.put_object()
.bucket(MIXED_BUCKET)
.bucket(bucket)
.key(&key)
.body(ByteStream::from_static(UPGRADE_READINESS_BODY))
.send(),
@@ -517,19 +706,20 @@ async fn prepare_previous_release_baseline(cluster: &mut RustFSTestClusterEnviro
// waits for pool metadata (#7473). First prove the elected writer can
// persist data, then restart each old process once with three peers still
// readable. This preparation ends before any current binary is started.
wait_for_upgrade_write_readiness(&clients[..1], "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_upgrade_write_readiness(&clients[..1], MIXED_BUCKET, "previous-seed", LISTING_CONVERGENCE_TIMEOUT).await?;
for node in [1, 2, 3, 0] {
cluster.stop_node(node)?;
cluster.start_node_from_binary(node, previous_binary).await?;
wait_for_upgrade_write_readiness(
std::slice::from_ref(&clients[node]),
MIXED_BUCKET,
&format!("previous-restart-{node}"),
LISTING_CONVERGENCE_TIMEOUT,
)
.await?;
}
wait_for_upgrade_write_readiness(&clients, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
wait_for_upgrade_write_readiness(&clients, MIXED_BUCKET, "previous-baseline", LISTING_CONVERGENCE_TIMEOUT).await?;
for (reader, client) in clients.iter().enumerate() {
assert_eq!(
read_object(client, MIXED_BUCKET, PREVIOUS_RELEASE_SEED_KEY, None).await?.1,
@@ -556,9 +746,10 @@ mod upgrade_write_readiness_tests {
#[tokio::test]
async fn waits_for_each_writer_after_metadata_is_ready() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket(MIXED_BUCKET);
let bucket = "distributed-upgrade-history";
target.create_bucket(bucket);
let client = fake_source_client(&target);
client.head_bucket().bucket(MIXED_BUCKET).send().await?;
client.head_bucket().bucket(bucket).send().await?;
let phase = "one-previous-node";
let first_key = format!(".upgrade-readiness/{phase}/node-0");
@@ -578,7 +769,7 @@ mod upgrade_write_readiness_tests {
// Metadata readiness does not prove that a data write can succeed.
let premature = client
.put_object()
.bucket(MIXED_BUCKET)
.bucket(bucket)
.key(&first_key)
.body(ByteStream::from_static(b"upgrade write readiness"))
.send()
@@ -586,15 +777,15 @@ mod upgrade_write_readiness_tests {
.expect_err("metadata readiness does not prove write readiness");
assert_eq!(premature.raw_response().map(|response| response.status().as_u16()), Some(503));
wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], phase, Duration::from_secs(5)).await?;
wait_for_upgrade_write_readiness(&[client.clone(), client.clone()], bucket, phase, Duration::from_secs(5)).await?;
assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &first_key), 3);
assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &second_key), 2);
assert!(target.has_object(MIXED_BUCKET, &first_key));
assert!(target.has_object(MIXED_BUCKET, &second_key));
assert!(target.has_object(bucket, &first_key));
assert!(target.has_object(bucket, &second_key));
assert!(
client
.list_objects_v2()
.bucket(MIXED_BUCKET)
.bucket(bucket)
.prefix(format!("{phase}/"))
.send()
.await?
@@ -624,9 +815,10 @@ mod upgrade_write_readiness_tests {
let phase = format!("permanent-{}", status.as_u16());
let key = format!(".upgrade-readiness/{phase}/node-0");
target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Status(status), 1);
let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), &phase, Duration::from_secs(5))
.await
.expect_err("a permanent error must not be retried into success");
let error =
wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, &phase, Duration::from_secs(5))
.await
.expect_err("a permanent error must not be retried into success");
assert!(error.to_string().contains("node 0 write readiness failed"), "{error}");
assert_eq!(target.count_requests(FakeTargetOperation::PutObject, &key), 1);
assert!(!target.has_object(MIXED_BUCKET, &key));
@@ -649,7 +841,7 @@ mod upgrade_write_readiness_tests {
FaultAction::Status(StatusCode::SERVICE_UNAVAILABLE),
10,
);
let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
let error = wait_for_upgrade_write_readiness(&[client], MIXED_BUCKET, phase, Duration::from_secs(1))
.await
.expect_err("persistent unavailability must exhaust the shared deadline");
// Transport scheduling consumes the same budget; do not require a
@@ -668,10 +860,134 @@ mod upgrade_write_readiness_tests {
let phase = "stalled";
let key = format!(".upgrade-readiness/{phase}/node-0");
target.inject_for_key(FakeTargetOperation::PutObject, &key, FaultAction::Stall(Duration::from_secs(30)), 1);
let error = wait_for_upgrade_write_readiness(&[client], phase, Duration::from_secs(1))
let error = wait_for_upgrade_write_readiness(std::slice::from_ref(&client), MIXED_BUCKET, phase, Duration::from_secs(1))
.await
.expect_err("a stalled request must not outlive the readiness deadline");
assert!(error.to_string().contains("deadline exceeded during PutObject"), "{error}");
let part = vec![2_u8; 64 * 1024];
let successful_key = "stalled/multipart/02/00";
mixed_multipart_before(
Instant::now() + MIXED_MULTIPART_UPLOAD_TIMEOUT,
2,
successful_key,
write_multipart(&client, MIXED_BUCKET, successful_key, std::slice::from_ref(&part)),
)
.await?;
assert_eq!(read_object(&client, MIXED_BUCKET, successful_key, None).await?.1, part);
let stalled_key = "stalled/multipart/02/01";
target.inject_for_key(
FakeTargetOperation::CompleteMultipartUpload,
stalled_key,
FaultAction::Stall(Duration::from_secs(30)),
1,
);
let error = {
let upload = write_multipart(&client, MIXED_BUCKET, stalled_key, std::slice::from_ref(&part));
tokio::pin!(upload);
tokio::select! {
result = &mut upload => panic!("stalled completion unexpectedly returned: {result:?}"),
observed = tokio::time::timeout(Duration::from_secs(10), async {
while target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key) == 0
|| !target.has_object(MIXED_BUCKET, stalled_key)
{
sleep(Duration::from_millis(10)).await;
}
}) => observed.expect("multipart completion must reach and commit on the fake peer"),
}
// An already committed object is not an acknowledged client success.
// Expire only after observing the real completion request, so host
// scheduling cannot turn this into a timeout before admission.
tokio::time::timeout(Duration::from_secs(1), mixed_multipart_before(Instant::now(), 2, stalled_key, upload))
.await
.expect("expired deadline must stop the in-flight completion")
.expect_err("unacknowledged completion must fail even if the peer committed")
};
assert!(error.to_string().contains("node 2"), "{error}");
assert!(error.to_string().contains(stalled_key), "{error}");
assert!(error.to_string().contains("acknowledgement deadline exceeded"), "{error}");
assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1);
let failure = error
.downcast_ref::<MultipartAcknowledgementDeadline>()
.expect("the timeout must retain typed writer/object context");
assert_eq!(failure.node, 2);
assert_eq!(failure.key, stalled_key);
let original_message = error.to_string();
// These four probe addresses share an in-process peer. This verifies
// bounded diagnostic fan-out, not a four-node RustFS cluster.
let mut probes = RustFSTestClusterEnvironment::new(MIXED_NODE_COUNT).await?;
probes.access_key = FAKE_ACCESS_KEY.to_owned();
probes.secret_key = FAKE_SECRET_KEY.to_owned();
for node in &mut probes.nodes {
node.url = target.endpoint().to_owned();
}
let log_dir = tempfile::tempdir()?;
target.inject_for_key(
FakeTargetOperation::HeadObject,
stalled_key,
FaultAction::Stall(Duration::from_secs(30)),
1,
);
let error = tokio::time::timeout(
Duration::from_secs(7),
observe_mixed_multipart_failure(&probes, phase, error, Some(log_dir.path())),
)
.await
.expect("the shared diagnostic budget must include artifact writing")
.expect_err("observing committed data cannot acknowledge the stalled upload");
assert_eq!(error.to_string(), original_message);
assert!(error.downcast_ref::<MultipartAcknowledgementDeadline>().is_some());
let snapshot: Value =
serde_json::from_slice(&tokio::fs::read(log_dir.path().join("mixed-multipart-failure.json")).await?)?;
assert_eq!(snapshot["key"], stalled_key);
assert_eq!(snapshot["writer_node"], 2);
let heads = snapshot["heads"]
.as_array()
.expect("failure artifact must retain every probe");
assert_eq!(heads.len(), MIXED_NODE_COUNT);
assert_eq!(
heads
.iter()
.filter(|head| head["timeout"] == true && head["observation"] == "unknown")
.count(),
1
);
assert_eq!(
heads
.iter()
.filter(|head| head["status"] == 200 && head["observation"] == "visible")
.count(),
3
);
assert_eq!(target.count_requests(FakeTargetOperation::HeadObject, stalled_key), MIXED_NODE_COUNT);
assert_eq!(target.count_requests(FakeTargetOperation::CompleteMultipartUpload, stalled_key), 1);
assert_eq!(snapshot["top_locks"]["observation"], "unknown", "the fake peer has no admin lock API");
let locks: MixedLockSnapshot = serde_json::from_value(serde_json::json!({
"total": 1, "truncated": true, "capability_note": "partial snapshot",
"credentials": { "secret_key": "must not persist" },
"locks": [{
"resource": format!("{MIXED_BUCKET}/{stalled_key}"), "owner": "writer", "type": "WRITE",
"elapsed_secs": 30, "ttl_secs": 1, "merged_config": { "secret_key": "must not persist" },
}],
}))?;
let safe_locks = serde_json::to_value(locks)?;
assert_eq!(safe_locks["total"], 1);
assert_eq!(safe_locks["truncated"], true);
assert_eq!(safe_locks["locks"][0]["owner"], "writer");
assert_eq!(safe_locks["locks"][0]["elapsed_secs"], 30);
assert!(safe_locks.get("credentials").is_none());
assert!(safe_locks["locks"][0].get("merged_config").is_none());
let mut body = br#"{"locks":[]}"#.to_vec();
body.resize(1024 * 1024, b' ');
let mut at_limit = reqwest::Response::from(http::Response::new(body.clone()));
assert!(read_mixed_lock_snapshot(&mut at_limit).await?.locks.is_empty());
body.push(b' ');
let mut oversized = reqwest::Response::from(http::Response::new(body));
assert_eq!(read_mixed_lock_snapshot(&mut oversized).await.err(), Some("body_limit_exceeded"));
target.shutdown().await;
Ok(())
}
@@ -2029,7 +2345,7 @@ fn layout_reported_parts(attributes: &aws_sdk_s3::operation::get_object_attribut
attributes.object_parts().map(|parts| parts.parts().len())
}
async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
async fn assert_layout_full_body(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
let label = case.label();
let head = layout_head(client, case).await?;
assert_eq!(
@@ -2046,7 +2362,12 @@ async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &st
let full = layout_get(client, case, None, None).await?.body.collect().await?.into_bytes();
assert_eq!(full.len(), case.body.len(), "{context}: {label}: full GET length");
assert!(full == case.body, "{context}: {label}: full GET body must equal the rc.5 upload");
Ok(())
}
async fn assert_layout_readable(client: &Client, case: &LayoutCase, context: &str) -> TestResult {
assert_layout_full_body(client, case, context).await?;
let label = case.label();
if case.is_multipart_layout() {
let first = case.part_sizes[0];
let range = format!("bytes={}-{}", first - 32, first + 31);
@@ -2181,9 +2502,9 @@ struct LayoutTransport {
uploaded_parts: Vec<i32>,
single_puts: usize,
completes: usize,
/// Raw per-key journal in target order: (sequence, operation, part number,
/// upload id), so duplicate drives can be told apart from retries.
journal: Vec<(u64, String, Option<i32>, Option<String>)>,
/// Allowlisted request identity and prepared response metadata. A prepared
/// response does not prove that the client received it.
journal: Vec<Value>,
}
/// Configure every layout bucket to replicate its existing objects to a fresh
@@ -2245,12 +2566,28 @@ async fn replicate_layouts(
journal: key_requests
.iter()
.map(|record| {
(
record.sequence,
format!("{:?}", record.operation),
record.part_number,
record.upload_id.as_ref().map(|id| id.chars().take(12).collect()),
)
serde_json::json!({
"sequence": record.sequence,
"operation": format!("{:?}", record.operation),
"part_number": record.part_number,
"upload_id": record.upload_id,
"requested_version_id": record.version_id,
"source_version_id": record.source_version_id,
"source_mtime": record.source_mtime,
"source_etag": record.source_etag,
"source_replication_request": record.source_replication_request,
"request_content_length": record.content_length,
"journaled_at_unix_millis": record.journaled_at_unix_millis,
"prepared_response": record.prepared_response.as_ref().map(|response| serde_json::json!({
"status": response.status,
"version_id": response.version_id,
"etag": response.etag,
"last_modified": response.last_modified,
"content_length": response.content_length,
"prepared_at_unix_millis": response.prepared_at_unix_millis,
"elapsed_secs": response.elapsed.as_secs_f64(),
})),
})
})
.collect(),
};
@@ -2309,10 +2646,11 @@ async fn direct_upgrade_from_rc5_preserves_multipart_layouts() -> TestResult {
"rc.5 wrote a legacy layout"
);
}
// The rc.5 writer must itself still read what it wrote, so a later
// failure is attributable to the upgrade rather than to the fixture.
// Prove the rc.5 writer persisted every original byte before upgrading.
// Its compressed cross-part range reader has a legacy limitation; the
// upgraded reader must still satisfy every range and partNumber assertion.
for case in &cases {
assert_layout_readable(&old_client, case, "rc.5").await?;
assert_layout_full_body(&old_client, case, "rc.5").await?;
}
// Upgrade in place.
+4 -3
View File
@@ -176,13 +176,14 @@ regex = { workspace = true }
path-absolutize = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
tokio-util = { workspace = true, features = ["io", "compat"] }
tokio-util = { workspace = true, features = ["io", "compat", "rt"] }
tokio-stream = { workspace = true, features = ["sync"] }
hmac = { workspace = true }
sha2 = { workspace = true }
hex-simd = { workspace = true }
tempfile.workspace = true
hyper = { workspace = true, features = ["http2", "http1", "server"] }
h2 = { workspace = true }
hyper-util = { workspace = true, features = ["tokio", "server-auto", "server-graceful", "tracing"] }
hyper-rustls = { workspace = true, default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs"] }
hostname.workspace = true
@@ -200,8 +201,8 @@ md-5.workspace = true
memmap2 = { workspace = true }
libc.workspace = true
# "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
# (backlog#1178); "fs" comes from the workspace default.
rustix = { workspace = true, features = ["process", "fs"] }
# (backlog#1178); "thread" provides cleanup affinity; "fs" is a workspace default.
rustix = { workspace = true, features = ["process", "fs", "thread"] }
rustfs-madmin.workspace = true
reqwest = { workspace = true }
aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+24 -15
View File
@@ -199,10 +199,10 @@ pub mod bucket {
get_object_lock_config, get_object_lock_config_state, get_on_demand_migration_config,
get_on_demand_migration_config_in, get_public_access_block_config, get_quota_config, get_replication_config,
get_request_payment_config, get_sse_config, get_tagging_config, get_versioning_config, get_website_config,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata, set_bucket_metadata,
update, update_bucket_targets_under_transaction_lock, update_config_with, update_if_incarnation,
update_if_incarnation_at, update_quota_if_incarnation, update_quota_if_incarnation_at, update_under_transaction_lock,
update_under_transaction_lock_at,
init_bucket_metadata_sys, list_bucket_targets, reload_bucket_metadata, remove_bucket_metadata,
remove_bucket_metadata_if_absent, set_bucket_metadata, update, update_bucket_targets_under_transaction_lock,
update_config_with, update_if_incarnation, update_if_incarnation_at, update_quota_if_incarnation,
update_quota_if_incarnation_at, update_under_transaction_lock, update_under_transaction_lock_at,
};
#[cfg(feature = "test-util")]
pub use crate::bucket::metadata_sys::{ConfigWriteLockProbe, test_support};
@@ -210,7 +210,8 @@ pub mod bucket {
pub mod migration {
pub use crate::bucket::migration::{
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
try_migrate_iam_config,
};
}
@@ -240,7 +241,10 @@ pub mod bucket {
}
pub mod quota {
pub use crate::bucket::quota::{BucketQuota, QuotaCheckResult, QuotaError, QuotaOperation};
pub use crate::bucket::quota::{
BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
QuotaOperation,
};
pub mod checker {
pub use crate::bucket::quota::checker::QuotaChecker;
@@ -362,10 +366,11 @@ pub mod config {
ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
is_server_config_corrupt_error, lookup_configs, read_config, read_config_no_lock, read_config_with_metadata,
read_config_without_migrate, read_config_without_migrate_no_lock, read_existing_server_config_no_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_server_config,
save_server_config_no_lock, save_server_config_snapshot, save_server_config_snapshot_with_generation,
server_config_path, try_migrate_server_config, with_config_object_read_lock, with_config_object_write_lock,
with_server_config_read_lock, with_server_config_write_lock,
read_server_config_snapshot, save_config, save_config_no_lock, save_config_with_opts, save_config_with_opts_quiet,
save_server_config, save_server_config_no_lock, save_server_config_snapshot,
save_server_config_snapshot_with_generation, server_config_path, try_migrate_server_config,
with_config_object_read_lock, with_config_object_write_lock, with_server_config_read_lock,
with_server_config_write_lock,
};
}
@@ -418,7 +423,9 @@ pub mod disk {
pub use crate::disk::os::{LocalPublicationPause, LocalPublicationStage};
pub use crate::disk::{
BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
CheckPartsResp, ConditionalFileUpdate, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION, DeleteOptions, DeleteRecoveryAction,
DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection, DeleteRecoveryPage, DeleteRecoveryRpcOperation,
DeleteRecoveryRpcReply, DeleteRecoveryState, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption,
DiskStore, FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, NsScannerOpenRequest, OldCurrentSize,
PartTransactionAction, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp, ReadOptions, RenameDataResp,
STORAGE_FORMAT_FILE, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
@@ -468,7 +475,7 @@ pub mod event {
pub mod global {
pub use crate::runtime::global::{
set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services,
update_erasure_type,
update_erasure_type, wait_for_detached_mutations,
};
}
@@ -515,13 +522,14 @@ pub mod integrity {
pub mod object {
pub use crate::object_api::{
BLOCK_SIZE_V2, ERASURE_ALGORITHM, EncryptionResolutionError, EncryptionResolutionErrorKind, GetObjectBodyCacheHook,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReader, NamespaceLockFence, ObjectEncryptionResolver,
ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions, PutObjReader, QuotaAdmission,
RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
GetObjectBodyCacheHookLookup, GetObjectBodySource, GetObjectReadCondition, GetObjectReadMetadata, GetObjectReader,
NamespaceLockFence, ObjectEncryptionResolver, ObjectInfo, ObjectLockConfigSnapshot, ObjectMutationHook, ObjectOptions,
PutObjReader, QuotaAdmission, RangedDecompressReader, ReadEncryptionMaterial, ReadEncryptionMode, ReadEncryptionRequest,
SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitStartError,
ScannerPublicationCommitState, ShardIntegrityWriteMode, StreamConsumer, WriteCommitGuard, WriteCompletion,
get_object_body_cache_plaintext_len, lookup_get_object_body_cache_hook, register_get_object_body_cache_hook,
register_object_mutation_hook, unregister_get_object_body_cache_hook, unregister_object_mutation_hook,
with_get_object_read_condition,
};
pub use crate::store::{
PrepareSelectObjectSnapshotError, PreparedGetObjectReader, SelectObjectSnapshot, SelectObjectSnapshotReadError,
@@ -586,6 +594,7 @@ pub mod set_disk {
#[cfg(feature = "test-util")]
pub mod test_util {
pub use crate::bucket::quota::reservation::fail_next_quota_ledger_save_for_test;
pub use crate::set_disk::ConditionalReadBenchmarkMetadataGuard;
pub use crate::set_disk::{MultipartCommitBarrier, MultipartCommitPause, PutObjectCommitBarrier, PutObjectCommitPause};
/// Keep a namespace commit pending until the returned owner is dropped.
@@ -87,6 +87,7 @@ use rustfs_filemeta::{
FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
is_restored_object_on_disk,
};
use rustfs_lock::{NamespaceLockGuard, error::LockError};
use rustfs_scanner_metrics::metrics::{
IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
};
@@ -190,6 +191,8 @@ const TIER_FREE_VERSION_RECOVERY_BASE_INTERVAL: StdDuration = StdDuration::from_
// before another full sweep can discover work persisted by another node.
const TIER_FREE_VERSION_RECOVERY_MAX_IDLE_INTERVAL: StdDuration = StdDuration::from_secs(10 * 60);
const TIER_FREE_VERSION_RECOVERY_JITTER_PERCENT: u64 = 10;
const TIER_FREE_VERSION_RECOVERY_LEADER_LOCK: &str = "tier-free-version-recovery/leader.lock";
const TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT: StdDuration = StdDuration::from_secs(1);
const DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS: i64 = 5;
const EXPIRY_WORKER_QUEUE_CAPACITY: usize = 1000;
const DEFAULT_MANUAL_TRANSITION_JOB_RECOVERY_LIMIT: usize = 100;
@@ -1441,13 +1444,24 @@ pub(crate) async fn enqueue_committed_free_versions(api: &ECStore, free_versions
queued
}
async fn enqueue_recovered_free_version_with_state(state: &Arc<RwLock<ExpiryState>>, oi: ObjectInfo) -> bool {
async fn enqueue_recovered_free_version_with_state(
state: &Arc<RwLock<ExpiryState>>,
oi: ObjectInfo,
cancel: &CancellationToken,
) -> bool {
let task = FreeVersionTask(oi);
let hash = task.op_hash();
let (wrkr, stats) = {
let state = state.read().await;
let state = select! {
biased;
_ = cancel.cancelled() => return false,
state = state.read() => state,
};
(state.get_worker_ch(hash), Arc::clone(&state.stats))
};
if cancel.is_cancelled() {
return false;
}
let Some(wrkr) = wrkr else {
stats.increment_missed_freevers_tasks();
stats.record_scanner_expiry_state();
@@ -1497,9 +1511,13 @@ fn set_recovered_free_version_enqueue_observer(
RecoveredFreeVersionEnqueueObserverGuard
}
pub async fn enqueue_recovered_free_version(api: &ECStore, oi: ObjectInfo) -> bool {
pub(super) async fn enqueue_recovered_free_version_with_cancel(
api: &ECStore,
oi: ObjectInfo,
cancel: &CancellationToken,
) -> bool {
let expiry_state = api.ctx.expiry_state();
let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi).await;
let queued = enqueue_recovered_free_version_with_state(&expiry_state, oi, cancel).await;
#[cfg(test)]
if let Some(observer) = RECOVERED_FREE_VERSION_ENQUEUE_OBSERVER
@@ -2399,6 +2417,16 @@ fn is_slow_down(err: &Error) -> bool {
matches!(err, Error::SlowDown)
}
/// Lifecycle expiry is idempotent when the exact object or version has already
/// disappeared between the scanner listing and the delete admission.
///
/// Keep this predicate narrower than bucket/volume "not found" helpers: an
/// unavailable bucket or disk is an operational failure and must not be
/// reported as a completed expiry.
fn is_lifecycle_expiry_already_gone(err: &Error) -> bool {
is_err_object_not_found(err) || is_err_version_not_found(err)
}
/// Resolves the expiry worker count from the single documented knob,
/// `RUSTFS_MAX_EXPIRY_WORKERS`: a set, parsable, non-zero value wins;
/// anything else falls back to `min(cpus, 16)`. The historical
@@ -2921,6 +2949,20 @@ fn resolve_tier_free_version_recovery_enabled(value: Result<String, env::VarErro
}
fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<()>) -> Option<JoinHandle<()>> {
spawn_tier_free_version_recovery_once_with_options(
api,
started,
TIER_FREE_VERSION_RECOVERY_LEADER_LOCK.to_string(),
tier_free_version_recovery_leader_retry_delay,
)
}
fn spawn_tier_free_version_recovery_once_with_options(
api: Arc<ECStore>,
started: &OnceLock<()>,
leader_lock_name: String,
retry_delay: fn() -> StdDuration,
) -> Option<JoinHandle<()>> {
if !tier_free_version_recovery_enabled() {
warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
@@ -2937,20 +2979,43 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<(
Some(tokio::spawn(async move {
let cancel_token = api.ctx.background_cancel_token().unwrap_or_default();
let expiry_state = api.ctx.expiry_state();
run_tier_free_version_recovery_loop(
let acquire_api = Arc::clone(&api);
let acquire_lock_name = Arc::<str>::from(leader_lock_name);
run_tier_free_version_recovery_coordinator(
cancel_token,
expiry_state,
jitter_tier_free_version_recovery_delay,
move |bucket_marker, object_marker, recovery_cancel| {
retry_delay,
move || {
let api = Arc::clone(&acquire_api);
let lock_name = Arc::clone(&acquire_lock_name);
async move {
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &lock_name).await?;
match lock.get_write_lock_quiet(TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT).await {
Ok(guard) => Ok(Some(guard)),
Err(LockError::Timeout { .. }) => Ok(None),
Err(err) => Err(err.into()),
}
}
},
move |leader_cancel| {
let api = Arc::clone(&api);
async move {
recover_tier_free_versions_with_cancel(
api,
DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
bucket_marker,
object_marker,
recovery_cancel,
run_tier_free_version_recovery_loop(
leader_cancel,
api.ctx.expiry_state(),
jitter_tier_free_version_recovery_delay,
move |bucket_marker, object_marker, recovery_cancel| {
let api = Arc::clone(&api);
async move {
recover_tier_free_versions_with_cancel(
api,
DEFAULT_FREE_VERSION_RECOVERY_LIMIT,
bucket_marker,
object_marker,
recovery_cancel,
)
.await
}
},
)
.await
}
@@ -2960,6 +3025,86 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<(
}))
}
fn tier_free_version_recovery_leader_retry_delay() -> StdDuration {
StdDuration::from_secs(rand::rng().random_range(15..=30))
}
async fn run_tier_free_version_recovery_coordinator<A, AF, R, RF>(
cancel: CancellationToken,
retry_delay: fn() -> StdDuration,
mut acquire: A,
mut recover: R,
) where
A: FnMut() -> AF,
AF: Future<Output = crate::error::Result<Option<NamespaceLockGuard>>>,
R: FnMut(CancellationToken) -> RF,
RF: Future<Output = ()>,
{
loop {
let acquired = select! {
biased;
_ = cancel.cancelled() => return,
result = acquire() => result,
};
match acquired {
Ok(Some(mut guard)) => {
let leader_cancel = cancel.child_token();
// Hold ownership across every page, cooldown and idle wait. A
// follower taking over starts a fresh sweep of persisted markers.
let recovery = recover(leader_cancel.clone());
tokio::pin!(recovery);
debug!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "free_version_recovery_leader_acquired",
lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
"Tier free-version recovery leader acquired"
);
select! {
biased;
_ = cancel.cancelled() => {
leader_cancel.cancel();
recovery.await;
}
_ = guard.lock_lost_notified() => {
debug!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "free_version_recovery_leader_lost",
lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
"Tier free-version recovery leader lost"
);
// Dropping the recovery future would detach its active
// walk. Cancel it and poll the same future through cleanup.
leader_cancel.cancel();
recovery.await;
}
_ = &mut recovery => {}
}
leader_cancel.cancel();
guard.release();
}
Ok(None) => {}
Err(err) => warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
state = "free_version_recovery_leader_acquire_failed",
lock_name = TIER_FREE_VERSION_RECOVERY_LEADER_LOCK,
error = %err,
"Tier free-version recovery leader acquisition failed"
),
}
select! {
biased;
_ = cancel.cancelled() => return,
_ = tokio::time::sleep(retry_delay()) => {}
}
}
}
async fn run_tier_free_version_recovery_loop<F, Fut>(
cancel_token: CancellationToken,
expiry_state: Arc<RwLock<ExpiryState>>,
@@ -2969,7 +3114,11 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
F: FnMut(Option<String>, Option<String>, CancellationToken) -> Fut,
Fut: Future<Output = crate::error::Result<FreeVersionRecoveryStats>>,
{
let recovery_notify = Arc::clone(&expiry_state.read().await.recovery_notify);
let recovery_notify = select! {
biased;
_ = cancel_token.cancelled() => return,
state = expiry_state.read() => Arc::clone(&state.recovery_notify),
};
let mut schedule = TierFreeVersionRecoverySchedule::default();
loop {
@@ -2992,7 +3141,11 @@ async fn run_tier_free_version_recovery_loop<F, Fut>(
schedule.record_success(&stats);
rustfs_io_metrics::record_stage_duration("lifecycle_free_version_recovery", elapsed.as_secs_f64() * 1000.0);
let (pending_tasks, active_tasks) = {
let state = expiry_state.read().await;
let state = select! {
biased;
_ = cancel_token.cancelled() => return,
state = expiry_state.read() => state,
};
(state.pending_tasks(), state.stats.active_tasks())
};
debug!(
@@ -4748,7 +4901,7 @@ async fn expire_transitioned_object_with_lock_lost_signal(
bucket_incarnation_id: Uuid,
lock_lost_signal: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
) -> Result<ObjectInfo, std::io::Error> {
let publication_guard = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id)
let publication_guard = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id)
.await
.ok_or_else(|| std::io::Error::other("lifecycle expiry is not allowed for this bucket"))?;
let snapshot = lifecycle_delete_config_snapshot(&api, oi)
@@ -4777,6 +4930,20 @@ async fn expire_transitioned_object_with_lock_lost_signal(
//audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
Ok(dobj)
}
Err(err) if is_lifecycle_expiry_already_gone(&err) => {
debug!(
event = EVENT_LIFECYCLE_DELETE_COMPLETED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
operation = "delete_restored_object",
reason = "already_gone",
"Lifecycle delete target was already absent"
);
crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
Ok(oi.clone())
}
Err(err) => Err(std::io::Error::other(err)),
};
}
@@ -4787,6 +4954,8 @@ async fn expire_transitioned_object_with_lock_lost_signal(
// pointing to a remote tier version that has already been removed. If this
// only creates a delete marker, remote cleanup must be driven by persisted
// free-version recovery rather than the visible delete result.
// Keep remote transitioned not-found errors visible: local metadata
// absence alone does not prove that the tier object was cleaned up.
let dobj = match api.delete_object(&oi.bucket, &oi.name, opts).await {
Ok(obj) => obj,
Err(e) => {
@@ -4838,10 +5007,14 @@ pub fn gen_transition_objname(bucket: &str) -> Result<String, Error> {
pub async fn transition_object(api: Arc<ECStore>, oi: &ObjectInfo, lae: LcAuditEvent) -> Result<(), Error> {
let time_ilm = Metrics::time_ilm(lae.event.action);
let (publication_guard, bucket_incarnation_id) = lifecycle_transition_publication_guard(&api, oi)
.await
.ok_or_else(|| Error::other("lifecycle transition is not allowed for this bucket"))?;
let etag = if let Some(etag) = &oi.etag { etag } else { "" };
let etag = etag.to_string();
let opts = ObjectOptions {
let mut opts = ObjectOptions {
transition: TransitionOptions {
status: lifecycle::TRANSITION_PENDING.to_string(),
tier: lae.event.storage_class,
@@ -4853,8 +5026,10 @@ pub async fn transition_object(api: Arc<ECStore>, oi: &ObjectInfo, lae: LcAuditE
versioned: BucketVersioningSys::prefix_enabled(&oi.bucket, &oi.name).await,
version_suspended: BucketVersioningSys::prefix_suspended(&oi.bucket, &oi.name).await,
mod_time: oi.mod_time,
expected_bucket_incarnation_id: Some(bucket_incarnation_id),
..Default::default()
};
opts.add_namespace_lock_guard(&publication_guard);
let result = api.transition_object(&oi.bucket, &oi.name, &opts).await;
time_ilm(1)();
result
@@ -5458,20 +5633,26 @@ pub async fn apply_transition_rule(event: &lifecycle::Event, src: &LcEventSrc, o
.await
}
async fn lifecycle_expiry_publication_guard(
async fn acquire_lifecycle_publication_guard(api: &ECStore, bucket: &str) -> Result<rustfs_lock::NamespaceLockGuard, Error> {
let lock = api
.new_ns_lock(bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await?;
let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
if guard.is_lock_lost() {
return Err(Error::other("table-bucket publication lock was lost before lifecycle admission"));
}
Ok(guard)
}
pub(crate) async fn lifecycle_expiry_publication_guard(
api: &ECStore,
oi: &ObjectInfo,
bucket: &str,
object: &str,
bucket_incarnation_id: Uuid,
) -> Option<rustfs_lock::NamespaceLockGuard> {
let result = async {
let lock = api
.new_ns_lock(&oi.bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await?;
let guard = lock.get_read_lock(get_lock_acquire_timeout()).await.map_err(Error::other)?;
if guard.is_lock_lost() {
return Err(Error::other("table-bucket publication lock was lost before lifecycle delete admission"));
}
if !metadata_boundary::lifecycle_expiry_allowed(api, &oi.bucket, bucket_incarnation_id).await? {
let result: std::result::Result<Option<rustfs_lock::NamespaceLockGuard>, Error> = async {
let guard = acquire_lifecycle_publication_guard(api, bucket).await?;
if !metadata_boundary::lifecycle_expiry_allowed(api, bucket, bucket_incarnation_id).await? {
return Ok(None);
}
Ok(Some(guard))
@@ -5484,8 +5665,8 @@ async fn lifecycle_expiry_publication_guard(
event = EVENT_LIFECYCLE_DELETE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
bucket,
object,
operation = "authorize_lifecycle_expiry",
error = %err,
"Lifecycle delete admission failed"
@@ -5495,6 +5676,34 @@ async fn lifecycle_expiry_publication_guard(
}
}
async fn lifecycle_transition_publication_guard(
api: &ECStore,
oi: &ObjectInfo,
) -> Option<(rustfs_lock::NamespaceLockGuard, Uuid)> {
let result: std::result::Result<(rustfs_lock::NamespaceLockGuard, Uuid), Error> = async {
let guard = acquire_lifecycle_publication_guard(api, &oi.bucket).await?;
let configs = metadata_boundary::get_expiry_configs(api, &oi.bucket).await?;
Ok((guard, configs.bucket_incarnation_id))
}
.await;
match result {
Ok(state) => Some(state),
Err(err) => {
warn!(
event = EVENT_LIFECYCLE_TIER_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
operation = "authorize_lifecycle_transition",
error = %err,
"Lifecycle transition admission failed"
);
None
}
}
}
pub async fn apply_expiry_on_transitioned_object(
api: Arc<ECStore>,
oi: &ObjectInfo,
@@ -5551,7 +5760,8 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
bucket_incarnation_id: Uuid,
lock_lost_signal: Option<Arc<rustfs_lock::distributed_lock::LockLostSignal>>,
) -> bool {
let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, oi, bucket_incarnation_id).await else {
let Some(publication_guard) = lifecycle_expiry_publication_guard(&api, &oi.bucket, &oi.name, bucket_incarnation_id).await
else {
return false;
};
let snapshot = match lifecycle_delete_config_snapshot(&api, oi).await {
@@ -5611,11 +5821,27 @@ async fn apply_expiry_on_non_transitioned_objects_with_lock_lost_signal(
} else {
None
};
// The delete owns the publication guard from here on, so a cancelled
// expiry worker cannot let table-bucket publication overtake it.
let mut dobj = match api
.delete_object_with_tier_delete_journal(&oi.bucket, &encode_dir_object(&oi.name), opts)
.delete_object_with_tier_delete_journal_and_guards(&oi.bucket, &encode_dir_object(&oi.name), opts, [publication_guard])
.await
{
Ok(dobj) => dobj,
Err(err) if is_lifecycle_expiry_already_gone(&err) => {
debug!(
event = EVENT_LIFECYCLE_DELETE_COMPLETED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket = %oi.bucket,
object = %oi.name,
operation = "delete_object",
reason = "already_gone",
"Lifecycle delete target was already absent"
);
crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
return true;
}
Err(e) => {
error!(
event = EVENT_LIFECYCLE_DELETE_FAILED,
@@ -5844,6 +6070,7 @@ pub async fn apply_lifecycle_action(event: &lifecycle::Event, src: &LcEventSrc,
#[cfg(test)]
mod tests {
use super::expiry_worker_count;
use super::lifecycle_transition_publication_guard;
use super::{
DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS, DEFAULT_TRANSITION_QUEUE_CAPACITY, DEFAULT_TRANSITION_WORKERS_ABSOLUTE_MAX,
DEFAULT_TRANSITION_WORKERS_CAP, EVENT_LIFECYCLE_EVALUATION_FAILED, EVENT_LIFECYCLE_EXPIRED_DETECTED,
@@ -5854,15 +6081,16 @@ mod tests {
cleanup_empty_multipart_sha_dirs_on_local_disks, cleanup_stale_multipart_uploads_once_at,
enqueue_recovered_free_version_with_state, enqueue_transition_for_existing_objects_scoped,
enqueue_transition_with_lifecycle, enqueue_transition_with_lifecycle_report, eval_action_from_lifecycle,
get_lock_acquire_timeout, jitter_tier_free_version_recovery_delay, lifecycle_action_blocked_by_replication,
lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object, lifecycle_replication_blocks_action,
lifecycle_rule_has_date_expiration, lifecycle_version_delete_target, manual_transition_duration_elapsed,
manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink, manual_transition_version_marker,
manual_transition_worker_failure_reason, mark_delete_opts_skip_decommissioned_on_remote_success,
merge_stale_multipart_candidate, persist_manual_transition_job_progress_if_owned,
persist_manual_transition_page_checkpoint, recover_manual_transition_job, recover_manual_transition_jobs,
resolve_tier_free_version_recovery_enabled, resolve_transition_queue_capacity, resolve_transition_queue_send_timeout,
resolve_transition_worker_count, resolve_transition_workers_absolute_max, run_tier_free_version_recovery_loop,
get_lock_acquire_timeout, is_lifecycle_expiry_already_gone, jitter_tier_free_version_recovery_delay,
lifecycle_action_blocked_by_replication, lifecycle_delete_all_versions_replication_scan, lifecycle_deleted_object,
lifecycle_replication_blocks_action, lifecycle_rule_has_date_expiration, lifecycle_version_delete_target,
manual_transition_duration_elapsed, manual_transition_has_more_after_limit, manual_transition_recovery_progress_sink,
manual_transition_version_marker, manual_transition_worker_failure_reason,
mark_delete_opts_skip_decommissioned_on_remote_success, merge_stale_multipart_candidate,
persist_manual_transition_job_progress_if_owned, persist_manual_transition_page_checkpoint,
recover_manual_transition_job, recover_manual_transition_jobs, resolve_tier_free_version_recovery_enabled,
resolve_transition_queue_capacity, resolve_transition_queue_send_timeout, resolve_transition_worker_count,
resolve_transition_workers_absolute_max, run_tier_free_version_recovery_coordinator, run_tier_free_version_recovery_loop,
select_restore_s3_location, set_lifecycle_observability_observer, set_recovered_free_version_enqueue_observer,
should_defer_date_expiry_for_recent_config_update, transitioned_cleanup_tuple, transitioned_object_delete_opts,
wait_for_tier_free_version_recovery,
@@ -5959,6 +6187,231 @@ mod tests {
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
fn recovery_test_lock() -> (Arc<rustfs_lock::NamespaceLock>, Vec<Arc<dyn rustfs_lock::LockClient>>) {
let clients: Vec<Arc<dyn rustfs_lock::LockClient>> = (0..3)
.map(|_| {
Arc::new(rustfs_lock::LocalClient::with_manager(Arc::new(rustfs_lock::GlobalLockManager::new())))
as Arc<dyn rustfs_lock::LockClient>
})
.collect();
let lock =
rustfs_lock::NamespaceLock::with_clients_and_quorum(format!("recovery-test-{}", Uuid::new_v4()), clients.clone(), 2);
(Arc::new(lock), clients)
}
async fn acquire_recovery_test_lock(
lock: Arc<rustfs_lock::NamespaceLock>,
owner: String,
) -> crate::error::Result<Option<rustfs_lock::NamespaceLockGuard>> {
// Keep this lease longer than the virtual-time cooldown/idle advances;
// production uses the namespace wrapper's renewable 30-second lease.
lock.lock_guard(
rustfs_lock::ObjectKey::new(crate::disk::RUSTFS_META_BUCKET, super::TIER_FREE_VERSION_RECOVERY_LEADER_LOCK),
&owner,
StdDuration::from_millis(5),
StdDuration::from_secs(3600),
)
.await
.map_err(Into::into)
}
fn recovery_test_retry_delay() -> StdDuration {
StdDuration::from_secs(15)
}
#[tokio::test(start_paused = true)]
#[serial]
async fn tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over() {
let (lock, _clients) = recovery_test_lock();
let (pages_tx, mut pages_rx) = tokio::sync::mpsc::unbounded_channel();
let mut tasks = Vec::new();
let mut cancels = Vec::new();
for node in 0..8 {
let cancel = CancellationToken::new();
cancels.push(cancel.clone());
let lock = Arc::clone(&lock);
let pages_tx = pages_tx.clone();
tasks.push(tokio::spawn(async move {
run_tier_free_version_recovery_coordinator(
cancel,
recovery_test_retry_delay,
move || acquire_recovery_test_lock(Arc::clone(&lock), format!("node-{node}")),
move |leader_cancel| {
let pages_tx = pages_tx.clone();
async move {
let mut pages = 0;
run_tier_free_version_recovery_loop(
leader_cancel,
ExpiryState::new(),
std::convert::identity,
move |bucket, object, _| {
pages += 1;
pages_tx.send((node, bucket, object)).expect("page observer should be open");
std::future::ready(Ok(free_version_recovery_stats(0, 0, pages == 1)))
},
)
.await;
}
},
)
.await;
}));
}
let (leader, bucket, object) = pages_rx.recv().await.expect("one coordinator should start recovery");
assert!(bucket.is_none() && object.is_none());
// The same leader must survive both a truncated page's cooldown and
// complete idle sweeps. All other coordinators keep probing the lock.
let mut observed_pages = 1;
for _ in 0..27 {
tokio::time::advance(StdDuration::from_secs(60)).await;
for _ in 0..100 {
tokio::task::yield_now().await;
}
while let Ok((node, bucket, object)) = pages_rx.try_recv() {
assert_eq!(node, leader, "followers must not repeat a namespace sweep");
if observed_pages == 1 {
assert_eq!(bucket.as_deref(), Some("bucket"));
assert_eq!(object.as_deref(), Some("object"));
} else {
assert!(bucket.is_none() && object.is_none(), "complete sweeps should reset their cursors");
}
observed_pages += 1;
}
}
assert!(observed_pages >= 7, "the leader should finish a full ten-minute idle wait");
cancels[leader].cancel();
let (successor, bucket, object) = tokio::time::timeout(StdDuration::from_secs(60), pages_rx.recv())
.await
.expect("a follower should take over after leader shutdown")
.expect("page observer should remain open");
assert_ne!(successor, leader);
assert!(bucket.is_none() && object.is_none(), "a new leader must restart discovery safely");
for cancel in cancels {
cancel.cancel();
}
for task in tasks {
task.await.expect("all coordinators should shut down cleanly");
}
}
#[tokio::test(start_paused = true)]
#[serial]
async fn tier_free_version_recovery_lock_loss_waits_for_active_page_cleanup() {
let (lock, _clients) = recovery_test_lock();
let cancel = CancellationToken::new();
let loop_cancel = cancel.clone();
let (page_tx, mut page_rx) = tokio::sync::mpsc::unbounded_channel();
let cleanup_started = Arc::new(tokio::sync::Notify::new());
let finish_cleanup = Arc::new(tokio::sync::Notify::new());
let observed_cleanup = Arc::clone(&cleanup_started);
let release_cleanup = Arc::clone(&finish_cleanup);
let task = tokio::spawn(async move {
run_tier_free_version_recovery_coordinator(
loop_cancel,
recovery_test_retry_delay,
move || acquire_recovery_test_lock(Arc::clone(&lock), "old-leader".to_string()),
move |leader_cancel| {
let page_tx = page_tx.clone();
let cleanup_started = Arc::clone(&cleanup_started);
let finish_cleanup = Arc::clone(&finish_cleanup);
async move {
run_tier_free_version_recovery_loop(
leader_cancel,
ExpiryState::new(),
std::convert::identity,
move |_, _, page_cancel| {
let cleanup_started = Arc::clone(&cleanup_started);
let finish_cleanup = Arc::clone(&finish_cleanup);
page_tx.send(()).expect("page observer should remain open");
async move {
page_cancel.cancelled().await;
cleanup_started.notify_one();
finish_cleanup.notified().await;
Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "cancelled page").into())
}
},
)
.await;
}
},
)
.await;
});
page_rx.recv().await.expect("leader should start an active page");
// Let authoritative backend leases expire before the next heartbeat
// can renew them. A failed refresh must cancel the active page.
tokio::time::advance(StdDuration::from_secs(3600)).await;
tokio::time::timeout(StdDuration::from_secs(1), observed_cleanup.notified())
.await
.expect("expired backend leases must trigger recovery cancellation");
assert!(!task.is_finished(), "lock loss must wait for the active page's cleanup");
assert!(page_rx.try_recv().is_err(), "the old leader must not start another page");
cancel.cancel();
release_cleanup.notify_one();
task.await.expect("coordinator should finish cleanup before exiting");
}
#[tokio::test(start_paused = true)]
async fn tier_free_version_recovery_coordinator_retries_without_scanning_after_lock_error() {
let cancel = CancellationToken::new();
let loop_cancel = cancel.clone();
let attempts = Arc::new(AtomicUsize::new(0));
let observed_attempts = Arc::clone(&attempts);
let task = tokio::spawn(async move {
run_tier_free_version_recovery_coordinator(
loop_cancel,
recovery_test_retry_delay,
move || {
observed_attempts.fetch_add(1, Ordering::SeqCst);
std::future::ready(Err(rustfs_lock::LockError::InsufficientNodes {
required: 2,
available: 1,
}
.into()))
},
|_| async { panic!("a node without ownership must never start discovery") },
)
.await;
});
tokio::task::yield_now().await;
assert_eq!(attempts.load(Ordering::SeqCst), 1);
tokio::time::advance(StdDuration::from_secs(14)).await;
assert_eq!(attempts.load(Ordering::SeqCst), 1, "lock errors must not cause a tight retry loop");
tokio::time::advance(StdDuration::from_secs(1)).await;
tokio::task::yield_now().await;
assert_eq!(attempts.load(Ordering::SeqCst), 2);
cancel.cancel();
task.await.expect("a follower should respond to shutdown during retry wait");
}
#[tokio::test]
async fn tier_free_version_recovery_enqueue_cancels_while_waiting_for_state_lock() {
let state = ExpiryState::new();
let (tx, mut rx) = tokio::sync::mpsc::channel(1);
let mut held = state.write().await;
held.tasks_tx.push(tx);
let cancel = CancellationToken::new();
let enqueue_cancel = cancel.clone();
let enqueue_state = Arc::clone(&state);
let enqueue = tokio::spawn(async move {
enqueue_recovered_free_version_with_state(&enqueue_state, ObjectInfo::default(), &enqueue_cancel).await
});
tokio::task::yield_now().await;
assert!(!enqueue.is_finished(), "enqueue should be blocked on the state lock");
cancel.cancel();
let queued = tokio::time::timeout(StdDuration::from_secs(1), enqueue)
.await
.expect("cancellation must not wait for the state write guard")
.expect("enqueue should return cleanly");
assert!(!queued);
assert_eq!(held.pending_tasks(), 0);
assert_eq!(held.stats.missed_free_vers_tasks(), 0);
assert!(rx.try_recv().is_err(), "a cancelled recovery must not enqueue a task");
drop(held);
assert!(!enqueue_recovered_free_version_with_state(&state, ObjectInfo::default(), &cancel).await);
assert!(rx.try_recv().is_err(), "cancellation must win even when the lock is available");
}
fn free_version_recovery_stats(enqueued: usize, failed: usize, truncated: bool) -> FreeVersionRecoveryStats {
FreeVersionRecoveryStats {
scanned: enqueued.saturating_add(failed),
@@ -7638,7 +8091,7 @@ mod tests {
assert_eq!(state.read().await.stats.pending_tasks(), 1);
// The single-slot queue is full for both enqueue paths.
assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone()).await);
assert!(!enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await);
assert_eq!(state.read().await.stats.pending_tasks(), 1);
assert_eq!(state.read().await.stats.missed_free_vers_tasks(), 1);
@@ -7670,7 +8123,7 @@ mod tests {
..Default::default()
};
let queued = enqueue_recovered_free_version_with_state(&state, oi).await;
let queued = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
let state = state.read().await;
assert!(!queued);
@@ -7799,8 +8252,8 @@ mod tests {
..Default::default()
};
let first = enqueue_recovered_free_version_with_state(&state, oi.clone()).await;
let second = enqueue_recovered_free_version_with_state(&state, oi).await;
let first = enqueue_recovered_free_version_with_state(&state, oi.clone(), &CancellationToken::new()).await;
let second = enqueue_recovered_free_version_with_state(&state, oi, &CancellationToken::new()).await;
let state = state.read().await;
assert!(first);
@@ -7901,7 +8354,7 @@ mod tests {
};
assert!(
super::enqueue_recovered_free_version(&ecstore, oi).await,
super::enqueue_recovered_free_version_with_cancel(&ecstore, oi, &CancellationToken::new()).await,
"the resized production worker queue should accept the task"
);
stop_tx.send(None).await.expect("worker stop signal should be delivered");
@@ -10099,6 +10552,26 @@ mod tests {
);
}
#[test]
fn lifecycle_expiry_already_gone_only_accepts_object_or_version_absence() {
assert!(is_lifecycle_expiry_already_gone(&Error::FileNotFound));
assert!(is_lifecycle_expiry_already_gone(&Error::ObjectNotFound(
"bucket".to_string(),
"object".to_string(),
)));
assert!(is_lifecycle_expiry_already_gone(&Error::FileVersionNotFound));
assert!(is_lifecycle_expiry_already_gone(&Error::VersionNotFound(
"bucket".to_string(),
"object".to_string(),
"version".to_string(),
)));
assert!(!is_lifecycle_expiry_already_gone(&Error::VolumeNotFound));
assert!(!is_lifecycle_expiry_already_gone(&Error::DiskNotFound));
assert!(!is_lifecycle_expiry_already_gone(&Error::ErasureReadQuorum));
assert!(!is_lifecycle_expiry_already_gone(&Error::MethodNotAllowed));
}
#[tokio::test]
async fn manual_transition_counts_already_transitioned_object() {
let lc = latest_transition_lifecycle();
@@ -12973,7 +13446,9 @@ mod tests {
.bucket_incarnation_id_from_disk(&bucket)
.await
.expect("bucket incarnation");
for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, false)] {
// Replaying the same scanner snapshot after the delete is an
// already-gone success and must not be reported as a failure.
for (attempt_incarnation, expected) in [(Uuid::new_v4(), false), (incarnation, true), (incarnation, true)] {
assert_eq!(
super::apply_expiry_on_non_transitioned_objects(
ecstore.clone(),
@@ -13028,6 +13503,171 @@ mod tests {
}
}
#[tokio::test]
#[serial]
async fn queued_noncurrent_batch_expiry_does_not_delete_from_table_bucket() {
let (_disk_paths, ecstore) = setup_test_env().await;
let bucket = format!("table-noncurrent-lifecycle-{}", Uuid::new_v4().simple());
let object = "tables/table-id/data/part-00001.parquet";
create_test_bucket(&ecstore, &bucket).await;
metadata_sys::update_in(
&ecstore.ctx,
&bucket,
BUCKET_VERSIONING_CONFIG,
b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
)
.await
.expect("bucket versioning should be enabled");
let mut old_reader = PutObjReader::from_vec(b"old table data".to_vec());
let old = ecstore
.put_object(
&bucket,
object,
&mut old_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("the historical table version should be created");
let old_version_id = old.version_id.expect("the historical version should have an identity");
let mut current_reader = PutObjReader::from_vec(b"current table data".to_vec());
ecstore
.put_object(
&bucket,
object,
&mut current_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("the current table version should be created");
let incarnation = ecstore
.bucket_incarnation_id_from_disk(&bucket)
.await
.expect("bucket incarnation should be available");
let publication_lock = ecstore
.new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await
.expect("table-bucket publication lock should be created");
let enable_guard = publication_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.expect("table-bucket enablement should acquire the publication lock");
let delete_store = ecstore.clone();
let delete_bucket = bucket.clone();
let delete_object = object.to_string();
let mut delete = tokio::spawn(async move {
let target = ObjectToDelete {
object_name: delete_object,
version_id: Some(old_version_id),
..Default::default()
};
crate::bucket::lifecycle::object_handlers_common::delete_object_versions(
&delete_store,
&delete_bucket,
std::slice::from_ref(&target),
lifecycle::Event::default(),
incarnation,
)
.await
});
tokio::task::yield_now().await;
assert!(
tokio::time::timeout(StdDuration::from_millis(100), &mut delete)
.await
.is_err(),
"noncurrent lifecycle cleanup must wait for the table-bucket publication fence"
);
let sys = metadata_sys::bucket_metadata_sys_of(&ecstore.ctx).expect("metadata system should be initialized");
let sys = sys.read().await.clone();
let mut metadata = (*sys.get(&bucket).await.expect("bucket metadata should exist")).clone();
metadata.table_bucket_config_json = br#"{"enabled":true}"#.to_vec();
sys.persist_and_set(metadata)
.await
.expect("table bucket marker should be persisted");
sys.reload_from_store(&bucket)
.await
.expect("table bucket marker should become authoritative");
drop(enable_guard);
let failed = tokio::time::timeout(StdDuration::from_secs(2), delete)
.await
.expect("noncurrent lifecycle cleanup should resume after the fence is released")
.expect("noncurrent lifecycle cleanup task should join");
assert_eq!(failed, 1, "table-bucket noncurrent cleanup must be rejected");
assert!(
ecstore
.get_object_info(
&bucket,
object,
&ObjectOptions {
version_id: Some(old_version_id.to_string()),
versioned: true,
..Default::default()
},
)
.await
.is_ok(),
"table data must remain after queued noncurrent cleanup is rejected"
);
}
#[tokio::test]
#[serial]
async fn lifecycle_transition_admission_holds_publication_reader() {
let (_disk_paths, ecstore) = setup_test_env().await;
let bucket = format!("lifecycle-transition-{}", Uuid::new_v4().simple());
let object = "tables/table-id/data/part-00001.parquet";
create_test_bucket(&ecstore, &bucket).await;
let object_info = ObjectInfo {
bucket: bucket.clone(),
name: object.to_string(),
..Default::default()
};
let publication_lock = ecstore
.new_ns_lock(&bucket, rustfs_common::table_catalog::TABLE_BUCKET_PUBLICATION_LOCK_PATH)
.await
.expect("table-bucket publication lock should be created");
let enable_guard = publication_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.expect("publication writer should be acquired");
let admission_store = ecstore.clone();
let admission_object = object_info.clone();
let mut admission =
tokio::spawn(async move { lifecycle_transition_publication_guard(&admission_store, &admission_object).await });
assert!(
tokio::time::timeout(StdDuration::from_millis(100), &mut admission)
.await
.is_err(),
"transition admission must wait for an in-progress table publication"
);
drop(enable_guard);
let (transition_guard, incarnation) = tokio::time::timeout(StdDuration::from_secs(2), admission)
.await
.expect("transition admission should resume after publication")
.expect("transition admission task should join")
.expect("transition admission should succeed");
assert_ne!(incarnation, Uuid::nil(), "transition admission must bind a bucket incarnation");
let blocked_writer = publication_lock.get_write_lock(StdDuration::from_millis(100)).await;
assert!(blocked_writer.is_err(), "transition admission must retain the publication read lock");
drop(transition_guard);
publication_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.expect("publication writer should proceed after transition admission releases its guard");
}
#[tokio::test]
#[serial]
async fn queued_delete_all_rechecks_a_same_id_rule_moved_into_the_future() {
@@ -13630,6 +14270,10 @@ mod tests {
.await;
}
fn fast_tier_free_version_recovery_leader_retry_delay() -> StdDuration {
StdDuration::from_millis(10)
}
#[tokio::test]
#[serial]
async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
@@ -13645,8 +14289,13 @@ mod tests {
seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
let started = OnceLock::new();
let recovery = super::spawn_tier_free_version_recovery_once(Arc::clone(&ecstore), &started)
.expect("production recovery entrypoint should start once");
let recovery = super::spawn_tier_free_version_recovery_once_with_options(
Arc::clone(&ecstore),
&started,
format!("tier-free-version-recovery/test/{}", Uuid::new_v4()),
fast_tier_free_version_recovery_leader_retry_delay,
)
.expect("production recovery entrypoint should start once");
let task = tokio::time::timeout(StdDuration::from_secs(30), async { recovery_rx.lock().await.recv().await })
.await
.expect("production recovery entrypoint should enqueue the seeded item")
@@ -13689,13 +14338,14 @@ mod tests {
};
let mut recovery_rx = recovery_rx.lock().await;
assert!(
super::enqueue_recovered_free_version(
super::enqueue_recovered_free_version_with_cancel(
&ecstore,
ObjectInfo {
bucket: "prefill".to_string(),
name: "prefill".to_string(),
..Default::default()
},
&CancellationToken::new(),
)
.await,
"the production recovery queue should accept its first task"
@@ -36,6 +36,10 @@ pub async fn delete_object_versions(
_lc_event: lifecycle::Event,
bucket_incarnation_id: Uuid,
) -> usize {
if to_del.is_empty() {
return 0;
}
if to_del.iter().any(|target| {
target.version_id.is_none()
|| (target.version_id.is_some_and(|version_id| version_id.is_nil()) && target.expected_identity.is_none())
@@ -52,6 +56,25 @@ pub async fn delete_object_versions(
return to_del.len();
}
let Some(publication_guard) = super::bucket_lifecycle_ops::lifecycle_expiry_publication_guard(
api,
bucket,
to_del[0].object_name.as_str(),
bucket_incarnation_id,
)
.await
else {
debug!(
event = EVENT_LIFECYCLE_CLEANUP_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_LIFECYCLE,
bucket,
target_count = to_del.len(),
reason = "publication_admission_rejected",
"Skipped lifecycle noncurrent version cleanup"
);
return to_del.len();
};
let delete_config_snapshot = match ReplicationObjectBridge::delete_request_config(api, bucket).await {
Ok(snapshot) => Arc::new(snapshot),
Err(err) => {
@@ -78,17 +101,13 @@ pub async fn delete_object_versions(
remaining = &[];
}
let (mut deleted_objs, errors) = api
.delete_objects(
bucket,
to_del.to_vec(),
ObjectOptions {
delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)),
expected_bucket_incarnation_id: Some(bucket_incarnation_id),
..Default::default()
},
)
.await;
let mut delete_options = ObjectOptions {
delete_replication_config_snapshot: Some(Arc::clone(&delete_config_snapshot)),
expected_bucket_incarnation_id: Some(bucket_incarnation_id),
..Default::default()
};
delete_options.add_namespace_lock_guard(&publication_guard);
let (mut deleted_objs, errors) = api.delete_objects(bucket, to_del.to_vec(), delete_options).await;
failed += errors.iter().filter(|err| err.is_some()).count();
for (i, deleted_obj) in deleted_objs.iter_mut().enumerate() {
@@ -20,7 +20,7 @@ use std::time::Duration;
use tokio::sync::mpsc;
use tokio_util::sync::CancellationToken;
use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version;
use crate::bucket::lifecycle::bucket_lifecycle_ops::enqueue_recovered_free_version_with_cancel;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::Result;
use crate::object_api::ObjectInfo;
@@ -201,7 +201,11 @@ pub(super) async fn recover_tier_free_versions_with_cancel(
return Err(tier_free_version_recovery_cancelled());
}
retry_cursor.visit(&oi);
if !record_recovered_free_version_enqueue(&mut stats, enqueue_recovered_free_version(&api, oi).await) {
let queued = enqueue_recovered_free_version_with_cancel(&api, oi, &cancel_token).await;
if cancel_token.is_cancelled() {
return Err(tier_free_version_recovery_cancelled());
}
if !record_recovered_free_version_enqueue(&mut stats, queued) {
let (bucket_marker, object_marker) = retry_cursor.retry_markers();
stats.truncated = true;
stats.next_bucket_marker = bucket_marker;
+161
View File
@@ -1210,6 +1210,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.notification_config = None;
if !self.notification_config_xml.is_empty()
&& let Err(e) = deserialize::<NotificationConfiguration>(&self.notification_config_xml)
.map(|c| self.notification_config = Some(c))
@@ -1224,6 +1225,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.lifecycle_config = None;
if !self.lifecycle_config_xml.is_empty()
&& let Err(e) =
deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
@@ -1238,6 +1240,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.object_lock_config = None;
if !self.object_lock_config_xml.is_empty()
&& let Err(e) =
deserialize::<ObjectLockConfiguration>(&self.object_lock_config_xml).map(|c| self.object_lock_config = Some(c))
@@ -1267,6 +1270,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.sse_config = None;
if !self.encryption_config_xml.is_empty()
&& let Err(e) =
deserialize::<ServerSideEncryptionConfiguration>(&self.encryption_config_xml).map(|c| self.sse_config = Some(c))
@@ -1281,6 +1285,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.tagging_config = None;
if !self.tagging_config_xml.is_empty()
&& let Err(e) = deserialize::<Tagging>(&self.tagging_config_xml).map(|c| self.tagging_config = Some(c))
{
@@ -1346,6 +1351,7 @@ impl BucketMetadata {
} else {
self.bucket_target_config = Some(BucketTargets::default());
}
self.cors_config = None;
if !self.cors_config_xml.is_empty()
&& let Err(e) = deserialize::<CORSConfiguration>(&self.cors_config_xml).map(|c| self.cors_config = Some(c))
{
@@ -1359,6 +1365,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.logging_config = None;
if !self.logging_config_xml.is_empty()
&& let Err(e) = deserialize::<BucketLoggingStatus>(&self.logging_config_xml).map(|c| self.logging_config = Some(c))
{
@@ -1372,6 +1379,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.website_config = None;
if !self.website_config_xml.is_empty()
&& let Err(e) = deserialize::<WebsiteConfiguration>(&self.website_config_xml).map(|c| self.website_config = Some(c))
{
@@ -1385,6 +1393,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.accelerate_config = None;
if !self.accelerate_config_xml.is_empty()
&& let Err(e) =
deserialize::<AccelerateConfiguration>(&self.accelerate_config_xml).map(|c| self.accelerate_config = Some(c))
@@ -1399,6 +1408,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.request_payment_config = None;
if !self.request_payment_config_xml.is_empty()
&& let Err(e) = deserialize::<RequestPaymentConfiguration>(&self.request_payment_config_xml)
.map(|c| self.request_payment_config = Some(c))
@@ -1413,6 +1423,7 @@ impl BucketMetadata {
"Failed to parse bucket metadata config"
);
}
self.public_access_block_config = None;
if !self.public_access_block_config_xml.is_empty()
&& let Err(e) = deserialize::<PublicAccessBlockConfiguration>(&self.public_access_block_config_xml)
.map(|c| self.public_access_block_config = Some(c))
@@ -1973,6 +1984,156 @@ mod test {
assert_eq!(bm.lifecycle_config_xml, malformed_xml, "raw bytes are retained");
}
const REPARSE_XML_FIXTURES: [(&str, &[u8]); 13] = [
(BUCKET_NOTIFICATION_CONFIG, br#"<NotificationConfiguration/>"#),
(
BUCKET_LIFECYCLE_CONFIG,
br#"<LifecycleConfiguration><Rule><ID>keep</ID><Status>Enabled</Status><Expiration><Days>30</Days></Expiration></Rule></LifecycleConfiguration>"#,
),
(
OBJECT_LOCK_CONFIG,
br#"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Mode>GOVERNANCE</Mode><Days>7</Days></DefaultRetention></Rule></ObjectLockConfiguration>"#,
),
(BUCKET_VERSIONING_CONFIG, br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#),
(
BUCKET_SSECONFIG,
br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#,
),
(BUCKET_TAGGING_CONFIG, br#"<Tagging><TagSet><Tag><Key>team</Key><Value>blue</Value></Tag></TagSet></Tagging>"#),
(
BUCKET_REPLICATION_CONFIG,
br#"<ReplicationConfiguration><Role>arn:aws:iam::123456789012:role/replication-role</Role><Rule><ID>copy</ID><Status>Enabled</Status><Prefix>documents/</Prefix><Destination><Bucket>arn:aws:s3:::destination-bucket</Bucket></Destination></Rule></ReplicationConfiguration>"#,
),
(
BUCKET_CORS_CONFIG,
br#"<CORSConfiguration><CORSRule><AllowedOrigin>https://example.test</AllowedOrigin><AllowedMethod>GET</AllowedMethod></CORSRule></CORSConfiguration>"#,
),
(BUCKET_LOGGING_CONFIG, br#"<BucketLoggingStatus/>"#),
(BUCKET_WEBSITE_CONFIG, br#"<WebsiteConfiguration><IndexDocument><Suffix>index.html</Suffix></IndexDocument></WebsiteConfiguration>"#),
(BUCKET_ACCELERATE_CONFIG, br#"<AccelerateConfiguration><Status>Enabled</Status></AccelerateConfiguration>"#),
(BUCKET_REQUEST_PAYMENT_CONFIG, br#"<RequestPaymentConfiguration><Payer>BucketOwner</Payer></RequestPaymentConfiguration>"#),
(
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG,
br#"<PublicAccessBlockConfiguration><BlockPublicAcls>true</BlockPublicAcls><IgnorePublicAcls>true</IgnorePublicAcls><BlockPublicPolicy>true</BlockPublicPolicy><RestrictPublicBuckets>true</RestrictPublicBuckets></PublicAccessBlockConfiguration>"#,
),
];
fn reparse_xml_state(bm: &BucketMetadata, config: &str) -> &'static str {
fn state<T>(raw: &[u8], parsed: &Option<T>) -> &'static str {
match ConfigState::of(raw, parsed) {
ConfigState::Absent => "absent",
ConfigState::Valid(_) => "valid",
ConfigState::Unreadable { .. } => "unreadable",
}
}
match config {
BUCKET_NOTIFICATION_CONFIG => state(&bm.notification_config_xml, &bm.notification_config),
BUCKET_LIFECYCLE_CONFIG => state(&bm.lifecycle_config_xml, &bm.lifecycle_config),
OBJECT_LOCK_CONFIG => state(&bm.object_lock_config_xml, &bm.object_lock_config),
BUCKET_VERSIONING_CONFIG => state(&bm.versioning_config_xml, &bm.versioning_config),
BUCKET_SSECONFIG => state(&bm.encryption_config_xml, &bm.sse_config),
BUCKET_TAGGING_CONFIG => state(&bm.tagging_config_xml, &bm.tagging_config),
BUCKET_REPLICATION_CONFIG => state(&bm.replication_config_xml, &bm.replication_config),
BUCKET_CORS_CONFIG => state(&bm.cors_config_xml, &bm.cors_config),
BUCKET_LOGGING_CONFIG => state(&bm.logging_config_xml, &bm.logging_config),
BUCKET_WEBSITE_CONFIG => state(&bm.website_config_xml, &bm.website_config),
BUCKET_ACCELERATE_CONFIG => state(&bm.accelerate_config_xml, &bm.accelerate_config),
BUCKET_REQUEST_PAYMENT_CONFIG => state(&bm.request_payment_config_xml, &bm.request_payment_config),
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG => state(&bm.public_access_block_config_xml, &bm.public_access_block_config),
_ => panic!("unknown XML config: {config}"),
}
}
// A persisted payload can change without going through the request decoder.
fn replace_xml_raw(bm: &mut BucketMetadata, config: &str, raw: Vec<u8>) {
match config {
BUCKET_NOTIFICATION_CONFIG => bm.notification_config_xml = raw,
BUCKET_LIFECYCLE_CONFIG => bm.lifecycle_config_xml = raw,
OBJECT_LOCK_CONFIG => bm.object_lock_config_xml = raw,
BUCKET_VERSIONING_CONFIG => bm.versioning_config_xml = raw,
BUCKET_SSECONFIG => bm.encryption_config_xml = raw,
BUCKET_TAGGING_CONFIG => bm.tagging_config_xml = raw,
BUCKET_REPLICATION_CONFIG => bm.replication_config_xml = raw,
BUCKET_CORS_CONFIG => bm.cors_config_xml = raw,
BUCKET_LOGGING_CONFIG => bm.logging_config_xml = raw,
BUCKET_WEBSITE_CONFIG => bm.website_config_xml = raw,
BUCKET_ACCELERATE_CONFIG => bm.accelerate_config_xml = raw,
BUCKET_REQUEST_PAYMENT_CONFIG => bm.request_payment_config_xml = raw,
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG => bm.public_access_block_config_xml = raw,
_ => panic!("unknown XML config: {config}"),
}
}
#[test]
fn reparse_xml_rejects_stale_views_after_corruption() {
let corrupt = b"<not-a-valid-document";
for (config, valid) in REPARSE_XML_FIXTURES {
let mut bm = BucketMetadata::new("reparse-corrupt");
bm.update_config(config, valid.to_vec()).expect("store valid XML");
bm.parse_all_configs().expect("parse valid XML");
assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: valid setup");
replace_xml_raw(&mut bm, config, corrupt.to_vec());
let before = bm.marshal_msg().expect("marshal corrupt raw metadata");
bm.parse_all_configs()
.expect("one corrupt config must not fail the metadata load");
assert_eq!(reparse_xml_state(&bm, config), "unreadable", "{config}: stale view accepted");
assert_eq!(bm.xml_config_unreadable_len(config), Some(corrupt.len()), "{config}: missing refusal");
assert_eq!(
bm.marshal_msg().expect("marshal reparsed metadata"),
before,
"{config}: persisted bytes changed"
);
}
}
#[test]
fn reparse_xml_clears_stale_views_after_raw_is_removed() {
for (config, valid) in REPARSE_XML_FIXTURES {
let mut bm = BucketMetadata::new("reparse-removed");
bm.update_config(config, valid.to_vec()).expect("store valid XML");
bm.parse_all_configs().expect("parse valid XML");
assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: valid setup");
replace_xml_raw(&mut bm, config, Vec::new());
let before = bm.marshal_msg().expect("marshal absent raw metadata");
bm.parse_all_configs().expect("parse removed XML");
assert_eq!(reparse_xml_state(&bm, config), "absent", "{config}: stale view survived removal");
assert!(!bm.xml_config_unreadable(config), "{config}: absence is not corruption");
assert_eq!(
bm.marshal_msg().expect("marshal reparsed metadata"),
before,
"{config}: persisted bytes changed"
);
}
}
#[test]
fn reparse_xml_recovers_after_corrupt_raw_is_repaired() {
for (config, valid) in REPARSE_XML_FIXTURES {
let mut bm = BucketMetadata::new("reparse-repaired");
bm.update_config(config, valid.to_vec()).expect("store valid XML");
bm.parse_all_configs().expect("parse valid XML");
replace_xml_raw(&mut bm, config, b"<not-a-valid-document".to_vec());
bm.parse_all_configs().expect("parse corrupt XML");
assert_eq!(reparse_xml_state(&bm, config), "unreadable", "{config}: corrupt setup");
replace_xml_raw(&mut bm, config, valid.to_vec());
let before = bm.marshal_msg().expect("marshal repaired raw metadata");
bm.parse_all_configs().expect("parse repaired XML");
assert_eq!(reparse_xml_state(&bm, config), "valid", "{config}: repaired config must become readable");
assert!(!bm.xml_config_unreadable(config), "{config}: repaired config is not corrupt");
assert_eq!(
bm.marshal_msg().expect("marshal reparsed metadata"),
before,
"{config}: persisted bytes changed"
);
}
}
#[test]
fn lifecycle_update_config_clears_parsed_config_on_delete() {
let mut bm = BucketMetadata::new("test-bucket");
+690 -26
View File
@@ -178,6 +178,13 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
Ok(ObjectLockConfigState::ConfirmedAbsent)
}
#[derive(Debug, Clone, PartialEq)]
pub(crate) struct CachedQuotaMetadataSnapshot {
pub(crate) quota: Option<BucketQuota>,
pub(crate) bucket_incarnation: Uuid,
pub(crate) quota_revision: OffsetDateTime,
}
/// Convert the persisted serving-layer configuration into the storage-level
/// [`DefaultRetention`](crate::bucket::object_lock::types::DefaultRetention)
/// the WORM evaluation code consumes (rustfs/backlog#1842). A rule without a
@@ -314,6 +321,33 @@ pub async fn remove_bucket_metadata(bucket: &str) -> Result<bool> {
Ok(lock.remove(bucket).await)
}
/// Remove a peer's cached metadata only when the bucket is absent under the
/// namespace read fence. A delayed delete notification must not withdraw the
/// metadata of a same-name bucket that has already been recreated.
pub async fn remove_bucket_metadata_if_absent(api: Arc<ECStore>, bucket: &str) -> Result<bool> {
if is_meta_bucketname(bucket) {
return Ok(false);
}
// Lock order: bucket namespace read -> metadata system read (released)
// -> metadata publish, matching peer reloads.
let namespace_lock = api.new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
let sys = bucket_metadata_sys_of(&api.ctx)?;
let sys = sys.read().await.clone();
if sys
.bucket_exists(bucket, &namespace_guard, "peer bucket metadata removal existence check")
.await?
{
return Ok(false);
}
await_bucket_namespace_operation(Some(&namespace_guard), bucket, "peer bucket metadata removal", async {
Ok(sys.remove(bucket).await)
})
.await
}
fn start_refresh_buckets_metadata_loop(sys: Arc<RwLock<BucketMetadataSys>>) {
let Some(cancel_token) = runtime_sources::background_services_cancel_token() else {
warn!("bucket metadata refresh loop skipped because background cancellation token is not initialized");
@@ -491,6 +525,36 @@ pub(crate) async fn get_bucket_incarnation_id_in(ctx: &crate::runtime::instance:
sys.get_bucket_incarnation_id_from_disk(bucket).await
}
/// Directory loss is recoverable only for a committed, non-retired generation.
/// The caller retains the bucket lifecycle fence through the ensuing reconstruction.
pub(crate) async fn validate_bucket_reconstruction_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
expected: Uuid,
) -> Result<()> {
let api = bucket_metadata_sys_of(ctx)?.read().await.object_store();
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
.await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let (metadata, persisted) = load_bucket_metadata_parse_with_presence(api.clone(), bucket, true).await?;
if !persisted
|| !metadata.bucket_incarnation_sidecar
|| expected.is_nil()
|| metadata.bucket_incarnation_id != expected
|| !metadata.bucket_creation_committed
{
return Err(Error::PreconditionFailed);
}
if crate::bucket::retirement::is_retired(api, bucket, expected).await? {
return Err(Error::BucketNotFound(bucket.to_owned()));
}
if guard.is_lock_lost() {
return Err(Error::PreconditionFailed);
}
Ok(())
}
/// Validate against disk while retaining an already-held Object Lock fence.
pub(crate) async fn get_bucket_incarnation_id_for_options_in(
ctx: &crate::runtime::instance::InstanceContext,
@@ -576,6 +640,27 @@ pub(crate) async fn update_in(
Box::pin(update_with_sys(require_bucket_metadata_sys_in(ctx)?, bucket, config_file, data)).await
}
/// Validate a caller's fence after acquiring the lifecycle and metadata
/// transaction locks, before changing the configuration.
pub(crate) async fn update_validated_in<F>(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
config_file: &str,
data: Vec<u8>,
validate: F,
) -> Result<OffsetDateTime>
where
F: FnOnce() -> Result<()> + Send,
{
Box::pin(async {
let sys = require_bucket_metadata_sys_in(ctx)?;
let guard = acquire_config_write_guard(sys.clone(), bucket).await?;
validate()?;
update_under_config_write_guard(sys, &guard, config_file, data, None).await
})
.await
}
pub async fn delete(bucket: &str, config_file: &str) -> Result<OffsetDateTime> {
delete_with_sys(get_bucket_metadata_sys()?, bucket, config_file).await
}
@@ -1132,7 +1217,7 @@ async fn acquire_transaction_lock_with_sys(
/// agree on, so renaming it would leave a mixed-version cluster with two
/// disjoint keys — and old and new nodes would stop excluding each other on
/// the very writes that are serialized today.
fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
pub(crate) fn bucket_metadata_transaction_lock_key(bucket: &str) -> String {
format!("bucket-targets/{bucket}/transaction.lock")
}
@@ -1271,14 +1356,34 @@ pub(crate) async fn get_object_lock_config_state_in(
pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
get_object_lock_config_and_incarnation_from_disk_with_guard_in(ctx, bucket, None).await
}
/// Bucket deletion already owns the namespace/lifecycle fences. Reuse its
/// live guard rather than recursively acquiring the bucket namespace lock.
/// The caller must also hold the metadata transaction read lock.
pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
guard: Option<&rustfs_lock::NamespaceLockGuard>,
) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
match bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
{
let authority = match guard {
Some(guard) => {
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_guard(bucket, guard)
.await?
}
None => {
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
}
};
match authority {
BucketMetadataAuthority::Authoritative(metadata)
if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
{
@@ -1331,13 +1436,60 @@ pub(crate) async fn get_quota_config_and_incarnation_from_disk_in(
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
match bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
quota_config_and_incarnation_from_authority(
bucket,
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?,
)
}
pub(crate) async fn get_quota_config_and_incarnation_from_disk_for_options_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
opts: &ObjectOptions,
transaction_guard: &Arc<rustfs_lock::NamespaceLockGuard>,
) -> Result<(Option<BucketQuota>, Uuid, OffsetDateTime)> {
let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() else {
return get_quota_config_and_incarnation_from_disk_in(ctx, bucket).await;
};
let sys = bucket_metadata_sys_of(ctx)?.read().await.clone();
let store = sys.object_store();
let snapshot_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, opts.expected_bucket_incarnation_id)
.filter(|guard| Arc::ptr_eq(guard, transaction_guard))
.ok_or_else(|| Error::other("quota snapshot does not hold this bucket's metadata transaction fence"))?;
if snapshot_guard.is_released() {
return Err(Error::other("quota snapshot metadata transaction fence was released"));
}
// The snapshot carries the caller's lifecycle fence, excluding deletion.
// Its transaction guard excludes config writes, so a same-name object
// owner need not recursively acquire the bucket namespace here.
let authority = sys
.read_authoritative_metadata_from_disk_under_guard(bucket, transaction_guard)
.await?;
if snapshot_guard.is_lock_lost()
|| snapshot_guard.is_released()
|| matches!(&authority, BucketMetadataAuthority::Authoritative(metadata)
if !snapshot.is_valid_for_destructive_put(store.id, bucket, metadata.bucket_incarnation_id))
{
return Err(Error::other("quota snapshot fence or bucket incarnation changed during the disk read"));
}
quota_config_and_incarnation_from_authority(bucket, authority)
}
fn quota_config_and_incarnation_from_authority(
bucket: &str,
authority: BucketMetadataAuthority,
) -> Result<(Option<BucketQuota>, Uuid, OffsetDateTime)> {
match authority {
BucketMetadataAuthority::Authoritative(metadata)
if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
{
if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
return Err(Error::other("persisted bucket quota configuration is invalid"));
}
Ok((
metadata.quota_config.clone(),
metadata.bucket_incarnation_id,
@@ -1352,6 +1504,15 @@ pub(crate) async fn get_quota_config_and_incarnation_from_disk_in(
}
}
pub(crate) async fn get_cached_quota_config_and_incarnation_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<Option<CachedQuotaMetadataSnapshot>> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
}
pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -2703,29 +2864,30 @@ impl BucketMetadataSys {
async fn read_authoritative_metadata_from_disk_under_transaction_lock(
&self,
bucket: &str,
) -> Result<BucketMetadataAuthority> {
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
self.read_authoritative_metadata_from_disk_under_guard(bucket, &namespace_guard)
.await
}
async fn read_authoritative_metadata_from_disk_under_guard(
&self,
bucket: &str,
guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<BucketMetadataAuthority> {
#[cfg(test)]
if self.object_lock_disk_read_errors.write().await.remove(bucket) {
return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
}
let namespace_lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let namespace_guard = namespace_lock
.get_read_lock(crate::set_disk::get_lock_acquire_timeout())
.await?;
match await_bucket_namespace_operation(
Some(&namespace_guard),
bucket,
"bucket metadata snapshot existence check",
async {
self.object_store()
.get_bucket_info_from_sets_at_read_quorum(
bucket,
&crate::storage_api_contracts::bucket::BucketOptions::default(),
)
.await
},
)
match await_bucket_namespace_operation(Some(guard), bucket, "bucket metadata snapshot existence check", async {
self.object_store()
.get_bucket_info_from_sets_at_read_quorum(bucket, &crate::storage_api_contracts::bucket::BucketOptions::default())
.await
})
.await
{
Ok(_) => {}
@@ -2734,7 +2896,7 @@ impl BucketMetadataSys {
}
let (metadata, persisted) = await_bucket_namespace_operation(
Some(&namespace_guard),
Some(guard),
bucket,
"bucket metadata authoritative snapshot",
load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
@@ -2866,6 +3028,27 @@ impl BucketMetadataSys {
}
}
async fn cached_quota_config_and_incarnation(&self, bucket: &str) -> Result<Option<CachedQuotaMetadataSnapshot>> {
let metadata = {
let metadata_map = self.metadata_map.read().await;
metadata_map.get(bucket).cloned()
};
let Some(metadata) = metadata else {
return Ok(None);
};
if !metadata.bucket_incarnation_sidecar || metadata.bucket_incarnation_id.is_nil() {
return Ok(None);
}
if !metadata.quota_config_json.is_empty() && metadata.quota_config.is_none() {
return Err(Error::other("persisted bucket quota configuration is invalid"));
}
Ok(Some(CachedQuotaMetadataSnapshot {
quota: metadata.quota_config.clone(),
bucket_incarnation: metadata.bucket_incarnation_id,
quota_revision: metadata.quota_config_updated_at,
}))
}
pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
let (bm, _) = self.get_config(bucket).await?;
@@ -4121,6 +4304,114 @@ mod tests {
sys.persist_new_and_set(bm).await.expect("initial metadata should persist");
}
#[tokio::test]
async fn reparse_xml_rmw_corruption_refuses_cached_and_reloaded_encryption() {
use crate::bucket::metadata::{BUCKET_SSECONFIG, is_unreadable_config_error};
use std::sync::atomic::{AtomicUsize, Ordering};
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
let bucket = "reparse-encryption-corrupt";
let valid = br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
let corrupt = b"<ServerSideEncryptionConfiguration";
persist_bucket_with_raw_config(&sys, &dirs, bucket, BUCKET_SSECONFIG, valid).await;
sys.get_sse_config(bucket).await.expect("valid encryption must be readable");
let updated = sys
.update_config_with(bucket, BUCKET_SSECONFIG, |bm| {
assert!(bm.sse_config.is_some(), "rewrite must start with parsed encryption");
Ok(corrupt.to_vec())
})
.await
.expect("persist future or corrupt raw encryption without failing the whole bucket");
let err = sys
.get_sse_config(bucket)
.await
.expect_err("cached old encryption must not hide corrupt raw");
assert!(is_unreadable_config_error(&err), "cached refusal must retain its type: {err}");
let (cached, _) = sys.get_config(bucket).await.expect("bucket remains readable");
assert_eq!(cached.encryption_config_xml, corrupt);
assert_eq!(cached.encryption_config_updated_at, updated);
sys.metadata_map.write().await.clear();
let err = sys
.get_sse_config(bucket)
.await
.expect_err("reloaded encryption must still refuse corrupt raw");
assert!(is_unreadable_config_error(&err), "reloaded refusal must retain its type: {err}");
let calls = AtomicUsize::new(0);
let err = sys
.update_config_with(bucket, BUCKET_SSECONFIG, |_| {
calls.fetch_add(1, Ordering::SeqCst);
Ok(Vec::new())
})
.await
.expect_err("another rewrite must refuse before it consumes corrupt encryption");
assert!(is_unreadable_config_error(&err));
assert_eq!(calls.load(Ordering::SeqCst), 0, "refused rewrite must not erase the original raw");
let disk = sys
.get_config_from_disk(bucket)
.await
.expect("read unchanged metadata from disk");
assert_eq!(disk.encryption_config_xml, corrupt);
assert_eq!(disk.encryption_config_updated_at, updated);
sys.update(bucket, BUCKET_SSECONFIG, valid.to_vec())
.await
.expect("explicit repair must remain possible");
sys.get_sse_config(bucket)
.await
.expect("repaired cached encryption must be readable");
sys.metadata_map.write().await.clear();
sys.get_sse_config(bucket).await.expect("repair must survive a disk reload");
}
#[tokio::test]
async fn reparse_xml_rmw_removal_clears_cached_and_reloaded_encryption() {
use crate::bucket::metadata::BUCKET_SSECONFIG;
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = BucketMetadataSys::new(ecstore);
let bucket = "reparse-encryption-removed";
let valid = br#"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"#;
persist_bucket_with_raw_config(&sys, &dirs, bucket, BUCKET_SSECONFIG, valid).await;
sys.get_sse_config(bucket).await.expect("valid encryption must be readable");
let updated = sys
.update_config_with(bucket, BUCKET_SSECONFIG, |bm| {
assert!(bm.sse_config.is_some(), "rewrite must start with parsed encryption");
Ok(Vec::new())
})
.await
.expect("removing encryption must persist");
assert_eq!(
sys.get_sse_config(bucket)
.await
.expect_err("old cached encryption must be gone"),
Error::ConfigNotFound
);
let (cached, _) = sys.get_config(bucket).await.expect("bucket remains readable");
assert!(cached.encryption_config_xml.is_empty());
assert_eq!(cached.encryption_config_updated_at, updated);
sys.metadata_map.write().await.clear();
assert_eq!(
sys.get_sse_config(bucket)
.await
.expect_err("removed encryption must stay absent after reload"),
Error::ConfigNotFound
);
let disk = sys
.get_config_from_disk(bucket)
.await
.expect("read removed encryption from disk");
assert!(disk.encryption_config_xml.is_empty());
assert_eq!(disk.encryption_config_updated_at, updated);
}
/// rustfs/backlog#1734: a read-modify-write of a stored config that cannot
/// be parsed must be refused before `mutate` runs. Otherwise `mutate` sees
/// the unreadable config as absent, rebuilds it from nothing, and the
@@ -4662,6 +4953,154 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn quota_admission_rejects_stale_cached_no_quota() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("scoped metadata system");
let bucket = "stale-no-quota-admission";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let (stale, _) = sys.read().await.get_config(bucket).await.expect("cache initial metadata");
store
.update_bucket_metadata_config(bucket, "quota.json", br#"{"quota":1}"#.to_vec())
.await
.expect("publish quota to authoritative storage");
// Model a peer that has not received the newer configuration snapshot.
sys.read().await.set(bucket.to_owned(), stale).await;
assert!(
get_cached_quota_config_and_incarnation_in(&store.ctx, bucket)
.await
.expect("read stale cache")
.expect("cached bucket identity")
.quota
.is_none()
);
temp_env::async_with_vars([("RUSTFS_QUOTA_BEGIN_SAFE_NO_QUOTA_CACHE_FAST_PATH", Some("true"))], async {
let error = crate::bucket::quota::reservation::begin(&store.ctx, bucket, "object", &ObjectOptions::default(), 0, 0)
.await
.err()
.expect("persisted quota must reject admission without its snapshot");
assert!(matches!(error, Error::PartMissingOrCorrupt), "{error:?}");
})
.await;
}
#[tokio::test]
#[serial]
async fn peer_reload_clears_negative_cache_after_remote_bucket_creation() {
let (dirs, node_a, node_b) = two_nodes_over_one_store().await;
let bucket = "cross-node-negative-cache-refresh";
let node_b_sys = node_b.read().await.clone();
assert_eq!(
node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap_err(),
Error::BucketNotFound(bucket.to_string()),
"the probe must populate node B's missing-bucket cache"
);
for dir in &dirs {
std::fs::create_dir_all(dir.path().join(bucket)).expect("physical bucket should exist before creation metadata");
}
let created = BucketMetadata::new(bucket);
let created_incarnation = created.bucket_incarnation_id;
node_a
.read()
.await
.persist_new_and_set(created)
.await
.expect("node A should publish the created bucket metadata");
node_b_sys
.reload_from_store(bucket)
.await
.expect("peer reload should clear the stale negative cache");
assert_eq!(
node_b_sys.get_bucket_incarnation_id(bucket).await.unwrap(),
created_incarnation,
"node B should resolve the newly created bucket immediately"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn peer_delete_clears_metadata_for_absent_bucket() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "peer-delete-absent-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("initial bucket creation");
let cached = sys.read().await.get(bucket).await.expect("initial cached metadata");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion");
// Model a peer that still caches the metadata after the physical delete.
sys.read().await.set(bucket.to_string(), cached).await;
assert!(
remove_bucket_metadata_if_absent(store.clone(), bucket)
.await
.expect("conditional peer delete"),
"the peer must evict the deleted bucket's cached metadata"
);
let sys = sys.read().await;
assert!(matches!(sys.get(bucket).await, Err(Error::ConfigNotFound)));
assert_eq!(
sys.get_bucket_incarnation_id(bucket)
.await
.expect_err("deleted bucket lookup"),
Error::BucketNotFound(bucket.to_string()),
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn delayed_peer_delete_preserves_recreated_bucket_metadata() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("metadata owner");
let bucket = "delayed-peer-delete-recreated-bucket";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("initial bucket creation");
let old_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("old incarnation");
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket recreation");
let new_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
assert_ne!(old_incarnation, new_incarnation);
let removed = remove_bucket_metadata_if_absent(store.clone(), bucket)
.await
.expect("conditional peer delete");
assert!(!removed, "a delayed delete must not remove metadata for a live recreated bucket");
assert_eq!(
sys.read()
.await
.get(bucket)
.await
.expect("recreated metadata should remain cached")
.bucket_incarnation_id,
new_incarnation,
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn stale_config_request_cannot_mutate_a_recreated_bucket() {
@@ -4736,6 +5175,231 @@ mod tests {
.expect("DeleteBucket should succeed");
}
#[tokio::test]
#[serial]
async fn quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "quota-same-name-owner";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("quota fixture bucket");
let snapshot = store
.object_lock_config_snapshot(bucket)
.await
.expect("guarded Object Lock snapshot");
let incarnation = store.bucket_incarnation_id(bucket).await.expect("fixture incarnation");
let transaction_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
.expect("snapshot's actual transaction guard");
let expected = get_quota_config_and_incarnation_from_disk_for_options_in(
&store.ctx,
bucket,
&ObjectOptions::default(),
&transaction_guard,
)
.await
.expect("a reader without a snapshot must retain the ordinary namespace path");
let sys = bucket_metadata_sys_of(&store.ctx).expect("fixture metadata system");
let sys = sys.read().await.clone();
let mut stale = (*sys.get(bucket).await.expect("cached fixture metadata")).clone();
let metadata_path = stale.save_file_path();
stale.quota_config = Some(BucketQuota {
quota: Some(123),
..Default::default()
});
sys.set(bucket.to_string(), Arc::new(stale)).await;
let namespace = store.new_ns_lock(bucket, bucket).await.expect("same-name object namespace");
let _owner = namespace
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await
.expect("same-name object write owner");
let opts = ObjectOptions {
object_lock_config_snapshot: Some(snapshot),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
assert_eq!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.expect("quota disk read must reuse the held lifecycle and transaction fences"),
expected,
"the cached quota must not replace disk authority"
);
crate::config::com::save_config_with_opts(
store.clone(),
&metadata_path,
b"corrupt bucket metadata".to_vec(),
&ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("inject committed metadata corruption");
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"a valid snapshot must not hide corrupt disk metadata"
);
crate::config::com::delete_config(store.clone(), &metadata_path)
.await
.expect("remove the persisted metadata while retaining its sidecar");
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"a valid snapshot must not promote fabricated metadata to authority"
);
}
#[tokio::test]
#[serial]
async fn quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "quota-snapshot-scope";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("quota fixture bucket");
let incarnation = store.bucket_incarnation_id(bucket).await.expect("fixture incarnation");
let snapshot = store
.object_lock_config_snapshot(bucket)
.await
.expect("guarded Object Lock snapshot");
let transaction_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
.expect("snapshot's actual transaction guard");
let mut opts = ObjectOptions {
object_lock_config_snapshot: Some(snapshot),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, "another-bucket", &opts, &transaction_guard)
.await
.is_err()
);
opts.expected_bucket_incarnation_id = Some(Uuid::new_v4());
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err()
);
opts.expected_bucket_incarnation_id = Some(incarnation);
let other_guard = Arc::new(
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket)
.await
.expect("distinct real transaction read guard"),
);
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &other_guard)
.await
.is_err(),
"the same namespace key does not prove that this is the snapshot's actual guard"
);
let (_other_dirs, other_store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(other_store.clone(), Vec::new()).await;
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&other_store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err()
);
opts.object_lock_config_snapshot = Some(Arc::new(crate::object_api::ObjectLockConfigSnapshot::new(
ObjectLockConfigState::ConfirmedAbsent,
)));
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"an unguarded snapshot must not authorize namespace reuse"
);
}
#[tokio::test]
#[serial]
async fn quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "quota-snapshot-loss";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("quota fixture bucket");
let sys = bucket_metadata_sys_of(&store.ctx).expect("fixture metadata system");
let sys = sys.read().await.clone();
let mut metadata = (*sys.get(bucket).await.expect("fixture metadata")).clone();
let incarnation = metadata.bucket_incarnation_id;
let _lifecycle_guard = store
.acquire_bucket_lifecycle_read_lock(bucket)
.await
.expect("real lifecycle owner");
let transaction_guard = Arc::new(
acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, bucket)
.await
.expect("real metadata transaction owner"),
);
let (lifecycle_fence, lost) = crate::object_api::NamespaceLockFence::loss_handle_for_test();
let snapshot = crate::object_api::ObjectLockConfigSnapshot::for_store_bucket_under_lifecycle_fence(
store.id,
bucket,
incarnation,
metadata.object_lock_config_updated_at,
ObjectLockConfigState::ConfirmedAbsent,
lifecycle_fence,
Arc::clone(&transaction_guard),
);
let opts = ObjectOptions {
object_lock_config_snapshot: Some(Arc::new(snapshot)),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
let mut read = Box::pin(get_quota_config_and_incarnation_from_disk_for_options_in(
&store.ctx,
bucket,
&opts,
&transaction_guard,
));
assert!(
futures::poll!(read.as_mut()).is_pending(),
"the authoritative disk read must be in flight"
);
lost.store(true, std::sync::atomic::Ordering::Release);
assert!(read.await.is_err(), "a lifecycle fence lost during the disk read must reject its result");
let snapshot = store
.object_lock_config_snapshot(bucket)
.await
.expect("fresh guarded snapshot");
let transaction_guard = snapshot
.metadata_transaction_guard_for(store.id, bucket, Some(incarnation))
.expect("snapshot's actual transaction guard");
let opts = ObjectOptions {
object_lock_config_snapshot: Some(snapshot),
expected_bucket_incarnation_id: Some(incarnation),
..Default::default()
};
metadata.bucket_incarnation_id = Uuid::new_v4();
save_bucket_incarnation(store.clone(), bucket, metadata.bucket_incarnation_id)
.await
.expect("inject a new disk incarnation");
metadata
.save_with_store_committed(store.clone())
.await
.expect("persist its matching metadata blob");
assert!(
get_quota_config_and_incarnation_from_disk_for_options_in(&store.ctx, bucket, &opts, &transaction_guard)
.await
.is_err(),
"a matching sidecar and blob from another incarnation must not reuse the old snapshot"
);
}
/// Writers on different nodes updating *different* config files of one
/// bucket must both survive. Each rewrites the whole metadata blob, so
/// without a lock spanning the read-modify-write the later save carries
+24
View File
@@ -273,6 +273,30 @@ where
let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
try_migrate_bucket_metadata_for_buckets(store, &buckets).await
}
/// Migrate the already discovered startup candidates without repeating a
/// majority-only namespace listing. Every migration write retains its quorum.
pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
> + ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
if buckets.is_empty() {
debug!("No migrating bucket metadata found");
return Ok(());
+40 -6
View File
@@ -32,7 +32,10 @@ pub enum QuotaType {
Hard,
}
pub(crate) const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
pub const QUOTA_RESERVATION_PROTOCOL_V1: u32 = 1;
/// Sharded durable reservations. Older nodes reject this protocol instead of
/// silently falling back to the single-ledger protocol.
pub const QUOTA_RESERVATION_PROTOCOL_V2: u32 = 2;
/// Bucket quota configuration. quota_type defaults to Hard when omitted.
#[derive(Debug, Default, Clone, PartialEq)]
@@ -89,7 +92,10 @@ impl<'de> Deserialize<'de> for BucketQuota {
D: Deserializer<'de>,
{
let wire = BucketQuotaWire::deserialize(deserializer)?;
let quota = if wire.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1) {
let quota = if matches!(
wire.reservation_protocol,
Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
) {
Some(
wire.reservation_quota
.ok_or_else(|| D::Error::custom("reservation_quota is required for reservation protocol v1"))?,
@@ -129,17 +135,33 @@ impl BucketQuota {
}
}
/// Creates a quota using the sharded durable reservation protocol.
/// Deployment must complete the mixed-version rollout before enabling it;
/// older nodes fail closed when they see protocol v2.
pub fn new_sharded(quota: u64) -> Self {
let mut config = Self::new(Some(quota));
config.reservation_protocol = Some(QUOTA_RESERVATION_PROTOCOL_V2);
config
}
pub fn get_quota_limit(&self) -> Option<u64> {
self.quota
}
pub fn uses_durable_reservations(&self) -> bool {
self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V1)
matches!(
self.reservation_protocol,
Some(QUOTA_RESERVATION_PROTOCOL_V1) | Some(QUOTA_RESERVATION_PROTOCOL_V2)
)
}
pub fn uses_sharded_reservations(&self) -> bool {
self.reservation_protocol == Some(QUOTA_RESERVATION_PROTOCOL_V2)
}
pub fn has_unsupported_reservation_protocol(&self) -> bool {
self.reservation_protocol
.is_some_and(|version| version != QUOTA_RESERVATION_PROTOCOL_V1)
.is_some_and(|version| !matches!(version, QUOTA_RESERVATION_PROTOCOL_V1 | QUOTA_RESERVATION_PROTOCOL_V2))
}
pub fn check_operation_allowed(&self, current_usage: u64, operation_size: u64) -> bool {
@@ -322,9 +344,21 @@ mod tests {
}
#[test]
fn unknown_reservation_protocol_does_not_activate_v1() {
fn sharded_quota_roundtrips_with_authoritative_reservation_quota() {
let quota = BucketQuota::new_sharded(4096);
let json = serde_json::to_value(&quota).expect("sharded quota should serialize");
assert_eq!(json["quota"], 0);
assert_eq!(json["reservation_protocol"], QUOTA_RESERVATION_PROTOCOL_V2);
assert_eq!(json["reservation_quota"], 4096);
let restored: BucketQuota = serde_json::from_value(json).expect("sharded quota should parse");
assert_eq!(restored.quota, Some(4096));
assert!(restored.uses_sharded_reservations());
}
#[test]
fn unknown_reservation_protocol_does_not_activate_any_known_protocol() {
let quota: BucketQuota =
serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":2,"reservation_quota":2048}"#)
serde_json::from_str(r#"{"quota":0,"quota_type":"Hard","reservation_protocol":3,"reservation_quota":2048}"#)
.expect("future protocol should remain parseable");
assert!(!quota.uses_durable_reservations());
File diff suppressed because it is too large Load Diff
@@ -448,6 +448,26 @@ fn is_version_id_format_mismatch(err: &SdkError<HeadObjectError>) -> bool {
is_version_id_mismatch(code, raw_status)
}
/// A version-addressed HEAD rejected because the target cannot address the
/// source version id (AWS S3: bodiless 400). Narrower than
/// [`is_version_id_format_mismatch`]: a bodiless 403 may be a genuine
/// `s3:GetObjectVersion` denial, so it is not accepted where a mutation or a
/// completion verdict follows instead of another non-destructive HEAD.
fn rejects_source_version_id(err: &SdkError<HeadObjectError>) -> bool {
is_version_id_format_mismatch(err) && !has_raw_status(err, 403)
}
/// Whether the target's latest version of `object` is a delete marker. For
/// targets that mint their own version ids, which cannot be asked about the
/// source marker id: S3 answers a versionless HEAD on such a key with 404
/// (405 on some targets) plus `x-amz-delete-marker: true`.
async fn latest_target_version_is_delete_marker(tgt_client: &TargetClient, object: &str) -> bool {
match head_object_for_worker(tgt_client, &tgt_client.bucket, object, None).await {
Ok(head) => head.delete_marker == Some(true),
Err(e) => raw_header_value(e.as_ref(), "x-amz-delete-marker").as_deref() == Some("true"),
}
}
async fn mark_replication_target_offline_if_needed(target_client: &Arc<TargetClient>, err: &(impl Display + ?Sized)) {
if is_replication_target_offline_error(err) {
ReplicationTargetStore::mark_target_offline(target_client).await;
@@ -1846,7 +1866,17 @@ async fn verify_resync_head_result(
// only absence proves completion. All other outcomes remain
// failed (backlog#862 / #799 B13).
let retryable = if roi.version_purge_status.is_empty() {
matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
match classify_delete_marker_head_error(&err) {
// A target that mints its own version ids never holds the
// source marker id; verify the marker on the latest version.
DeleteMarkerHeadOutcome::Ambiguous => {
let mints_own = rejects_source_version_id(&err)
|| ReplicationTargetStore::version_identity_capability(&target_client.arn)
.version_addressing_unreliable();
!(mints_own && latest_target_version_is_delete_marker(target_client, &roi.name).await)
}
DeleteMarkerHeadOutcome::Exists | DeleteMarkerHeadOutcome::Missing => false,
}
} else {
!matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
};
@@ -3567,7 +3597,16 @@ async fn replicate_delete_to_target(
}
}
if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
// A target that mints its own version ids never holds the source marker
// id, so a version-addressed pre-check cannot succeed there; the marker
// is created by the versionless DELETE below.
let version_addressable =
!ReplicationTargetStore::version_identity_capability(&tgt_client.arn).version_addressing_unreliable();
if version_addressable
&& !is_version_purge
&& dobj.delete_object.delete_marker
&& dobj.delete_object.delete_marker_version_id.is_some()
{
match head_object_for_worker(
tgt_client.as_ref(),
&tgt_client.bucket,
@@ -3576,6 +3615,11 @@ async fn replicate_delete_to_target(
)
.await
{
// The verdict is not known yet (e.g. no PUT since restart) and the
// target rejected the source id: learn it as the PUT path does.
Err(e) if rejects_source_version_id(e.as_ref()) => {
ReplicationTargetStore::record_version_identity_capability(&tgt_client.arn, VersionIdentityCapability::MintsOwn);
}
Ok(head) if head.delete_marker == Some(true) => {
// Some targets expose an existing marker as a successful HEAD
// with the marker bit instead of the usual 405. Treat that as
@@ -6675,6 +6719,93 @@ mod tests {
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
}
/// AWS S3 answers a HEAD for a version id it did not mint with a bodiless
/// 400. On a target with no verdict yet, that teaches `MintsOwn` and the
/// marker is still created with one versionless DELETE.
#[tokio::test]
async fn live_delete_marker_worker_learns_mints_own_from_bodiless_400() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.starts_with("HEAD ") {
empty_response("400 Bad Request")
} else if line.starts_with("DELETE ") {
empty_response("204 No Content")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
register_test_target(&target).await;
let dobj = marker_creation_dobj(&target.arn);
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
assert_eq!(
ReplicationTargetStore::version_identity_capability(&target.arn),
VersionIdentityCapability::MintsOwn
);
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 2, "{seen:?}");
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
assert!(seen[1].starts_with("DELETE "), "{seen:?}");
assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
}
/// A bodiless 403 on the version-addressed HEAD may be a genuine
/// GetObjectVersion denial on an id-adopting target. It must fail closed:
/// no DELETE, and no `MintsOwn` verdict cached for the ARN.
#[tokio::test]
async fn live_delete_marker_worker_keeps_bodiless_403_failure_closed() {
let (endpoint, server) = spawn_scripted_target_server(1, |line| {
if line.starts_with("HEAD ") {
empty_response("403 Forbidden")
} else if line.starts_with("DELETE ") {
empty_response("204 No Content")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
register_test_target(&target).await;
let dobj = marker_creation_dobj(&target.arn);
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
assert_ne!(
ReplicationTargetStore::version_identity_capability(&target.arn),
VersionIdentityCapability::MintsOwn
);
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 1, "a denied HEAD must not trigger DELETE: {seen:?}");
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
}
/// A target already known to mint its own version ids is never asked for
/// the source marker id: the marker is created by the versionless DELETE.
#[tokio::test]
async fn live_delete_marker_worker_skips_version_head_on_mints_own_target() {
let (endpoint, server) = spawn_scripted_target_server(1, |line| {
if line.starts_with("DELETE ") {
empty_response("204 No Content")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
register_test_target(&target).await;
ReplicationTargetStore::record_version_identity_capability(&target.arn, VersionIdentityCapability::MintsOwn);
let dobj = marker_creation_dobj(&target.arn);
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 1, "no version-addressed HEAD on a MintsOwn target: {seen:?}");
assert!(seen[0].starts_with("DELETE "), "{seen:?}");
assert!(!seen[0].contains("versionId="), "marker creation must remain versionless: {seen:?}");
}
/// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
/// 405, so the SDK reports no error code. That is the marker having
/// propagated, not a failure (rustfs/backlog#2479).
@@ -6765,6 +6896,76 @@ mod tests {
server.join().expect("test HTTP server should finish");
}
/// AWS answers `HEAD ?versionId=<source marker id>` with a bodiless 400.
/// The marker is verified through the latest version instead: a
/// versionless 404 with `x-amz-delete-marker: true` proves it propagated.
#[tokio::test]
async fn resync_verification_accepts_latest_delete_marker_when_target_rejects_source_id() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.contains("versionId=") {
empty_response("400 Bad Request")
} else {
"HTTP/1.1 404 Not Found\r\nX-Amz-Delete-Marker: true\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
.to_string()
}
});
let target = test_target_client(endpoint);
let roi = delete_marker_roi();
let mut st = TargetReplicationResyncStatus::default();
let head_result =
head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
assert!(err.is_none(), "{err:?}");
assert_eq!((size, st.replicated_count, st.failed_count), (0, 1, 0));
let seen = server.join().expect("test HTTP server should finish");
assert!(!seen[1].contains("versionId="), "the re-check must address the latest version: {seen:?}");
}
/// A live latest version on a target that rejects the source id means the
/// marker has not propagated: the verification stays failed.
#[tokio::test]
async fn resync_verification_fails_delete_marker_when_latest_target_version_is_live() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.contains("versionId=") {
empty_response("400 Bad Request")
} else {
empty_response("200 OK")
}
});
let target = test_target_client(endpoint);
let roi = delete_marker_roi();
let mut st = TargetReplicationResyncStatus::default();
let head_result =
head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
assert!(err.is_some(), "a live latest version must not verify the marker");
assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
server.join().expect("test HTTP server should finish");
}
/// A bodiless 403 may be a genuine permission denial: it is not evidence
/// that the target mints its own ids, so no latest-version re-check runs.
#[tokio::test]
async fn resync_verification_keeps_delete_marker_failed_on_bodiless_403() {
let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("403 Forbidden"));
let target = test_target_client(endpoint);
let roi = delete_marker_roi();
let mut st = TargetReplicationResyncStatus::default();
let head_result =
head_object_for_worker(target.as_ref(), &target.bucket, &roi.name, roi.version_id.map(|v| v.to_string())).await;
let (size, err) = verify_resync_head_result(head_result, &roi, &mut st, &target).await;
assert!(err.is_some(), "a 403 must not count as a propagated delete marker");
assert_eq!((size, st.replicated_count, st.failed_count), (0, 0, 1));
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 1, "{seen:?}");
}
/// Serves exactly `requests` connections, answering each from
/// `respond(request_line)`, and returns the request lines it saw. Reads
/// the whole request (headers plus `Content-Length` body, honoring
@@ -12,9 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(test)]
use crate::disk::DiskIoClass;
use crate::disk::disk_store::{get_drive_walkdir_peek_timeout, get_drive_walkdir_stall_timeout};
use crate::disk::error::DiskError;
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions};
use crate::disk::{self, DiskAPI, DiskStore, WalkDirOptions, current_disk_io_context, with_optional_disk_io_context};
use futures::future::join_all;
use metrics::counter;
use rustfs_filemeta::{MetaCacheEntries, MetaCacheEntry, MetacacheReader, is_io_eof};
@@ -245,6 +247,8 @@ pub struct ListPathRawOptions {
pub(crate) test_fallback_reader_behaviors: Vec<TestReaderBehavior>,
#[cfg(test)]
pub(crate) peek_timeout: Option<Duration>,
#[cfg(test)]
pub(crate) test_disk_io_context_observed: Option<Arc<AtomicBool>>,
// pub agreed: Option<Arc<dyn Fn(MetaCacheEntry) + Send + Sync>>,
// pub partial: Option<Arc<dyn Fn(MetaCacheEntries, &[Option<Error>]) + Send + Sync>>,
// pub finished: Option<Arc<dyn Fn(&[Option<Error>]) + Send + Sync>>,
@@ -276,6 +280,8 @@ impl Clone for ListPathRawOptions {
test_fallback_reader_behaviors: self.test_fallback_reader_behaviors.clone(),
#[cfg(test)]
peek_timeout: self.peek_timeout,
#[cfg(test)]
test_disk_io_context_observed: self.test_disk_io_context_observed.clone(),
..Default::default()
}
}
@@ -344,6 +350,7 @@ async fn list_path_raw_inner(
let log_path = opts.path.clone();
let mut jobs = JoinSet::new();
let disk_io_context = current_disk_io_context();
let mut readers = Vec::with_capacity(opts.disks.len());
let fds = Arc::new(TokioMutex::new(opts.fallback_disks.iter().flatten().cloned().collect::<VecDeque<_>>()));
#[cfg(test)]
@@ -364,11 +371,19 @@ async fn list_path_raw_inner(
let test_fallbacks_clone = test_fallbacks.clone();
let cancel_rx_clone = cancel_rx.clone();
let producer_errs_clone = producer_errs.clone();
let disk_io_context_clone = disk_io_context.clone();
let (rd, wr) = tokio::io::duplex(64);
readers.push(MetacacheReader::new(rd));
jobs.spawn(async move {
jobs.spawn(with_optional_disk_io_context(disk_io_context_clone, async move {
let mut wr = PublishedBytesWriter::new(wr);
#[cfg(test)]
if let Some(observer) = &opts_clone.test_disk_io_context_observed {
observer.store(
crate::disk::current_disk_io_context().is_some_and(|context| context.class == DiskIoClass::Background),
std::sync::atomic::Ordering::Release,
);
}
#[cfg(test)]
let test_primary_error = if let Some(behavior) = opts_clone.test_reader_behaviors.get(disk_idx).cloned() {
match behavior {
TestReaderBehavior::Eof => return Ok(()),
@@ -652,7 +667,7 @@ async fn list_path_raw_inner(
// warn!("list_path_raw: while need_fallback done");
Ok(())
});
}));
}
let revjob_rx = rx.clone();
@@ -1168,6 +1183,68 @@ mod tests {
assert_eq!(err, DiskError::ErasureReadQuorum);
}
#[tokio::test]
async fn list_path_raw_joinset_producers_inherit_scanner_disk_io_class() {
let observed = Arc::new(AtomicBool::new(false));
crate::disk::with_background_disk_io(
CancellationToken::new(),
list_path_raw(
CancellationToken::new(),
ListPathRawOptions {
disks: vec![None],
min_disks: 1,
test_reader_behaviors: vec![TestReaderBehavior::Eof],
test_disk_io_context_observed: Some(observed.clone()),
..Default::default()
},
),
)
.await
.expect("empty test producer should complete the listing");
assert!(
observed.load(std::sync::atomic::Ordering::Acquire),
"the per-disk JoinSet producer must retain the scanner background class"
);
}
#[tokio::test]
async fn scanner_joinset_walk_uses_the_local_disk_background_gate() {
let root = tempfile::tempdir().expect("local disk root should be created");
let endpoint = crate::disk::endpoint::Endpoint::try_from(root.path().to_str().expect("disk path should be UTF-8"))
.expect("endpoint should parse");
let local = Arc::new(
crate::disk::local::LocalDisk::new(&endpoint, false)
.await
.expect("local disk should open"),
);
let wrapper = crate::disk::disk_store::LocalDiskWrapper::new(local, false);
wrapper.make_volume("bucket").await.expect("bucket volume should be created");
let disk = Arc::new(crate::disk::Disk::Local(Box::new(wrapper.clone())));
let cancellation = CancellationToken::new();
crate::disk::with_background_disk_io(
cancellation.clone(),
list_path_raw(
cancellation,
ListPathRawOptions {
disks: vec![Some(disk)],
bucket: "bucket".to_string(),
min_disks: 1,
recursive: true,
..Default::default()
},
),
)
.await
.expect("scanner metacache walk should complete");
assert!(
wrapper.background_io_admissions_for_tests() > 0,
"the LocalDiskWrapper walk should acquire background admission inside its JoinSet producer"
);
}
#[tokio::test]
async fn list_path_raw_returns_typed_quorum_error_for_multiple_drive_failures() {
let err = list_path_raw(
+184 -30
View File
@@ -24,10 +24,11 @@ use crate::cluster::rpc::{gen_tonic_replay_scope_headers, gen_tonic_signature_he
#[cfg(test)]
use crate::cluster::rpc::{tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers};
use crate::disk::error::{DiskError, Error as DiskErrorType, RpcStatusError};
#[cfg(test)]
use crate::runtime::sources as runtime_sources;
use http::{Request as HttpRequest, Response as HttpResponse, Uri};
use rustfs_protos::{
ChannelClass, create_new_channel, get_channel_for_class,
ChannelClass, get_channel_for_class,
proto_gen::node_service::{
heal_control_service_client::HealControlServiceClient, node_service_client::NodeServiceClient,
scanner_control_service_client::ScannerControlServiceClient,
@@ -53,7 +54,7 @@ use super::context_propagation::{inject_request_id_into_metadata, inject_trace_c
/// 3. Subsequent calls will attempt fresh connections
/// 4. If node is still down, connection will fail fast (3s timeout)
pub async fn node_service_time_out_client(
addr: &String,
addr: &str,
interceptor: TonicInterceptor,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
// Default to the latency-sensitive control channel; bulk `bytes` RPCs opt in via the
@@ -66,12 +67,9 @@ pub(crate) async fn scanner_control_time_out_client(
interceptor: TonicInterceptor,
) -> crate::error::Result<ScannerControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr)
.await
.map_err(|err| crate::error::Error::other(err.to_string()))?,
};
let channel = get_channel_for_class(addr, ChannelClass::Control)
.await
.map_err(|err| crate::error::Error::other(err.to_string()))?;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
let limit = rustfs_protos::scoped_dirty_usage::SCOPED_DIRTY_USAGE_MAX_REQUEST_BYTES as usize;
Ok(ScannerControlServiceClient::with_interceptor(channel, interceptor)
@@ -84,10 +82,7 @@ pub async fn heal_control_time_out_client(
interceptor: TonicInterceptor,
) -> Result<HealControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(HealControlServiceClient::with_interceptor(channel, interceptor)
@@ -100,10 +95,7 @@ pub async fn tier_mutation_control_time_out_client(
interceptor: TonicInterceptor,
) -> Result<TierMutationControlServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match runtime_sources::cached_node_channel(addr).await {
Some(channel) => channel,
None => create_new_channel(addr).await?,
};
let channel = get_channel_for_class(addr, ChannelClass::Control).await?;
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
Ok(TierMutationControlServiceClient::with_interceptor(channel, interceptor)
@@ -118,22 +110,12 @@ pub async fn tier_mutation_control_time_out_client(
/// from lock/health RPCs; everything else uses [`ChannelClass::Control`]. When isolation is
/// disabled the two classes resolve to the same cached channel, i.e. legacy behavior.
pub async fn node_service_time_out_client_for_class(
addr: &String,
addr: &str,
interceptor: TonicInterceptor,
class: ChannelClass,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let channel = match class {
ChannelClass::Control => match runtime_sources::cached_node_channel(addr).await {
Some(channel) => {
debug!("Using cached gRPC channel for: {}", addr);
channel
}
// No cached connection, create new one.
None => create_new_channel(addr).await?,
},
ChannelClass::Bulk => get_channel_for_class(addr, ChannelClass::Bulk).await?,
};
let channel = get_channel_for_class(addr, class).await?;
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
@@ -142,8 +124,24 @@ pub async fn node_service_time_out_client_for_class(
.max_encoding_message_size(max_message_size))
}
pub(crate) async fn node_service_delete_recovery_client(
addr: &str,
interceptor: TonicInterceptor,
timeout: std::time::Duration,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, DiskError> {
let interceptor = interceptor.with_rpc_audience(addr).map_err(DiskError::from)?;
let channel = rustfs_protos::create_channel_with_timeout(addr, timeout)
.await
.map_err(|error| DiskError::RemoteClientUnavailable(error.to_string()))?;
let channel = ReplayScopeChannel::new(channel, interceptor.replay_scope_audience());
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
Ok(NodeServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size))
}
pub async fn node_service_time_out_client_no_auth(
addr: &String,
addr: &str,
) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>, Box<dyn Error>> {
node_service_time_out_client(addr, TonicInterceptor::NoOp(NoOpInterceptor)).await
}
@@ -355,6 +353,13 @@ fn apply_peer_replay_response(
}
}
fn rename_data_grpc_stage(path: &str) -> bool {
matches!(
path,
"/node_service.NodeService/RenameData" | "/node_service.NodeService/RenameDataAtIncarnation"
)
}
impl<S, ReqBody, ResBody> Service<HttpRequest<ReqBody>> for ReplayScopeChannel<S>
where
S: Service<HttpRequest<ReqBody>, Response = HttpResponse<ResBody>>,
@@ -372,6 +377,12 @@ where
}
fn call(&mut self, mut request: HttpRequest<ReqBody>) -> Self::Future {
let observe_rename_data = rustfs_io_metrics::put_stage_metrics_enabled() && rename_data_grpc_stage(request.uri().path());
let request_scope_started = if observe_rename_data {
rustfs_io_metrics::put_stage_timer()
} else {
None
};
let authenticated = self.audience.as_ref().is_some_and(|_| {
request
.headers()
@@ -406,11 +417,30 @@ where
}
}
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REQUEST_SCOPE,
request_scope_started,
);
let audience = self.audience.clone();
let future = self.inner.call(request);
Box::pin(async move {
let response = future.await?;
let response = if observe_rename_data {
rustfs_io_metrics::observe_put_stage_future(
future,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_TRANSPORT_CALL_FIRST_PENDING_TO_READY,
)
.await?
} else {
future.await?
};
let replay_response_started = if observe_rename_data {
rustfs_io_metrics::put_stage_timer()
} else {
None
};
if let (Some(audience), Some(challenge)) = (audience, challenge) {
let response_state = verify_tonic_peer_replay_capabilities_response(&audience, challenge, response.headers());
if let Err(error) = &response_state
@@ -428,6 +458,10 @@ where
}
apply_peer_replay_response(audience, sent_state, response_state);
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_REPLAY_RESPONSE,
replay_response_started,
);
Ok(response)
})
}
@@ -546,6 +580,126 @@ mod tests {
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{Registry, layer::SubscriberExt};
#[tokio::test]
async fn delete_recovery_channel_uses_its_scoped_deadline() {
use bytes::Bytes;
use http_body_util::StreamBody;
use hyper::{body::Frame, service::service_fn};
use hyper_util::rt::{TokioExecutor, TokioIo};
use rustfs_protos::proto_gen::node_service::{RecoverDeleteTransactionRequest, RecoverDeleteTransactionResponse};
use tonic_prost::prost::Message;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("bind recovery peer");
let addr = format!("http://{}", listener.local_addr().expect("peer address"));
let server = tokio::spawn(async move {
loop {
let (stream, _) = listener.accept().await.expect("accept recovery connection");
tokio::spawn(async move {
let service = service_fn(|_request: HttpRequest<hyper::body::Incoming>| async {
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
let payload = RecoverDeleteTransactionResponse {
success: true,
protocol_version: 1,
result: Bytes::new(),
error: None,
}
.encode_to_vec();
let mut data = vec![0];
data.extend_from_slice(&u32::try_from(payload.len()).expect("small response").to_be_bytes());
data.extend_from_slice(&payload);
let mut trailers = http::HeaderMap::new();
trailers.insert("grpc-status", "0".parse().expect("status"));
let frames: Vec<Result<Frame<Bytes>, std::convert::Infallible>> =
vec![Ok(Frame::data(data.into())), Ok(Frame::trailers(trailers))];
let mut response = HttpResponse::new(StreamBody::new(futures::stream::iter(frames)));
response
.headers_mut()
.insert("content-type", "application/grpc".parse().expect("content type"));
Ok::<_, std::convert::Infallible>(response)
});
let _ = hyper::server::conn::http2::Builder::new(TokioExecutor::new())
.serve_connection(TokioIo::new(stream), service)
.await;
});
}
});
let request = || RecoverDeleteTransactionRequest {
disk: "disk".into(),
volume: "bucket".into(),
protocol_version: 1,
operation: Bytes::new(),
};
// Seed an existing short-lived control channel to reproduce the inherited deadline.
let control = tonic::transport::Endpoint::from_shared(addr.clone())
.expect("test endpoint")
.timeout(std::time::Duration::from_millis(50))
.connect()
.await
.expect("control connect");
rustfs_common::cache_connection(addr.clone(), control).await;
let mut foreground = node_service_time_out_client(&addr, TonicInterceptor::NoOp(NoOpInterceptor))
.await
.expect("control client");
let error = foreground
.recover_delete_transaction(request())
.await
.expect_err("short control deadline");
assert!(matches!(error.code(), tonic::Code::Cancelled | tonic::Code::DeadlineExceeded));
// The 2-second test override bypasses the production range parser, which rejects 2.
let mut recovery = node_service_delete_recovery_client(
&addr,
TonicInterceptor::NoOp(NoOpInterceptor),
std::time::Duration::from_secs(2),
)
.await
.expect("scoped recovery client");
assert!(
recovery
.recover_delete_transaction(request())
.await
.expect("scoped deadline must survive control deadline")
.into_inner()
.success
);
server.abort();
rustfs_protos::evict_failed_connection(&addr).await;
}
#[tokio::test]
async fn rpc_client_factories_do_not_reuse_https_channels_without_tls_generation() {
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("reserve local address");
let addr = format!("https://{}", listener.local_addr().expect("local address"));
drop(listener);
for factory in 0..4 {
let placeholder = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy();
runtime_sources::cache_test_node_channel(addr.clone(), placeholder).await;
let interceptor = TonicInterceptor::NoOp(NoOpInterceptor);
let _result = match factory {
0 => scanner_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
1 => heal_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
2 => tier_mutation_control_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
_ => node_service_time_out_client(&addr, interceptor)
.await
.map(|_| ())
.map_err(|e| e.to_string()),
};
assert!(
!runtime_sources::test_node_channel_is_cached(&addr).await,
"factory {factory} must evict an unbound HTTPS cache entry before attempting a fresh dial"
);
}
}
#[derive(Clone)]
struct EpochProofService {
audience: String,
@@ -289,13 +289,9 @@ async fn client(
>,
NetworkPeerProbeError,
> {
node_service_time_out_client_for_class(
&address.to_owned(),
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
class,
)
.await
.map_err(|_| NetworkPeerProbeError::Unreachable)
node_service_time_out_client_for_class(address, TonicInterceptor::Signature(gen_tonic_signature_interceptor()), class)
.await
.map_err(|_| NetworkPeerProbeError::Unreachable)
}
fn ping_request(payload: &[u8]) -> PingRequest {
@@ -329,6 +325,24 @@ fn map_status(status: tonic::Status) -> NetworkPeerProbeError {
match status.code() {
Code::Unavailable | Code::Unknown => NetworkPeerProbeError::Unreachable,
Code::DeadlineExceeded => NetworkPeerProbeError::TimedOut,
Code::Cancelled | Code::Internal => {
let mut source = std::error::Error::source(&status);
while let Some(error) = source {
if error
.downcast_ref::<h2::Error>()
.is_some_and(|error| matches!(error.reason(), Some(h2::Reason::CANCEL | h2::Reason::NO_ERROR)))
{
return NetworkPeerProbeError::Unreachable;
}
source = if let Some(error) = error.downcast_ref::<std::io::Error>() {
// io::Error::source skips its immediate custom error.
error.get_ref().map(|inner| inner as &(dyn std::error::Error + 'static))
} else {
error.source()
};
}
NetworkPeerProbeError::ProtocolFailure
}
_ => NetworkPeerProbeError::ProtocolFailure,
}
}
@@ -339,6 +353,35 @@ mod tests {
use crate::layout::endpoint::Endpoint;
use crate::layout::endpoints::{Endpoints, PoolEndpoints};
#[test]
fn peer_transport_resets_are_unreachable_without_reclassifying_application_errors() {
for reason in [h2::Reason::CANCEL, h2::Reason::NO_ERROR] {
let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
assert!(matches!(status.code(), Code::Cancelled | Code::Internal));
let code = status.code();
assert_eq!(map_status(status), NetworkPeerProbeError::Unreachable);
let mut wrapped = tonic::Status::new(code, "wrapped transport reset");
wrapped.set_source(Arc::new(std::io::Error::other(h2::Error::from(reason))));
assert_eq!(map_status(wrapped), NetworkPeerProbeError::Unreachable);
}
for reason in [h2::Reason::PROTOCOL_ERROR, h2::Reason::COMPRESSION_ERROR] {
let status = tonic::Status::from_error(Box::new(h2::Error::from(reason)));
assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
}
for status in [
tonic::Status::cancelled("peer cancelled the application request"),
tonic::Status::internal("peer rejected the probe"),
tonic::Status::unauthenticated("authentication rejected"),
tonic::Status::permission_denied("authorization rejected"),
] {
assert_eq!(map_status(status), NetworkPeerProbeError::ProtocolFailure);
}
assert_eq!(
map_status(tonic::Status::deadline_exceeded("probe deadline elapsed")),
NetworkPeerProbeError::TimedOut
);
}
fn pacing(duration: Duration) -> Arc<DiagnosticPacing> {
NetworkPeerProbeClient::from_endpoint_pools(&topology())
.with_diagnostic_pacing(Instant::now(), duration)
@@ -640,6 +640,7 @@ pub struct PeerLiveEventsBatch {
pub struct ScannerPublicationLease {
pub token: Uuid,
pub movement_generation: u64,
pub observational_only: bool,
/// Stable storage owner identity. This is distinct from the activity
/// session and is bound into both acquire and release proofs.
pub owner_id: String,
@@ -658,6 +659,7 @@ fn validate_scanner_publication_lease_response_fields(
response: &ScannerPublicationLeaseResponse,
expected_session_id: &str,
expected_generation: u64,
expected_observational_only: bool,
) -> Result<(Uuid, String)> {
if !response.success {
return Err(Error::other(
@@ -674,6 +676,9 @@ fn validate_scanner_publication_lease_response_fields(
if response.session_id != expected_session_id {
return Err(Error::other("peer returned a different scanner publication lease session"));
}
if response.observational_only != expected_observational_only {
return Err(Error::other("peer returned a different scanner publication lease purpose"));
}
let owner_id = Uuid::parse_str(&response.owner_id)
.ok()
.filter(|owner_id| !owner_id.is_nil())
@@ -2418,6 +2423,7 @@ impl PeerRestClient {
&self,
expected_session_id: &str,
expected_generation: u64,
observational_only: bool,
) -> Result<ScannerPublicationLease> {
let request_started = std::time::Instant::now();
self.finalize_result(
@@ -2434,6 +2440,7 @@ impl PeerRestClient {
ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
expected_session_id: expected_session_id.to_string(),
token: Bytes::new(),
observational_only,
});
let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
.map_err(|_| Error::other("scanner publication lease request is too large to authenticate"))?;
@@ -2444,11 +2451,16 @@ impl PeerRestClient {
.map_err(|_| Error::other("scanner publication lease response is too large to authenticate"))?;
verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
.map_err(|_| Error::other("peer returned an invalid scanner publication lease proof"))?;
let (token, owner_id) =
validate_scanner_publication_lease_response_fields(&response, expected_session_id, expected_generation)?;
let (token, owner_id) = validate_scanner_publication_lease_response_fields(
&response,
expected_session_id,
expected_generation,
observational_only,
)?;
Ok(ScannerPublicationLease {
token,
movement_generation: response.movement_generation,
observational_only,
owner_id,
session_id: response.session_id,
expires_at: scanner_publication_lease_deadline(
@@ -2483,6 +2495,7 @@ impl PeerRestClient {
ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
expected_session_id: lease.session_id.clone(),
token: lease.token.as_bytes().to_vec().into(),
observational_only: lease.observational_only,
});
let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
.map_err(|_| Error::other("scanner publication lease validation request is too large to authenticate"))?;
@@ -2494,8 +2507,12 @@ impl PeerRestClient {
)?;
verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
.map_err(|_| Error::other("peer returned an invalid scanner publication lease validation proof"))?;
let (token, owner_id) =
validate_scanner_publication_lease_response_fields(&response, &lease.session_id, lease.movement_generation)?;
let (token, owner_id) = validate_scanner_publication_lease_response_fields(
&response,
&lease.session_id,
lease.movement_generation,
lease.observational_only,
)?;
if token != lease.token {
return Err(Error::other("peer returned a different scanner publication lease token"));
}
@@ -3001,15 +3018,16 @@ mod tests {
response_proof: Bytes::new(),
owner_id: Uuid::new_v4().to_string(),
session_id: "session-a".to_string(),
observational_only: false,
};
assert!(validate_scanner_publication_lease_response_fields(&response, "session-a", 7).is_ok());
assert!(validate_scanner_publication_lease_response_fields(&response, "session-a", 7, false).is_ok());
let stale_generation = ScannerPublicationLeaseResponse {
movement_generation: 6,
..response.clone()
};
let error = validate_scanner_publication_lease_response_fields(&stale_generation, "session-a", 7)
let error = validate_scanner_publication_lease_response_fields(&stale_generation, "session-a", 7, false)
.expect_err("a response from an older movement generation must be rejected");
assert!(error.to_string().contains("different scanner publication lease generation"));
@@ -3017,9 +3035,21 @@ mod tests {
session_id: "session-b".to_string(),
..response
};
let error = validate_scanner_publication_lease_response_fields(&stale_session, "session-a", 7)
let error = validate_scanner_publication_lease_response_fields(&stale_session, "session-a", 7, false)
.expect_err("a response from an older scanner session must be rejected");
assert!(error.to_string().contains("different scanner publication lease session"));
let observational = ScannerPublicationLeaseResponse {
observational_only: true,
..stale_session
};
assert!(
validate_scanner_publication_lease_response_fields(&observational, "session-b", 7, true).is_ok(),
"the signed purpose echo should validate an observation lease"
);
let error = validate_scanner_publication_lease_response_fields(&observational, "session-b", 7, false)
.expect_err("a peer must not upgrade an observational lease to authoritative");
assert!(error.to_string().contains("different scanner publication lease purpose"));
}
#[test]
+418 -39
View File
@@ -14,7 +14,8 @@
use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_time_out_client, node_service_time_out_client_for_class, node_service_time_out_client_no_auth,
node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
node_service_time_out_client_no_auth,
};
use crate::cluster::rpc::internode_data_transport::{
InternodeDataTransport, NsScannerCapabilityRequest, NsScannerStreamRequest, ReadStreamRequest, WalkDirStreamRequest,
@@ -22,10 +23,11 @@ use crate::cluster::rpc::internode_data_transport::{
};
use crate::disk::error::{Error, Result};
use crate::disk::{
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DeleteOptions, DiskAPI, DiskInfo,
DiskInfoOptions, DiskLocation, DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq,
ReadMultipleResp, ReadOptions, RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
batch_read_version_one_by_one,
BatchReadVersionReq, BatchReadVersionResp, CheckPartsResp, ConditionalFileUpdate, DELETE_RECOVERY_PROTOCOL_VERSION,
DeleteOptions, DeleteRecoveryAction, DeleteRecoveryCursor, DeleteRecoveryDigest, DeleteRecoveryInspection,
DeleteRecoveryPage, DeleteRecoveryRpcOperation, DeleteRecoveryRpcReply, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation,
DiskOption, FileInfoVersions, FileReader, FileWriter, PartTransactionAction, ReadMultipleReq, ReadMultipleResp, ReadOptions,
RenameDataResp, SnapshotLeaseToken, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, batch_read_version_one_by_one,
disk_store::{
DEFAULT_RUSTFS_DRIVE_ACTIVE_MONITORING, ENV_RUSTFS_DRIVE_ACTIVE_MONITORING, SKIP_IF_SUCCESS_BEFORE,
get_drive_active_check_interval, get_drive_active_check_timeout, get_drive_disk_info_timeout, get_drive_list_dir_timeout,
@@ -59,10 +61,10 @@ use rustfs_protos::proto_gen::node_service::{
CompareAndUpdateFileRequest, DeletePathsRequest, DeleteRequest, DeleteVersionRequest, DeleteVersionsRequest,
DeleteVersionsResponse, DeleteVolumeRequest, DiskInfoRequest, ListDirRequest, ListVolumesRequest, MakeVolumeRequest,
MakeVolumesRequest, PreparePartTransactionRequest, ReadAllRequest, ReadMetadataRequest, ReadMultipleRequest,
ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RenameDataRequest, RenameFileRequest,
SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest,
SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest, WriteMetadataRequest,
node_service_client::NodeServiceClient,
ReadMultipleResponse, ReadPartsRequest, ReadVersionRequest, ReadXlRequest, RecoverDeleteTransactionRequest,
RenameDataRequest, RenameFileRequest, SettlePartTransactionRequest, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
SnapshotLeaseRequest, SnapshotLeaseResponse, StatVolumeRequest, UpdateMetadataRequest, VerifyFileRequest, WriteAllRequest,
WriteMetadataRequest, node_service_client::NodeServiceClient,
};
use serde::{Serialize, de::DeserializeOwned};
use std::{
@@ -142,6 +144,45 @@ fn decode_delete_versions_errors(response: DeleteVersionsResponse, expected_len:
.collect()
}
fn decode_delete_recovery_batch_errors(codes: Vec<Option<u32>>, expected_len: usize) -> Result<Vec<Option<DiskError>>> {
if codes.len() != expected_len {
return Err(Error::other("malformed recoverable batch delete response length"));
}
codes
.into_iter()
.map(|code| match code {
None => Ok(None),
Some(code) => DiskError::from_u32(code)
.map(Some)
.ok_or_else(|| Error::other("invalid recoverable batch delete error code")),
})
.collect()
}
fn validate_delete_recovery_inspection(
volume: &str,
object: &str,
transaction: Uuid,
inspection: &DeleteRecoveryInspection,
) -> Result<()> {
if inspection.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| inspection.id.object != object
|| inspection.id.transaction != transaction
|| transaction.is_nil()
{
return Err(Error::other("delete recovery inspection does not match request"));
}
if let Some(journal) = &inspection.journal
&& (journal.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| journal.volume != volume
|| journal.object != object
|| journal.transaction != transaction)
{
return Err(Error::other("delete recovery journal does not match request"));
}
Ok(())
}
fn snapshot_lease_token_from_response(response: SnapshotLeaseResponse) -> Result<SnapshotLeaseToken> {
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -858,6 +899,75 @@ fn spawn_control_channel_prewarm(addr: String) {
}
impl RemoteDisk {
async fn recover_delete_transaction(
&self,
volume: &str,
operation: DeleteRecoveryRpcOperation,
) -> Result<DeleteRecoveryRpcReply> {
let payload = rmp_serde::to_vec_named(&operation)?;
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
}
let settle_timeout = if matches!(&operation, DeleteRecoveryRpcOperation::Settle { .. }) {
Some(crate::disk::get_delete_recovery_timeout()?)
} else {
None
};
self.execute_with_timeout(
|| async {
let mut client = if let Some(timeout) = settle_timeout {
if let Some(error) = self.offline_bypass_error() {
return Err(error);
}
node_service_delete_recovery_client(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
timeout,
)
.await?
} else {
self.get_client().await?
};
let mut request = Request::new(RecoverDeleteTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_owned(),
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
operation: payload.into(),
});
let canonical_body = rustfs_protos::canonical_recover_delete_transaction_request_body(request.get_ref())
.map_err(|_| Error::other("recover_delete_transaction request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &canonical_body).map_err(Error::other)?;
// UNIMPLEMENTED is deliberately propagated: older peers cannot settle transactions.
let response = client
.recover_delete_transaction(request)
.await
.map_err(|status| {
if status.code() == Code::Unimplemented {
DiskError::UnsupportedDisk
} else {
status.into()
}
})?
.into_inner();
if response.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION {
return Err(DiskError::UnsupportedDisk);
}
if !response.success {
return Err(response
.error
.ok_or_else(|| Error::other("missing delete recovery error"))?
.into());
}
if response.error.is_some() {
return Err(Error::other("malformed delete recovery success"));
}
rustfs_protos::decode_delete_recovery_payload(&response.result).map_err(Into::into)
},
settle_timeout.unwrap_or_else(get_max_timeout_duration),
)
.await
}
async fn rename_file_with_durability(
&self,
src_volume: &str,
@@ -1920,6 +2030,12 @@ fn encode_msgpack<T: Serialize>(value: &T) -> Result<Vec<u8>> {
encode_msgpack_with_capacity(value, MSGPACK_ENCODE_CAPACITY_HINT)
}
// Optional fields are omitted by serde; named maps keep rollback flags and
// transaction UUIDs independent of field order across peer versions.
fn encode_delete_options_msgpack(value: &DeleteOptions) -> Result<Vec<u8>> {
Ok(rmp_serde::to_vec_named(value)?)
}
fn encode_file_info_msgpack(value: &FileInfo) -> Result<Vec<u8>> {
encode_msgpack_with_capacity(value, FILE_INFO_MSGPACK_ENCODE_CAPACITY_HINT)
}
@@ -2163,6 +2279,7 @@ impl RemoteDisk {
self.execute_with_timeout_for_op(
"rename_data",
|| async {
let prepare_started = rustfs_io_metrics::put_stage_timer();
let file_info = compat_json(fi)?;
let file_info_bin = encode_file_info_msgpack(fi)?;
let mut client = self.get_client().await?;
@@ -2190,24 +2307,67 @@ impl RemoteDisk {
} else {
attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
prepare_started,
);
let response = if incarnation_bound {
// Older peers return Unimplemented before mutation; never downgrade.
client.rename_data_at_incarnation(request).await?
} else {
client.rename_data(request).await?
}
.into_inner();
let rpc_started = rustfs_io_metrics::put_stage_timer();
let response = rustfs_io_metrics::observe_put_stage_future(
async {
if incarnation_bound {
// Older peers return Unimplemented before mutation; never downgrade.
client.rename_data_at_incarnation(request).await
} else {
client.rename_data(request).await
}
},
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
)
.await?;
let response_process_started = rustfs_io_metrics::put_stage_timer();
let response_headers_started = rustfs_io_metrics::put_stage_timer();
let _ = response.metadata().len();
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_HEADERS,
response_headers_started,
);
let into_inner_started = rustfs_io_metrics::put_stage_timer();
let response = response.into_inner();
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_INTO_INNER,
into_inner_started,
);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC,
rpc_started,
);
let success_check_started = rustfs_io_metrics::put_stage_timer();
if !response.success {
return Err(response.error.unwrap_or_default().into());
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_SUCCESS_CHECK,
success_check_started,
);
let decode_started = rustfs_io_metrics::put_stage_timer();
let rename_data_resp = decode_msgpack_or_json::<RenameDataResp>(
&response.rename_data_resp_bin,
&response.rename_data_resp,
"RenameDataResp",
)?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_DECODE,
decode_started,
);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RESPONSE_PROCESS,
response_process_started,
);
Ok(rename_data_resp)
},
@@ -2566,7 +2726,7 @@ impl DiskAPI for RemoteDisk {
// `_bin` support for DeleteVersion is new (grpc-optimization P2); always dual-write
// JSON + msgpack until its fallback counter has read zero across a release window.
let file_info_bin = encode_file_info_msgpack(&fi)?;
let opts_bin = encode_msgpack(&opts)?;
let opts_bin = encode_delete_options_msgpack(&opts)?;
let conditional_marker = opts.expected_delete_marker.is_some();
let file_info = serde_json::to_string(&fi)?;
let opts = serde_json::to_string(&opts)?;
@@ -2638,7 +2798,7 @@ impl DiskAPI for RemoteDisk {
// `_bin` support for DeleteVersions is new (grpc-optimization P2); always dual-write JSON +
// msgpack until its fallback counter has read zero across a release window.
let opts_bin = match encode_msgpack(&opts) {
let opts_bin = match encode_delete_options_msgpack(&opts) {
Ok(opts_bin) => opts_bin,
Err(err) => {
let mut errors = Vec::with_capacity(versions.len());
@@ -3617,6 +3777,138 @@ impl DiskAPI for RemoteDisk {
}
#[tracing::instrument(level = "trace", skip_all)]
async fn delete_version_recoverable(
&self,
volume: &str,
object: &str,
file_info: FileInfo,
force_del_marker: bool,
opts: DeleteOptions,
) -> Result<()> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::ApplyVersion {
object: object.to_owned(),
file_info: Box::new(file_info),
force_del_marker,
opts,
},
)
.await?
{
DeleteRecoveryRpcReply::AppliedVersion => Ok(()),
_ => Err(Error::other("unexpected recoverable delete reply")),
}
}
async fn delete_versions_recoverable(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
let expected_len = versions.len();
let result = match self
.recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ApplyVersions { versions, opts })
.await
{
Ok(DeleteRecoveryRpcReply::AppliedVersions(codes)) => decode_delete_recovery_batch_errors(codes, expected_len),
Ok(_) => Err(Error::other("unexpected recoverable batch delete reply")),
Err(error) => Err(error),
};
match result {
Ok(errors) => errors,
Err(error) => vec![Some(error); expected_len],
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
object: Option<&str>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Discover {
object: object.map(str::to_owned),
cursor,
limit,
},
)
.await?
{
DeleteRecoveryRpcReply::Discovered(page) => {
if object.is_some_and(|object| page.transactions.iter().any(|id| id.object != object)) {
return Err(Error::other("delete recovery discovery escaped requested object"));
}
if page.transactions.len() > usize::try_from(limit).map_err(|_| Error::other("invalid discovery limit"))?
|| page
.transactions
.iter()
.any(|id| id.transaction.is_nil() || id.object.is_empty())
{
return Err(Error::other("malformed delete discovery page"));
}
Ok(page)
}
_ => Err(Error::other("unexpected delete discovery reply")),
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Inspect {
object: object.to_owned(),
transaction,
},
)
.await?
{
DeleteRecoveryRpcReply::Inspected(inspection) => {
validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
Ok(inspection)
}
_ => Err(Error::other("unexpected delete inspection reply")),
}
}
async fn settle_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
match self
.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::Settle {
object: object.to_owned(),
transaction,
expected_fingerprint,
action,
},
)
.await?
{
DeleteRecoveryRpcReply::Settled(inspection) => {
validate_delete_recovery_inspection(volume, object, transaction, &inspection)?;
Ok(inspection)
}
_ => Err(Error::other("unexpected delete settlement reply")),
}
}
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
self.execute_with_timeout(
|| async {
@@ -4030,6 +4322,45 @@ mod tests {
static INIT: Once = Once::new();
#[test]
fn recoverable_batch_delete_rejects_invalid_error_codes_and_length() {
let errors = decode_delete_recovery_batch_errors(vec![None, Some(DiskError::FileNotFound.to_u32())], 2)
.expect("valid per-item errors");
assert!(errors[0].is_none());
assert_eq!(errors[1], Some(DiskError::FileNotFound));
assert!(decode_delete_recovery_batch_errors(vec![None], 2).is_err());
assert!(decode_delete_recovery_batch_errors(vec![Some(0)], 1).is_err());
assert!(decode_delete_recovery_batch_errors(vec![Some(u32::MAX)], 1).is_err());
}
#[test]
fn delete_recovery_inspection_rejects_wrong_transaction_object_and_protocol() {
let transaction = Uuid::new_v4();
let inspection = DeleteRecoveryInspection {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
id: crate::disk::DeleteTransactionId {
object: "object".into(),
transaction,
},
state: crate::disk::DeleteRecoveryState::Absent,
journal: None,
current_digest: None,
current_versions: Vec::new(),
backup_digest: None,
rollback_data_complete: false,
commit_data_safe: false,
explicit_delete_safe: false,
fingerprint: [0; 32],
};
validate_delete_recovery_inspection("bucket", "object", transaction, &inspection).expect("matching inspection");
assert!(validate_delete_recovery_inspection("bucket", "other", transaction, &inspection).is_err());
assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::new_v4(), &inspection).is_err());
assert!(validate_delete_recovery_inspection("bucket", "object", Uuid::nil(), &inspection).is_err());
let mut incompatible = inspection;
incompatible.protocol_version += 1;
assert!(validate_delete_recovery_inspection("bucket", "object", transaction, &incompatible).is_err());
}
#[test]
fn compare_and_update_wire_outcome_fails_closed() {
assert_eq!(
@@ -4063,32 +4394,19 @@ mod tests {
}
#[test]
fn retired_marker_options_preserve_legacy_positional_wire_shape() {
fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
#[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
struct LegacyDeleteOptions {
recursive: bool,
immediate: bool,
undo_write: bool,
#[serde(default)]
undo_delete: bool,
old_data_dir: Option<Uuid>,
#[serde(default, skip_serializing_if = "Option::is_none")]
expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
}
let legacy = LegacyDeleteOptions {
recursive: false,
immediate: false,
undo_write: false,
undo_delete: false,
old_data_dir: None,
};
let original = encode_msgpack(&legacy).unwrap();
let current = encode_msgpack(&DeleteOptions::default()).unwrap();
assert_eq!(current, original, "ordinary deletes must retain the older peer's positional payload");
assert_eq!(rmp_serde::from_slice::<LegacyDeleteOptions>(&current).unwrap(), legacy);
assert!(
rmp_serde::from_slice::<DeleteOptions>(&original)
.unwrap()
.expected_delete_marker
.is_none()
);
let transaction = Uuid::new_v4();
let mut marker = FileInfo {
deleted: true,
version_id: Some(Uuid::new_v4()),
@@ -4097,12 +4415,69 @@ mod tests {
};
marker.set_delete_marker_incarnation(Uuid::new_v4());
let marker = rustfs_filemeta::MetaDeleteMarker::from(marker);
for expected_delete_marker in [None, Some(marker)] {
for (undo_write, undo_delete) in [(true, false), (false, true)] {
let legacy = LegacyDeleteOptions {
recursive: false,
immediate: false,
undo_write,
undo_delete,
old_data_dir: Some(transaction),
expected_delete_marker: expected_delete_marker.clone(),
};
// Older senders use positional arrays. Their unchanged field prefix and
// missing optional fields must also decode on the named-map sender's peer.
for (encoding, payload) in [
("positional", rmp_serde::to_vec(&legacy).expect("encode legacy positional options")),
("named", rmp_serde::to_vec_named(&legacy).expect("encode legacy named options")),
] {
let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode legacy options on current peer");
assert_eq!(decoded.undo_write, undo_write, "{encoding} undo-write flag");
assert_eq!(decoded.undo_delete, undo_delete, "{encoding} undo-delete flag");
assert_eq!(decoded.old_data_dir, Some(transaction), "{encoding} transaction UUID");
assert_eq!(decoded.expected_delete_marker, expected_delete_marker, "{encoding} marker precondition");
assert!(decoded.delete_recovery_bucket_incarnation.is_none(), "{encoding} omitted incarnation");
assert!(!decoded.recoverable_delete, "{encoding} omitted recovery flag");
}
for incarnation in [None, Some(Uuid::new_v4())] {
let current = DeleteOptions {
undo_write,
undo_delete,
old_data_dir: Some(transaction),
expected_delete_marker: expected_delete_marker.clone(),
delete_recovery_bucket_incarnation: incarnation,
recoverable_delete: true,
..Default::default()
};
let payload = encode_delete_options_msgpack(&current).expect("encode current named options");
let decoded: DeleteOptions = rmp_serde::from_slice(&payload).expect("decode current options on current peer");
assert_eq!(decoded.undo_write, undo_write);
assert_eq!(decoded.undo_delete, undo_delete);
assert_eq!(decoded.old_data_dir, Some(transaction));
assert_eq!(decoded.expected_delete_marker, expected_delete_marker);
assert_eq!(decoded.delete_recovery_bucket_incarnation, incarnation);
assert!(decoded.recoverable_delete);
let decoded: LegacyDeleteOptions =
rmp_serde::from_slice(&payload).expect("decode current options on legacy peer");
assert_eq!(decoded, legacy, "unknown recovery fields cannot shift rollback or marker values");
}
}
}
}
#[test]
fn malformed_nonempty_delete_options_do_not_fall_back_to_json() {
let options = DeleteOptions {
expected_delete_marker: Some(marker.clone()),
undo_write: true,
old_data_dir: Some(Uuid::new_v4()),
..Default::default()
};
let decoded: DeleteOptions = rmp_serde::from_slice(&encode_msgpack(&options).unwrap()).unwrap();
assert_eq!(decoded.expected_delete_marker, Some(marker));
let json = serde_json::to_string(&options).expect("valid compatibility JSON");
assert!(decode_msgpack_or_json::<DeleteOptions>(&[0xc1], &json, "DeleteOptions").is_err());
let decoded: DeleteOptions =
decode_msgpack_or_json(&[], &json, "DeleteOptions").expect("absent MessagePack permits JSON compatibility");
assert!(decoded.undo_write);
assert_eq!(decoded.old_data_dir, options.old_data_dir);
}
#[test]
@@ -4785,6 +5160,8 @@ mod tests {
cleanup_data_dir: Some(Uuid::new_v4()),
sign: Some(vec![0x14, 0x35]),
old_current_size: Some(crate::disk::OldCurrentSize::Present(64 * 1024)),
old_current_source_checked: false,
old_current_source: None,
};
let json = serde_json::to_string(&response).expect("legacy rename_data JSON response should encode");
let decode_before = rustfs_io_metrics::internode_metrics::global_internode_metrics().msgpack_json_decode_total_for_test();
@@ -5187,6 +5564,8 @@ mod tests {
cleanup_data_dir: Some(Uuid::new_v4()),
sign: Some(vec![1_u8; 32]),
old_current_size: Some(crate::disk::OldCurrentSize::Present(4096)),
old_current_source_checked: false,
old_current_source: None,
};
let json = serde_json::to_vec(&response).expect("rename data response json should encode");
let named_msgpack = rmp_serde::encode::to_vec_named(&response).expect("rename data response named msgpack should encode");
+75 -1
View File
@@ -738,7 +738,12 @@ where
.filter(|etag| !etag.is_empty())
.map(str::to_owned)
.ok_or_else(|| Error::other("data usage snapshot has no ETag"))?;
let mut data_usage_info = parse_usage_snapshot(&reader.read_all().await?)?;
let mut data_usage_info = parse_usage_snapshot(
&reader
.read_all()
.await
.map_err(|err| map_data_usage_metadata_read_error(err, object))?,
)?;
populate_backward_compatible_usage_maps(&mut data_usage_info);
validate_complete_usage_snapshot(&mut data_usage_info);
Ok(Some((data_usage_info, revision)))
@@ -2997,6 +3002,61 @@ mod tests {
}
}
#[derive(Debug)]
struct UsageSnapshotReadFailureStore;
struct UsageSnapshotFailureReader;
impl tokio::io::AsyncRead for UsageSnapshotFailureReader {
fn poll_read(
self: std::pin::Pin<&mut Self>,
_cx: &mut std::task::Context<'_>,
_buf: &mut tokio::io::ReadBuf<'_>,
) -> std::task::Poll<std::io::Result<()>> {
std::task::Poll::Ready(Err(std::io::Error::other(Error::VolumeNotFound)))
}
}
#[async_trait::async_trait]
impl crate::storage_api_contracts::object::ObjectIO for UsageSnapshotReadFailureStore {
type Error = Error;
type RangeSpec = crate::storage_api_contracts::range::HTTPRangeSpec;
type HeaderMap = http::HeaderMap;
type ObjectOptions = ObjectOptions;
type ObjectInfo = ObjectInfo;
type GetObjectReader = crate::object_api::GetObjectReader;
type PutObjectReader = PutObjReader;
async fn get_object_reader(
&self,
_bucket: &str,
_object: &str,
_range: Option<Self::RangeSpec>,
_headers: Self::HeaderMap,
_opts: &Self::ObjectOptions,
) -> Result<Self::GetObjectReader, Self::Error> {
Ok(crate::object_api::GetObjectReader {
stream: Box::new(UsageSnapshotFailureReader),
object_info: ObjectInfo {
etag: Some("usage-revision".to_string()),
..Default::default()
},
buffered_body: None,
body_source: Default::default(),
})
}
async fn put_object(
&self,
_bucket: &str,
_object: &str,
_data: &mut Self::PutObjectReader,
_opts: &Self::ObjectOptions,
) -> Result<Self::ObjectInfo, Self::Error> {
unimplemented!("the read-failure fixture does not write")
}
}
/// Minimal ObjectIO backing `load_data_usage_cache` tests: records the keys
/// read and fails the first N reads with a transient (non-absence) error.
#[derive(Debug, Default)]
@@ -3161,6 +3221,20 @@ mod tests {
}
}
#[tokio::test]
async fn data_usage_removal_maps_missing_system_volume_during_snapshot_stream_read() {
let store = UsageSnapshotReadFailureStore;
let error = load_data_usage_for_bucket_removal(&store, "bucket-metadata/.usage.json")
.await
.expect_err("a system volume disappearing during body read must remain a retryable storage error");
assert_eq!(
error,
Error::InsufficientReadQuorum(RUSTFS_META_BUCKET.to_string(), "bucket-metadata/.usage.json".to_string())
);
}
#[tokio::test]
async fn load_data_usage_cache_retries_a_transient_failure() {
let name = "usage-cache";
@@ -15,7 +15,9 @@
use crate::cluster::rpc::{
ScannerBucketListing, TonicInterceptor, gen_tonic_signature_interceptor, node_service_time_out_client,
};
use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_from_backend_cached};
use crate::data_usage::{
DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_admin_data_usage_from_backend_cached as load_server_info_usage_snapshot,
};
use crate::error::{Error, Result};
use crate::{
disk::endpoint::{Endpoint, EndpointType},
@@ -42,7 +44,11 @@ use std::{
use time::OffsetDateTime;
use tokio::time::timeout;
use tonic::Request;
use tracing::warn;
use tracing::{error, trace, warn};
const EVENT_SERVER_PING: &str = "server_ping";
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_DIAGNOSTICS: &str = "diagnostics";
use shadow_rs::shadow;
@@ -157,11 +163,22 @@ async fn is_server_resolvable(endpoint: &Endpoint) -> Result<()> {
let response: PingResponse = client.ping(request).await?.into_inner();
let ping_response_body = flatbuffers::root::<PingBody>(&response.body);
if let Err(e) = ping_response_body {
eprintln!("{e}");
if flatbuffers::root::<PingBody>(&response.body).is_err() {
error!(
event = EVENT_SERVER_PING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
result = "invalid_response",
"Server ping response decoding failed"
);
} else {
println!("ping_resp:body(flatbuffer): {ping_response_body:?}");
trace!(
event = EVENT_SERVER_PING,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DIAGNOSTICS,
result = "success",
"Server ping completed"
);
}
Ok(())
@@ -294,7 +311,7 @@ pub async fn get_server_info(get_pools: bool) -> InfoMessage {
if let Some(store) = runtime_sources::object_store_handle() {
mode = ITEM_ONLINE;
apply_data_usage_result(
load_data_usage_from_backend_cached(store.clone()).await,
load_server_info_usage_snapshot(store.clone()).await,
&mut buckets,
&mut objects,
&mut versions,
@@ -967,6 +984,93 @@ mod tests {
assert_eq!(usage.size, 6);
}
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
usage_snapshot_bootstrap_pending: true,
usage_snapshot_converged: Some(false),
..Default::default()
};
let observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(2)),
scanner_epoch: Some(1),
scanner_cycle: Some(1),
buckets_count: 1,
objects_total_count: 3,
versions_total_count: 4,
delete_markers_total_count: 1,
objects_total_size: 6,
buckets_usage: [(
"bucket".to_string(),
BucketUsageInfo {
objects_count: 3,
versions_count: 4,
delete_markers_count: 1,
size: 6,
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(primary.snapshot_identity()),
..Default::default()
};
let primary_bytes = serde_json::to_vec(&primary).expect("bootstrap should encode");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
.await
.expect("bootstrap should persist");
save_config(
store.clone(),
DATA_USAGE_OBSERVED_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&observed).expect("observation should encode"),
)
.await
.expect("observation should persist");
invalidate_data_usage_snapshot_cache().await;
let displayed = super::load_server_info_usage_snapshot(store.clone())
.await
.expect("server info should load the complete observation");
assert_eq!(displayed.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(displayed), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert!(objects.error.is_none());
assert_eq!(
(buckets.count, objects.count, versions.count, delete_markers.count, usage.size),
(1, 3, 4, 1, 6)
);
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("quota should retain its bootstrap state");
assert!(!quota.is_complete_bucket_usage_snapshot());
assert_eq!(
read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("primary should remain readable"),
primary_bytes
);
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
+1
View File
@@ -182,6 +182,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_DELETE_MARKER: &str = "delete_marker"
pub(crate) const GET_METADATA_CACHE_REASON_DIST_ERASURE: &str = "dist_erasure";
pub(crate) const GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS: &str = "incl_free_versions";
pub(crate) const GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM: &str = "insufficient_cached_quorum";
pub(crate) const GET_METADATA_CACHE_REASON_BODY_LAYOUT_REQUIRED: &str = "body_layout_required";
pub(crate) const GET_METADATA_CACHE_REASON_META_BUCKET: &str = "meta_bucket";
pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_io_metrics::record_object_lock_acquire_failure;
use rustfs_lock::{LockError, NamespaceLockGuard, NamespaceLockWrapper};
use std::time::{Duration, Instant};
use tracing::Span;
@@ -21,6 +22,26 @@ const EVENT_OBJECT_NAMESPACE_LOCK: &str = "object_namespace_lock";
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
pub(crate) fn record_lock_acquire_failure(op: &'static str, bucket: &str, object: &str, mode: &'static str, error: &LockError) {
let resource_class = if bucket == crate::disk::RUSTFS_META_MULTIPART_BUCKET {
"multipart_metadata"
} else if bucket == crate::disk::RUSTFS_META_BUCKET || bucket == ".minio.sys" {
"system_metadata"
} else if object.is_empty() || object == bucket {
"bucket_namespace"
} else {
"object_namespace"
};
let reason = match error {
LockError::Timeout { .. } => "timeout",
LockError::AlreadyLocked { .. } => "contention",
LockError::QuorumNotReached { .. } | LockError::InsufficientNodes { .. } => "quorum",
LockError::Network { .. } | LockError::Internal { .. } => "transport",
_ => "other",
};
record_object_lock_acquire_failure(op, mode, resource_class, reason);
}
/// Diagnostic lifetime only; it never owns or changes a namespace lock.
/// Keep it alongside the real guard when ownership moves to a commit tail.
#[derive(Default)]
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+27 -4
View File
@@ -131,6 +131,15 @@ pub fn build_write_quorum_failure_summary(
fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
let Some(dominant_error) = dominant_error else {
// The reducer also returns None when every failure is ignored for
// quorum voting. That case contains no successful nil entry.
if !errors.is_empty() && errors.iter().all(Option::is_some) {
return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
"disk_not_found"
} else {
"all_ignored"
};
}
return "nil_dominated";
};
@@ -387,9 +396,9 @@ mod tests {
}
#[test]
fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
// All disks offline (all DiskNotFound). The dominant error must be None and
// the label must be "nil_dominated" rather than falling back to "other_error".
fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
// Ignoring unavailable disks for quorum reduction must not imply a
// successful nil entry in the diagnostic label.
let errors = vec![
Some(Error::DiskNotFound),
Some(Error::DiskNotFound),
@@ -397,7 +406,21 @@ mod tests {
];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.dominant_error_label, "nil_dominated");
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "disk_not_found");
}
#[test]
fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "all_ignored");
assert_eq!(
reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
Some(Error::ErasureWriteQuorum)
);
}
#[test]
-28
View File
@@ -223,12 +223,6 @@ pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()
std::fs::rename(from, to)
}
#[cfg(any(not(windows), test))]
#[tracing::instrument(level = "debug", skip_all)]
pub async fn read_file(path: impl AsRef<Path>) -> io::Result<Vec<u8>> {
fs::read(path.as_ref()).await
}
// Bucket existence cache - reduces statx syscalls for repeated bucket checks
/// Cache for bucket directory existence checks.
@@ -614,28 +608,6 @@ mod tests {
assert_eq!(content, "test content");
}
#[tokio::test]
async fn test_read_file() {
let temp_dir = TempDir::new().unwrap();
let file_path = temp_dir.path().join("test_read.txt");
let test_content = b"This is test content for reading";
tokio::fs::write(&file_path, test_content).await.unwrap();
// Read file
let read_content = read_file(&file_path).await.unwrap();
assert_eq!(read_content, test_content);
}
#[tokio::test]
async fn test_read_file_nonexistent() {
let temp_dir = TempDir::new().unwrap();
let file_path = temp_dir.path().join("nonexistent.txt");
// Should fail for non-existent file
assert!(read_file(&file_path).await.is_err());
}
#[tokio::test]
async fn test_same_file() {
let temp_dir = TempDir::new().unwrap();
+706
View File
@@ -0,0 +1,706 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Per-local-disk foreground-priority admission for scanner I/O.
use std::collections::VecDeque;
use std::future::Future;
use std::io;
use std::pin::Pin;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::task::{Context, Poll};
use std::time::Duration;
use metrics::{counter, gauge, histogram};
use rustfs_rio::ChunkReader;
use tokio::io::{AsyncRead, ReadBuf};
use tokio::sync::Notify;
use tokio::time::Instant;
use tokio_util::sync::CancellationToken;
const BACKGROUND_FOREGROUND_BURST: u32 = 8;
const BACKGROUND_MAX_WAIT: Duration = Duration::from_millis(100);
tokio::task_local! {
static DISK_IO_CONTEXT: DiskIoContext;
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum DiskIoClass {
Foreground,
Background,
}
#[derive(Clone, Debug)]
pub(crate) struct DiskIoContext {
pub(crate) class: DiskIoClass,
pub(crate) cancellation: CancellationToken,
pub(crate) admission: Option<Arc<DiskIoAdmission>>,
pub(crate) permit: Option<DiskIoPermit>,
}
/// Mark disk I/O issued by `future` as background work for local-disk admission.
#[doc(hidden)]
pub async fn with_background_disk_io<F: Future>(cancellation: CancellationToken, future: F) -> F::Output {
with_disk_io_context(
DiskIoContext {
class: DiskIoClass::Background,
cancellation,
admission: None,
permit: None,
},
future,
)
.await
}
pub(crate) async fn with_disk_io_context<F: Future>(context: DiskIoContext, future: F) -> F::Output {
DISK_IO_CONTEXT.scope(context, future).await
}
pub(crate) async fn with_disk_io_permit<F: Future>(permit: Option<DiskIoPermit>, future: F) -> F::Output {
let Some(permit) = permit else {
return future.await;
};
let mut context = current_disk_io_context().unwrap_or_else(|| DiskIoContext {
class: DiskIoClass::Foreground,
cancellation: CancellationToken::new(),
admission: None,
permit: None,
});
context.admission = Some(permit.admission());
with_disk_io_context(
DiskIoContext {
permit: Some(permit),
..context
},
future,
)
.await
}
pub(crate) async fn with_optional_disk_io_context<F: Future>(context: Option<DiskIoContext>, future: F) -> F::Output {
match context {
Some(context) => with_disk_io_context(context, future).await,
None => future.await,
}
}
pub(crate) fn current_disk_io_context() -> Option<DiskIoContext> {
DISK_IO_CONTEXT.try_with(Clone::clone).ok()
}
/// Acquire one low-level permit only when the current walker explicitly owns a gate.
/// Ordinary wrapper calls use their outer operation permit instead.
pub(crate) async fn acquire_context_disk_io_permit() -> io::Result<Option<DiskIoPermit>> {
let Some(context) = current_disk_io_context() else {
return Ok(None);
};
let Some(admission) = context.admission else {
return Ok(None);
};
if let Some(permit) = context.permit
&& permit.is_for(&admission)
{
return Ok(Some(permit));
}
match context.class {
DiskIoClass::Foreground => Ok(Some(admission.foreground())),
DiskIoClass::Background => Ok(Some(admission.background(&context.cancellation).await?)),
}
}
#[derive(Debug, Default)]
struct AdmissionState {
foreground_completions: u32,
next_background_ticket: u64,
background_queue: VecDeque<u64>,
background_front_since: Option<Instant>,
}
/// One gate is shared by all handles to a local disk, including reconnects.
/// Foreground I/O never waits on this gate. Background I/O runs exclusively
/// while the disk is idle, or after a bounded foreground burst/wait.
#[derive(Debug)]
pub(crate) struct DiskIoAdmission {
state: Mutex<AdmissionState>,
changed: Notify,
disk: String,
foreground_active: AtomicUsize,
background_active: AtomicBool,
background_waiters: AtomicUsize,
#[cfg(test)]
background_admissions: AtomicUsize,
}
impl DiskIoAdmission {
pub(crate) fn new(disk: impl Into<String>) -> Arc<Self> {
Arc::new(Self {
state: Mutex::new(AdmissionState::default()),
changed: Notify::new(),
disk: disk.into(),
foreground_active: AtomicUsize::new(0),
background_active: AtomicBool::new(false),
background_waiters: AtomicUsize::new(0),
#[cfg(test)]
background_admissions: AtomicUsize::new(0),
})
}
#[cfg(test)]
pub(crate) fn background_admissions_for_tests(&self) -> usize {
self.background_admissions.load(Ordering::Relaxed)
}
#[cfg(test)]
pub(crate) fn background_active_for_tests(&self) -> bool {
self.background_active.load(Ordering::Acquire)
}
#[cfg(test)]
fn foreground_active_for_tests(&self) -> usize {
self.foreground_active.load(Ordering::Acquire)
}
pub(crate) fn foreground(self: &Arc<Self>) -> DiskIoPermit {
self.foreground_active.fetch_add(1, Ordering::AcqRel);
if self.background_waiters.load(Ordering::Acquire) > 0 {
self.record_state();
}
DiskIoPermit {
lease: Arc::new(DiskIoPermitLease {
admission: self.clone(),
class: DiskIoClass::Foreground,
}),
}
}
pub(crate) async fn background(self: &Arc<Self>, cancellation: &CancellationToken) -> std::io::Result<DiskIoPermit> {
let started_at = Instant::now();
let ticket = {
let mut state = self.state.lock().unwrap_or_else(|error| error.into_inner());
let ticket = state.next_background_ticket;
state.next_background_ticket = state.next_background_ticket.wrapping_add(1);
if state.background_queue.is_empty() && !self.background_active.load(Ordering::Acquire) {
state.background_front_since = Some(Instant::now());
}
state.background_queue.push_back(ticket);
self.background_waiters.fetch_add(1, Ordering::Release);
ticket
};
self.record_state();
let mut waiter = BackgroundWaiter {
admission: self.clone(),
ticket,
waiting: true,
};
loop {
let notified = self.changed.notified();
tokio::pin!(notified);
notified.as_mut().enable();
let granted = {
let mut state = self.state.lock().unwrap_or_else(|error| error.into_inner());
let is_front = state.background_queue.front() == Some(&ticket);
let bounded_wait_elapsed = is_front
&& state
.background_front_since
.is_some_and(|front_since| front_since.elapsed() >= BACKGROUND_MAX_WAIT);
if is_front
&& !self.background_active.load(Ordering::Acquire)
&& (self.foreground_active.load(Ordering::Acquire) == 0
|| state.foreground_completions >= BACKGROUND_FOREGROUND_BURST
|| bounded_wait_elapsed)
{
self.background_active.store(true, Ordering::Release);
state.background_queue.pop_front();
state.background_front_since = None;
self.background_waiters.fetch_sub(1, Ordering::AcqRel);
state.foreground_completions = 0;
true
} else {
false
}
};
if granted {
waiter.waiting = false;
#[cfg(test)]
self.background_admissions.fetch_add(1, Ordering::Relaxed);
histogram!("rustfs_disk_io_admission_wait_seconds", "disk" => self.disk.clone(), "class" => "background")
.record(started_at.elapsed().as_secs_f64());
counter!("rustfs_disk_io_admission_total", "disk" => self.disk.clone(), "class" => "background").increment(1);
self.record_state();
return Ok(DiskIoPermit {
lease: Arc::new(DiskIoPermitLease {
admission: self.clone(),
class: DiskIoClass::Background,
}),
});
}
let remaining = {
let state = self.state.lock().unwrap_or_else(|error| error.into_inner());
if state.background_queue.front() == Some(&ticket) {
state
.background_front_since
.map(|front_since| BACKGROUND_MAX_WAIT.saturating_sub(front_since.elapsed()))
} else {
None
}
};
if let Some(remaining) = remaining.filter(|remaining| !remaining.is_zero()) {
tokio::select! {
_ = cancellation.cancelled() => {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "background disk I/O admission cancelled"));
}
_ = tokio::time::sleep(remaining) => {
self.changed.notify_waiters();
}
_ = &mut notified => {}
}
} else {
tokio::select! {
_ = cancellation.cancelled() => {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "background disk I/O admission cancelled"));
}
_ = &mut notified => {}
}
}
}
}
fn record_state(&self) {
let (foreground_active, background_waiters) = {
let state = self.state.lock().unwrap_or_else(|error| error.into_inner());
(
if state.background_queue.is_empty() {
0
} else {
self.foreground_active.load(Ordering::Acquire)
},
state.background_queue.len(),
)
};
let foreground_active = u32::try_from(foreground_active).unwrap_or(u32::MAX);
let background_waiters = u32::try_from(background_waiters).unwrap_or(u32::MAX);
gauge!("rustfs_disk_io_admission_active", "disk" => self.disk.clone(), "class" => "foreground")
.set(f64::from(foreground_active));
gauge!("rustfs_disk_io_admission_active", "disk" => self.disk.clone(), "class" => "background").set(
if self.background_active.load(Ordering::Acquire) {
1.0
} else {
0.0
},
);
gauge!("rustfs_disk_io_admission_waiters", "disk" => self.disk.clone(), "class" => "background")
.set(f64::from(background_waiters));
}
}
struct BackgroundWaiter {
admission: Arc<DiskIoAdmission>,
ticket: u64,
waiting: bool,
}
impl Drop for BackgroundWaiter {
fn drop(&mut self) {
if !self.waiting {
return;
}
let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
if let Some(index) = state.background_queue.iter().position(|ticket| *ticket == self.ticket) {
state.background_queue.remove(index);
if index == 0 {
state.background_front_since =
if state.background_queue.is_empty() || self.admission.background_active.load(Ordering::Acquire) {
None
} else {
Some(Instant::now())
};
}
self.admission.background_waiters.fetch_sub(1, Ordering::AcqRel);
}
drop(state);
self.admission.record_state();
self.admission.changed.notify_waiters();
}
}
#[derive(Debug, Clone)]
pub(crate) struct DiskIoPermit {
lease: Arc<DiskIoPermitLease>,
}
impl DiskIoPermit {
pub(crate) fn is_for(&self, admission: &Arc<DiskIoAdmission>) -> bool {
Arc::ptr_eq(&self.lease.admission, admission)
}
fn admission(&self) -> Arc<DiskIoAdmission> {
self.lease.admission.clone()
}
}
#[derive(Debug)]
struct DiskIoPermitLease {
admission: Arc<DiskIoAdmission>,
class: DiskIoClass,
}
pub(crate) fn spawn_blocking_with_disk_io_permit<T, F>(
io_permit: Option<DiskIoPermit>,
operation: F,
) -> tokio::task::JoinHandle<T>
where
T: Send + 'static,
F: FnOnce() -> T + Send + 'static,
{
tokio::task::spawn_blocking(move || {
let _io_permit = io_permit;
operation()
})
}
type AdmissionWait = Pin<Box<dyn Future<Output = io::Result<DiskIoPermit>> + Send>>;
impl Drop for DiskIoPermitLease {
fn drop(&mut self) {
match self.class {
DiskIoClass::Foreground => {
self.admission.foreground_active.fetch_sub(1, Ordering::AcqRel);
let record_state = self.admission.background_waiters.load(Ordering::Acquire) > 0;
if record_state {
let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
if !state.background_queue.is_empty() {
state.foreground_completions = state.foreground_completions.saturating_add(1);
}
}
if record_state {
self.admission.record_state();
self.admission.changed.notify_waiters();
}
}
DiskIoClass::Background => {
let mut state = self.admission.state.lock().unwrap_or_else(|error| error.into_inner());
self.admission.background_active.store(false, Ordering::Release);
state.background_front_since = if state.background_queue.is_empty() {
None
} else {
Some(Instant::now())
};
drop(state);
self.admission.record_state();
self.admission.changed.notify_waiters();
}
}
}
}
pub(crate) struct AdmissionReader<R> {
inner: R,
admission: Arc<DiskIoAdmission>,
context: DiskIoContext,
wait: Mutex<Option<AdmissionWait>>,
permit: Option<DiskIoPermit>,
}
impl<R> AdmissionReader<R> {
pub(crate) fn new(inner: R, admission: Arc<DiskIoAdmission>, context: DiskIoContext) -> Self {
Self {
inner,
admission,
context,
wait: Mutex::new(None),
permit: None,
}
}
fn poll_permit(&mut self, cx: &mut Context<'_>) -> Poll<io::Result<()>> {
if self.permit.is_some() {
return Poll::Ready(Ok(()));
}
if self.context.class == DiskIoClass::Foreground {
self.permit = Some(self.admission.foreground());
return Poll::Ready(Ok(()));
}
let mut wait = self.wait.lock().unwrap_or_else(|error| error.into_inner());
if wait.is_none() {
let admission = self.admission.clone();
let cancellation = self.context.cancellation.clone();
*wait = Some(Box::pin(async move { admission.background(&cancellation).await }));
}
let Some(wait_future) = wait.as_mut() else {
return Poll::Ready(Err(io::Error::other("background disk I/O admission waiter is missing")));
};
let result = wait_future.as_mut().poll(cx);
match result {
Poll::Ready(Ok(permit)) => {
wait.take();
drop(wait);
self.permit = Some(permit);
Poll::Ready(Ok(()))
}
Poll::Ready(Err(error)) => {
wait.take();
Poll::Ready(Err(error))
}
Poll::Pending => Poll::Pending,
}
}
fn finish_poll<T>(&mut self, result: Poll<io::Result<T>>) -> Poll<io::Result<T>> {
if result.is_ready() {
self.permit = None;
}
result
}
}
impl<R: AsyncRead + Unpin> AsyncRead for AdmissionReader<R> {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
match self.poll_permit(cx) {
Poll::Ready(Ok(())) => {}
Poll::Ready(Err(error)) => return Poll::Ready(Err(error)),
Poll::Pending => return Poll::Pending,
}
let result = Pin::new(&mut self.inner).poll_read(cx, buf);
self.finish_poll(result)
}
}
impl<R: ChunkReader + Unpin> ChunkReader for AdmissionReader<R> {
fn poll_read_chunk(mut self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<bytes::Bytes>>> {
match self.poll_permit(cx) {
Poll::Ready(Ok(())) => {}
Poll::Ready(Err(error)) => return Poll::Ready(Err(error)),
Poll::Pending => return Poll::Pending,
}
let result = Pin::new(&mut self.inner).poll_read_chunk(cx, max);
self.finish_poll(result)
}
}
pub(crate) struct BoxedChunkReader(pub(crate) rustfs_rio::ChunkReaderBox);
impl AsyncRead for BoxedChunkReader {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Pin::new(&mut self.0).poll_read(cx, buf)
}
}
impl ChunkReader for BoxedChunkReader {
fn poll_read_chunk(self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<bytes::Bytes>>> {
Pin::new(self.get_mut().0.as_mut()).poll_read_chunk(cx, max)
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
struct BlockingWorker(tokio::task::JoinHandle<()>);
async fn wait_for_background_waiters(admission: &DiskIoAdmission, expected: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let waiters = admission
.state
.lock()
.unwrap_or_else(|error| error.into_inner())
.background_queue
.len();
if waiters == expected {
return;
}
tokio::task::yield_now().await;
}
})
.await
.expect("background admission waiter should reach expected count");
}
async fn wait_for_foreground_active(admission: &DiskIoAdmission, expected: usize) {
tokio::time::timeout(Duration::from_secs(1), async {
loop {
let active = admission.foreground_active.load(Ordering::Acquire);
if active == expected {
return;
}
tokio::task::yield_now().await;
}
})
.await
.expect("foreground disk read should retain its permit while pending");
}
#[tokio::test]
async fn foreground_bypasses_background_and_background_progress_is_bounded() {
let admission = DiskIoAdmission::new("disk-a");
let foreground = admission.foreground();
let background_admission = admission.clone();
let cancellation = CancellationToken::new();
let background = tokio::spawn(async move { background_admission.background(&cancellation).await });
wait_for_background_waiters(&admission, 1).await;
for _ in 0..BACKGROUND_FOREGROUND_BURST {
drop(admission.foreground());
}
let background_permit = tokio::time::timeout(Duration::from_secs(1), background)
.await
.expect("background admission should progress after the foreground burst")
.expect("background waiter task should complete")
.expect("background admission should not be cancelled");
assert_eq!(
admission.foreground_active.load(Ordering::Acquire),
1,
"foreground operations remain admitted while one background operation runs"
);
drop(background_permit);
drop(foreground);
}
#[tokio::test]
async fn cancellation_removes_a_queued_background_waiter() {
let admission = DiskIoAdmission::new("disk-a");
let background = admission
.background(&CancellationToken::new())
.await
.expect("first background operation should be admitted");
let cancellation = CancellationToken::new();
let waiter_admission = admission.clone();
let waiter_cancellation = cancellation.clone();
let waiter = tokio::spawn(async move { waiter_admission.background(&waiter_cancellation).await });
wait_for_background_waiters(&admission, 1).await;
cancellation.cancel();
let error = tokio::time::timeout(Duration::from_secs(1), waiter)
.await
.expect("cancelled background waiter should stop")
.expect("background waiter task should complete")
.expect_err("cancelled admission must not start disk I/O");
assert_eq!(error.kind(), io::ErrorKind::Interrupted);
assert_eq!(
admission
.state
.lock()
.unwrap_or_else(|error| error.into_inner())
.background_queue
.len(),
0
);
drop(background);
}
#[tokio::test]
async fn background_admission_is_isolated_per_disk() {
let disk_a = DiskIoAdmission::new("disk-a");
let disk_b = DiskIoAdmission::new("disk-b");
let _disk_a_permit = disk_a
.background(&CancellationToken::new())
.await
.expect("disk A background operation should be admitted");
let _disk_b_permit = tokio::time::timeout(Duration::from_secs(1), disk_b.background(&CancellationToken::new()))
.await
.expect("disk A must not block disk B")
.expect("disk B background operation should be admitted");
}
#[tokio::test]
async fn streamed_read_holds_admission_until_disk_read_resolves() {
let admission = DiskIoAdmission::new("disk-a");
let (mut writer, reader) = tokio::io::duplex(8);
let mut reader = AdmissionReader::new(
reader,
admission.clone(),
DiskIoContext {
class: DiskIoClass::Foreground,
cancellation: CancellationToken::new(),
admission: None,
permit: None,
},
);
let read = tokio::spawn(async move {
let mut bytes = [0; 4];
reader
.read_exact(&mut bytes)
.await
.expect("streamed disk read should complete");
bytes
});
wait_for_foreground_active(&admission, 1).await;
writer.write_all(b"rust").await.expect("test bytes should be written");
assert_eq!(read.await.expect("reader task should finish"), *b"rust");
wait_for_foreground_active(&admission, 0).await;
}
#[tokio::test]
async fn blocking_worker_keeps_permit_after_caller_timeout() {
let admission = DiskIoAdmission::new("disk-a");
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let worker = spawn_blocking_with_disk_io_permit(Some(admission.foreground()), move || {
started_tx.send(()).expect("worker start should be observed");
std::thread::sleep(Duration::from_millis(100));
});
started_rx.await.expect("blocking worker should start");
assert!(tokio::time::timeout(Duration::from_millis(10), worker).await.is_err());
assert_eq!(admission.foreground_active_for_tests(), 1);
wait_for_foreground_active(&admission, 0).await;
}
#[tokio::test]
async fn cloned_permit_remains_active_until_all_workers_finish() {
let admission = DiskIoAdmission::new("disk-a");
let permit = admission.foreground();
let worker_permit = permit.clone();
drop(permit);
assert_eq!(admission.foreground_active_for_tests(), 1);
drop(worker_permit);
assert_eq!(admission.foreground_active_for_tests(), 0);
}
#[tokio::test]
async fn scoped_foreground_permit_is_inherited_by_blocking_worker() {
let admission = DiskIoAdmission::new("disk-a");
let permit = admission.foreground();
let worker = with_disk_io_permit(Some(permit), async {
let worker_permit = acquire_context_disk_io_permit()
.await
.expect("context permit should be available")
.expect("scoped permit should be inherited");
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let worker = spawn_blocking_with_disk_io_permit(Some(worker_permit), move || {
started_tx.send(()).expect("worker start should be observed");
std::thread::sleep(Duration::from_millis(100));
});
started_rx.await.expect("blocking worker should start");
BlockingWorker(worker)
})
.await;
assert_eq!(admission.foreground_active_for_tests(), 1);
assert!(tokio::time::timeout(Duration::from_millis(10), worker.0).await.is_err());
wait_for_foreground_active(&admission, 0).await;
}
}
File diff suppressed because it is too large Load Diff
+75 -17
View File
@@ -47,6 +47,33 @@ use tokio::fs;
use tracing::{info, warn};
use uuid::Uuid;
fn transitioned_old_current_source(
dst_meta_existed: bool,
dst_meta_unparsable: bool,
xlmeta: &FileMeta,
volume: &str,
path: &str,
version_id: Uuid,
) -> (bool, Option<FileInfo>) {
if dst_meta_unparsable {
return (false, None);
}
if !dst_meta_existed {
return (true, None);
}
let Ok(old_current) = xlmeta.into_fileinfo(volume, path, "", false, false, true) else {
return (false, None);
};
if old_current.version_id.unwrap_or_default() != version_id {
return (true, None);
}
if old_current.transition_status == rustfs_filemeta::TRANSITION_COMPLETE {
(true, Some(old_current))
} else {
(true, None)
}
}
/// Hold later repair publications after admitting one baseline object. The
/// fixture arms this on one replacement disk before rejoining the cluster.
#[cfg(feature = "e2e-test-hooks")]
@@ -214,7 +241,7 @@ pub(super) async fn lock_rename_commit_directories(
publication_root: &os::PublicationRoot,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<os::RenameCommitGuard> {
#[cfg(windows)]
#[cfg(any(windows, target_os = "linux"))]
let result = {
let source_parent = source_parent.to_path_buf();
let destination_parent = destination_parent.to_path_buf();
@@ -222,7 +249,7 @@ pub(super) async fn lock_rename_commit_directories(
let publication_root = publication_root.clone();
os::run_blocking_namespace_operation(mutation_lease, move || {
let result = os::prepare_rename_commit_guard(&source_parent, &destination_parent, &base_dir, &publication_root);
#[cfg(test)]
#[cfg(all(test, windows))]
if result.is_ok() {
run_destination_commit_directory_preparation(&destination_parent);
}
@@ -230,7 +257,7 @@ pub(super) async fn lock_rename_commit_directories(
})
.await
};
#[cfg(not(windows))]
#[cfg(not(any(windows, target_os = "linux")))]
let result = {
let _ = mutation_lease;
os::prepare_rename_commit_guard(source_parent, destination_parent, base_dir, publication_root)
@@ -244,12 +271,12 @@ pub(super) async fn lock_rename_commit_directories(
result.map_err(to_file_error).map_err(DiskError::from)
}
async fn read_rename_destination_metadata(
async fn prepare_rename_destination_metadata(
file_path: &Path,
rename_commit_guard: &os::RenameCommitGuard,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<Option<Bytes>> {
#[cfg(windows)]
#[cfg(any(windows, target_os = "linux"))]
let result = {
let file_path = file_path.to_path_buf();
let rename_commit_guard = rename_commit_guard.clone();
@@ -258,10 +285,10 @@ async fn read_rename_destination_metadata(
})
.await
};
#[cfg(not(windows))]
#[cfg(not(any(windows, target_os = "linux")))]
let _ = (rename_commit_guard, mutation_lease);
#[cfg(not(windows))]
let result = match super::super::fs::read_file(file_path).await {
#[cfg(not(any(windows, target_os = "linux")))]
let result = match fs::read(file_path).await {
Ok(data) => Ok(Some(data)),
Err(err) if err.kind() == ErrorKind::NotFound => Ok(None),
Err(err) => Err(err),
@@ -398,7 +425,7 @@ impl LocalDisk {
// xl.meta path
let src_file_path = self.io_get_object_path(src_volume, format!("{}/{}", src_path, STORAGE_FORMAT_FILE).as_str())?;
let dst_file_path = self.io_get_object_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
// data_dir path
let has_data_dir_path = {
@@ -453,8 +480,7 @@ impl LocalDisk {
fs::create_dir_all(src_file_parent).await.map_err(to_file_error)?;
}
// Acquire the common trees before reading destination metadata. On
// Windows this pins the object directory identity across metadata
// preparation, data publication, rollback backup, and final commit.
// Windows this pins the object directory identity across publication.
let rename_commit_guard = lock_rename_commit_directories(
src_file_parent,
dst_file_parent,
@@ -472,9 +498,12 @@ impl LocalDisk {
source_volume_dir: &src_volume_dir,
data_paths: has_data_dir_path.as_ref(),
};
let has_dst_buf =
read_rename_destination_metadata(commit.destination_metadata, &commit.directory_guard, commit.mutation_lease.clone())
.await?;
let has_dst_buf = prepare_rename_destination_metadata(
commit.destination_metadata,
&commit.directory_guard,
commit.mutation_lease.clone(),
)
.await?;
if no_inline {
// Non-inline: read xl.meta, parse, write, rename data dir, rename xl.meta
@@ -508,6 +537,14 @@ impl LocalDisk {
}
let version_id = fi.version_id.unwrap_or_default();
let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
has_dst_buf.is_some(),
dst_meta_unparsable,
&xlmeta,
dst_volume,
dst_path,
version_id,
);
let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
let rollback_data_dir = has_old_data_dir.or_else(|| {
@@ -798,7 +835,9 @@ impl LocalDisk {
&& let Some(parent) = dst_file_path.parent()
{
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dst_dir_group_commit(parent, Some(commit.mutation_lease.clone())).await {
if let Err(err) =
os::fsync_commit_directory(parent, &commit.directory_guard, commit.mutation_lease.clone(), None).await
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
fsync_started,
@@ -897,6 +936,8 @@ impl LocalDisk {
cleanup_data_dir: has_old_data_dir,
sign: version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
})
} else {
// Inline metadata preparation is blocking. The transaction lease is
@@ -910,6 +951,8 @@ impl LocalDisk {
None
};
let dst_path_for_failpoint = dst_path.to_string();
let dst_volume_for_source = dst_volume.to_string();
let dst_path_for_source = dst_path.to_string();
#[cfg(windows)]
let source_parent = src_file_parent.to_path_buf();
let rename_commit_guard_for_preparation = commit.directory_guard.clone();
@@ -954,6 +997,14 @@ impl LocalDisk {
};
let version_id = fi.version_id.unwrap_or_default();
let (old_current_source_checked, old_current_source) = transitioned_old_current_source(
has_dst_buf.is_some(),
dst_meta_unparsable,
&xlmeta,
&dst_volume_for_source,
&dst_path_for_source,
version_id,
);
let old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
let old_version_exists = xlmeta.find_version(Some(version_id)).is_ok();
let rollback_data_dir = old_data_dir.or_else(|| {
@@ -1003,6 +1054,8 @@ impl LocalDisk {
old_data_dir,
version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
staged_rollback_path,
has_dst_buf.is_none(),
prepared_metadata_source,
@@ -1022,6 +1075,8 @@ impl LocalDisk {
cleanup_data_dir,
version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
mut local_rollback_path,
destination_was_absent,
prepared_metadata_source,
@@ -1147,10 +1202,11 @@ impl LocalDisk {
&& let Some(dst_parent) = dst_file_path.parent()
{
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dst_dir_group_commit_or_namespace_file_sync_limit(
if let Err(err) = os::fsync_commit_directory(
dst_parent,
&commit.directory_guard,
commit.mutation_lease.clone(),
admission,
Some(admission),
)
.await
{
@@ -1262,6 +1318,8 @@ impl LocalDisk {
cleanup_data_dir,
sign: version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
})
}
}
File diff suppressed because it is too large Load Diff
+674
View File
@@ -14,6 +14,7 @@
// #730: disk abstractions still carry staged health and direct-I/O migration paths.
pub(crate) mod cleanup_runtime;
pub mod disk_store;
pub mod endpoint;
pub mod error;
@@ -22,9 +23,15 @@ pub mod error_reduce;
pub mod format;
pub mod fs;
pub mod health_state;
mod io_admission;
pub mod local;
pub mod os;
#[cfg(test)]
pub(crate) use io_admission::DiskIoClass;
pub use io_admission::with_background_disk_io;
pub(crate) use io_admission::{acquire_context_disk_io_permit, current_disk_io_context, with_optional_disk_io_context};
pub const RUSTFS_META_BUCKET: &str = ".rustfs.sys";
pub const MIGRATING_META_BUCKET: &str = ".minio.sys";
pub const RUSTFS_META_MULTIPART_BUCKET: &str = ".rustfs.sys/multipart";
@@ -224,6 +231,105 @@ pub enum Disk {
}
impl Disk {
pub(crate) async fn delete_versions_with_namespace_owner(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<Error>> {
match self {
Self::Local(disk) => {
disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
.await
}
Self::Remote(disk) => {
let _owner = namespace_owner;
disk.delete_versions(volume, versions, opts).await
}
}
}
pub(crate) async fn delete_versions_recoverable_with_namespace_owner(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
mut opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<Error>> {
if let Err(error) = validate_delete_recovery_options(&opts) {
return vec![Some(error); versions.len()];
}
if versions.iter().any(|version| version.name.starts_with('/')) {
return vec![Some(DiskError::InvalidPath); versions.len()];
}
opts.recoverable_delete = true;
match self {
Self::Local(disk) => {
disk.delete_versions_with_namespace_owner(volume, versions, opts, namespace_owner)
.await
}
Self::Remote(disk) => {
let _owner = namespace_owner;
disk.delete_versions_recoverable(volume, versions, opts).await
}
}
}
pub(crate) async fn settle_delete_transaction_with_namespace_owner(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<DeleteRecoveryInspection> {
match self {
Self::Local(disk) => {
disk.settle_delete_transaction_with_namespace_owner(
volume,
object,
transaction,
expected_fingerprint,
action,
namespace_owner,
)
.await
}
Self::Remote(disk) => {
let _owner = namespace_owner;
disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
}
}
}
pub(crate) async fn delete_version_recoverable_with_namespace_owner(
&self,
volume: &str,
path: &str,
fi: FileInfo,
force_del_marker: bool,
mut opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
validate_delete_recovery_options(&opts)?;
if path.starts_with('/') {
return Err(DiskError::InvalidPath);
}
opts.recoverable_delete = true;
match self {
Self::Local(disk) => {
disk.delete_version_with_namespace_owner(volume, path, fi, force_del_marker, opts, namespace_owner)
.await
}
Self::Remote(disk) => {
disk.delete_version_recoverable(volume, path, fi, force_del_marker, opts)
.await
}
}
}
pub(crate) async fn set_disk_id_state(&self, id: Option<Uuid>) -> Result<()> {
match self {
Disk::Local(local_disk) => {
@@ -251,6 +357,70 @@ impl Disk {
#[async_trait::async_trait]
impl DiskAPI for Disk {
async fn delete_version_recoverable(
&self,
volume: &str,
object: &str,
file_info: FileInfo,
force_del_marker: bool,
opts: DeleteOptions,
) -> Result<()> {
self.delete_version_recoverable_with_namespace_owner(volume, object, file_info, force_del_marker, opts, None)
.await
}
async fn delete_versions_recoverable(
&self,
volume: &str,
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
match self {
Self::Local(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
Self::Remote(disk) => disk.delete_versions_recoverable(volume, versions, opts).await,
}
}
async fn discover_delete_transactions(
&self,
volume: &str,
object: Option<&str>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
match self {
Self::Local(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
Self::Remote(disk) => disk.discover_delete_transactions(volume, object, cursor, limit).await,
}
}
async fn inspect_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
match self {
Self::Local(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
Self::Remote(disk) => disk.inspect_delete_transaction(volume, object, transaction).await,
}
}
async fn settle_delete_transaction(
&self,
volume: &str,
object: &str,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
match self {
Self::Local(disk) => {
disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
}
Self::Remote(disk) => {
disk.settle_delete_transaction(volume, object, transaction, expected_fingerprint, action)
.await
}
}
}
fn to_string(&self) -> String {
match self {
Disk::Local(local_disk) => local_disk.to_string(),
@@ -1240,6 +1410,51 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
Err(DiskError::MethodNotAllowed)
}
async fn delete(&self, volume: &str, path: &str, opt: DeleteOptions) -> Result<()>;
async fn discover_delete_transactions(
&self,
_volume: &str,
_object: Option<&str>,
_cursor: Option<DeleteRecoveryCursor>,
_limit: u32,
) -> Result<DeleteRecoveryPage> {
Err(DiskError::MethodNotAllowed)
}
async fn delete_version_recoverable(
&self,
_volume: &str,
_object: &str,
_file_info: FileInfo,
_force_del_marker: bool,
_opts: DeleteOptions,
) -> Result<()> {
Err(DiskError::MethodNotAllowed)
}
async fn delete_versions_recoverable(
&self,
_volume: &str,
versions: Vec<FileInfoVersions>,
_opts: DeleteOptions,
) -> Vec<Option<DiskError>> {
vec![Some(DiskError::MethodNotAllowed); versions.len()]
}
async fn inspect_delete_transaction(
&self,
_volume: &str,
_object: &str,
_transaction: Uuid,
) -> Result<DeleteRecoveryInspection> {
Err(DiskError::MethodNotAllowed)
}
async fn settle_delete_transaction(
&self,
_volume: &str,
_object: &str,
_transaction: Uuid,
_expected_fingerprint: DeleteRecoveryDigest,
_action: DeleteRecoveryAction,
) -> Result<DeleteRecoveryInspection> {
Err(DiskError::MethodNotAllowed)
}
// VerifyFile
async fn verify_file(&self, volume: &str, path: &str, fi: &FileInfo) -> Result<CheckPartsResp>;
// CheckParts
@@ -1562,6 +1777,14 @@ pub struct RenameDataResp {
/// "cannot vote", never as `Absent`.
#[serde(default)]
pub old_current_size: Option<OldCurrentSize>,
/// Whether this disk decoded enough of the overwritten current version to
/// decide if a transitioned source has to be returned.
#[serde(default)]
pub old_current_source_checked: bool,
/// Transitioned old-current source used by PUT free-version cleanup receipt.
/// Empty when the old current is not transitioned or no source was known.
#[serde(default)]
pub old_current_source: Option<FileInfo>,
}
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
@@ -1576,6 +1799,269 @@ pub struct DeleteOptions {
/// Remote calls carrying it must use DeleteRetiredMarker, never DeleteVersion.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub expected_delete_marker: Option<rustfs_filemeta::MetaDeleteMarker>,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delete_recovery_bucket_incarnation: Option<Uuid>,
#[serde(default)]
pub recoverable_delete: bool,
}
pub const DELETE_RECOVERY_PROTOCOL_VERSION: u32 = 1;
pub(crate) fn get_delete_recovery_timeout() -> Result<std::time::Duration> {
let value = std::env::var("RUSTFS_DELETE_RECOVERY_TIMEOUT_SECS");
match value {
Ok(value) => parse_delete_recovery_timeout(Some(&value)),
Err(std::env::VarError::NotPresent) => parse_delete_recovery_timeout(None),
Err(std::env::VarError::NotUnicode(_)) => Err(DiskError::other("invalid delete recovery timeout configuration")),
}
}
fn parse_delete_recovery_timeout(value: Option<&str>) -> Result<std::time::Duration> {
let seconds = match value {
None => 900,
Some(value) => value
.parse::<u64>()
.ok()
.filter(|seconds| (30..=86400).contains(seconds))
.ok_or_else(|| DiskError::other("delete recovery timeout must be an integer between 30 and 86400 seconds"))?,
};
Ok(std::time::Duration::from_secs(seconds))
}
pub(crate) fn validate_delete_recovery_options(opts: &DeleteOptions) -> Result<()> {
if opts.old_data_dir.is_none_or(|id| id.is_nil())
|| opts.undo_write
|| opts.undo_delete
|| opts.recursive
|| opts.immediate
|| opts.expected_delete_marker.is_some()
|| opts.delete_recovery_bucket_incarnation.is_some_and(|id| id.is_nil())
{
return Err(DiskError::FileCorrupt);
}
Ok(())
}
pub type DeleteRecoveryDigest = [u8; 32];
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteTransactionId {
pub object: String,
pub transaction: Uuid,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryCursor {
pub token: Uuid,
}
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryPage {
pub transactions: Vec<DeleteTransactionId>,
pub next: Option<DeleteRecoveryCursor>,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryJournal {
pub protocol_version: u32,
pub volume: String,
pub object: String,
pub transaction: Uuid,
pub bucket_incarnation: Option<Uuid>,
pub original_digest: Option<DeleteRecoveryDigest>,
pub target_digest: Option<DeleteRecoveryDigest>,
pub data_dirs: Vec<Uuid>,
pub deleted_versions: Vec<Uuid>,
pub requests: Vec<FileInfo>,
pub original_versions: Vec<DeleteRecoveryVersionIdentity>,
#[serde(default)]
pub predecessors: Vec<Uuid>,
}
impl DeleteRecoveryJournal {
/// Checks source generations only; the coordinator must supply a certified
/// decision and the receiver must retain its namespace/incarnation fences.
pub(crate) fn matches_adoption_source(&self, current: &[DeleteRecoveryVersionIdentity]) -> bool {
if self.requests.is_empty() {
return false;
}
let mut originals = std::collections::HashMap::with_capacity(self.original_versions.len());
for version in &self.original_versions {
originals.entry(version.version_id.unwrap_or_default()).or_insert(version);
}
let mut locals = std::collections::HashMap::with_capacity(current.len());
for version in current {
locals.entry(version.version_id.unwrap_or_default()).or_insert(version);
}
self.requests.iter().all(|request| {
if request.deleted && request.mark_deleted {
return current.first() == self.original_versions.first();
}
let id = request.version_id.unwrap_or_default();
let Some(original) = originals.get(&id) else {
return false;
};
match locals.get(&id) {
Some(local) => local == original,
None => !id.is_nil(),
}
})
}
/// Matches immutable local preparation intent against an independently
/// inspected same-transaction commit receipt. This match is not authority
/// by itself; the coordinator must supply the certified receipt.
pub(crate) fn matches_prepared_authority(&self, authority: &Self) -> bool {
if self.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| authority.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| self.transaction.is_nil()
|| self.transaction != authority.transaction
|| self.volume != authority.volume
|| self.object != authority.object
|| self.bucket_incarnation != authority.bucket_incarnation
|| self.bucket_incarnation.is_some_and(|id| id.is_nil())
|| self.requests.is_empty()
|| self.requests != authority.requests
|| self.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
!= authority.deleted_versions.iter().collect::<std::collections::HashSet<_>>()
{
return false;
}
let sources: std::collections::HashMap<_, _> = self
.original_versions
.iter()
.map(|version| (version.version_id.unwrap_or_default(), version))
.collect();
let certified: std::collections::HashMap<_, _> = authority
.original_versions
.iter()
.map(|version| (version.version_id.unwrap_or_default(), version))
.collect();
if sources.len() != self.original_versions.len() || certified.len() != authority.original_versions.len() {
return false;
}
self.requests.iter().all(|request| {
let (local, remote) = if request.deleted && request.mark_deleted {
(self.original_versions.first(), authority.original_versions.first())
} else {
let id = request.version_id.unwrap_or_default();
(sources.get(&id).copied(), certified.get(&id).copied())
};
match (local, remote) {
(Some(local), Some(remote)) => local == remote,
(None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
_ => false,
}
})
}
}
#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryVersionIdentity {
pub version_id: Option<Uuid>,
pub mod_time: Option<OffsetDateTime>,
pub data_dir: Option<Uuid>,
pub deleted: bool,
pub metadata_digest: DeleteRecoveryDigest,
}
impl DeleteRecoveryVersionIdentity {
pub(crate) fn from_file_info(info: FileInfo) -> error::Result<Self> {
local::delete_recovery::generation(info)
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum DeleteRecoveryState {
Absent,
NewPrepared,
QuorumCommitted,
CommitFinished,
LegacyAmbiguous,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct DeleteRecoveryInspection {
pub protocol_version: u32,
pub id: DeleteTransactionId,
pub state: DeleteRecoveryState,
pub journal: Option<DeleteRecoveryJournal>,
pub current_digest: Option<DeleteRecoveryDigest>,
pub current_versions: Vec<DeleteRecoveryVersionIdentity>,
pub backup_digest: Option<DeleteRecoveryDigest>,
pub rollback_data_complete: bool,
pub commit_data_safe: bool,
pub explicit_delete_safe: bool,
pub fingerprint: DeleteRecoveryDigest,
}
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum DeleteRecoveryAction {
PublishCommit,
FinishCommit,
FinalizeCommit,
Rollback,
DiscardLegacy,
AdoptCommit { authority: Box<DeleteRecoveryJournal> },
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub enum DeleteRecoveryRpcOperation {
Discover {
object: Option<String>,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
},
Inspect {
object: String,
transaction: Uuid,
},
Settle {
object: String,
transaction: Uuid,
expected_fingerprint: DeleteRecoveryDigest,
action: DeleteRecoveryAction,
},
ApplyVersion {
object: String,
file_info: Box<FileInfo>,
force_del_marker: bool,
opts: DeleteOptions,
},
ApplyVersions {
versions: Vec<FileInfoVersions>,
opts: DeleteOptions,
},
}
impl DeleteRecoveryRpcOperation {
/// Full metadata application and adoption share the existing mutation envelope.
pub fn max_wire_payload_bytes(&self) -> usize {
match self {
Self::ApplyVersion { .. }
| Self::ApplyVersions { .. }
| Self::Settle {
action: DeleteRecoveryAction::AdoptCommit { .. },
..
} => rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES,
_ => rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES,
}
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub enum DeleteRecoveryRpcReply {
Discovered(DeleteRecoveryPage),
Inspected(DeleteRecoveryInspection),
Settled(DeleteRecoveryInspection),
AppliedVersion,
AppliedVersions(Vec<Option<u32>>),
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -1866,6 +2352,8 @@ mod tests {
undo_delete: false,
old_data_dir: Some(Uuid::new_v4()),
expected_delete_marker: None,
delete_recovery_bucket_incarnation: None,
recoverable_delete: false,
};
assert!(opts.recursive);
@@ -2038,6 +2526,8 @@ mod tests {
cleanup_data_dir: Some(uuid),
sign: Some(signature.clone()),
old_current_size: Some(OldCurrentSize::Present(42)),
old_current_source_checked: false,
old_current_source: None,
};
assert_eq!(resp.old_data_dir, Some(uuid));
@@ -2058,6 +2548,8 @@ mod tests {
cleanup_data_dir: Some(Uuid::new_v4()),
sign: Some(vec![0x01, 0x02, 0x03]),
old_current_size,
old_current_source_checked: false,
old_current_source: None,
};
let encoded = rmp_serde::encode::to_vec_named(&resp).expect("named msgpack should encode");
@@ -2095,6 +2587,8 @@ mod tests {
assert_eq!(decoded.cleanup_data_dir, None);
assert_eq!(decoded.sign, legacy.sign);
assert_eq!(decoded.old_current_size, None);
assert!(!decoded.old_current_source_checked);
assert!(decoded.old_current_source.is_none());
}
/// Test constants
@@ -2290,3 +2784,183 @@ mod tests {
}
}
}
#[cfg(test)]
mod delete_recovery_timeout_tests {
use super::*;
#[test]
fn recovery_timeout_defaults_and_validates_range() {
assert_eq!(
parse_delete_recovery_timeout(None).expect("default timeout"),
std::time::Duration::from_secs(900)
);
for seconds in ["30", "900", "86400"] {
assert!(parse_delete_recovery_timeout(Some(seconds)).is_ok());
}
for value in ["", "invalid", "0", "2", "29", "86401", "-1", "18446744073709551616"] {
assert!(parse_delete_recovery_timeout(Some(value)).is_err());
}
}
}
#[cfg(test)]
mod delete_recovery_wire_limit_tests {
use super::*;
#[test]
fn boxed_apply_version_preserves_existing_wire_payload() {
#[derive(Serialize)]
enum PriorOperation<'a> {
ApplyVersion {
object: &'a str,
file_info: &'a FileInfo,
force_del_marker: bool,
opts: &'a DeleteOptions,
},
}
let file_info = FileInfo {
name: "object".into(),
version_id: Some(Uuid::new_v4()),
metadata: std::collections::HashMap::from([("metadata".into(), "value".into())]),
..Default::default()
};
let opts = DeleteOptions {
old_data_dir: Some(Uuid::new_v4()),
recoverable_delete: true,
delete_recovery_bucket_incarnation: Some(Uuid::new_v4()),
..Default::default()
};
let prior = PriorOperation::ApplyVersion {
object: "object",
file_info: &file_info,
force_del_marker: true,
opts: &opts,
};
let encoded = rmp_serde::to_vec_named(&prior).expect("encode the previous unboxed request shape");
let operation = DeleteRecoveryRpcOperation::ApplyVersion {
object: "object".into(),
file_info: Box::new(file_info.clone()),
force_del_marker: true,
opts: opts.clone(),
};
assert_eq!(rmp_serde::to_vec_named(&operation).expect("encode boxed request"), encoded);
assert_eq!(
serde_json::to_value(&operation).expect("boxed request JSON"),
serde_json::to_value(prior).expect("previous request JSON")
);
let decoded: DeleteRecoveryRpcOperation =
rustfs_protos::decode_delete_recovery_payload(&encoded).expect("decode the previous payload");
assert_eq!(rmp_serde::to_vec_named(&decoded).expect("re-encode decoded prior request"), encoded);
}
#[test]
fn adoption_and_inspection_reply_support_large_version_journals() {
let transaction = Uuid::new_v4();
let request = FileInfo {
metadata: std::collections::HashMap::from([("metadata".into(), "x".repeat(2048))]),
..Default::default()
};
let journal = DeleteRecoveryJournal {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
volume: "bucket".into(),
object: "object".into(),
transaction,
bucket_incarnation: None,
original_digest: None,
target_digest: None,
data_dirs: Vec::new(),
deleted_versions: Vec::new(),
requests: vec![request; 1000],
original_versions: Vec::new(),
predecessors: Vec::new(),
};
let operation = DeleteRecoveryRpcOperation::Settle {
object: "object".into(),
transaction,
expected_fingerprint: [0; 32],
action: DeleteRecoveryAction::AdoptCommit {
authority: Box::new(journal.clone()),
},
};
let encoded = rmp_serde::to_vec_named(&operation).expect("encode large adoption");
assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
assert!(encoded.len() < operation.max_wire_payload_bytes());
let decoded: DeleteRecoveryRpcOperation =
rustfs_protos::decode_delete_recovery_payload(&encoded).expect("large adoption can decode");
assert_eq!(decoded.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_PAYLOAD_BYTES);
let reply = DeleteRecoveryRpcReply::Inspected(DeleteRecoveryInspection {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
id: DeleteTransactionId {
object: "object".into(),
transaction,
},
state: DeleteRecoveryState::QuorumCommitted,
journal: Some(journal),
current_digest: None,
current_versions: Vec::new(),
backup_digest: None,
rollback_data_complete: false,
commit_data_safe: true,
explicit_delete_safe: false,
fingerprint: [0; 32],
});
let encoded = rmp_serde::to_vec_named(&reply).expect("encode large inspection");
assert!(encoded.len() > rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
assert!(rustfs_protos::decode_delete_recovery_payload::<DeleteRecoveryRpcReply>(&encoded).is_ok());
let inspect = DeleteRecoveryRpcOperation::Inspect {
object: "object".into(),
transaction,
};
assert_eq!(inspect.max_wire_payload_bytes(), rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
}
}
#[cfg(test)]
mod delete_recovery_adoption_tests {
use super::*;
#[test]
fn source_matching_preserves_non_null_absence_and_rejects_foreign_or_missing_null() {
let id = Uuid::new_v4();
let source = DeleteRecoveryVersionIdentity {
version_id: Some(id),
mod_time: None,
data_dir: None,
deleted: false,
metadata_digest: [1; 32],
};
let mut journal = DeleteRecoveryJournal {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
volume: "bucket".into(),
object: "object".into(),
transaction: Uuid::new_v4(),
bucket_incarnation: None,
original_digest: None,
target_digest: None,
data_dirs: Vec::new(),
deleted_versions: vec![id],
requests: vec![FileInfo {
version_id: Some(id),
..Default::default()
}],
original_versions: vec![source.clone()],
predecessors: Vec::new(),
};
assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
let mut foreign = source.clone();
foreign.metadata_digest = [2; 32];
assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
journal.original_versions.push(foreign.clone());
assert!(
journal.matches_adoption_source(&[source.clone(), foreign.clone()]),
"retain original first-match semantics"
);
assert!(!journal.matches_adoption_source(&[foreign, source]));
journal.requests[0].version_id = None;
journal.original_versions[0].version_id = None;
assert!(!journal.matches_adoption_source(&[]), "null absence does not establish its generation");
assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
journal.requests.clear();
assert!(!journal.matches_adoption_source(&journal.original_versions));
}
}
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More