Compare commits

...

94 Commits

Author SHA1 Message Date
overtrue 3a10879774 chore(protocols): narrow the SessionDiag blanket to its one unread field
The last item-level bare allow of backlog#1823 step 10. `SessionDiag` itself is live — `sftp/server.rs` constructs one per accepted connection and `wedge_watchdog` reads `session_id`, `peer` and `last_activity_ms` off it — so the struct-level blanket was covering exactly one field: `accepted_at`, which is written at accept time and never read back. The allow moves onto that field with a reason.

The three remaining `#![allow(dead_code)]` in this crate (`sftp/test_support.rs`, `common/dummy_storage.rs`) are module-root blankets in test-support files, which belong to steps 1-5 rather than step 10.

Refs backlog#1823
2026-08-19 17:04:01 +08:00
houseme b97fb02180 docs(operations): land the heal/scanner MinIO audit baseline with closure results (backlog#1862) (#6258)
docs(operations): land the heal/scanner MinIO audit baseline with closure results

Move the comprehensive heal/scanner vs MinIO analysis (2026-08-16) into
docs/operations/ so it finally enters the tree — the docs/ root is
ignored by the gitignore whitelist, which is why the baseline the audit
issue referenced as "to be merged with a PR" never landed. Append §9
closure results: all 14 backlog sub-issues (#1865-#1878) closed with the
per-item PR map, two further misjudgment corrections (HS-17 was already
implemented; HS-14's MinIO idle semantics drifted upstream), HS-12/HS-18
audit conclusions, and the registered follow-ups.

Backlog issue: rustfs/backlog#1862

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 05:51:23 +00:00
Zhengchao An f7073d0191 refactor(admin): route plugin handler auth through authorize_admin_request (#6247) 2026-08-19 05:43:39 +00:00
houseme d404e1bb8a refactor(heal,scanner): clean up dead heal/scanner code, flags, and metrics (HS-09/10/19/20) (#6256) 2026-08-19 13:11:46 +08:00
Zhengchao An ceb6f779fb chore(rustfs): adjudicate 37 bare dead_code allows in the four densest files (#6254) 2026-08-19 13:11:32 +08:00
Zhengchao An e4eae22a70 test(crypto): replace the one-file key scan with a repo-wide guard (#6255) 2026-08-19 13:10:53 +08:00
houseme d030719dbc docs(scanner): record heal/scanner MinIO parity decisions (backlog#1878 HS-14/16/18) (#6250)
* refactor(scanner): drop the always-None single-disk default cycle hook

single_disk_default_cycle_secs returned None for every maintenance
feature combination, so the single-disk startup path already resolved
its default cycle from the speed preset (60s at 'default'). Remove the
never-wired hook and its pin tests, keep the explicit reset, and record
the decision: no special single-disk cycle override without measured
cold-start ILM latency evidence; clean-idle backoff already stretches
idle cadence (backlog#1878 HS-16).

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(operations): add heal/scanner MinIO parity decision notes

Document the HS-14/16/18 decision batch from backlog#1878: the scanner
idle throttling semantics matrix (RUSTFS_SCANNER_IDLE_MODE x speed
preset x foreground read backoff) side by side with MinIO's current
static idle_speed switch as verified against upstream master, the
migration warnings for env names and value vocabularies, the bitrot
cycle default divergence (30d vs off), the stale-multipart / tmp / trash
three-stage cleanup comparison with the crash-residue window grading,
and the single-disk default cycle decision.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 05:01:39 +00:00
Zhengchao An 1741f79d7d test(audit,heal): assert three more smoke tests (#6248)
`audit_runtime_facade_stops_empty_replay_workers` called the stop path and checked nothing, the same shape as the notify facade test in the previous commit. It now asserts the worker manager is empty afterwards and that a second call — which shutdown paths make — stays harmless.

The two heal timestamp tests bound their fields to `_`. Both timestamps come from `SystemTime::now().duration_since(UNIX_EPOCH).unwrap_or_default()`, so a pre-epoch clock yields 0; binding to `_` could not tell that apart from a real reading, which is precisely what the tests said they were guarding. They now require the value to be past 2020-01-01, and `last_update` not to predate `start_time`.

`test_config_parsing_with_multiple_instances` is left as it was — see the issue comment. Asserting on it turned up something bigger than a missing assertion.

Refs backlog#1836
2026-08-19 12:20:54 +08:00
唐小鸭 b8686b471a fix(site-replication): clear pending_remove on join, surface failures (#6244) 2026-08-19 03:41:05 +00:00
Zhengchao An 19cbba7ec8 refactor(app): resolve the bucket default SSE in one place (#6236) 2026-08-19 11:02:18 +08:00
cxymds 13a9d19505 fix(admin): report refreshed lock lease TTLs (#6230) 2026-08-19 11:02:07 +08:00
Zhengchao An aa6b9001f1 test(e2e): drop 79 no-op serial markers from six more suites (#6239) 2026-08-19 11:01:38 +08:00
Zhengchao An 728efcec89 test(ecstore): assert the error conversions, and stop the census over-reporting (#6241) 2026-08-19 11:01:20 +08:00
Zhengchao An 7b5389d2f9 test(io-schedule): drop 24 no-op serial markers from the IO scheduler unit tests (#6243) 2026-08-19 11:00:20 +08:00
Zhengchao An fa49f0ee4f test(crypto): replace the one-file key scan with a repo-wide guard (#6246) 2026-08-19 10:59:54 +08:00
Zhengchao An 09fe561443 refactor(data-usage): own SizeSummary once, with the scanner's semantics (#6237)
`SizeSummary` and `ReplTargetSizeSummary` existed in both `rustfs-data-usage` and `rustfs-scanner`, and the two copies had drifted three ways: four size fields were `usize` in one and `i64` in the other, only the scanner's carried `tier_stats`, and — the difference that matters — the scanner's `add` saturated while the data-usage copy used plain `+=`, which panics on overflow in a debug build and wraps in a release one.

The data-usage copy is now the only definition and takes the scanner's shape and semantics, since that is the side a test already pinned (`MAX + 1 == MAX`). An equivalent saturation test now guards it in its new home. The scanner re-exports both types alongside the ones it already re-exported.

`DataUsageEntry::add_sizes` and `BucketUsageInfo::add_size_summary` are removed. Both took a `SizeSummary` and had no callers anywhere — they were the duplicate fold paths, and `apply_scanner_size_summary` is now the only one.

`actions_accounting` stays in the scanner as the `ScannerSizeSummaryExt` extension trait: it needs `ObjectInfo`, which sits above `rustfs-data-usage`, and an inherent impl on a foreign type is not allowed. The three call sites are unchanged.

Refs backlog#1828
2026-08-19 02:34:30 +00:00
Zhengchao An e3d7892404 test(io-metrics): assert what the remaining smoke tests only called (#6238)
Eight tests in this crate called a recorder and asserted nothing. Five of them were worse than that: every `record_*` in `list_objects_metrics` returns early unless `get_stage_metrics_enabled()` is true, and that flag defaults to false, so those tests only ever exercised the early return — never the code their names describe.

They now run against a local `DebuggingRecorder` with the flag on, and each asserts the boundary it is named for: an empty page reports the scan count as its amplification instead of dividing by zero, a zero read quorum is recorded rather than skipped, index serving divides verification attempts by returned objects, and the `-1` whole-directory sentinel reaches the limit histogram unclamped.

`msgpack_json_fallback_counter_records_without_panicking` has no in-struct total to check, so it now asserts the emission: two direction/message pairs must land in two separate series, which a dropped label would collapse into one.

The two process-sampler tests discarded their snapshots. They now assert what cannot differ between callers — a process has one start time and one descriptor limit regardless of which entry point or which sampler observed it, and the status enum must match its numeric projection.

This clears io-metrics from the census (`scripts/find_assertless_tests.py`), taking the tree from 61 candidates to 53.

Refs backlog#1836
2026-08-19 10:32:09 +08:00
hector 7f2c0f1dfb fix(package): write release checksum entries with GitHub asset names (#6234) 2026-08-19 10:26:41 +08:00
Zhengchao An bde6736213 fix(ecstore): classify a missing data-usage cache by the error that arrives (#6233)
`is_data_usage_cache_absent` matched `FileNotFound | VolumeNotFound`, but `SetDisks::get_object_reader` runs its failures through `to_object_err`, which rewrites those to `ObjectNotFound` and `BucketNotFound` before they reach the caller. The classifier therefore never matched in production: a cache object that simply does not exist was treated as a transient failure, retried five times with backoff, and then reported as an error instead of an empty cache. Admin server-info resolves one cache per erasure set, so that is roughly 1.5s of pointless backoff per set on any cluster whose scanner has not written a cache yet.

The same rewrite is why the pre-existing `FileNotFound | VolumeNotFound` arm in the old loop never fired either, which left the legacy-key fallback beside it unreachable — it only ever returned an empty cache through the catch-all break.

The classifier now covers the rewritten variants as well as the raw pair, the test store reports absence the way `to_object_err` does, and a new test pins which variants actually arrive.

Refs backlog#1828
2026-08-19 10:25:56 +08:00
Zhengchao An cd9c96a03c test(e2e): fold eleven post-object accept cases into one table-driven test (#6207) 2026-08-19 01:32:37 +00:00
houseme 4b676ef1ed perf(server): skip output layer work on common GET paths (#6232)
Avoid fixed response-layer work on the ordinary GET path by bypassing CORS request cloning when no Origin header is present and by only splitting/rebuilding compatibility responses when their target conditions match.

Add service-level regression tests for CORS, S3 error, Iceberg REST, ObjectAttributes, and bodyless-status compatibility paths.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 01:32:01 +00:00
houseme c7c5a8df6a test(heal): cover privileged mount readiness (#6231)
Add Linux-only ignored replacement readiness tests for independent mount admission and same-device sibling rejection.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 01:27:55 +00:00
Zhengchao An 612a5927b6 fix(ecstore): make the data-usage cache load actually retry (#6229)
* fix(ecstore): make the data-usage cache load actually retry

`load_data_usage_cache` wrapped its read in `while retries < 5`, but every arm of the match inside broke out of the loop, so `retries` was never incremented and the random sleep below it was unreachable: the loop always ran exactly once. The fallback arm compounded this by re-matching the *outer* error after the legacy-key read failed, which meant its second arm could not be reached either.

The read now goes through `rustfs_utils::retry::retry_with_backoff`. A key that is absent under both the prefixed and the legacy name still yields an empty cache without retrying, since retrying a definitive absence cannot turn it into a hit. A transient failure is retried with capped, jittered backoff and surfaces as an error once the attempts are exhausted, instead of being reported as an empty cache — the sole caller already maps `Err` to `usage_error = DATA_USAGE_UNAVAILABLE`, so a read failure now says "unavailable" rather than "zero usage".

`load_data_usage_cache` is generic over `ObjectIO` rather than taking `&SetDisks`, which is what makes the retry and fallback ordering testable at all; being untestable is why the inert loop survived. The call site passes `as_ref()` instead of cloning the `Arc` it immediately borrowed.

Refs backlog#1828

* fix(ecstore): route the load bound through the storage-api contracts

The generic bound named `rustfs_storage_api::ObjectIO` directly, which the architecture guard rejects: ecstore modules must reach storage-api symbols through `crates/ecstore/src/storage_api_contracts`. The bound is now the crate's own `EcstoreObjectIO` alias, which pins the same associated types in one place.

That alias is `pub(crate)`, so `load_data_usage_cache` becomes `pub(crate)` too rather than exposing a crate-private bound on a public signature. Nothing outside ecstore called it — its only caller is `diagnostics/admin_server_info.rs`, and it was never re-exported from the crate root.

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-19 00:50:59 +00:00
Zhengchao An 5355210070 fix(sse): read objects that MinIO encrypted (#6191)
* fix(sse): read objects that MinIO encrypted

RustFS could not read a single MinIO-encrypted object. Two independent blockers, and backlog#1638 could only argue them statically because the fixtures the interop tests consume are generated, not checked in — so those tests had never once run. With the fixture lab working, both are now measured, fixed and covered.

The detection gate required `x-amz-server-side-encryption` to be present. MinIO never persists it: `crypto.S3.CreateMetadata` writes only the `X-Minio-Internal-*` family and the public header is synthesized onto the response by `DecryptObjectInfo`. Every MinIO object therefore fell out of the managed path and failed with "encrypted object metadata is incomplete". The scheme is now inferred from which sealed-key slot is present, which is self-consistent by construction: the slot decides both which header the unseal reads and which domain string the sealing key is derived under, so an inference that disagreed with the slot could not silently derive a wrong key. Inferring from the KMS key id would NOT be safe — MinIO writes `-S3-Kms-Key-Id` on SSE-S3 objects too, which the fixtures show and a mutation test pins.

Past the gate, the data key itself could not be unwrapped. Its wire format is `sealed_bytes || iv[16] || nonce[12]` — the randomness trails the ciphertext rather than leading it — with a per-ciphertext sealing key of `HMAC-SHA256(master, iv)` and the encryption context bound as associated data (`internal/kms/secret-key.go`). Note this is not the `{"aead":...}` JSON that backlog#1638's analysis described: current MinIO writes the raw layout and treats JSON only as a legacy encoding, normalizing it into the same byte order. Both are decoded here, in a decoder of their own — `LocalSseDekEnvelope`'s `deny_unknown_fields` is untouched, since loosening it to admit MinIO's shape would also admit malformed RustFS envelopes that backlog#1567 requires to keep failing closed.

Routing between the two decoders cannot key on metadata: RustFS's own writer fills MinIO's slots while storing a RustFS envelope in them, so neither the slot nor the header name distinguishes writers. It keys on the data key's own shape instead, recognizing the two strict RustFS JSON shapes positively and leaving only the remainder to MinIO — so neither decoder is ever handed the other's format. Three round-trip tests caught an earlier slot-based attempt doing exactly that.

Fail-closed is preserved throughout: a scheme that cannot be established still returns None, and the read plan independently classifies the object as encrypted from its markers and refuses to serve it without material, so no path degrades into returning ciphertext as plaintext.

The interop harness also gets a provider reset. The DEK provider is cached process-wide, so a case that ran earlier kept serving its master key to every later case — which silently made the wrong-key negative test unable to fail. It fails correctly now, and the whole suite is meaningful for the first time.

Refs rustfs/backlog#1638.

* fix(sse): gate the MinIO data-key trait method behind rio-v2

The method's only call site sits in the rio-v2 branch of the managed read path, so a build without that feature carried a trait method nothing could reach — a warning under default features and, with -D warnings, a hard failure of the sftp lane. The declaration now carries the same gate its implementation and its sibling decrypt_legacy_sse_dek already had.

Verified against the lane that caught it (cargo clippy -p rustfs --features sftp --all-targets -- -D warnings, clean), plus the default build and the rio-v2 interop suite (4 passed).

Refs rustfs/backlog#1638.

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-19 00:33:52 +00:00
houseme b648dea340 fix(ecstore): group inline dst dir fsync (#6228)
Route strict inline rename_data dst-parent fsync through the default-off group-commit helper when enabled while preserving the namespace file-sync limited path by default.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 07:55:15 +08:00
Zhengchao An 40c15c769b docs(architecture): describe io-core by its surviving surface (#6227)
The Code Map entry still summarised io-core as "buffer pool, storage profiling, admission control", which predates #6201 removing eight zero-consumer modules. The crate now exposes pool, io_profile, config, backpressure, deadlock_detector, lock_optimizer, and progress, so the summary names the policy, lock, and progress helpers as well.

Refs backlog#1824
2026-08-19 07:09:00 +08:00
houseme ff9ac1013a feat(ecstore): add default-off dst dir fsync group commit (#6226)
* feat(ecstore): add dst dir fsync group commit

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): tidy dst dir fsync group open

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-19 06:24:45 +08:00
Zhengchao An 850445e957 fix(e2e): add missing serial_test import in replication_extension_test (#6224) 2026-08-19 06:24:15 +08:00
houseme 50c39fec45 perf(get): emit accept-ranges with static header (#6225)
Avoid constructing the typed Accept-Ranges string on the GetObject output path. Inject the static header after CORS wrapping so the final S3 response remains unchanged while the hot path avoids one fixed per-GET allocation/conversion.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 17:20:16 +00:00
hector c86a94a2dc fix(package): declare /etc/default/rustfs as a deb conffile (#6220)
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-18 15:56:52 +00:00
houseme 905082893f fix(e2e): import serial test attribute (#6222) 2026-08-18 23:12:39 +08:00
houseme eed0ca3612 perf(ecstore): validate local IO paths with openat2 (#6221)
Use Linux openat2 with RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS for LocalDisk I/O path validation while keeping the existing lstat walk as the public-path and unsupported-kernel fallback. Add focused regression coverage for traversal, symlink swaps, missing leaves, recreated parents, high-cardinality prefixes, final symlink leaves, and concurrent validation.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 22:40:43 +08:00
GatewayJ 91c97f3416 chore(deps): update s3s to upstream main (#6203)
* deps: update s3s to upstream main

* deps: refresh s3s upstream revision

* deps: pin s3s to latest upstream main

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 21:46:56 +08:00
唐小鸭 1d056d7605 test(e2e): pin bounded physical reads for compressed multipart range GETs (#6167)
Byte-exactness tests stay green if the compressed range seek regresses into
decoding from byte zero: the returned bytes are still correct and only the
read amplification explodes. Assert the cost side as well.

The observation reuses rustfs_io_get_object_shard_read_observed_bytes_total,
already emitted per shard read by the erasure layer, so no production code is
instrumented. The OTLP collector learns to accumulate a second counter, keyed
by its path/role/outcome labels rather than by data-point position, which is
not stable across exports.

Two failure modes the assertions guard against:

- With RUSTFS_OBS_METER_INTERVAL=1, treating one unchanged sample as settled
  measures a delta of zero, because the range read's counter has not been
  exported yet. Settling now requires several consecutive equal samples.
- An upper bound alone passes vacuously on a zero delta, so a lower bound
  turns "measured nothing" into a failure instead of a green run.

Refs rustfs/rustfs#5957, backlog#1848.
2026-08-18 21:46:16 +08:00
Zhengchao An 8315c23d49 test(kms): move the Vault KV2 doc guard into check_fips_wording.sh (#6215)
test(kms): move the Vault KV2 Transit-wrapping doc guard into check_fips_wording.sh

`test_vault_kv2_sources_do_not_claim_transit_wrapping` asserted that four
`include_str!`-pinned files never describe the Vault KV2 backend as wrapping key
material through Vault's Transit engine. The invariant is a documentation-claim
invariant with no behavioral twin by construction, and the test form was weak in
both directions: it saw only four files (the same prose in a fifth file passed
silently) and it stopped compiling — rather than reporting a violation — as soon
as one of them was renamed.

Move the four literals verbatim into `scripts/check_fips_wording.sh`, which
already guards the adjacent cryptographic over-claim class (unsupported FIPS
validation wording) and is anchored to the same policy document. The guard now
greps every file under `crates/kms` for the same four case-sensitive literals and
separately reports a moved pinned source instead of failing to build.

`check_fips_wording.sh` previously ran only in `make pre-commit` / `pre-pr`, so
wire it into the Quick Checks job of both CI workflows to keep the invariant's
failure visibility at least as strong as the deleted test's.
2026-08-18 21:46:00 +08:00
唐小鸭 1cf0f7af15 feat(replication): split oversized hot-path functions, proxy unreplicated reads, and fail SSE-C passthrough closed (#6170)
* refactor(replication): split four oversized hot-path functions into focused helpers

Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:

- resync_bucket (522 lines -> 61-line step sequence): leader lock,
  target resolution, walk/collector/worker spawning, and dispatch loop
  extracted into focused helpers; pure decision helpers (DTO builders,
  HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
  seeding, read/stat option builders, skip-path notes, target HEAD
  action resolution, and the multipart/single-put payload transport
  extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
  ledger load, per-entry replay (delete/object/metadata), and retained
  entry resolution extracted; retry bookkeeping semantics preserved
  exactly (inner continue-paths push inside helpers, outer Missed push
  stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
  IAM item type.

No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).

* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)

* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets

Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.

Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
  into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
  ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
  value is never re-proxied. Outbound client proxy calls send the marker
  as "true"; replication worker convergence HEADs send it as "false" so
  a peer's proxy layer cannot answer a convergence check by proxying
  back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
  when the marker is set, versioning is suspended, or no replication
  config; otherwise filter_target_arns -> TargetClient lookup, skipping
  targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
  the three tagging calls. Proxy calls never send the replication-check
  SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
  the target performs real decryption. Conditional (If-*) headers are
  not forwarded (MinIO parity); Range and part_number are, with
  parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
  MinIO-aligned (one total per proxied request, one failed when no
  target served it). The previous misattributed counters — replication
  worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
  removed; ReplProxyMetric now maps the tagging counters instead of
  dropping them.

e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.

Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.

Refs rustfs/backlog#1675 (P1-5)

* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).

Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
  BucketTargetSys per target ARN with a recording timestamp. Entries reset
  whenever the target is rebuilt, edited, or removed (arn_remotes_map
  lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
  an expired verdict in either direction is re-earned through the audit, so
  an Unsupported target recovers automatically after an upgrade (at most one
  wasted PUT+HEAD audit per bad target per TTL window) and a Supported
  verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
  targets never receive the PUT — the attempt fails immediately into the
  normal MRF retry channel with a "run ?replication-check to re-probe" hint.
  Unknown or expired verdicts are audited: after the PUT the worker HEADs
  the replica back through the replication-check channel (source version id
  mapped through resolve_read_api_version_id, so null-version objects audit
  correctly) and requires SSE-C evidence (the echoed customer-algorithm
  header); missing evidence records Unsupported and fails the attempt.
  Convergence HEADs are audited the same way, so a broken ciphertext replica
  from an earlier attempt can never launder itself into COMPLETED via an
  ETag match. The gate/evidence policy is pure (replication_target_boundary,
  staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
  carrying the live transport-header shape, HEAD-back for evidence, and a
  machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
  The probe verdict is synced into the runtime capability cache. Unlike
  VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
  overall — it is a capability limit, not a broken replication contract,
  and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
  transport headers, echoes SSE-C evidence); the new
  drop_unlisted_replication_headers mode models MinIO. The journal records
  whether a request carried transport headers.

Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.

Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.

Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
  A hypothetical target that preserves that one header while dropping other
  transport headers (partial-drop) would pass the audit; no known target
  behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
  to different target nodes until the rollout completes; the TTL bounds how
  long each stale verdict persists.

New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.

* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)

* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)

Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.

Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
  replicate_delete_outcome, target_delete_version_id,
  delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
  SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
  ssec_passthrough_evidence_present (param-demoted to the echoed
  customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
  filemeta-independence contract)

ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
  resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
  capability enum so existing consumer paths keep compiling

Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.

* chore(replication): retire the datatypes.rs relay early

README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.

* chore(arch): extend migration guards to the M2-moved decision contracts

The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:

- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
  helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
  replicate_delete_outcome, resync_existing_delete_replication_info,
  target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
  classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
  fns/enums (ssec_passthrough_evidence_present deliberately excluded:
  ECStore keeps a thin HeadObjectOutput adapter under that name).

Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.

Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
2026-08-18 21:45:38 +08:00
Zhengchao An 27d23b6135 test(io-metrics): assert record helper emissions; fix census heuristic (#6217)
* test(io-metrics): assert lib.rs record helper emissions

The 29 assertion-less record_* smoke tests in io-metrics/src/lib.rs called
their helpers and checked nothing; because METRICS_ENABLED defaults to
false they did not even reach the emission bodies. Replace them with six
DebuggingRecorder tests that enable the gate, pin every metric name the
helpers own, and pin the derived values, branch selection and label
mapping (rustfs/backlog#1836).

* test(tooling): anchor assertless-census delegation tokens to name segments
2026-08-18 21:21:34 +08:00
houseme 127b662f3f feat(app): add opt-in small GET body once path (#6216)
Use the merged s3s single-chunk StreamingBlob support for exact-length materialized GET bodies when RUSTFS_GET_SMALL_BODY_ONCE_ENABLE is enabled.

Keep the default path unchanged and fall back to the guarded MemoryTrackedBytesStream on length mismatch.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 11:53:06 +00:00
Zhengchao An 8a3c66e655 test(admin): replace a source-text guard with a behavior test (#6212) 2026-08-18 18:28:46 +08:00
Zhengchao An 9852e53b4c test(lifecycle,scanner): drop 47 no-op #[serial] markers (backlog#1846 T1) (#6213)
Second batch of the #[serial] sweep started in #6209. nextest is the
repository's authoritative runner and executes every test in its own
process, so serial_test's in-process mutex cannot serialize tests against
each other -- docs/testing/README.md documents this, and the mechanism
that actually serializes across the process boundary is a
.config/nextest.toml [test-groups] entry with max-threads = 1.

Unlike the first batch (e2e, process-isolated by construction), these are
in-crate unit tests that could genuinely share process state under the
`cargo test` fallback runner, where #[serial] IS still effective. Every
marker was therefore reviewed individually and removed only where the
test provably touches neither the process environment nor a process-global.

Removed (47, pure deletions, no test bodies touched):

  crates/lifecycle/src/core.rs   35
  crates/scanner/src/scanner.rs  12

The lifecycle removals are all validate_* / filter_rules_* /
has_active_rules_* / noncurrent_versions_expiration_limit_* tests that
build a local BucketLifecycleConfiguration and call a &self method
walking only that value. The scanner removals are pure duration
arithmetic (randomized_cycle_delay_for, initial_scanner_delay_for with an
explicit Some(secs), the bitrot-disabled early return of
scanner_clean_idle_max_interval) and background_heal_info_for_scan_complete
/ _for_scan_result field comparisons over locally built values.

Retained deliberately -- see the PR body for the full list and reasons:

  crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs  101 (all)
  crates/lifecycle/src/core.rs                                  44
  crates/scanner/src/scanner.rs                                 53

bucket_lifecycle_ops.rs keeps every marker: its test module caches a
process-wide `static STALE_MULTIPART_TEST_ENV: OnceLock<(Vec<PathBuf>,
Arc<ECStore>)>`, and its own reregister_env_local_disks helper documents
in-tree that sibling #[serial] tests reset and reshape the shared
local-disk registry for each other. That sharing is real, so the markers
stay.

No test was renamed, added, or deleted; no reserved migration-gate name
substring is affected; no .config/nextest.toml entry references any of
the 47 removed tests.
2026-08-18 18:28:13 +08:00
Zhengchao An a38743caf5 chore(zip): trim the unused extract and create surface (#6210)
rustfs-zip has one workspace consumer, and it uses only
CompressionFormat::{from_extension, extension, get_decoder} and
ArchiveLimits. Remove the tar/zip extract, zip create, and in-memory
compress helpers together with the types and dependencies that only
served them. Trimming public API is semver-major once the stable tag is
cut, so it costs least now.
2026-08-18 18:27:01 +08:00
houseme 382ae9529e feat(ecstore): instrument rename sync tail metrics (#6205)
Add default-off PUT stage attribution for the rename_data sync tail so strict durability probes can split queue wait, fdatasync, directory fsync, rename, per-disk wait, and quorum wait without changing commit ordering or S3-visible behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 17:20:55 +08:00
Zhengchao An 68547ed7ea test(e2e): drop 187 no-op serial markers from the three densest e2e suites (#6209)
test(e2e): drop no-op serial markers from the three densest e2e suites

serial_test's #[serial] is an in-process mutex. cargo-nextest, this repo's
authoritative runner, executes every test in its own process, so the mutex is
never contended and the attribute is a documented no-op -- see the "Serial
execution & nextest profiles" section of docs/testing/README.md and the header
of .config/nextest.toml. Cross-process serialization is provided only by a
[test-groups] entry with max-threads = 1.

Remove 187 such markers (plus 3 now-unused imports) from the three
marker-densest modules of crates/e2e_test:

  multipart_auth_test.rs             85
  replication_extension_test.rs      68
  object_lock/object_lock_test.rs    34

None of these modules is covered by any [test-groups] entry, so the markers
were carrying no isolation for any lane. Every test in all three files builds
its own server via RustFSTestEnvironment::new(), which gives a UUID temp dir
and a uniquely allocated port -- the .config/nextest.toml comment on
replication_extension_test already states this explicitly ("parallel-safe by
construction"). No test mutates process env, binds a fixed port, or touches
process-global state, so nothing here needed temp_env or a test-group instead.

Pure deletion: 190 lines removed, 0 added, no test renamed, no behaviour
changed. serial_test stays in Cargo.toml -- 338 markers across 90 other files
in the crate still use it.

Refs: backlog#1846 (T1).
2026-08-18 17:05:30 +08:00
Zhengchao An f06a9c9cba fix(deps): record heal's bytes and crc-fast in the lockfile (#6211) 2026-08-18 17:04:55 +08:00
Zhengchao An bd296eff9e chore(io-core): drop eight zero-consumer modules (#6201) 2026-08-18 08:14:05 +00:00
houseme a5800033bd feat(heal): incremental status cursors and typed overlap policy (HS-06) (#6206)
* feat(heal): incremental heal status cursors and typed overlap policy (HS-06)

Incremental results: every retained result item now carries a monotonic
sequence number. The status query accepts a client cursor (sinceSeq on
the admin wire, Option<u64> internally) and returns only newer items,
plus nextSeq (the next cursor) and minSeq (the oldest retained
sequence). A cursor that fell behind the 1024-item retention window is
flagged through the existing truncated signal together with minSeq so
the client can restart from it. Sequencing survives task completion:
the completion archive stores the seq-stamped window. None keeps the
exact legacy full-snapshot behavior, so existing clients see no change.

Typed overlap handling for admin starts: RUSTFS_HEAL_OVERLAP_POLICY
(merge default | minio_error). Under minio_error, an admin start whose
path overlaps an active or queued task rejects with typed
already-running / overlapping-paths admission reasons (surfaced through
reason_label in the admin error body, sharing the existing
OperationAborted site because the s3s footprint ratchet forbids new
s3_error! sites); an exact duplicate start rejects with
already-running instead of silently merging. Scanner/autoheal/
read-repair sources never take the rejection path.

forceStart semantics now match MinIO for admin requests: an admin
forceStart first cancels the overlapping active admin task, then
admits the replacement.

Wire: the heal-control Query command grows an optional sinceSeq
(defaulted and skipped when absent, so older peers stay compatible);
the admin handler accepts the sinceSeq query parameter; the local
channel query gains the same cursor.

Tests: seq monotonicity and incremental slicing, window slide moving
minSeq with lagging-cursor flags, overlap matrix (same/containing/
contained/disjoint x policy x source), forceStart cancel-then-admit,
and the completion-archive window handoff.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: fmt after main merge

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-08-18 16:09:30 +08:00
cxymds a4ea36b298 fix(ecstore): single-flight remote disk recovery (#6096)
* fix(ecstore): single-flight remote disk recovery

* test(ecstore): cover remote recovery review cases

* test(ecstore): exercise recovery through disk slot

* test(ecstore): match format reads exactly

* test(ecstore): cover recovery teardown races

* style(ecstore): format recovery race tests

* fix(ecstore): remove unused health snapshot helper

* fix(ecstore): group recovery monitor test state

* fix(protos): preserve production source in compatibility checks
2026-08-18 14:49:53 +08:00
cxymds 4f68f117ba fix(lock): reap expired local lease guards (#6094)
* fix(lock): reap expired local lease guards

* fix(lock): reject refresh after guard expiry

* test(lock): isolate expired refresh regression

* style(lock): format expired refresh assertion
2026-08-18 14:49:39 +08:00
cxymds 0f30a75fdb fix(ecstore): resume remote shard reads once (#6091)
* fix(ecstore): preserve CopyObject producer errors

* fix(ecstore): resume remote shard reads once

* fix(app): resume preserved relocation I/O errors

* fix(ecstore): reserve remote read recovery budget
2026-08-18 14:49:19 +08:00
Zhengchao An 355c8d2e22 fix(admin): classify missing kms config by error variant (#6196) 2026-08-18 05:40:10 +00:00
Zhengchao An 60eb139db9 refactor: import x-amz-checksum header names from the shared constants (#6193)
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-18 05:27:38 +00:00
hector 9ef059c908 ci(package): auto-trigger DEB/RPM packaging on releases and upload to GitHub release assets (#6202) 2026-08-18 05:18:13 +00:00
Zhengchao An 51497cb533 fix(ecstore): give peer REST failures op and bucket context (#6200) 2026-08-18 04:57:44 +00:00
Zhengchao An c7a29ec0a7 chore(storage): drop dead backpressure and lock-optimizer wrappers (#6195)
Neither rustfs/src/storage/backpressure.rs nor rustfs/src/storage/lock_optimizer.rs had a production caller: their only non-self references were the pub mod lines in storage/mod.rs and a cfg(test) module, so the object transfer path never applied this backpressure and never took these lock shortcuts.

The six removed tests in concurrent_fix_test.rs duplicated tests that lived inside the deleted files; the shared primitives they shadowed keep their own coverage in rustfs-io-core.
2026-08-18 12:51:44 +08:00
Zhengchao An deb0edb7cc chore: adjudicate 26 bare dead_code allows across five crates (#6187)
Remove every bare `#[allow(dead_code)]` in io-core, object-capacity, targets, rio, and scanner. Each allow was stripped first and clippy was then asked which ones the compiler actually missed, so the verdicts rest on the diagnostic rather than on inspection.

23 were inert: they sat on `pub fn`s inside `pub mod`s, where `dead_code` does not apply, or on scanner integration-test helpers that the tests in the same file do call.

The remaining 3 are in rio's private `compress_index` module and the code behind them is deleted rather than annotated. `remove_index_headers` is dead and also wrong — after skipping the 4-byte chunk header it matches against `S2_INDEX_TRAILER` where `S2_INDEX_HEADER` sits, so it returns `None` for every well-formed index; rio-v2 carries the correct equivalent that is actually in use. `restore_index_headers` is its unreachable counterpart, likewise duplicated live in rio-v2. `Index::reset` is a private method with no caller.

Refs backlog#1823
2026-08-18 12:45:42 +08:00
houseme a08de9229b feat(heal): wire MRF intents with durable repair journal (HS-01) (#6189)
* feat(common): add MRF intent channel and Mrf request source (HS-01)

Introduce the producer-facing half of the mission repair feed: a global
bounded (8192) channel carrying lightweight MrfIntent values from IO
error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and
config constants for queue/journal sizing. Delivery is strictly
non-blocking (try_send, drop-on-full) so it can sit on decode-failure
and partial-write paths without adding latency. HealRequestSource grows
a 'mrf' variant so admission accounting can attribute replayed intents.

Part of backlog#1865 (option a: wire HealEvent-style intents with a
durable retry ledger).

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(heal): add MRF queue, durable journal, and intent consumer (HS-01)

Consumer half of the mission repair feed: a bounded pending queue
(100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable
journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending
snapshot, and a consumer task that batches intents off the global
channel, translates them into prioritized heal requests (decode
failure -> Urgent ECDecode, metadata corruption -> High Metadata,
partial write -> Normal object heal), and retries full admissions with
a 5s backoff and a 3-attempt ceiling.

Durability: every journal record carries its own CRC32 and a
format/version header, so a torn tail truncates cleanly at replay; the
journal is deleted after a successful replay and when the pending set
drains (mirroring MinIO's post-replay list.bin unlink). Losing the last
500 ms flush window is acceptable: replayed duplicates merge via the
manager dedup key and read-repair remains the safety net.

Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total
{reason}, _replayed_total, _journal_bytes, _journal_fsync_total.
The consumer is wired at heal runtime bootstrap right after manager
start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off).

Tests: unit tests for the dual ceiling, record roundtrip, torn-tail
truncation, and the priority mapping; integration tests against a real
4-disk ECStore proving channel intents reach the manager queue as
Urgent/mrf-attributed requests and journal replay arms intents, drops
torn tails, and removes the file.

Part of backlog#1865 (option a).

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ecstore,scanner): deliver MRF intents from error paths (HS-01)

Wire the three production delivery points, each a single non-blocking
try_send next to the existing in-memory heal paths, which stay as the
fast path:

- read.rs decode-error branch: DecodeFailure intent beside the existing
  read-repair submit, so an Urgent ECDecode request survives restarts
  even when the Low-priority read-repair request was dropped or lost.
- add_partial: PartialWrite intent, giving partial-write recovery a
  durable Normal-priority object heal across restarts.
- scanner_folder metadata-corruption classification: MetadataCorruption
  intent beside the existing High-priority scanner heal request.

All three are on error paths only: zero cost on healthy IO.

Part of backlog#1865 (option a).

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: include mrf heal source counts

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: keep node heal status wire compatibility

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 12:43:27 +08:00
Zhengchao An abffa5cf1b chore(storage): drop dead io-schedule metrics and helpers (#6199) 2026-08-18 04:35:36 +00:00
Zhengchao An b825c54850 refactor(admin): route kms management auth through shared gate (#6194) 2026-08-18 12:21:49 +08:00
houseme de9145e87a feat(storage): add default-off PUT admission gate (#6197)
Add an experimental fixed-count foreground PutObject admission gate for #1882 Phase 0 validation. The gate is default-off, returns SlowDown before body ingest when saturated, and keeps the admission permit with the spawned store commit owner until store PUT returns.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 04:15:19 +00:00
houseme 84bd76a3ce chore(deps): refresh cargo dependencies (#6198)
Update workspace Cargo dependency requirements and lockfile after cargo update/upgrade, including rumqttc-next 0.34.0 and MQTT API compatibility adjustments.

Verification:

- cargo update --verbose

- cargo upgrade --verbose

- cargo update -p rumqttc-next --precise 0.34.0 --verbose

- cargo tree --invert rumqttc-next --locked

- cargo metadata --locked --no-deps --format-version 1

- cargo fmt --all --check

- cargo check -p rustfs-targets --all-targets --locked

- cargo test -p rustfs-targets mqtt --locked

- make pre-pr

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 04:11:53 +00:00
houseme 9a2d06b370 test(heal): lock heal vs delete/overwrite race invariants (HS-12) (#6183)
* test(heal): add concurrency invariants for heal vs delete/overwrite races (HS-12)

Audit conclusion for backlog#1874: RustFS does not need a persistent
object-level healing marker (MinIO x-minio-healing) because every path
that can touch the same (bucket, object) commit surface serializes on
the same namespace write lock, and the heal lock guard spans the whole
rename commit including the HEAL_RENAME_INCOMPLETE partial path.

Lock the conclusion in with two race regression tests:

- heal_racing_version_delete_never_resurrects_the_deleted_version:
  shard damage is injected on the doomed version so a Deep heal has real
  reconstruction work while a versioned DELETE runs concurrently; the
  deleted version must stay deleted and the survivor intact.
- heal_racing_unversioned_overwrites_preserves_the_last_commit:
  unversioned overwrites (activating the post-commit tail that deletes
  the replaced data dir without the ns lock) race a Deep heal in a
  loop; the final current version must be exactly the last commit.

Also adds docs/operations/heal-concurrency-safety-notes-zh.md with the
full intersection matrix (17 intersections), lock-coverage argument,
and the residual-window classification (commit tail races are
fail-into-retry safe; bare prefix delete has zero production callers;
admin no_lock is an explicit operator opt-in).

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: remove redundant heal etag clone

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 02:01:04 +00:00
Zhengchao An 00de43528c fix(ecstore): describe peer bucket RPC failures with no details (#6190)
heal_bucket, list_bucket, get_bucket_info and delete_bucket returned Error::other("") when a peer answered success=false without an error payload, so operators saw a bare "io error " after quorum reduction. Route all five bucket RPCs through peer_failure_without_details, which names the operation and bucket while staying identical across the peers of one operation so reduce_errs keeps grouping them into a single dominant error.
2026-08-18 01:55:29 +00:00
houseme 35a30cd614 feat(scanner): emit excess alerts as S3 notification events (HS-04) (#6176)
* feat(scanner): emit excess alerts as S3 notification events

The excess-versions / excess-version-size / excess-folders alerts were
metrics-and-logs only; consoles and external auditors had no way to hear
them (rustfs/backlog#1868, HS-04). MinIO emits s3:ObjectManyVersions /
s3:ObjectLargeVersions / s3:PrefixManyFolders for the same conditions —
RustFS carries those as EventName::Scanner* with s3:Scanner:* wire names
that already existed unpublished.

The three alert sites now also dispatch through the standard event
pipeline (send_event via the storage_api owner facade), carrying the
actual values and thresholds in req_params and UserAgent "Scanner".
Without a cooldown a single over-threshold object would re-emit on every
~60s scan cycle, so emissions are edge-held per (kind, bucket, object)
for 24h (RUSTFS_SCANNER_ALERT_COOLDOWN_SECS, 0 = every cycle), backed by
a process-global map with a 4096-key hard cap that clears rather than
grows. Metrics and structured logs stay level-triggered every cycle;
only the notification events are held back. A restart resets the
cooldown deliberately: one re-emission per still-hot key buys back
visibility after the restarts that accompany incident response.

Tests pin the edge-hold semantics (first fires, immediate re-check held,
independent keys, cooldown expiry re-fires, zero cooldown always emits,
hard bound) in one sequential test for the process-global map, and pin
the emitted wire names against EventName's canonical string forms so a
subscribed bucket notification can never silently stop matching.
docs/operations/scanner-excess-alerts.md documents the three events,
the metric-vs-event cadence difference, and the HS-15 threshold deltas
(alert_excess_folders 65538 vs MinIO 50000 is deliberate: Proxmox
Backup Server chunk layout compatibility).

Closes rustfs/backlog#1868.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(operations): split scanner excess alerts into English and Chinese pages

The page shipped Chinese-only; keep it as scanner-excess-alerts_zh.md and
add a faithful English translation at the original path, cross-linked at
the top of both.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 08:46:32 +08:00
houseme 360bceafce feat(heal): add progress and trace observability (#6179)
* feat(heal): track erasure set progress baseline

Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.

Keep usage-cache failures observational so heal execution continues without a baseline.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(heal): skip filtered erasure set versions

Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.

Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(heal): wire abandoned data-dir cleanup check

Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.

Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add heal scanner trace bus

Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.

Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): stream heal trace events from admin API

Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): emit heal trace events

Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): emit scanner trace events

Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(heal): route data usage loader through storage api

Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(heal): avoid lifecycle snapshots on ordinary heal pages

Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(heal): update bug-fix mocks for lifecycle snapshots

Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(rustfs): sync heal storage mock signature

Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): allocate smoke ports across nextest processes

Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 08:29:29 +08:00
Zhengchao An 7cb91a0190 chore(ecstore): adjudicate 32 bare dead_code allows (#6173)
Replace every bare `#[allow(dead_code)]` in ecstore with either a deletion or a per-item allow carrying a `reason`. Blanket allows at module, struct, and impl level silence the lint for future members too, so each is narrowed to the members that are actually dead.

Delete the dead cluster in `config/heal.rs` (`Config`, its three methods, `RUSTFS_BITROT_CYCLE_IN_MONTHS`, `parse_bitrot_config`) rather than annotate it: it has no callers and is unreachable outside the crate, and `parse_bitrot_config` would panic on its disabled path via `Duration::from_secs_f64(-1.0)`. `DEFAULT_KVS` stays, since the config registry uses it.

Correct two `reason` strings on `Checksum::new` and `PutObjReader::md5_current_hex_string`, which are methods but carried a field-only rationale.

Refs backlog#1823

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-17 23:51:56 +00:00
hector beb6e1383e feat(helm): add TLSRoute passthrough support for gateway api (#6169)
Add an optional TLS passthrough listener to the Gateway API support. When gatewayApi.listeners.tls.enabled is true, the Gateway gets a TLS listener with tls.mode: Passthrough and a TLSRoute is rendered to the RustFS service so TLS terminates at the backend (end-to-end encryption).

Refs rustfs/rustfs#3862.
2026-08-18 01:21:15 +08:00
houseme 59b7d13095 feat(scanner): expose prefix-level bucket usage via admin API (HS-08) (#6171)
feat(scanner): expose prefix-level bucket usage via admin API

The scanner's per-bucket, per-set usage caches already hold a path-keyed
prefix tree, but dui() flattened it only to bucket names — consoles and
operators had no way to ask "what does this prefix hold" without an S3
listing sweep (rustfs/backlog#1872, MinIO loadPrefixUsageFromBackend
parity).

Add:

- data-usage: prefix_usage_in_cache — a shared aggregation over the
  entry map (arbitrary prefix, full counters, one-level sub-prefix
  breakdown with names recovered from the literal-path cache keys),
  hardened like the scanner's checked flatten: cycles, dangling child
  links, over-deep trees, and overflowing counters yield None rather
  than unbounded recursion or wrapped totals.
- ecstore: ECStore::all_set_disks — iterate every erasure set so a
  query can read each set's own cache copy; the hash-routed store path
  would always land on one set.
- scanner: bucket_prefix_usage — per-set loads (5s budget each, a slow
  set degrades to not-reporting instead of stalling the caller),
  merged across sets with partial/compacted/truncated flags, served
  from a bounded 30s cache (128 entries, hard-capped) that bucket
  writes invalidate through the dirty-usage hook.
- admin: GET /rustfs/admin/v3/usage/{bucket}?prefix=&max-entries=
  behind the same any-of gate as datausageinfo (DataUsageInfoAdminAction
  OR ListBucketAction), rejecting unknown query parameters and
  clamping max-entries to 1..=10000. Route registered in the policy
  table (deferred MultipleActions, matching datausageinfo) and the
  route matrix test.

Closes rustfs/backlog#1872.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 11:40:56 +00:00
唐小鸭 e0b87b0e7e fix(site-replication): admit only verifiable peer-edit fences (#6123) 2026-08-17 09:47:36 +00:00
houseme 984c705713 docs(ecstore): fix bitrot comment typo (#6168)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 08:24:54 +00:00
houseme 23b17c2d5a feat(madmin): add a SigV4-signed admin client for heal and scanner APIs (HS-05) (#6166)
feat(madmin): add a SigV4-signed admin client for heal and scanner APIs

The madmin crate held only wire types; automation and mc-style tooling
had no way to drive the heal/scanner admin surface without hand-rolled
HTTP. Add `AdminClient`, which signs with the same rustfs-signer path
the server authenticates (UNSIGNED-PAYLOAD marker, matching RustFS peer
admin calls) and wraps:

- heal_start / heal_status / heal_stop over POST /rustfs/admin/v3/heal/
  (bucket/prefix path params percent-encoded per segment; stop models
  the server's two cancel branches: token-scoped task status vs
  path-scoped start-success receipt);
- background_heal_status, scanner_status (freshness typed), plus
  ilm_expiry_status / replacement_recovery_status passthroughs;
- a public get_json escape hatch for endpoints not wrapped yet.

Wire types follow the madmin-go model (SDK-owned mirrors pinned by
round-trip tests): HealOpts with serde defaults so partial settings
objects decode, HealScanMode accepting both the numeric and name
encodings, and status structs that type the fields operators branch on
while flattening unknown nested payloads verbatim so server additions
cannot break the client. Errors map to a closed AdminClientError enum
(InvalidEndpoint / Transport / HttpStatus with body / Decode).

Tests cover wire round-trips, path building, both stop branches, error
mapping, and — via a dependency-free raw-TCP test server — that signed
requests carry a SigV4 Authorization header, the right method/path/
query, and the expected JSON body.

Closes rustfs/backlog#1869 (first increment; single-sourcing the wire
structs server-side and an embedded-server e2e roundtrip are noted as
follow-ups there).

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 15:04:49 +08:00
houseme 89e2513205 feat(ecstore): pin bitrot algorithms with a startup self-test (HS-11) (#6165)
feat(ecstore): pin bitrot algorithms with a startup self-test

A drifted HighwayHash implementation fails silently: every shard reads
back corrupt, heal rewrites healthy data, and cross-platform clusters
disagree about which copy is good. Mirror MinIO's bitrotSelfTest by
verifying, once at process start:

- known-answer digests for HighwayHash256S / HighwayHash256SLegacy over
  a deterministic 4096-byte xorshift64* payload, plus the externally
  verifiable FIPS SHA-256 "abc" vector guarding the HashAlgorithm
  plumbing itself;
- an end-to-end roundtrip per streaming variant (encode -> size formula
  -> bitrot_verify -> BitrotReader read-back), over full blocks and a
  partial tail;
- tamper detection: one flipped byte in the final data block and one in
  the leading hash must both be rejected as a hash mismatch, not by an
  incidental read error.

The check costs microseconds and runs inline in
init_background_service_runtime before any shard can be written or
verified. Outcome surfaces as one structured bitrot_selftest log event,
the rustfs_bitrot_selftest_status gauge (1=passed / 0=failed / 2=skipped),
a bitrotSelftest field on the admin server-info response, and
RUSTFS_BITROT_SELFTEST_STRICT=on turns a failure into a startup error
(MinIO Fatal parity; the default only degrades the status so a bad build
cannot brick an existing fleet on upgrade).

Closes rustfs/backlog#1873 (HS-11).

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 15:04:23 +08:00
唐小鸭 d091554ffe fix(kms): resolve Vault auth from the environment at startup and add Kubernetes auth (#6095) 2026-08-17 14:12:39 +08:00
唐小鸭 c04ee41cf0 feat(site-replication): drain the retry queue from the reconcile tick (#6131) 2026-08-17 14:12:04 +08:00
Zhengchao An a9691b6797 chore: adjudicate 19 bare dead_code allows across six leaf crates (#6161)
backlog#1823 step 10, batch 1 of the repo-wide item-allow sweep. 227 bare #[allow(dead_code)] remain across 83 files; this takes the 19 in utils, notify, checksums, policy, keystone and trusted-proxies, which are small enough to verify end to end.

Removing all 19 first, before writing any reason, matters: 8 of them suppress nothing. Every allow in utils, one in policy and three in notify sit on items that are publicly reachable, so dead_code never applied to them — the same shape as the swift module and kms's dek.rs. Writing a reason onto a no-op allow would dress noise up as considered judgement, so those are simply deleted.

Three items are genuinely dead and go with their allows: notify's new_target_id_set, the AWS metadata fetcher's get_metadata_token, and policy's empty `pub struct Value;`, none of which is referenced anywhere in the tree.

The remaining eight keep an allow, now saying why the item survives rather than who calls it. Two are exercised only by their own crate's tests (checksums' MD5_HEADER_NAME, policy's is_match_as_pattern_prefix). Four are fields written but never read back: keystone's verify_ssl, parsed from config after the reqwest client is already built; keystone's client handle, which keeps the Keystone client alive for the mapper's lifetime; the AWS IMDS endpoint, kept beside the client while requests build their own URLs; and notify's rules_map, whose own comment retains it for snapshot-time judgements no code performs.

checksums' Md5 needed the most care. Crc32, Sha256 and seven others each have an arm in ChecksumAlgorithm::into_impl, and Md5 has none, which reads like a missing algorithm. It is not: ChecksumAlgorithm has no Md5 variant at all. S3 carries Content-MD5 as its own header, separate from the x-amz-checksum-* family, and this impl exists so both paths share the Checksum trait. The reason records that, so the next reader does not re-derive it.

One measurement note for anyone continuing this sweep: cargo does not re-emit warnings for cached compilations, so a per-crate loop of `cargo check -p <crate>` under-reports. checksums showed zero that way while actually carrying three. Touch the sources and check the crates in one invocation, then attribute by path.

Verification: the six crates are warning-free under cargo check --tests; clippy --lib --tests -D warnings clean; cargo nextest run 1096 passed; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 10).
2026-08-17 11:34:47 +08:00
Zhengchao An 7db3882777 chore(obs): adjudicate 19 bare dead_code allows (#6162)
backlog#1823 step 10, batch 2. Eighteen of the nineteen suppress nothing and are deleted; one was real and keeps an allow that now says why.

Rotation::Never is constructed only by the rolling-appender tests at rolling.rs:456, 477 and 498, so the lib target reports it as never constructed. Its allow is restored with that reason.

Finding it corrected the method used for batch 1. Removing all nineteen and running cargo check -p rustfs-obs --tests reported zero warnings even after touching every source file, while clippy --lib --tests -D warnings caught Rotation::Never. cargo's warning output is not a reliable completeness check — it does not re-emit for cached compilations, and touching the sources did not cover the lib target here. Later batches should treat clippy -D warnings as the gate; batch 1's six crates were re-checked under clippy and are clean.

Taken with #6086, which cleared this crate's 44 module-level blankets and left six real items, obs has now had 63 dead-code suppressions examined, of which seven were suppressing anything at all. The rest sat on items that are publicly reachable, where dead_code never applied — the same shape as the swift module and kms's dek.rs.

Verification: clippy --lib --tests -D warnings clean in the default, gpu and pyroscope lanes; cargo nextest run -p rustfs-obs 324 passed; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 10).
2026-08-17 11:32:23 +08:00
houseme 3f3e3f4f05 perf(get): avoid memory body stream wrapper (#6163)
Use MemoryTrackedBytesStream directly as an s3s ByteStream so in-memory GET bodies avoid the generic StreamingBlob::wrap adapter while preserving exact remaining length, request lifecycle tracking, and length-mismatch failure semantics.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 10:45:24 +08:00
houseme 01e0af6312 perf(io-metrics): avoid get handoff label allocations (#6160)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 08:26:23 +08:00
Zhengchao An 3377688dab ci(arch): ratchet ecstore module-level lint blankets (#6155)
backlog#1823 step 9. The step 2 burn-down cleared every module-level #![allow(dead_code)] from ecstore, but nothing stops the next PR from adding one back, and the other module-level blankets (unused_variables, unused_must_use, clippy::all) were never counted at all.

Two rules land in check_architecture_migration_rules.sh:

ecstore must carry zero module-level #![allow(dead_code)]. The count is asserted at zero rather than registered, since there is nothing left to grandfather; a genuinely unused item takes an item-level allow with a reason, which is what step 2 produced roughly 350 times.

Every other module-level blanket must match scripts/ecstore-module-lint-register.txt exactly — 94 entries across 33 files, nearly all of them in the MinIO-ported client module.

The exact match is the point. A "no new entries" rule lets the register rot into an amnesty list, which is the failure mode backlog#1834 found in the layer-dependency baseline: rebuilt at 29 entries, 2 more added by a later PR, zero retired. Here, removing a blanket costs one line in the register, so it can only shrink, and adding one shows up as a register line a reviewer has to accept.

Verified by injection, since a guard that cannot fail is worse than no guard: adding a dead_code blanket, adding an unregistered clippy::all blanket, and deleting a registered blanket without updating the register each produce the expected failure, and the tree passes once reverted.

make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 9).
2026-08-17 08:08:36 +08:00
Henry Guo 890ddea94b fix(table-catalog): prevent object catalog lock reentry (#6144)
* fix(table-catalog): prevent object catalog lock reentry

* test(table-catalog): distinguish unlocked object reads

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-17 08:06:53 +08:00
Zhengchao An 9f02ca6c36 fix(ecstore): resolve nine unused bindings in set_disk write and heal paths (#6158)
backlog#1823 step 1, the diagnosis half. Temporarily removing set_disk/mod.rs's #![allow(unused_variables)] surfaced nine bindings. The issue asks that values computed and then dropped on write/quorum paths be diagnosed before being underscored, and that turned out to matter: only four were plain leftovers.

Two errors were bound and then left out of the log they were bound for. complete_multipart_upload's checksum failures read `if let Err(err) = ...` and then log part_id, bucket and object with no `err` anywhere in the message, so a checksum failure in production told you which part failed but not why. Both messages now carry the error.

One is a lock guard. heal's write_lock_guard holds a namespace write lock for the rest of the scope; renaming it to a bare `_` would drop it immediately and release the lock. It is now `_write_lock_guard`, with a comment saying why it must not be `_`.

One was kept alive by a corpse. `errors` in read_multiple_files is read by nothing except two commented-out debug! lines directly below it; the binding and the commented lines go together.

One is a cfg split. heal's disk_index is read only inside the #[cfg(test)] fault-injection branch, so underscoring it would break the test build; a `#[cfg(not(test))] let _ = disk_index;` covers the non-test lane instead.

The remaining four are genuine leftovers: an unused enumerate index in list_object_parts, a discarded error in a heal reader loop, an inner binding shadowing its own iterator variable, and delete_object's write_quorum.

That last one is worth a separate look: delete_object asks get_object_info_and_quorum for a write quorum and never uses it, because delete_object_version below recomputes its own as disks.len() / 2 + 1. The two are not the same number — one comes from the object's erasure configuration, the other is a plain majority of the disk array. Pre-existing behaviour, untouched here.

The blankets stay for now. Removing #![allow(unused_imports)] exposes 76 unused imports in set_disk/mod.rs, and they cannot be removed per-lane: cargo fix, working from the lib lane, produced 54 compile errors in the test lane. That needs its own pass with both lanes checked per import.

Verification: cargo check -p rustfs-ecstore --tests and --features test-util --tests both warning-free; clippy --lib --tests -D warnings clean; cargo nextest run -p rustfs-ecstore 4101 passed; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 1).
2026-08-17 08:06:37 +08:00
Zhengchao An 33cd11472a refactor(rustfs): move module switches below the layer boundary (#6154)
backlog#1834 PR5. Whether the scanner, heal, audit and notify modules are on gets read from infra (storage helpers, node-service RPC) and from interface (admin handlers), but the switches lived in startup_background (composition) and server (interface). Every one of those reads was an upward edge carried in the layer-dependency baseline.

The env-derived scanner/heal predicates and the audit/notify state cells now live in rustfs/src/module_switches.rs, at the bottom of the layer order, so the same reads are ordinary downward edges. startup_background and server import from there; server keeps re-exporting the getters for its own consumers.

The issue's plan was to move is/refresh_audit/notify_module_enabled as a group. Moving refresh_* wholesale would have dragged resolve_audit_module_state and resolve_notify_module_state — server-side configuration logic — down into infra, which breaks more layering than it fixes. State and resolution are split instead: module_switches owns the atomics plus is_*/set_* accessors, and server's refresh_* keeps the configuration logic and publishes through the setter.

That leaves storage/helper.rs's test module importing refresh_* from server, so two infra->interface edges stay. Those tests assert that a configuration change takes effect through refresh, which a plain setter would no longer exercise; the edges are worth more than the two baseline lines.

Baseline drops 44 -> 36 lines, deletions only:

- 4 interface/infra -> composition edges for ENV_SCANNER_ENABLED, scanner_enabled_from_env and heal_enabled_from_env
- 2 infra -> interface edges for is_audit_module_enabled and is_notify_module_enabled
- cycle|composition<->infra and cycle|composition<->interface

The two cycles were not expected to go until whole subsystems moved out; clearing composition's inbound upward edges dissolved both, leaving three of the original five.

Verification: scripts/check_layer_dependencies.sh passes, cargo check -p rustfs warning-free, make pre-commit exit 0.
2026-08-17 08:05:35 +08:00
Zhengchao An 3ff250f1cd chore(protocols): drop 43 no-op dead_code allows from swift (#6157)
backlog#1823 step 8, partial. The swift module carries 43 #[allow(dead_code)] attributes, most with a comment naming a consumer: "Used by handler", "Handler integration: GET container", "Used by handler and object.rs".

Every one of them suppresses nothing. crates/protocols/src/lib.rs declares `pub mod swift`, and swift/mod.rs declares all 22 submodules `pub mod`, so every item is publicly reachable and dead_code never applied to it. Removing all 43 leaves the warning count at zero, in both the default and --features swift lanes.

That is also why those comments survived. They assert who calls the item — a claim the compiler normally settles on its own — and the compiler had been silenced by the visibility chain.

The rest of step 8 needs a decision this PR does not make. Downgrading the 22 submodules to `pub(crate) mod` does restore detection, and it surfaces 39 real items, 16 of them the whole of sync.rs: SyncConfig, SyncStatus, SyncQueueEntry, ConflictResolution and every function and constant around them, i.e. Swift container sync is built and never wired.

But the `pub mod` chain is load-bearing. Six integration tests under crates/protocols/tests are separate crates that import the submodules directly (swift::quota, swift::slo, swift::symlink, swift::sync, swift::tempurl, swift::container), and the downgrade fails to compile them. Restoring dead-code detection for this module therefore depends on first deciding whether those tests move in-crate — which is a testing-strategy call, not a cleanup one.

Verification: cargo check -p rustfs-protocols warning-free in the default lane and with --features swift (lib and --tests); clippy --features swift --lib --tests -D warnings clean; cargo nextest run -p rustfs-protocols --features swift 441 passed; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 8).
2026-08-17 08:03:29 +08:00
Zhengchao An d795729585 fix(scanner): canonicalize temp_dir for drive field assertion on macOS (#6159) 2026-08-17 07:11:25 +08:00
Henry Guo 9e6e02ea09 fix(table-catalog): assign fresh schema IDs on create (#6146)
* fix(table-catalog): assign fresh schema IDs on create

* fix(table-catalog): accept negative create schema IDs

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-08-17 01:06:25 +08:00
houseme 39274fc37c feat(ecstore): default bounded metadata fanout (#6156)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 00:56:20 +08:00
houseme 33eff4c3c4 test(ecstore): add metadata slow-tail fault hook (#6150)
Add a diagnostic metadata-only read_version delay hook for GET data-read fanout so bounded/default behavior can be compared under controlled slow-tail metadata responses.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-16 23:36:46 +08:00
Zhengchao An a2f16aa066 test(tier): pin the compressed transitioned read against its stored bytes (#6151)
#6107 routed the transitioned read through the object's own ReadPlan so a tiered SSE object stops serving ciphertext. Compression rides that same plan and got fixed with it, but nothing pins it: revert the routing and a compressed object that ILM moved to a warm tier returns its stored (compressed) bytes under the compressed size, with every existing test still green.

The gap is easy to reopen because transition genuinely uploads the stored representation — the upload side is correct and the read side is the only place that can decode it. These tests state that contract at the boundary where it broke.

Four cases, all through SetDisks::get_object_reader against a mock warm tier:

- a full GET of a transitioned compressed object returns the plaintext and publishes the plaintext size (the test also asserts the remote copy holds the compressed bytes, so it fails loudly if the upload side ever changes instead);
- a ranged GET returns that plaintext slice, with the range deliberately starting past the compressed size so a range still measured in stored coordinates cannot produce it;
- a restore read still receives the stored bytes under the stored size — restore_request_active holds it on the Plain branch, and decompressing there would write plaintext under compressed metadata;
- a plain transitioned object still reads back byte-identical, full and ranged.

Verified as guards, not decoration: forcing the tiered read back onto the Plain branch turns the two compressed tests red and leaves the plain and restore tests green.

What these do not pin, so the gap stays recorded rather than implied covered: the fixture carries no compression index, so part.index stays None and the plan's storage offset is always 0 — the compressed-offset translation itself is still untested, as are multipart compressed objects, partNumber reads, and the encrypted tiered read that #6107 targeted.
2026-08-16 15:27:59 +00:00
Zhengchao An 4c8b9f87e1 chore(ecstore): annotate the two dead fns the blanket removals missed (#6152) 2026-08-16 22:42:51 +08:00
Zhengchao An 3272730c13 fix(ecstore): silence two dead_code warnings left on main (#6153) 2026-08-16 22:42:45 +08:00
Zhengchao An 1862112d0c chore: drop the remaining product-code dead_code blankets (#6149) 2026-08-16 14:18:00 +00:00
Zhengchao An cd0ac02879 test(interop): let the MinIO fixture lab build from registry mirrors (#6148) 2026-08-16 14:05:27 +00:00
Zhengchao An 6cf9cf7bb5 chore(ecstore): drop the bucket dead_code blanket (#6147)
* chore(ecstore): drop the bucket dead_code blanket

The last blanket of the backlog#1823 burn-down, and the largest: 71 items across lifecycle, replication, metadata, quota, object lock and bucket utils. Four are deleted.

Deleted, all trivial:

- check_valid_object_name and check_valid_object_name_prefix, a pair that only calls into each other with no external caller. Worth stating plainly so nobody reads this as a validation gap: object names are validated through check_object_name_for_length_and_slash, which is live; this pair is a second, unwired entry point.
- DEFAULT_HEALTH_CHECK_RELOAD_DURATION, a lone unused constant.
- The LifecycleReplicationConfig alias, which orphaned a re-export in replication/mod.rs that goes with it.

Everything else is kept, in four groups, because the blanket here was hiding structure rather than rot:

Windows platform gating. WINDOWS_RESERVED_NAMES, the two reason constants and object_name_has_windows_incompatible_segment are called from inside the #[cfg(target_os = "windows")] block in check_object_name_for_length_and_slash (utils.rs:228-255), so they only read as dead on non-Windows hosts. As with the Linux gating in the disk root, this cannot be adjudicated locally: cargo check for both x86_64-pc-windows-msvc and x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a cross C toolchain. CI covers both.

Declared boundary surface. The *_boundary.rs and *_bridge.rs files carry the replication split plan's contracts, which scripts/check_architecture_migration_rules.sh pins through the EcstoreReplicationBoundaryImports section of the split-plan doc. Their unused items are declarations, not leftovers.

test-util seams. ConfigWriteLockProbe with install/wait_until_attempted follows the same pattern as the barriers in the services and set_disk roots.

MinIO-parity tier/lifecycle entry points that this port never wired: apply_lifecycle_action, get_transitioned_object_reader, recover_tier_free_versions, delete_object_from_remote_tier, abort_tier_delete_journal_entry and the replication pool's worker-management surface. These are complete, substantial machinery with no caller — the same shape as data_usage's local_snapshot feature. Removing them is a product decision, so they are made explicit here rather than deleted.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. Note that clippy is what caught the orphaned re-export above: cargo check and pre-commit both treat unused_imports as a warning.

Ref rustfs/backlog#1823 (step 2, final root).

* chore(ecstore): correct inaccurate dead_code reasons in the bucket root

Six items were labelled 'asserted by this file's tests' or as MinIO-parity
entry points while having no caller at all - free get_bucket_acl_config and
created_at only reach their own live methods (production goes through
created_at_in), BucketVersioningSys::get_in, utils::serialize_content and
ServiceType have no reference anywhere, and with_transition_queue_env_async
is an unused test fixture, not a tier entry point. Name what each one is so
the next reader does not assume coverage that is not there.

Ref rustfs/backlog#1823.
2026-08-16 21:39:04 +08:00
Zhengchao An f1f86ee9d0 chore(ecstore): drop the set_disk dead_code blanket (#6141)
* chore(ecstore): drop the set_disk dead_code blanket

Removing the blanket exposes 39 items; exactly one is deleted. The low share is a finding, not caution: unlike the disk root, where platform gating made local adjudication impossible, here the items were checked and nearly all of them are live.

Deleted: HealEntryResult, the only item with no reference anywhere.

What the checks turned up, in the order the warnings suggest deleting them:

SetDisks::rename_data looked like the head of a dead chain feeding into_legacy_tuple and RenameDataLegacyTuple. It is not: production goes through rename_data_owned, and rename_data itself has test callers at mod.rs:5809 and 5880. The chain below it is therefore live through the tests, and inferring "this is dead, so its callee is dead" would have removed three working items.

create_bitrot_readers_until_quorum, read_multiple_files and map_cleanup_join_result all have callers inside their files' test modules, so they only look dead in the lib target.

TransitionCommitBarrier and TransitionUploadedSaveProbe, with their install/wait_until_paused/release surfaces, are installed by tests behind #[cfg(all(test, feature = "test-util"))].

ctx.rs's SetDisksCtx accessors are the split seam left by the SetDisks god-object break-up (backlog#815).

heal_object_dir's two apparent references are comments, and they document an index-alignment contract that live code maintains for it, so they stay as they are.

Worth a maintainer decision: the metadata early-stop switch has a complete percentage-rollout facet — ENV_RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT, get_metadata_early_stop_rollout_pct and should_use_metadata_early_stop — with no caller, no test and no documentation, while its sibling enable flag is live. It is kept with an allow that says so rather than removed, since a rollout knob is a product call.

One placement note for anyone adding allows near heal code: check_logging_guardrails.sh requires #[instrument(level = "trace")] to sit immediately before async fn heal_object_dir, so the allow goes above the instrument attribute. Putting it between the two drops the guard's match count and fails the check.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 2).

* chore(ecstore): fix duplicated and inaccurate dead_code reasons in set_disk

format_lock_error carried the same #[allow] twice. Five items in the
locking/heal roots were labelled 'asserted by this file's tests' while
having no reference at all - heal_object_dir's only two references are
comments, as this branch's own notes point out. Say what each item
actually is instead, so the next reader does not assume test coverage
that is not there.

Ref rustfs/backlog#1823.

* chore(ecstore): correct the bounded_spare_disk_index dead_code reason

The mod.rs copy is an unused test fixture, not something this module's
tests assert; the namesake that is exercised lives in the io_primitives
test module.

Ref rustfs/backlog#1823.
2026-08-16 21:38:46 +08:00
Zhengchao An 1eef0de003 chore(ecstore): drop the disk dead_code blanket (#6139)
* chore(ecstore): drop the disk dead_code blanket

Removing the blanket exposes 36 items in the lowest storage layer: 7 deleted, 29 kept with reasoned item-level allows. That is the smallest deletion share of this burn-down, and the reason is a verification limit rather than a judgement call.

disk/local.rs carries 141 `#[cfg(target_os = "linux")]` sites — the densest platform gating in the tree, because O_DIRECT and io_uring only exist there. The direct-I/O cluster (six ENV_RUSTFS_OBJECT_DIRECT_IO_* constants plus is_direct_io_read_enabled, is_direct_io_write_enabled, get_direct_io_read_threshold, direct_write_staging_capacity, direct_write_tail_split and DIRECT_WRITE_STAGING_BYTES) reads as dead on macOS purely because its production callers at local.rs:1766, 3114 and 4605 sit inside Linux-gated blocks. direct_write_staging_capacity even documents itself as "Platform-independent (no O_DIRECT), so it is unit-tested on any host".

Deleting those would leave every local check green — 4096 tests pass, clippy is clean, make pre-commit exits 0 — and break the Linux build in CI, because all four local lanes compile for aarch64-apple-darwin. Cross-checking locally is not available either: cargo check --target x86_64-unknown-linux-gnu fails in the aws-lc-sys build script for want of a Linux C cross-compiler. Their allows name the platform reason so the next reader on a non-Linux host does not repeat the investigation.

Deleted, all in files with no target_os gating at all (os.rs, disk_store.rs):

- HealthDiskCtxKey and HealthDiskCtxValue with its private log_success. Note that DiskHealthTracker::log_success is a different method of the same name and is live from cluster/rpc/peer_s3_client.rs and remote_disk.rs — the two have to be told apart by type, not by name.
- LocalDiskWrapper::new_with_health and check_id.
- os.rs file_exists and lock_destination_directory_for_path_access.

Kept with allows: DiskHealthTracker's set_faulty, mark_offline, waiting_count and last_success have test callers in remote_disk.rs, so they only look dead in the lib target. to_disk_error, remove_all and sync_dir_files are asserted by their own files' tests. The reclaim, mmap and path-cache field groups are written but never read back.

Placement follows the same rule as the earlier roots: per-method allows inside impl DiskHealthTracker and impl LocalDisk, since both are mostly live and a block-level allow would be a smaller version of the blanket this issue removes. Struct-level allows are used only where the warning covers that struct's own fields. The three cached_read_env! functions take their allow inside the macro invocation, before the fn line, because the macro forwards $(#[$meta:meta])* onto the generated item.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4096 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0. The Linux lane is not covered locally and is left to CI.

Ref rustfs/backlog#1823 (step 2).

* chore(ecstore): correct two dead_code reasons in the disk root

check_valid_path and reject_symlink_components have no caller at all -
not even a test - so 'asserted by this file's tests' misreads them as
covered. Both are method wrappers over live free functions; say that
instead.

Ref rustfs/backlog#1823.
2026-08-16 21:38:37 +08:00
405 changed files with 26692 additions and 13895 deletions
+7 -2
View File
@@ -66,10 +66,15 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
./scripts/check_s3s_footprint.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
@echo "📣 Checking FIPS wording guard..."
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
./scripts/check_fips_wording.sh
.PHONY: embedded-secrets-check
embedded-secrets-check: ## Check no private key material or credential literal is committed
@echo "🔑 Checking embedded secret material guard..."
./scripts/check_embedded_secrets.sh
.PHONY: log-analyzer-rules-check
log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
@echo "🩺 Checking log-analyzer rule anchors..."
+3 -3
View File
@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
@echo "✅ Fast development checks passed!"
+1
View File
@@ -34,6 +34,7 @@ script-tests: ## Run shell script tests
./scripts/test_exact_1mib_handoff_abba.sh
./scripts/test_pinned_paired_abba_bench.sh
./scripts/test_manual_transition_runbooks.sh
./scripts/check_embedded_secrets.sh --self-test
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+11
View File
@@ -87,6 +87,13 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
# Serialize the default-off dst-dir fsync group-commit tests. They use
# process-global test hooks/registry to deterministically freeze fsync batches;
# no retries, just one at a time under nextest too.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
test-group = 'ecstore-serial-flaky'
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
# e2e-reliability test-group note above). The matching ci-profile override is at
# the end of the file, after [profile.ci] is declared.
@@ -188,6 +195,10 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
test-group = 'ecstore-serial-flaky'
# ---------------------------------------------------------------------------
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
# ---------------------------------------------------------------------------
+6
View File
@@ -117,6 +117,12 @@ jobs:
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
run: ./scripts/check_fips_wording.sh
- name: Check no embedded secret material
run: ./scripts/check_embedded_secrets.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
+6
View File
@@ -152,6 +152,12 @@ jobs:
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
run: ./scripts/check_fips_wording.sh
- name: Check no embedded secret material
run: ./scripts/check_embedded_secrets.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
+96 -11
View File
@@ -15,28 +15,35 @@
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2.
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - release published: automatically package when a GitHub release is published
# - workflow_dispatch: manual trigger with optional tag/run_id
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback (backfill / re-run) with optional tag/run_id
#
# Flow:
# 1. Find the Build workflow run for the release tag
# 1. Resolve the triggering Build workflow run for the release tag
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 5. Upload all packages to Cloudflare R2
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
contents: read
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
on:
release:
types: [ published ]
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
tag:
@@ -49,13 +56,26 @@ on:
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.release.tag_name || github.event.inputs.tag || github.run_id }}
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
@@ -75,8 +95,8 @@ jobs:
set -euo pipefail
# Determine tag
if [[ "${{ github.event_name }}" == "release" ]]; then
TAG="${{ github.event.release.tag_name }}"
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
TAG="${HEAD_BRANCH}"
elif [[ -n "$INPUT_TAG" ]]; then
TAG="$INPUT_TAG"
else
@@ -93,6 +113,11 @@ jobs:
BUILD_RUN_ID="$INPUT_RUN_ID"
echo "Using explicit build run ID: $BUILD_RUN_ID"
elif [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Use the Build and Release run that triggered this workflow
BUILD_RUN_ID="${WORKFLOW_RUN_ID}"
echo "Using triggering workflow run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
# Find the build run that produced this tag
echo "Looking for build run for tag: $TAG"
@@ -265,6 +290,12 @@ jobs:
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
@@ -456,6 +487,60 @@ jobs:
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.resolve.outputs.tag }}"
DEB_FILE="${{ steps.deb.outputs.deb_file }}"
RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber 2>/dev/null || true
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
touch "$checksum_file"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'
# (e.g. rustfs_1.0.0~rc.2_amd64.deb is stored as
# rustfs_1.0.0.rc.2_amd64.deb), so checksum entries must
# reference the name as stored on the release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
(cd "$(dirname "$f")" && "$checksum_cmd" -- "$github_base") >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# Summary
summary:
name: Summary
+4 -4
View File
@@ -31,7 +31,7 @@ HTTP request
→ storage/ecfs (erasure coding, encryption, checksums)
→ ecstore (disk pool selection, data distribution)
→ rio (reader pipeline: encrypt → compress → hash → write)
→ io-core (zero-copy I/O, buffer pool, direct I/O)
→ io-core (buffer pool, storage profiling, admission control)
→ local disk / remote disk via RPC
```
@@ -55,7 +55,7 @@ rustfs/ # Workspace root (virtual manifest)
├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members)
│ ├── ecstore/ # Erasure-coded storage engine
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
│ ├── io-core/ # Buffer pool, storage profiling, backpressure/deadlock policy, lock optimizer, operation progress
│ ├── io-metrics/ # I/O metrics collection
│ ├── common/ # Shared runtime state, globals, data usage types
│ ├── config/ # Configuration types and parsing
@@ -302,7 +302,7 @@ The binary (`main.rs`) boots in this order:
│ │ │
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ ecstore │ │ rio │ │ io-core │
│ (core) │ │ (readers) │ │ (zero-copy)
│ (core) │ │ (readers) │ │ (buffers)
└─────┬──────┘ └─────────────┘ └─────────────┘
┌─────┬──┼──┬─────┬──────┐
@@ -314,7 +314,7 @@ The binary (`main.rs`) boots in this order:
- **"Where does S3 PutObject go?"**
`server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
`ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
`ecstore` distributes → `rio` encrypts/compresses → `io-core` supplies buffers
- **"Where are bucket policies enforced?"**
`app/bucket_usecase` calls into `crates/policy/`
Generated
+115 -109
View File
@@ -964,9 +964,9 @@ dependencies = [
[[package]]
name = "aws-sdk-kms"
version = "1.114.0"
version = "1.115.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0b7d906608ee41e7ddea9983577ba82200435644d567d63dc34e822e088b453"
checksum = "d5b034f8b7ceadb873d0bc607c30bb4b0be68e09a84c837174e7c2c6878ff882"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -990,9 +990,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.141.0"
version = "1.142.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9f9420d3a2467eed22ed3635ca653653162c386a0b0f65c78189f9bd3c1379e"
checksum = "f9e15a5c55e05f4b0b7e483160b3c85cccdf77cff02c95504f3e71d460855cd2"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1027,9 +1027,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sso"
version = "1.105.0"
version = "1.106.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ffd0fbe7873cb548a7aa60f9573c268fff94155397fd4f14dc9f1ecaaab8516"
checksum = "2d0efcee834347b6705eca3eea2defd88242f43774f55d7326604222e3c86260"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1053,9 +1053,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ssooidc"
version = "1.107.0"
version = "1.108.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "175763eb222a46377df7aa257a3bca980ab3e96703fefc8f4d0b8da6ad2e254c"
checksum = "a59312a04cf19c962cfee32b64ecfee758f8786407ff6da5b30fff46ae96f201"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1079,9 +1079,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sts"
version = "1.110.0"
version = "1.111.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dd8b14781dfbff48984017d57167b6ea0b6471c6920ec52b44a2677c7feb3c13"
checksum = "120e7eb63457a9e547f9986fe3b273f77c43679da4d04f46359fa881c5e19b6e"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1598,7 +1598,7 @@ version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
]
[[package]]
@@ -1617,7 +1617,7 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
]
[[package]]
@@ -1858,9 +1858,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.4.2"
version = "1.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e"
checksum = "509591b7bcd67f4ef775afad7662703b4935daaa6ec0e5605cfb1090b32a2b6d"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -1968,7 +1968,7 @@ version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
dependencies = [
"crypto-common 0.1.6",
"crypto-common 0.1.7",
"inout 0.1.4",
]
@@ -2428,7 +2428,7 @@ version = "0.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
"rand_core 0.6.4",
"subtle",
"zeroize",
@@ -2453,11 +2453,11 @@ dependencies = [
[[package]]
name = "crypto-common"
version = "0.1.6"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
"typenum",
]
@@ -3664,7 +3664,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer 0.10.4",
"const-oid 0.9.6",
"crypto-common 0.1.6",
"crypto-common 0.1.7",
"subtle",
]
@@ -3924,7 +3924,7 @@ dependencies = [
"crypto-bigint 0.5.5",
"digest 0.10.7",
"ff 0.13.1",
"generic-array 0.14.9",
"generic-array 0.14.7",
"group 0.13.0",
"hkdf 0.12.4",
"pem-rfc7468 0.7.0",
@@ -4148,9 +4148,9 @@ dependencies = [
[[package]]
name = "find-msvc-tools"
version = "0.1.10"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de"
checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890"
[[package]]
name = "findshlibs"
@@ -4369,9 +4369,9 @@ dependencies = [
[[package]]
name = "generic-array"
version = "0.14.9"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
@@ -4380,11 +4380,11 @@ dependencies = [
[[package]]
name = "generic-array"
version = "1.4.4"
version = "1.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab4e5aa225bc56696909483320f0ff9b600f1a971b52e07a17d70f3d9b43254b"
checksum = "337d46834ee672ab3e48caca2cb0c78cc174fb12b3a68d0d88f99a0519a5e36e"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
"rustversion",
"typenum",
]
@@ -4726,9 +4726,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.15"
version = "0.4.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27"
dependencies = [
"atomic-waker",
"bytes",
@@ -5028,9 +5028,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62e810bedda5a467ef5c9b5c8a20763fefebc89b63ef36f7ee44a143085204a2"
checksum = "dce755d457a63bdd0c95e4c91511daad1b58b33209543b7f38027b676f387e5e"
dependencies = [
"arc-swap",
"async-channel",
@@ -5062,9 +5062,9 @@ dependencies = [
[[package]]
name = "hotpath-macros"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "01bdc59bfc1a9984bee2ff5da63b2f6fccbaa57cd9a4119d709524632bddf341"
checksum = "a903af89a8429cb07790c3818bc15270b394f80af1bc254e5ccf9c7de2961770"
dependencies = [
"proc-macro2",
"quote",
@@ -5073,15 +5073,15 @@ dependencies = [
[[package]]
name = "hotpath-macros-meta"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9216e8a01abe1e1671c376dc8736fb1bf772d7a889538d25f9e1200120ced38"
checksum = "bcc0ab94ffbb2ee77f4a897df02b5a137a10cf24d69bda936e59aff4dd456e61"
[[package]]
name = "hotpath-meta"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f22a9d20435fb79511b19dae37b3607224cd98f342a410702d84657cc38fc72f"
checksum = "053481f6cec8f775a3276c7f6e2f21123111d28261e4edc15ea7421c445964bb"
dependencies = [
"hotpath-macros-meta",
]
@@ -5280,9 +5280,9 @@ dependencies = [
[[package]]
name = "icu_collections"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c"
checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513"
dependencies = [
"displaydoc",
"potential_utf",
@@ -5294,9 +5294,9 @@ dependencies = [
[[package]]
name = "icu_locale_core"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29"
checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb"
dependencies = [
"displaydoc",
"litemap",
@@ -5307,9 +5307,9 @@ dependencies = [
[[package]]
name = "icu_normalizer"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4"
checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f"
dependencies = [
"icu_collections",
"icu_normalizer_data",
@@ -5321,16 +5321,17 @@ dependencies = [
[[package]]
name = "icu_normalizer_data"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38"
checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0"
[[package]]
name = "icu_properties"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de"
checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148"
dependencies = [
"displaydoc",
"icu_collections",
"icu_locale_core",
"icu_properties_data",
@@ -5341,15 +5342,15 @@ dependencies = [
[[package]]
name = "icu_properties_data"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14"
checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa"
[[package]]
name = "icu_provider"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421"
checksum = "92a7ed671a6aad807a8651a2e1782a6598fda9ce5185dd8158549e95a91c6428"
dependencies = [
"displaydoc",
"icu_locale_core",
@@ -5417,7 +5418,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
dependencies = [
"block-padding 0.3.3",
"generic-array 0.14.9",
"generic-array 0.14.7",
]
[[package]]
@@ -5968,9 +5969,9 @@ dependencies = [
[[package]]
name = "libredox"
version = "0.1.19"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2026a5056764a10b2bf5d56488cba40da507f5493a6a429340e2004d9ed085fa"
checksum = "28d0a00925a9f930d679b6789b721e3a7f9ed110f41b86d2497caa780c3a070a"
dependencies = [
"libc",
]
@@ -6033,9 +6034,9 @@ checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "litemap"
version = "0.8.2"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0"
checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae"
[[package]]
name = "local-ip-address"
@@ -6482,9 +6483,9 @@ dependencies = [
[[package]]
name = "mqttbytes-core-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ff7ae19c74aba9e0ed6e4071cd52aa364e020076fa3cc6ef17e43662f756f3c"
checksum = "366b6ba2b4209ca4bc5ac731ccddf570d09831981eed07e5fbd63564cf0cf1aa"
dependencies = [
"bytes",
"thiserror 2.0.20",
@@ -6885,7 +6886,7 @@ version = "5.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
dependencies = [
"base64 0.21.7",
"base64 0.22.1",
"chrono",
"getrandom 0.2.17",
"http 1.5.0",
@@ -7344,9 +7345,9 @@ dependencies = [
[[package]]
name = "pageant"
version = "0.2.1"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4f3a5ae18f65a85c67a77d18d42d3606c07948e3c17c1e5f74852b26589e88a5"
checksum = "3adadc44070da6f464b0918655a12f5792c156e088d8c4082d13e27d94c3e791"
dependencies = [
"base16ct 1.0.0",
"byteorder",
@@ -7728,9 +7729,9 @@ dependencies = [
[[package]]
name = "pkg-config"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "plotters"
@@ -7836,9 +7837,9 @@ dependencies = [
[[package]]
name = "potential_utf"
version = "0.1.5"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564"
checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661"
dependencies = [
"zerovec",
]
@@ -8046,7 +8047,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck 0.5.0",
"itertools 0.10.5",
"itertools 0.14.0",
"log",
"multimap",
"once_cell",
@@ -8066,7 +8067,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [
"heck 0.5.0",
"itertools 0.10.5",
"itertools 0.14.0",
"log",
"multimap",
"petgraph 0.8.3",
@@ -8087,7 +8088,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.10.5",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8100,7 +8101,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [
"anyhow",
"itertools 0.10.5",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8216,7 +8217,7 @@ dependencies = [
"reqwest",
"serde_json",
"smallvec",
"spin 0.12.2",
"spin 0.12.3",
"symbolic-demangle",
"tempfile",
"thiserror 2.0.20",
@@ -8285,9 +8286,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.16"
version = "0.11.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
checksum = "04759210543be93709136e28212294a659ef5001836ff4eab4d663e4529bba83"
dependencies = [
"aws-lc-rs",
"bytes",
@@ -8553,9 +8554,9 @@ dependencies = [
[[package]]
name = "redis"
version = "1.5.0"
version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3257df217f7eab0044627a268c9cc6cdb60c0c421c88f83ac41c4e31520b6b84"
checksum = "e37a4ca5c6ca42aa3e6df2fd32b987a65d32a4c2159a6f3fe0fd1df306a2658f"
dependencies = [
"arc-swap",
"arcstr",
@@ -8567,7 +8568,7 @@ dependencies = [
"futures-channel",
"futures-util",
"itoa",
"num-bigint 0.4.8",
"num-bigint 0.5.1",
"percent-encoding",
"pin-project-lite",
"rustls",
@@ -8868,9 +8869,9 @@ dependencies = [
[[package]]
name = "rumqttc-core-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d7d9205738dd41a2546e82d27a634d07d8b303dcf7558565ff70caf3ceb0f9c"
checksum = "249896ab27ed630590971738264baa8f722f18965d2e387c706c40a3c2a572cc"
dependencies = [
"async-tungstenite",
"futures-io",
@@ -8886,18 +8887,18 @@ dependencies = [
[[package]]
name = "rumqttc-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed1bad2180ff539da671da9a996152a921bc5316eb6d8a9cc3bd441653138b08"
checksum = "477c9bbfba8f3aecc7aad31c6de2eacb75822efaa18e7aeecb8d3d8e534fbf07"
dependencies = [
"rumqttc-v5-next",
]
[[package]]
name = "rumqttc-v5-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "229576cbedfa9089f90c17c9454e9429ac1e89cdd223bac5cb39d837593f79bc"
checksum = "3dfa6ddcc7a7dd5688f9bf78d8f81cb94f367bce56c055d8d94cf81ecb0518bf"
dependencies = [
"async-tungstenite",
"bytes",
@@ -8920,9 +8921,9 @@ dependencies = [
[[package]]
name = "russh"
version = "0.62.6"
version = "0.62.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b41043523e0edcbd4e31d00903e26f12994f63b21bae9904f7405c1ed92752a5"
checksum = "9decb68e4e44e1079700e54f17c8f23806ec53d7e0db73ab1c71d9dabc666812"
dependencies = [
"aes 0.9.2",
"aws-lc-rs",
@@ -8945,7 +8946,7 @@ dependencies = [
"enum_dispatch",
"flate2",
"futures",
"generic-array 1.4.4",
"generic-array 1.4.5",
"getrandom 0.4.3",
"ghash",
"hex-literal",
@@ -9280,6 +9281,7 @@ dependencies = [
"s3s",
"serde",
"serde_json",
"smallvec",
"tokio",
"tonic",
"tracing",
@@ -9463,6 +9465,7 @@ dependencies = [
"tokio-stream",
"tokio-util",
"tonic",
"tonic-prost",
"tower",
"tracing",
"tracing-core",
@@ -9490,7 +9493,7 @@ dependencies = [
"parking_lot",
"rayon",
"smallvec",
"spin 0.12.2",
"spin 0.12.3",
]
[[package]]
@@ -9536,6 +9539,8 @@ version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"base64 0.23.1",
"bytes",
"crc-fast",
"futures",
"hotpath",
"http 1.5.0",
@@ -9608,7 +9613,6 @@ version = "1.0.0-rc.2"
dependencies = [
"bytes",
"hotpath",
"memmap2",
"rustfs-io-metrics",
"thiserror 2.0.20",
"tokio",
@@ -9825,14 +9829,19 @@ name = "rustfs-madmin"
version = "1.0.0-rc.2"
dependencies = [
"hotpath",
"http 1.5.0",
"humantime",
"hyper",
"jiff",
"reqwest",
"rmp-serde",
"rustfs-signer",
"s3s",
"serde",
"serde_json",
"sysinfo",
"time",
"tokio",
]
[[package]]
@@ -10246,6 +10255,7 @@ dependencies = [
"rustfs-ecstore",
"rustfs-filemeta",
"rustfs-lock",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-utils",
"s3s",
@@ -10479,15 +10489,10 @@ dependencies = [
name = "rustfs-zip"
version = "1.0.0-rc.2"
dependencies = [
"astral-tokio-tar",
"async-compression",
"criterion",
"hotpath",
"tempfile",
"thiserror 2.0.20",
"tokio",
"tokio-stream",
"zip",
]
[[package]]
@@ -10673,7 +10678,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "s3s"
version = "0.14.1"
source = "git+https://github.com/rustfs/s3s.git?rev=d7028511a53f69d41ed3c69f36899f9b1aede647#d7028511a53f69d41ed3c69f36899f9b1aede647"
source = "git+https://github.com/rustfs/s3s.git?rev=d358a68783096df1db0c3e314127f2704603b29e#d358a68783096df1db0c3e314127f2704603b29e"
dependencies = [
"arc-swap",
"arrayvec",
@@ -10701,6 +10706,7 @@ dependencies = [
"numeric_cast",
"pin-project-lite",
"quick-xml",
"regex",
"serde",
"serde_json",
"serde_urlencoded",
@@ -10826,7 +10832,7 @@ checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc"
dependencies = [
"base16ct 0.2.0",
"der 0.7.10",
"generic-array 0.14.9",
"generic-array 0.14.7",
"pkcs8 0.10.2",
"subtle",
"zeroize",
@@ -11390,9 +11396,9 @@ checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3"
[[package]]
name = "spin"
version = "0.12.2"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8abadc99fd9c7bbb7d0ca2b31d72a067d0c0dcd7aad25ab8cac71ba91417694b"
checksum = "0134f9043ed38b087ac4f7d4af44c79e2c9e5094421fe3164f435ce585953b10"
dependencies = [
"lock_api",
]
@@ -11968,9 +11974,9 @@ dependencies = [
[[package]]
name = "tinystr"
version = "0.8.3"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d"
checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643"
dependencies = [
"displaydoc",
"zerovec",
@@ -12644,9 +12650,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.24.0"
version = "1.24.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239"
checksum = "2cefc03fd367c0c6d4305de1b312cf00248c4114f4a0418ce6a6af769e3b0bd9"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -13161,9 +13167,9 @@ dependencies = [
[[package]]
name = "writeable"
version = "0.6.3"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc"
[[package]]
name = "x509-cert"
@@ -13343,9 +13349,9 @@ dependencies = [
[[package]]
name = "zerotrie"
version = "0.2.4"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf"
checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f"
dependencies = [
"displaydoc",
"yoke",
@@ -13354,9 +13360,9 @@ dependencies = [
[[package]]
name = "zerovec"
version = "0.11.6"
version = "0.11.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239"
checksum = "94b5c6b5976d66c1d703c4fd17d3f5e43c8cedaacf604961b171adc7130896d8"
dependencies = [
"yoke",
"zerofrom",
@@ -13365,13 +13371,13 @@ dependencies = [
[[package]]
name = "zerovec-derive"
version = "0.11.3"
version = "0.11.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555"
checksum = "9f212a141d820099d57ffafb9569be9617a6f27d3dc881fbee8fb56642f917a9"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
+9 -9
View File
@@ -228,9 +228,9 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.10.1" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.114.0" }
aws-sdk-s3 = { default-features = false, version = "1.141.0" }
aws-sdk-sts = { default-features = false, version = "1.110.0" }
aws-sdk-kms = { default-features = false, version = "1.115.0" }
aws-sdk-s3 = { default-features = false, version = "1.142.0" }
aws-sdk-sts = { default-features = false, version = "1.111.0" }
aws-smithy-http-client = { default-features = false, version = "1.3.0" }
aws-smithy-runtime-api = { version = "1.14.0" }
aws-smithy-types = { version = "1.6.2" }
@@ -284,13 +284,13 @@ rayon = "1.12.0"
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
redis = { version = "1.5.0" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.6.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d7028511a53f69d41ed3c69f36899f9b1aede647" }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d358a68783096df1db0c3e314127f2704603b29e" }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
@@ -313,7 +313,7 @@ tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.24.0" }
uuid = { version = "1.24.1" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
@@ -341,7 +341,7 @@ libunftp = { version = "0.23.0" }
unftp-core = "0.1.0"
suppaftp = { version = "10.0.1" }
rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.62.6" }
russh = { version = "0.62.7" }
russh-sftp = "2.4.0"
# WebDAV
@@ -350,7 +350,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" }
libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] }
hotpath = { version = "0.23.2", default-features = false }
hotpath = { version = "0.23.3", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+1
View File
@@ -40,6 +40,7 @@ mak = "mak"
gae = "gae"
GAE = "GAE"
thr = "thr"
mis = "mis"
# s3-tests original test names (cannot be changed)
nonexisted = "nonexisted"
consts = "consts"
+9 -2
View File
@@ -236,12 +236,19 @@ async fn audit_pipeline_reports_empty_runtime_snapshots() {
}
#[tokio::test]
async fn audit_runtime_facade_stops_empty_replay_workers() {
async fn stopping_audit_replay_workers_is_a_no_op_when_there_are_none() {
let registry = Arc::new(Mutex::new(AuditRegistry::new()));
let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
let facade = AuditRuntimeFacade::new(registry, replay_workers);
let facade = AuditRuntimeFacade::new(registry, Arc::clone(&replay_workers));
facade.stop_replay_workers().await;
// The stop path takes the manager's workers and hands them to the adapter,
// so an empty facade must leave it empty rather than wedge it, and a second
// call — which shutdown paths make — must stay harmless (rustfs/backlog#1836).
assert!(replay_workers.read().await.is_empty());
facade.stop_replay_workers().await;
assert!(replay_workers.read().await.is_empty());
}
#[tokio::test]
-1
View File
@@ -11,7 +11,6 @@
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#![allow(dead_code)]
use base64_simd::STANDARD;
+4 -1
View File
@@ -38,7 +38,10 @@ pub const XXHASH_3_HEADER_NAME: &str = "x-amz-checksum-xxhash3";
pub const XXHASH_64_HEADER_NAME: &str = "x-amz-checksum-xxhash64";
pub const XXHASH_128_HEADER_NAME: &str = "x-amz-checksum-xxhash128";
#[allow(dead_code)]
#[allow(
dead_code,
reason = "Content-MD5 wire name, resolved by header_name() below and asserted by this crate's tests (backlog#1823)"
)]
pub(crate) static MD5_HEADER_NAME: &str = "content-md5";
pub const CHECKSUM_ALGORITHMS_IN_PRIORITY_ORDER: [&str; 5] =
+8 -2
View File
@@ -476,13 +476,19 @@ impl Checksum for Xxhash64 {
}
}
#[allow(dead_code)]
#[derive(Debug, Default)]
#[allow(
dead_code,
reason = "Content-MD5 is not a ChecksumAlgorithm variant and has no arm in into_impl: S3 carries it as its own header, separate from the x-amz-checksum-* family. This impl exists so the two paths share the Checksum trait, and is asserted by this crate's tests (backlog#1823)"
)]
struct Md5 {
hasher: md5::Md5,
}
#[allow(dead_code)]
#[allow(
dead_code,
reason = "Content-MD5 is not a ChecksumAlgorithm variant and has no arm in into_impl: S3 carries it as its own header, separate from the x-amz-checksum-* family. This impl exists so the two paths share the Checksum trait, and is asserted by this crate's tests (backlog#1823)"
)]
impl Md5 {
fn update(&mut self, bytes: &[u8]) {
use md5::Digest;
+1
View File
@@ -42,6 +42,7 @@ chrono = { workspace = true, features = ["serde"] }
jiff = { workspace = true, features = ["serde"] }
metrics = { workspace = true }
serde = { workspace = true, features = ["derive"] }
smallvec = { workspace = true }
rmp-serde = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
tracing = { workspace = true }
+27
View File
@@ -224,6 +224,13 @@ pub struct HealOpts {
pub enum HealAdmissionDropReason {
QueueFull,
PolicyDropped,
/// HS-06: an admin heal start overlaps (same bucket with mutually
/// containing prefixes, or the same erasure set) an already running or
/// queued task. Only produced when RUSTFS_HEAL_OVERLAP_POLICY=minio_error.
AlreadyRunning,
/// HS-06: same as [`Self::AlreadyRunning`] but for paths that merely
/// contain (or are contained by) the active task's path.
OverlappingPaths,
}
impl HealAdmissionDropReason {
@@ -231,6 +238,8 @@ impl HealAdmissionDropReason {
match self {
Self::QueueFull => "queue_full",
Self::PolicyDropped => "policy_dropped",
Self::AlreadyRunning => "already_running",
Self::OverlappingPaths => "overlapping_paths",
}
}
}
@@ -287,6 +296,9 @@ pub enum HealRequestSource {
Scanner,
AutoHeal,
ReadRepair,
/// Mission Repair Feed: intents delivered by error paths and replayed
/// from the durable MRF journal.
Mrf,
}
impl HealRequestSource {
@@ -297,6 +309,7 @@ impl HealRequestSource {
Self::Scanner => "scanner",
Self::AutoHeal => "auto_heal",
Self::ReadRepair => "read_repair",
Self::Mrf => "mrf",
}
}
}
@@ -313,6 +326,9 @@ pub enum HealChannelCommand {
Query {
heal_path: String,
client_token: String,
/// Incremental result cursor (HS-06): only items with a sequence
/// greater than this are returned; `None` keeps the full snapshot.
since_seq: Option<u64>,
response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
},
/// Cancel heal task
@@ -518,10 +534,21 @@ async fn receive_heal_channel_response(
/// Send heal query request
pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
query_heal_status_since(heal_path, client_token, None).await
}
/// Incremental heal query (HS-06): pass the client's last seen sequence
/// number to receive only newer result items.
pub async fn query_heal_status_since(
heal_path: String,
client_token: String,
since_seq: Option<u64>,
) -> Result<HealChannelResponse, String> {
let (response_tx, response_rx) = oneshot::channel();
send_heal_command(HealChannelCommand::Query {
heal_path,
client_token,
since_seq,
response_tx,
})
.await?;
+2
View File
@@ -17,8 +17,10 @@ pub mod globals;
pub mod heal_channel;
pub mod last_minute;
pub mod metrics;
pub mod mrf_channel;
mod readiness;
pub mod table_catalog;
pub mod trace_bus;
pub use globals::*;
pub use readiness::{GlobalReadiness, SystemStage};
+203
View File
@@ -0,0 +1,203 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Mission Repair Feed (MRF) intent channel.
//!
//! Producers on error paths (read decode failure, scanner metadata
//! corruption, partial-write recovery) hand a lightweight [`MrfIntent`] to the
//! heal crate through a global bounded channel. Delivery is strictly
//! non-blocking: `try_send_mrf_intent` never awaits and drops the intent
//! (counting it) when the channel is full or uninitialized — losing one heal
//! hint is always preferred over stalling an IO path. Durable replay of
//! unconsumed intents is the consumer's job (see `rustfs-heal`
//! `heal::mrf_queue`), mirroring MinIO's `.heal/mrf/list.bin`.
use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, Ordering},
};
use tokio::sync::mpsc;
use uuid::Uuid;
/// Bounded capacity of the global MRF channel. Backpressure is resolved by
/// dropping (and counting) intents, never by blocking the producer.
const MRF_CHANNEL_CAPACITY: usize = 8192;
/// Why an intent was produced. Drives the heal priority mapping on the
/// consumer side (DecodeFailure -> Urgent, MetadataCorruption -> High,
/// PartialWrite -> Normal).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MrfKind {
/// Erasure decode failed while serving a read (read path).
DecodeFailure,
/// Scanner classified object metadata as corrupt.
MetadataCorruption,
/// A write left the object with fewer committed shards than the set size.
PartialWrite,
}
impl MrfKind {
pub const fn as_str(self) -> &'static str {
match self {
MrfKind::DecodeFailure => "decode-failure",
MrfKind::MetadataCorruption => "metadata-corruption",
MrfKind::PartialWrite => "partial-write",
}
}
}
/// One repair intent. Kept deliberately small so the in-memory queue and the
/// journal stay bounded; `bucket`/`object` are `Arc<str>` so re-arming an
/// intent never re-allocates the strings.
#[derive(Clone, Debug)]
pub struct MrfIntent {
pub bucket: Arc<str>,
pub object: Arc<str>,
/// Version the intent targets, as raw UUID bytes.
pub version_id: Option<[u8; 16]>,
pub kind: MrfKind,
pub enqueued_at_ms: u64,
/// Times this intent has already been offered to the heal manager.
/// Dropped by the consumer once it reaches `MRF_MAX_ATTEMPTS`.
pub attempts: u8,
}
/// Consumer-side retry ceiling before an intent is given up on.
pub const MRF_MAX_ATTEMPTS: u8 = 3;
impl MrfIntent {
/// Rough in-memory footprint used by the queue's byte budget.
pub fn estimated_bytes(&self) -> usize {
// Struct + strings + version bytes; buckets and objects are usually
// far below this bound, so rounding up keeps the budget conservative.
64 + self.bucket.len() + self.object.len()
}
}
static GLOBAL_MRF_SENDER: OnceLock<mpsc::Sender<MrfIntent>> = OnceLock::new();
/// Delivery kill-switch, set from `RUSTFS_HEAL_MRF_ENABLE`. Producers check
/// this before touching the channel so the disabled path stays allocation- and
/// sync-free.
static MRF_DELIVERY_ENABLED: AtomicBool = AtomicBool::new(true);
/// Override delivery (used at heal-runtime startup from configuration).
pub fn set_mrf_delivery_enabled(enabled: bool) {
MRF_DELIVERY_ENABLED.store(enabled, Ordering::Relaxed);
}
/// Whether producers currently deliver intents.
pub fn mrf_delivery_enabled() -> bool {
MRF_DELIVERY_ENABLED.load(Ordering::Relaxed)
}
/// Create the global MRF channel and return the consumer half. Fails if the
/// channel is already initialized (the heal runtime is a singleton).
pub fn init_mrf_channel() -> Result<mpsc::Receiver<MrfIntent>, &'static str> {
let (sender, receiver) = mpsc::channel(MRF_CHANNEL_CAPACITY);
GLOBAL_MRF_SENDER
.set(sender)
.map_err(|_| "MRF channel sender already initialized")?;
Ok(receiver)
}
/// Best-effort, non-blocking intent delivery from an error path.
///
/// Returns `true` when the intent was accepted into the channel. `false`
/// means the intent was dropped (feature disabled, channel not yet
/// initialized, or channel full) — callers must not retry or await; the
/// existing read-repair / scanner heal paths remain the safety net.
///
/// This runs on IO error paths, so it stays synchronous and cheap: one
/// bounded allocation for the two `Arc<str>` handles plus the channel slot.
pub fn try_send_mrf_intent(kind: MrfKind, bucket: &str, object: &str, version_id: Option<Uuid>) -> bool {
if !mrf_delivery_enabled() {
return false;
}
let Some(sender) = GLOBAL_MRF_SENDER.get() else {
return false;
};
let intent = MrfIntent {
bucket: Arc::from(bucket),
object: Arc::from(object),
version_id: version_id.map(|vid| *vid.as_bytes()),
kind,
enqueued_at_ms: unix_now_ms(),
attempts: 0,
};
sender.try_send(intent).is_ok()
}
fn unix_now_ms() -> u64 {
// Kept trivial: the timestamp is diagnostic metadata only; wall-clock
// failure would be a bug rather than something to handle here.
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn intents_estimate_is_conservative() {
let intent = MrfIntent {
bucket: Arc::from("bucket"),
object: Arc::from("object"),
version_id: Some([0u8; 16]),
kind: MrfKind::DecodeFailure,
enqueued_at_ms: 0,
attempts: 0,
};
assert!(intent.estimated_bytes() >= intent.bucket.len() + intent.object.len());
}
#[tokio::test]
async fn try_send_delivers_and_respects_capacity() {
let mut receiver = init_mrf_channel().expect("first initialization should succeed");
assert!(init_mrf_channel().is_err(), "double initialization must fail");
assert!(try_send_mrf_intent(MrfKind::DecodeFailure, "b", "o", Some(Uuid::nil())));
let intent = receiver.recv().await.expect("intent should arrive");
assert_eq!(intent.kind, MrfKind::DecodeFailure);
assert_eq!(intent.bucket.as_ref(), "b");
// Disable delivery: producers become no-ops.
set_mrf_delivery_enabled(false);
assert!(!try_send_mrf_intent(MrfKind::PartialWrite, "b", "o", None));
set_mrf_delivery_enabled(true);
// Fill the bounded channel past capacity: excess intents are dropped,
// never blocking.
let mut accepted = 0;
for _ in 0..(MRF_CHANNEL_CAPACITY + 64) {
if try_send_mrf_intent(MrfKind::PartialWrite, "b", "o", None) {
accepted += 1;
}
}
assert_eq!(accepted, MRF_CHANNEL_CAPACITY);
}
#[test]
fn try_send_without_channel_is_false() {
// This test may run after the tokio test above in the same process;
// the singleton semantics make a clean "uninitialized" case hard, so
// assert the flag-off behavior only.
set_mrf_delivery_enabled(false);
assert!(!try_send_mrf_intent(MrfKind::MetadataCorruption, "b", "o", None));
set_mrf_delivery_enabled(true);
}
}
+333
View File
@@ -0,0 +1,333 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use smallvec::SmallVec;
use std::{
sync::{
Arc, OnceLock,
atomic::{AtomicUsize, Ordering},
},
time::{Duration, SystemTime},
};
use tokio::sync::broadcast;
const DEFAULT_TRACE_BUS_CAPACITY: usize = 1024;
const TRACE_ATTR_INLINE_CAPACITY: usize = 8;
static GLOBAL_TRACE_BUS: OnceLock<TraceBus> = OnceLock::new();
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TraceKind {
Heal,
Scanner,
}
impl TraceKind {
pub const fn as_str(self) -> &'static str {
match self {
Self::Heal => "heal",
Self::Scanner => "scanner",
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TraceFunc {
HealTask,
HealBucket,
HealObject,
HealCheckAbandonedParts,
HealErasureSetPage,
ScannerFolder,
ScannerIlmAction,
ScannerHealCandidate,
Dropped,
}
impl TraceFunc {
pub const fn as_str(self) -> &'static str {
match self {
Self::HealTask => "heal.Task",
Self::HealBucket => "heal.Bucket",
Self::HealObject => "heal.Object",
Self::HealCheckAbandonedParts => "heal.CheckAbandonedParts",
Self::HealErasureSetPage => "heal.ErasureSetPage",
Self::ScannerFolder => "scanner.Folder",
Self::ScannerIlmAction => "scanner.IlmAction",
Self::ScannerHealCandidate => "scanner.HealCandidate",
Self::Dropped => "trace.Dropped",
}
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum TraceVal {
Bool(bool),
U64(u64),
I64(i64),
Str(Arc<str>),
}
impl From<bool> for TraceVal {
fn from(value: bool) -> Self {
Self::Bool(value)
}
}
impl From<u64> for TraceVal {
fn from(value: u64) -> Self {
Self::U64(value)
}
}
impl From<i64> for TraceVal {
fn from(value: i64) -> Self {
Self::I64(value)
}
}
impl From<&str> for TraceVal {
fn from(value: &str) -> Self {
Self::Str(Arc::from(value))
}
}
impl From<String> for TraceVal {
fn from(value: String) -> Self {
Self::Str(Arc::from(value))
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TraceAttr {
pub key: &'static str,
pub value: TraceVal,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TraceEvent {
pub kind: TraceKind,
pub func: TraceFunc,
pub time: SystemTime,
pub bucket: Option<Arc<str>>,
pub object: Option<Arc<str>>,
pub duration: Duration,
pub bytes: u64,
pub attrs: SmallVec<[TraceAttr; TRACE_ATTR_INLINE_CAPACITY]>,
}
impl TraceEvent {
pub fn new(kind: TraceKind, func: TraceFunc) -> Self {
Self {
kind,
func,
time: SystemTime::now(),
bucket: None,
object: None,
duration: Duration::ZERO,
bytes: 0,
attrs: SmallVec::new(),
}
}
pub fn with_bucket(mut self, bucket: impl Into<Arc<str>>) -> Self {
self.bucket = Some(bucket.into());
self
}
pub fn with_object(mut self, object: impl Into<Arc<str>>) -> Self {
self.object = Some(object.into());
self
}
pub fn with_duration(mut self, duration: Duration) -> Self {
self.duration = duration;
self
}
pub fn with_bytes(mut self, bytes: u64) -> Self {
self.bytes = bytes;
self
}
pub fn with_attr(mut self, key: &'static str, value: impl Into<TraceVal>) -> Self {
self.attrs.push(TraceAttr {
key,
value: value.into(),
});
self
}
}
#[derive(Debug)]
pub struct TraceBus {
sender: broadcast::Sender<Arc<TraceEvent>>,
subscriber_count: Arc<AtomicUsize>,
}
impl TraceBus {
pub fn new(capacity: usize) -> Self {
let capacity = capacity.max(1);
let (sender, _receiver) = broadcast::channel(capacity);
Self {
sender,
subscriber_count: Arc::new(AtomicUsize::new(0)),
}
}
pub fn subscriber_count(&self) -> usize {
self.subscriber_count.load(Ordering::Acquire)
}
pub fn subscribe(&self) -> TraceSubscription {
let receiver = self.sender.subscribe();
self.subscriber_count.fetch_add(1, Ordering::AcqRel);
TraceSubscription {
receiver,
subscriber_count: Arc::clone(&self.subscriber_count),
}
}
pub fn emit(&self, build: impl FnOnce() -> TraceEvent) -> bool {
if self.subscriber_count() == 0 {
return false;
}
self.sender.send(Arc::new(build())).is_ok()
}
}
impl Default for TraceBus {
fn default() -> Self {
Self::new(DEFAULT_TRACE_BUS_CAPACITY)
}
}
#[derive(Debug)]
pub struct TraceSubscription {
receiver: broadcast::Receiver<Arc<TraceEvent>>,
subscriber_count: Arc<AtomicUsize>,
}
impl TraceSubscription {
pub async fn recv(&mut self) -> Result<Arc<TraceEvent>, broadcast::error::RecvError> {
self.receiver.recv().await
}
pub fn try_recv(&mut self) -> Result<Arc<TraceEvent>, broadcast::error::TryRecvError> {
self.receiver.try_recv()
}
}
impl Drop for TraceSubscription {
fn drop(&mut self) {
self.subscriber_count.fetch_sub(1, Ordering::AcqRel);
}
}
pub fn global_trace_bus() -> &'static TraceBus {
GLOBAL_TRACE_BUS.get_or_init(TraceBus::default)
}
pub fn subscribe_trace_events() -> TraceSubscription {
global_trace_bus().subscribe()
}
pub fn trace_emit(build: impl FnOnce() -> TraceEvent) -> bool {
global_trace_bus().emit(build)
}
pub fn trace_subscriber_count() -> usize {
global_trace_bus().subscriber_count()
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::AtomicUsize;
#[test]
fn trace_emit_skips_builder_without_subscribers() {
let bus = TraceBus::new(4);
let built = AtomicUsize::new(0);
let sent = bus.emit(|| {
built.fetch_add(1, Ordering::Relaxed);
TraceEvent::new(TraceKind::Heal, TraceFunc::HealTask)
});
assert!(!sent);
assert_eq!(built.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn trace_subscriber_receives_event() {
let bus = TraceBus::new(4);
let mut subscription = bus.subscribe();
assert!(bus.emit(|| {
TraceEvent::new(TraceKind::Heal, TraceFunc::HealObject)
.with_bucket("bucket")
.with_object("object")
.with_duration(Duration::from_millis(7))
.with_bytes(11)
.with_attr("dry", true)
}));
let event = subscription
.recv()
.await
.expect("subscriber should receive emitted trace event");
assert_eq!(event.kind, TraceKind::Heal);
assert_eq!(event.func, TraceFunc::HealObject);
assert_eq!(event.bucket.as_deref(), Some("bucket"));
assert_eq!(event.object.as_deref(), Some("object"));
assert_eq!(event.duration, Duration::from_millis(7));
assert_eq!(event.bytes, 11);
assert_eq!(
event.attrs.as_slice(),
&[TraceAttr {
key: "dry",
value: TraceVal::Bool(true)
}]
);
}
#[test]
fn trace_subscription_drop_decrements_count() {
let bus = TraceBus::new(4);
let subscription = bus.subscribe();
assert_eq!(bus.subscriber_count(), 1);
drop(subscription);
assert_eq!(bus.subscriber_count(), 0);
}
#[tokio::test]
async fn lagged_subscriber_drops_events_without_blocking_publishers() {
let bus = TraceBus::new(2);
let mut subscription = bus.subscribe();
for index in 0_u64..4 {
assert!(bus.emit(|| { TraceEvent::new(TraceKind::Scanner, TraceFunc::ScannerFolder).with_attr("index", index) }));
}
let err = subscription
.recv()
.await
.expect_err("receiver should observe lag instead of blocking publishers");
assert!(matches!(err, broadcast::error::RecvError::Lagged(_)));
}
}
+2 -3
View File
@@ -14,9 +14,8 @@
//! Shared backpressure policy type.
//!
//! The runtime backpressure implementation (byte-watermark pipes and
//! monitors) lives in `rustfs/src/storage/backpressure.rs`; this module only
//! carries the watermark policy type that implementation shares.
//! This module only carries the watermark policy; the admission primitive it
//! projects into lives in `rustfs-io-core`.
use rustfs_io_core::BackpressureConfig as CoreBackpressureConfig;
+37
View File
@@ -177,3 +177,40 @@ pub const DEFAULT_HEAL_MAINLINE_WRITE_UTILIZATION_HIGH_PERCENT: usize = 80;
/// Default foreground pressure recheck delay for heal scheduler, in milliseconds.
pub const DEFAULT_HEAL_MAINLINE_MAX_SLEEP_MS: u64 = 250;
/// Environment variable that toggles the MRF (mission repair feed) intent
/// pipeline: error paths deliver repair intents to the heal runtime, and
/// unconsumed intents are replayed from the durable journal after a restart.
pub const ENV_HEAL_MRF_ENABLE: &str = "RUSTFS_HEAL_MRF_ENABLE";
/// Environment variable for the MRF in-memory queue capacity (intent count).
pub const ENV_HEAL_MRF_QUEUE_SIZE: &str = "RUSTFS_HEAL_MRF_QUEUE_SIZE";
/// Environment variable for the MRF journal byte budget. The journal is
/// compacted once its on-disk size crosses this bound.
pub const ENV_HEAL_MRF_JOURNAL_MAX_BYTES: &str = "RUSTFS_HEAL_MRF_JOURNAL_MAX_BYTES";
/// Environment variable for the MRF journal replay batch size (intents per
/// replay push round).
pub const ENV_HEAL_MRF_REPLAY_BATCH: &str = "RUSTFS_HEAL_MRF_REPLAY_BATCH";
/// Default behavior keeps the MRF intent pipeline enabled.
pub const DEFAULT_HEAL_MRF_ENABLE: bool = true;
/// Default MRF queue capacity (matches MinIO's 100k MRF list ceiling).
pub const DEFAULT_HEAL_MRF_QUEUE_SIZE: usize = 100_000;
/// Default MRF journal byte budget (8 MiB), mirroring the channel payload cap.
pub const DEFAULT_HEAL_MRF_JOURNAL_MAX_BYTES: usize = 8 * 1024 * 1024;
/// Default MRF replay batch size.
pub const DEFAULT_HEAL_MRF_REPLAY_BATCH: usize = 256;
/// Environment variable selecting how admin heal starts behave when the
/// requested path overlaps an already running or queued heal: `merge`
/// (default, keep today's dedup/merge semantics) or `minio_error` (return a
/// typed already-running / overlapping-paths rejection like madmin).
pub const ENV_HEAL_OVERLAP_POLICY: &str = "RUSTFS_HEAL_OVERLAP_POLICY";
/// Default overlap policy: merge duplicate/overlapping requests.
pub const DEFAULT_HEAL_OVERLAP_POLICY: &str = "merge";
+25
View File
@@ -234,6 +234,31 @@ pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_A
/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
/// Enable foreground PutObject request admission.
///
/// This is an experimental, default-off foreground write backpressure gate for
/// strict commit tail investigations. When disabled, PUTs follow the legacy
/// path and only the existing request counters are updated.
pub const ENV_PUT_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE";
pub const DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE: bool = false;
/// Maximum foreground PutObject requests admitted concurrently per process.
///
/// The limit is used only when [`ENV_PUT_FOREGROUND_ADMISSION_ENABLE`] is true.
/// A value of `0` disables the gate even when the enable flag is present, so a
/// partially configured rollout cannot reject every PUT.
pub const ENV_PUT_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT";
pub const DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT: usize = 0;
/// Time in milliseconds a foreground PutObject waits for an admission permit.
///
/// Once this timeout expires the request fails before body ingest/storage
/// mutation with S3 `SlowDown`/503. `0` means fail fast when the limit is full.
pub const ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
/// Environment variable for minimum GetObject timeout in seconds.
///
/// When dynamic timeout calculation is enabled, this is the minimum timeout
-9
View File
@@ -228,15 +228,6 @@ pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4;
/// Default object interval for cooperative scanner yields.
pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
/// Compatibility flag kept for Patch 3 rollback windows.
///
/// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
/// When this flag is enabled, RustFS logs a warning and continues to use enqueue-based heal.
pub const ENV_SCANNER_INLINE_HEAL_ENABLE: &str = "RUSTFS_SCANNER_INLINE_HEAL_ENABLE";
/// Default inline scanner heal compatibility mode.
pub const DEFAULT_SCANNER_INLINE_HEAL_ENABLE: bool = false;
/// Scanner speed preset controlling throttling behavior.
///
/// Each preset defines three parameters:
-8
View File
@@ -203,14 +203,6 @@ mod tests {
assert!(result.is_err());
}
#[test]
fn test_source_does_not_embed_private_key() {
let source = include_str!("license_token.rs");
let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
assert!(!source.contains(&forbidden));
}
#[test]
fn test_parse_signed_license_token_rejects_invalid_token() {
let mut rng = rand::rng();
+374 -55
View File
@@ -317,15 +317,15 @@ pub struct SizeSummary {
/// Number of delete markers
pub delete_markers: usize,
/// Replicated size
pub replicated_size: usize,
pub replicated_size: i64,
/// Replicated count
pub replicated_count: usize,
/// Pending size
pub pending_size: usize,
pub pending_size: i64,
/// Failed size
pub failed_size: usize,
pub failed_size: i64,
/// Replica size
pub replica_size: usize,
pub replica_size: i64,
/// Replica count
pub replica_count: usize,
/// Pending count
@@ -334,19 +334,21 @@ pub struct SizeSummary {
pub failed_count: usize,
/// Replication target stats
pub repl_target_stats: HashMap<String, ReplTargetSizeSummary>,
/// Per-tier accounting, keyed by storage class or remote tier name
pub tier_stats: HashMap<String, TierStats>,
}
/// Replication target size summary
#[derive(Debug, Default, Clone)]
pub struct ReplTargetSizeSummary {
/// Replicated size
pub replicated_size: usize,
pub replicated_size: i64,
/// Replicated count
pub replicated_count: usize,
/// Pending size
pub pending_size: usize,
pub pending_size: i64,
/// Failed size
pub failed_size: usize,
pub failed_size: i64,
/// Pending count
pub pending_count: usize,
/// Failed count
@@ -710,28 +712,6 @@ impl DataUsageEntry {
self.children.insert(hash.key());
}
pub fn add_sizes(&mut self, summary: &SizeSummary) {
self.size += summary.total_size;
self.versions += summary.versions;
self.delete_markers += summary.delete_markers;
self.obj_sizes.add(summary.total_size as u64);
self.obj_versions.add(summary.versions as u64);
let replication_stats = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
replication_stats.replica_size += summary.replica_size as u64;
replication_stats.replica_count += summary.replica_count as u64;
for (arn, st) in &summary.repl_target_stats {
let tgt_stat = replication_stats.targets.entry(arn.to_string()).or_default();
tgt_stat.pending_size += st.pending_size as u64;
tgt_stat.failed_size += st.failed_size as u64;
tgt_stat.replicated_size += st.replicated_size as u64;
tgt_stat.replicated_count += st.replicated_count as u64;
tgt_stat.failed_count += st.failed_count as u64;
tgt_stat.pending_count += st.pending_count as u64;
}
}
pub fn merge(&mut self, other: &DataUsageEntry) {
self.objects += other.objects;
self.versions += other.versions;
@@ -870,6 +850,157 @@ pub struct DataUsageCacheInfo {
pub snapshot_complete: bool,
}
/// Prefix-level usage over a raw entry map — the shared core behind
/// [`DataUsageCache::prefix_usage`], usable by any cache-shaped reader (the
/// scanner's writer-side cache has the same map type).
///
/// Cache keys are cleaned literal paths (`bucket/pre/fix`), so sub-prefix
/// names come straight off the child keys — no reverse mapping exists or is
/// needed. A compacted prefix carries its aggregate but no children, which
/// the `compacted` flag reports so callers can say why the breakdown is
/// empty. `truncated` is set when the breakdown exceeded `max_entries` and
/// was cut (largest first).
pub fn prefix_usage_in_cache(
cache: &HashMap<String, DataUsageEntry>,
bucket: &str,
prefix: &str,
max_entries: usize,
) -> Option<PrefixUsageQuery> {
let prefix = prefix.trim_matches('/');
let root = if prefix.is_empty() {
bucket.to_string()
} else {
format!("{bucket}/{prefix}")
};
let entry = cache.get(&hash_path(&root).key())?.clone();
let usage = PrefixUsageSummary::from_entry(&flatten_entry(cache, &entry, 0)?);
let child_prefix = format!("{root}/");
let mut sub_prefixes: Vec<PrefixUsageEntry> = entry
.children
.iter()
.filter_map(|child_key| {
let child = cache.get(child_key)?;
let child_flat = flatten_entry(cache, child, 1)?;
// Child keys are literal `bucket/pre/name` paths; a trailing
// slash marks a directory object and is display-only here.
let name = child_key
.strip_prefix(child_prefix.as_str())
.unwrap_or(child_key.as_str())
.trim_end_matches('/')
.to_string();
Some(PrefixUsageEntry {
prefix: name,
usage: PrefixUsageSummary::from_entry(&child_flat),
})
})
.collect();
sub_prefixes.sort_by(|left, right| {
right
.usage
.size
.cmp(&left.usage.size)
.then_with(|| left.prefix.cmp(&right.prefix))
});
let truncated = sub_prefixes.len() > max_entries;
sub_prefixes.truncate(max_entries);
Some(PrefixUsageQuery {
usage,
compacted: entry.compacted,
truncated,
sub_prefixes,
})
}
/// Maximum subtree depth [`flatten_entry`] will walk before declaring the
/// cache corrupt — the same bound the scanner's checked flatten uses.
const PREFIX_USAGE_MAX_DEPTH: usize = 1024;
/// Flatten one entry's subtree into an aggregate: the free-function twin of
/// [`DataUsageCache::flatten`], carrying the scanner checked-flatten
/// hardening so a corrupt cache (cycles, over-deep trees, overflowing
/// counters) yields `None` instead of unbounded recursion or wrapped totals.
fn flatten_entry(cache: &HashMap<String, DataUsageEntry>, root: &DataUsageEntry, depth: usize) -> Option<DataUsageEntry> {
if depth > PREFIX_USAGE_MAX_DEPTH {
return None;
}
let mut flattened = DataUsageEntry::default();
if !flattened.checked_merge(root) {
return None;
}
flattened.compacted = root.compacted;
// The root itself is not pre-seeded: it is merged above, and a corrupt
// child edge pointing back at the root's own key is still terminated by
// the visited set on first encounter.
let mut visited: HashSet<&str> = HashSet::new();
let mut pending: Vec<(&String, usize)> = root.children.iter().map(|child| (child, depth + 1)).collect();
while let Some((key, child_depth)) = pending.pop() {
if child_depth > PREFIX_USAGE_MAX_DEPTH || !visited.insert(key.as_str()) {
return None;
}
let entry = cache.get(key)?;
if !flattened.checked_merge(entry) {
return None;
}
pending.extend(entry.children.iter().map(|child| (child, child_depth + 1)));
}
flattened.children.clear();
Some(flattened)
}
/// Flattened counters of one prefix subtree, as returned by
/// [`DataUsageCache::prefix_usage`].
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct PrefixUsageSummary {
pub size: u64,
pub objects: u64,
pub versions: u64,
pub delete_markers: u64,
}
impl PrefixUsageSummary {
fn from_entry(entry: &DataUsageEntry) -> Self {
Self {
size: entry.size as u64,
objects: entry.objects as u64,
versions: entry.versions as u64,
delete_markers: entry.delete_markers as u64,
}
}
/// Add another set's counters into this one (entries are partitioned by
/// set, so per-set results sum).
pub fn merge(&mut self, other: &Self) {
self.size = self.size.saturating_add(other.size);
self.objects = self.objects.saturating_add(other.objects);
self.versions = self.versions.saturating_add(other.versions);
self.delete_markers = self.delete_markers.saturating_add(other.delete_markers);
}
}
/// One first-level sub-prefix row of a [`PrefixUsageQuery`].
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize)]
pub struct PrefixUsageEntry {
pub prefix: String,
pub usage: PrefixUsageSummary,
}
/// Result of [`DataUsageCache::prefix_usage`].
#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct PrefixUsageQuery {
pub usage: PrefixUsageSummary,
/// The prefix entry was compacted by the scanner: its aggregate is valid
/// but no sub-prefix breakdown exists on disk.
pub compacted: bool,
/// The breakdown had more entries than `max_entries`; the largest remain.
pub truncated: bool,
pub sub_prefixes: Vec<PrefixUsageEntry>,
}
/// Read-only projection of a scanner-written `.usage-cache.bin` file.
///
/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
@@ -997,6 +1128,21 @@ impl DataUsageCache {
}
}
/// Prefix-level usage for one bucket subtree, plus the one-level
/// breakdown below it (rustfs/backlog#1872, MinIO
/// `loadPrefixUsageFromBackend` parity and beyond: arbitrary prefixes and
/// full counters instead of first-level sizes only).
///
/// Cache keys are cleaned literal paths (`bucket/pre/fix`), so sub-prefix
/// names come straight off the child keys — no reverse mapping exists or
/// is needed. A compacted prefix carries its aggregate but no children,
/// which the `compacted` flag reports so callers can say why the
/// breakdown is empty. `truncated` is set when the breakdown exceeded
/// `max_entries` and was cut (largest first).
pub fn prefix_usage(&self, bucket: &str, prefix: &str, max_entries: usize) -> Option<PrefixUsageQuery> {
prefix_usage_in_cache(&self.cache, bucket, prefix, max_entries)
}
pub fn force_compact(&mut self, limit: usize) {
if self.cache.len() < limit {
return;
@@ -1556,14 +1702,6 @@ impl BucketUsageInfo {
}
/// Add size summary to this bucket usage
pub fn add_size_summary(&mut self, summary: &SizeSummary) {
self.size += summary.total_size as u64;
self.versions_count += summary.versions as u64;
self.delete_markers_count += summary.delete_markers as u64;
self.replica_size += summary.replica_size as u64;
self.replica_count += summary.replica_count as u64;
}
/// Merge another BucketUsageInfo into this one
pub fn merge(&mut self, other: &BucketUsageInfo) {
self.size += other.size;
@@ -1609,29 +1747,32 @@ impl SizeSummary {
Self::default()
}
/// Add another SizeSummary to this one
/// Add another SizeSummary to this one.
///
/// Saturating throughout: a scan that overflows a counter should report the
/// ceiling rather than panic in a debug build or wrap in a release one.
pub fn add(&mut self, other: &SizeSummary) {
self.total_size += other.total_size;
self.versions += other.versions;
self.delete_markers += other.delete_markers;
self.replicated_size += other.replicated_size;
self.replicated_count += other.replicated_count;
self.pending_size += other.pending_size;
self.failed_size += other.failed_size;
self.replica_size += other.replica_size;
self.replica_count += other.replica_count;
self.pending_count += other.pending_count;
self.failed_count += other.failed_count;
self.total_size = self.total_size.saturating_add(other.total_size);
self.versions = self.versions.saturating_add(other.versions);
self.delete_markers = self.delete_markers.saturating_add(other.delete_markers);
self.replicated_size = self.replicated_size.saturating_add(other.replicated_size);
self.replicated_count = self.replicated_count.saturating_add(other.replicated_count);
self.pending_size = self.pending_size.saturating_add(other.pending_size);
self.failed_size = self.failed_size.saturating_add(other.failed_size);
self.replica_size = self.replica_size.saturating_add(other.replica_size);
self.replica_count = self.replica_count.saturating_add(other.replica_count);
self.pending_count = self.pending_count.saturating_add(other.pending_count);
self.failed_count = self.failed_count.saturating_add(other.failed_count);
// Merge replication target stats
for (target, stats) in &other.repl_target_stats {
let entry = self.repl_target_stats.entry(target.clone()).or_default();
entry.replicated_size += stats.replicated_size;
entry.replicated_count += stats.replicated_count;
entry.pending_size += stats.pending_size;
entry.failed_size += stats.failed_size;
entry.pending_count += stats.pending_count;
entry.failed_count += stats.failed_count;
entry.replicated_size = entry.replicated_size.saturating_add(stats.replicated_size);
entry.replicated_count = entry.replicated_count.saturating_add(stats.replicated_count);
entry.pending_size = entry.pending_size.saturating_add(stats.pending_size);
entry.failed_size = entry.failed_size.saturating_add(stats.failed_size);
entry.pending_count = entry.pending_count.saturating_add(stats.pending_count);
entry.failed_count = entry.failed_count.saturating_add(stats.failed_count);
}
}
}
@@ -1898,6 +2039,126 @@ mod tests {
);
}
/// Build a cache shaped like `bucket/{a,b/{c,d}},bucket/loose` with
/// distinct counters so aggregation is observable.
fn prefix_usage_fixture_cache() -> DataUsageCache {
let mut cache = DataUsageCache::default();
let mut insert = |path: &str, parent: &str, size: usize, objects: usize, versions: usize, delete_markers: usize| {
cache.replace(
path,
parent,
DataUsageEntry {
size,
objects,
versions,
delete_markers,
..Default::default()
},
);
};
insert("bucket", "", 0, 0, 0, 0);
insert("bucket/a", "bucket", 100, 1, 1, 0);
insert("bucket/b", "bucket", 0, 0, 0, 0);
insert("bucket/b/c", "bucket/b", 200, 2, 2, 1);
insert("bucket/b/d", "bucket/b", 40, 1, 3, 0);
insert("bucket/loose", "bucket", 10, 1, 1, 1);
cache
}
#[test]
fn prefix_usage_aggregates_bucket_root_and_one_level_below() {
let cache = prefix_usage_fixture_cache();
let root = cache
.prefix_usage("bucket", "", 100)
.expect("root query must find the bucket entry");
assert_eq!(root.usage.size, 350, "root aggregate flattens the whole subtree");
assert_eq!(root.usage.objects, 5);
assert_eq!(root.usage.versions, 7);
assert_eq!(root.usage.delete_markers, 2);
assert!(!root.compacted);
assert!(!root.truncated);
// Breakdown is one level: b (240) before a (100) before loose (10),
// each flattened to its own subtree total.
let names: Vec<(&str, u64)> = root
.sub_prefixes
.iter()
.map(|entry| (entry.prefix.as_str(), entry.usage.size))
.collect();
assert_eq!(names, vec![("b", 240), ("a", 100), ("loose", 10)]);
}
#[test]
fn prefix_usage_drills_into_arbitrary_prefixes() {
let cache = prefix_usage_fixture_cache();
let b = cache.prefix_usage("bucket", "b", 100).expect("nested prefix must resolve");
assert_eq!(b.usage.size, 240);
assert_eq!(b.usage.versions, 5);
let names: Vec<&str> = b.sub_prefixes.iter().map(|entry| entry.prefix.as_str()).collect();
assert_eq!(names, vec!["c", "d"]);
// Prefix slashes are normalized away.
let slashed = cache.prefix_usage("bucket", "/b/", 100).expect("slash-insensitive lookup");
assert_eq!(slashed.usage.size, 240);
assert!(cache.prefix_usage("bucket", "absent", 100).is_none(), "unknown prefix must be a miss");
assert!(cache.prefix_usage("other", "", 100).is_none(), "unknown bucket must be a miss");
}
#[test]
fn prefix_usage_reports_and_respects_truncation() {
let cache = prefix_usage_fixture_cache();
let capped = cache.prefix_usage("bucket", "", 2).expect("root query");
assert!(capped.truncated, "three children capped to two must flag truncation");
let names: Vec<&str> = capped.sub_prefixes.iter().map(|entry| entry.prefix.as_str()).collect();
assert_eq!(names, vec!["b", "a"], "largest prefixes survive the cut");
}
#[test]
fn prefix_usage_marks_compacted_entries() {
let mut cache = DataUsageCache::default();
cache.replace(
"bucket",
"",
DataUsageEntry {
size: 999,
objects: 9,
compacted: true,
..Default::default()
},
);
let compacted = cache.prefix_usage("bucket", "", 100).expect("compacted root resolves");
assert!(compacted.compacted, "compaction must be visible to callers");
assert_eq!(compacted.usage.size, 999);
assert!(compacted.sub_prefixes.is_empty(), "a compacted entry carries no children");
}
#[test]
fn prefix_usage_rejects_cyclic_and_dangling_caches() {
// A self-referencing child (corrupt cache) must yield a miss for the
// whole query, not unbounded recursion.
let mut cache = prefix_usage_fixture_cache();
if let Some(entry) = cache.cache.get_mut("bucket/b") {
entry.children.insert("bucket/b".to_string());
}
assert!(cache.prefix_usage("bucket", "b", 100).is_none(), "a cyclic subtree must be rejected");
// The unaffected sibling still answers.
assert!(cache.prefix_usage("bucket", "a", 100).is_some());
// A child key with no entry (dangling link) is rejected rather than
// silently dropped: half a tree would under-report usage.
let mut dangling = prefix_usage_fixture_cache();
if let Some(entry) = dangling.cache.get_mut("bucket/b") {
entry.children.insert("bucket/b/ghost".to_string());
}
assert!(
dangling.prefix_usage("bucket", "b", 100).is_none(),
"a dangling child link must be rejected"
);
}
#[test]
fn hash_path_uses_portable_slash_semantics() {
for (input, expected) in [
@@ -2057,6 +2318,64 @@ mod tests {
assert_eq!(usage1.versions_count, 15);
}
#[test]
fn size_summary_add_saturates_instead_of_overflowing() {
// The scanner folds one summary per object into a per-prefix total, so a
// counter at its ceiling must stay there rather than panic in a debug
// build or wrap in a release one (backlog#1828).
let mut summary = SizeSummary {
total_size: usize::MAX,
versions: usize::MAX,
replicated_size: i64::MAX,
pending_size: i64::MAX,
failed_size: i64::MAX,
replica_size: i64::MAX,
..Default::default()
};
summary.repl_target_stats.insert(
"arn".to_string(),
ReplTargetSizeSummary {
replicated_size: i64::MAX,
pending_size: i64::MAX,
failed_size: i64::MAX,
..Default::default()
},
);
let mut increment = SizeSummary {
total_size: 1,
versions: 1,
replicated_size: 1,
pending_size: 1,
failed_size: 1,
replica_size: 1,
..Default::default()
};
increment.repl_target_stats.insert(
"arn".to_string(),
ReplTargetSizeSummary {
replicated_size: 1,
pending_size: 1,
failed_size: 1,
..Default::default()
},
);
summary.add(&increment);
assert_eq!(summary.total_size, usize::MAX);
assert_eq!(summary.versions, usize::MAX);
assert_eq!(summary.replicated_size, i64::MAX);
assert_eq!(summary.pending_size, i64::MAX);
assert_eq!(summary.failed_size, i64::MAX);
assert_eq!(summary.replica_size, i64::MAX);
let target = summary.repl_target_stats.get("arn").expect("target survives the merge");
assert_eq!(target.replicated_size, i64::MAX);
assert_eq!(target.pending_size, i64::MAX);
assert_eq!(target.failed_size, i64::MAX);
}
#[test]
fn test_size_summary_add() {
let mut summary1 = SizeSummary::new();
-5
View File
@@ -38,7 +38,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use std::io::Read;
use std::process::{Command, Stdio};
@@ -162,7 +161,6 @@ mod tests {
/// A fully authenticated but non-admin credential must be rejected with
/// `403 AccessDenied` on an admin API, while the root credential succeeds.
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn non_admin_credential_denied_on_admin_api() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -198,7 +196,6 @@ mod tests {
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn non_admin_credential_denied_on_manual_transition_run() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -326,7 +323,6 @@ mod tests {
/// credential is accepted and the old one is rejected, on both the S3 data
/// plane and the admin plane.
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn root_credential_rotation_takes_effect() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -389,7 +385,6 @@ mod tests {
/// runtime. We capture the child's stdout/stderr directly (the shared
/// harness inherits stdio) and poll for the warning until it appears.
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn default_credentials_emit_startup_warning() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -33,7 +33,6 @@ use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::{Client, Config};
use reqwest::StatusCode;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
@@ -178,7 +177,6 @@ async fn assert_admin_status(
}
#[tokio::test]
#[serial]
async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestResult {
init_logging();
@@ -348,7 +346,6 @@ async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestRe
/// Full user -> policy -> service-account lifecycle, proving each management
/// call takes effect on the data plane, not just that the endpoint answers 200.
#[tokio::test]
#[serial]
async fn test_admin_user_policy_service_account_crud_lifecycle() -> TestResult {
init_logging();
@@ -573,7 +570,6 @@ async fn test_admin_user_policy_service_account_crud_lifecycle() -> TestResult {
/// non-admin credential with 403 AccessDenied (sec-4 assertion pattern; the
/// gate implementation itself is owned by sec-4 / admin_auth_test).
#[tokio::test]
#[serial]
async fn test_admin_iam_endpoints_deny_non_admin_credential() -> TestResult {
init_logging();
@@ -21,7 +21,6 @@ use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serde::Deserialize;
use serial_test::serial;
use std::error::Error;
use std::process::Command;
use tokio::time::{Duration, sleep, timeout};
@@ -100,7 +99,6 @@ fn offline_server_count(info: &InfoMessage) -> usize {
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn test_single_admin_timeout_does_not_immediately_mark_peer_offline() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -18,7 +18,6 @@
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::types::PublicAccessBlockConfiguration;
use serial_test::serial;
use tracing::info;
async fn setup_public_bucket(
@@ -73,7 +72,6 @@ async fn anonymous_get_object(
/// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
/// and no PublicAccessBlock configuration exists (ConfigNotFound).
#[tokio::test]
#[serial]
async fn test_anonymous_access_allowed_when_public_access_block_missing() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -100,7 +98,6 @@ async fn test_anonymous_access_allowed_when_public_access_block_missing() -> Res
/// Anonymous GetObject should be denied when RestrictPublicBuckets is true.
#[tokio::test]
#[serial]
async fn test_anonymous_access_denied_when_restrict_public_buckets_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -137,7 +134,6 @@ async fn test_anonymous_access_denied_when_restrict_public_buckets_enabled()
/// Anonymous GetObject should succeed when PublicAccessBlock exists but
/// RestrictPublicBuckets is explicitly false.
#[tokio::test]
#[serial]
async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -176,7 +172,6 @@ async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
/// reaches authorization through a fallback branch, and that branch has to apply the
/// same public-access gate as a direct grant.
#[tokio::test]
#[serial]
async fn ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -18,13 +18,11 @@
//! completely inert with default configuration.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use serial_test::serial;
use tracing::info;
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
#[tokio::test]
#[serial]
async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -87,7 +85,6 @@ async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResu
}
#[tokio::test]
#[serial]
async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -133,7 +130,6 @@ async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
}
#[tokio::test]
#[serial]
async fn api_rate_limit_stays_inert_by_default() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -24,7 +24,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::{pre_sign_v4, sign_v4};
use s3s::Body;
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::error::Error;
use std::io::{Cursor, Write};
@@ -339,7 +338,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_put_allows_content_encoding_by_default() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -367,7 +365,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_put_rejects_content_encoding_when_strict_mode_enabled() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -391,7 +388,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_put_with_aws_chunked_does_not_persist_content_encoding_by_default()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -427,7 +423,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_put_with_aws_chunked_and_effective_encoding_roundtrips_by_default()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -463,7 +458,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_put_with_aws_chunked_allowed_when_strict_mode_enabled() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -498,7 +492,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_put_with_aws_chunked_and_effective_encoding_rejects_when_strict_mode_enabled()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -529,7 +522,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_download_roundtrip_with_http_compression_enabled() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -591,7 +583,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_multipart_roundtrip_preserves_bytes() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -687,7 +678,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_multipart_get_ignores_empty_conditional_etag_headers() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -723,7 +713,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_multipart_with_aws_chunked_and_effective_encoding_roundtrips_by_default()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -753,7 +742,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_multipart_with_aws_chunked_allowed_when_strict_mode_enabled() -> Result<(), Box<dyn Error + Send + Sync>>
{
init_logging();
@@ -783,7 +771,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_archive_multipart_with_aws_chunked_and_effective_encoding_rejects_when_strict_mode_enabled()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -816,7 +803,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_presigned_get_and_reverse_proxy_preserve_multipart_bytes() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -24,11 +24,9 @@ mod tests {
};
use http::Method;
use http::header::CONTENT_TYPE;
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn test_dummy_bucket_compatibility_endpoints() {
init_logging();
info!("Starting test: dummy-compat bucket APIs should match S3-compatible behavior");
@@ -236,7 +234,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_dummy_bucket_compatibility_endpoints_no_such_bucket() {
init_logging();
info!("Starting test: dummy-compat bucket APIs should return NoSuchBucket for missing bucket");
@@ -392,7 +389,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_dummy_bucket_endpoints_http_contracts() {
init_logging();
info!("Starting test: dummy-compat bucket API HTTP contracts");
@@ -18,7 +18,6 @@
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::{Client, Config};
use serial_test::serial;
use tracing::info;
async fn create_user(
@@ -51,7 +50,6 @@ fn create_user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key:
}
#[tokio::test]
#[serial]
async fn test_bucket_policy_authenticated_user() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if !crate::common::awscurl_available() {
@@ -35,7 +35,6 @@ mod tests {
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use rustfs_data_usage::DataUsageInfo;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
use tracing::info;
@@ -59,7 +58,6 @@ mod tests {
/// 3. Query admin data usage API
/// 4. Verify object count > 0
#[tokio::test]
#[serial]
async fn test_bucket_object_count_updates_after_put() -> TestResult {
init_logging();
info!("RT-09: bucket object count updates after PUT");
@@ -126,7 +124,6 @@ mod tests {
/// Regression pattern: stats remain unchanged after objects are deleted
/// (rustfs#5615).
#[tokio::test]
#[serial]
async fn test_bucket_object_count_updates_after_delete() -> TestResult {
init_logging();
info!("RT-09b: bucket object count updates after DELETE");
@@ -220,7 +217,6 @@ mod tests {
/// Regression pattern: DataUsageInfo undercounts versioned bucket versions
/// and delete markers (rustfs#3898).
#[tokio::test]
#[serial]
async fn test_versioned_bucket_stats_count_all_versions() -> TestResult {
init_logging();
info!("RT-09c: versioned bucket stats count all versions");
@@ -26,7 +26,6 @@ mod tests {
use base64::Engine;
use md5::{Digest as Md5Digest, Md5};
use rustfs_rio::{Checksum, ChecksumType as RioChecksumType};
use serial_test::serial;
use sha2::Sha256;
use tracing::info;
@@ -90,7 +89,6 @@ mod tests {
/// PutObject with Content-MD5: upload succeeds and GetObject returns same content.
#[tokio::test]
#[serial]
async fn test_put_object_with_content_md5() {
init_logging();
info!("TEST: PutObject with Content-MD5");
@@ -126,7 +124,6 @@ mod tests {
/// PutObject with x-amz-checksum-sha256: upload succeeds and GetObject returns same content.
#[tokio::test]
#[serial]
async fn test_put_object_with_checksum_sha256() {
init_logging();
info!("TEST: PutObject with x-amz-checksum-sha256");
@@ -164,7 +161,6 @@ mod tests {
/// PutObject with a SHA256 checksum that does NOT match the body must be
/// rejected (BadDigest / checksum mismatch), NOT accepted with HTTP 200.
#[tokio::test]
#[serial]
async fn test_put_object_rejects_mismatched_sha256() {
init_logging();
info!("TEST: PutObject rejects mismatched x-amz-checksum-sha256 (issue #4341)");
@@ -212,7 +208,6 @@ mod tests {
/// After PutObject with a correct SHA256 checksum, HeadObject with
/// ChecksumMode=ENABLED must return that stored base64 SHA256 digest.
#[tokio::test]
#[serial]
async fn test_head_object_returns_stored_sha256() {
init_logging();
info!("TEST: HeadObject returns stored SHA256 with ChecksumMode=ENABLED (issue #4341)");
@@ -258,7 +253,6 @@ mod tests {
/// Multipart upload with checksum: CreateMultipartUpload, UploadPart(s) with checksum_sha256, CompleteMultipartUpload; then GetObject verifies content.
/// Uses part size >= 5MB (server minimum) for two parts.
#[tokio::test]
#[serial]
async fn test_multipart_upload_with_checksum() {
init_logging();
info!("TEST: MultipartUpload with checksum (checksum_sha256 on parts)");
@@ -356,7 +350,6 @@ mod tests {
/// Regression test for issue #2282:
/// CRC64NVME full-object checksum should match between direct PutObject and multipart upload.
#[tokio::test]
#[serial]
async fn test_crc64nvme_matches_between_put_object_and_multipart_upload() {
init_logging();
info!("TEST: CRC64NVME matches between direct PutObject and multipart upload");
@@ -492,7 +485,6 @@ mod tests {
/// value is rejected with BadDigest and nothing is stored. Full HEAD/GET header
/// echo round-trip is additionally exercised by the boto3+awscrt e2e.
#[tokio::test]
#[serial]
async fn test_additional_checksums_verify_on_write() {
init_logging();
info!("TEST: additional checksums (XXHash3/64/128, SHA-512, MD5) verify-on-write");
@@ -16,7 +16,6 @@ use crate::common::RustFSTestClusterEnvironment;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use bytes::Bytes;
use serial_test::serial;
use std::sync::Arc;
use tokio::sync::Barrier;
use tracing::{info, warn};
@@ -135,7 +134,6 @@ async fn run_race_iteration(
}
#[tokio::test]
#[serial]
async fn test_conditional_put_race_cluster() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
crate::common::init_logging();
info!("Starting conditional PUT race test with auto cluster");
@@ -192,7 +190,6 @@ async fn test_conditional_put_race_cluster() -> Result<(), Box<dyn std::error::E
}
#[tokio::test]
#[serial]
async fn test_conditional_put_basic_cluster() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
crate::common::init_logging();
info!("Starting basic conditional PUT test with auto cluster");
@@ -31,7 +31,6 @@
//! (toxiproxy / socket proxy) and 5GiB large-object budgets.
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use serial_test::serial;
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
@@ -58,7 +57,6 @@ async fn put_get_roundtrip(cluster: &RustFSTestClusterEnvironment, key: &str, pa
/// 4 nodes x 2 drives, single pool: the multi-drive layout boots and round-trips.
#[tokio::test]
#[serial]
async fn cluster_multidrive_single_pool_smoke() -> TestResult {
crate::common::init_logging();
@@ -81,7 +79,6 @@ async fn cluster_multidrive_single_pool_smoke() -> TestResult {
/// Two single-node pools, 2 drives each: the multi-pool layout boots and
/// round-trips. Every pool is a distinct erasure pool (`pool_idx` 0 and 1).
#[tokio::test]
#[serial]
async fn cluster_two_pool_smoke() -> TestResult {
crate::common::init_logging();
+79 -4
View File
@@ -32,6 +32,7 @@ use rustfs_signer::sign_v4;
use s3s::Body;
use std::ffi::OsStr;
use std::fs as stdfs;
use std::io::ErrorKind;
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Stdio};
use std::sync::Once;
@@ -51,6 +52,11 @@ pub(crate) const FAST_DATA_USAGE_SCANNER_ENV: &[(&str, &str)] =
&[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")];
pub const TEST_BUCKET: &str = "e2e-test-bucket";
const RUSTFS_FULL_FEATURE: &str = "full";
const TEST_PORT_MIN: u16 = 20_000;
const TEST_PORT_RANGE: u16 = 40_000;
const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);
fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
@@ -67,6 +73,64 @@ fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
}
struct PortAllocatorGuard;
impl PortAllocatorGuard {
async fn acquire() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
loop {
match stdfs::create_dir(TEST_PORT_LOCK_DIR) {
Ok(()) => return Ok(Self),
Err(err) if err.kind() == ErrorKind::AlreadyExists => {
remove_stale_port_allocator_lock();
sleep(Duration::from_millis(10)).await;
}
Err(err) => return Err(err.into()),
}
}
}
}
impl Drop for PortAllocatorGuard {
fn drop(&mut self) {
let _ = stdfs::remove_dir(TEST_PORT_LOCK_DIR);
}
}
fn advance_test_port(port: u16) -> u16 {
let offset = (port - TEST_PORT_MIN + 1) % TEST_PORT_RANGE;
TEST_PORT_MIN + offset
}
fn seeded_test_port() -> u16 {
let offset = (Uuid::new_v4().as_u128() % u128::from(TEST_PORT_RANGE)) as u16;
TEST_PORT_MIN + offset
}
fn read_next_test_port() -> u16 {
stdfs::read_to_string(TEST_PORT_COUNTER_PATH)
.ok()
.and_then(|value| value.trim().parse::<u16>().ok())
.filter(|port| (TEST_PORT_MIN..TEST_PORT_MIN + TEST_PORT_RANGE).contains(port))
.unwrap_or_else(seeded_test_port)
}
fn remove_stale_port_allocator_lock() {
let Ok(metadata) = stdfs::metadata(TEST_PORT_LOCK_DIR) else {
return;
};
let Ok(modified) = metadata.modified() else {
return;
};
if modified.elapsed().is_ok_and(|elapsed| elapsed > TEST_PORT_LOCK_STALE_AFTER) {
let _ = stdfs::remove_dir(TEST_PORT_LOCK_DIR);
}
}
fn write_next_test_port(port: u16) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
stdfs::write(TEST_PORT_COUNTER_PATH, port.to_string())?;
Ok(())
}
pub(crate) fn capture_command_logs(
command: &mut Command,
log_path: Option<&str>,
@@ -508,10 +572,21 @@ impl RustFSTestEnvironment {
/// Find an available port for the test
pub async fn find_available_port() -> Result<u16, Box<dyn std::error::Error + Send + Sync>> {
use std::net::TcpListener;
let listener = TcpListener::bind("127.0.0.1:0")?;
let port = listener.local_addr()?.port();
drop(listener);
Ok(port)
let _guard = PortAllocatorGuard::acquire().await?;
let mut next_port = read_next_test_port();
for _ in 0..TEST_PORT_RANGE {
let port = next_port;
next_port = advance_test_port(next_port);
write_next_test_port(next_port)?;
if let Ok(listener) = TcpListener::bind(("127.0.0.1", port)) {
drop(listener);
return Ok(port);
}
}
Err("no available E2E test port found".into())
}
/// Kill any existing RustFS processes
-7
View File
@@ -3,7 +3,6 @@
use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use serial_test::serial;
use std::fs;
use std::path::PathBuf;
use std::process::Command;
@@ -102,7 +101,6 @@ async fn start_rustfs_with_compression(env: &mut RustFSTestEnvironment) -> Resul
}
#[tokio::test]
#[serial]
async fn test_compression_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting compression roundtrip test");
@@ -230,7 +228,6 @@ async fn fetch_range(
/// (rustfs/rustfs#5957: multipart uploads previously bypassed disk compression
/// entirely).
#[tokio::test]
#[serial]
async fn test_compression_multipart_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting multipart compression roundtrip test");
@@ -349,7 +346,6 @@ const MPU_HIGH_RATIO_BUCKET: &str = "compression-mpu-high-ratio-bucket";
/// reproduced the mid-payload Pending truncation (rustfs/rustfs#5957). Every GET shape must return
/// the exact original bytes, and the stored size must show the data really was compressed.
#[tokio::test]
#[serial]
async fn test_compression_multipart_high_ratio_binary_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting multipart high-ratio binary compression roundtrip test");
@@ -446,7 +442,6 @@ const MPU_COPY_RANGE_LEN: usize = 5 * 1024 * 1024;
/// range must be decompressed on read and re-compressed into the destination part, so the final
/// object has to match "source prefix + uploaded tail" byte for byte.
#[tokio::test]
#[serial]
async fn test_compression_multipart_upload_part_copy_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting multipart upload-part-copy compression roundtrip test");
@@ -570,7 +565,6 @@ const MPU_THREE_PARTS_TAIL_SIZE: usize = 512 * 1024;
/// Three-part upload with uneven part sizes: each partNumber GET must map back to exactly one
/// compressed part stream, and a suffix range must resolve inside the trailing part.
#[tokio::test]
#[serial]
async fn test_compression_multipart_three_parts_part_number_gets() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting three-part multipart compression partNumber test");
@@ -689,7 +683,6 @@ async fn start_rustfs_with_compression_and_sse(
/// shape must still return the original plaintext bytes. Physical size must shrink because the
/// compression runs before encryption.
#[tokio::test]
#[serial]
async fn test_compression_multipart_sse_s3_roundtrip() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
use aws_sdk_s3::types::ServerSideEncryption;
@@ -18,7 +18,6 @@
//! concurrency — a queued connection is served only after a held one closes.
use crate::common::{RustFSTestEnvironment, init_logging};
use serial_test::serial;
use std::time::Duration;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
@@ -57,7 +56,6 @@ async fn read_response_head(stream: &mut TcpStream, dur: Duration) -> Option<Str
}
#[tokio::test]
#[serial]
async fn connection_cap_releases_permits_on_close() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -89,7 +87,6 @@ async fn open_and_stall(addr: &str) -> std::io::Result<TcpStream> {
}
#[tokio::test]
#[serial]
async fn connection_cap_blocks_excess_connections_until_permits_free() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -33,7 +33,6 @@
//! serve the unauthenticated console endpoints at all.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
@@ -58,7 +57,6 @@ async fn wait_for_console_ready(console_base: &str) -> Result<reqwest::Response,
}
#[tokio::test]
#[serial]
async fn test_console_over_the_wire_smoke() -> TestResult {
init_logging();
@@ -22,12 +22,10 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use tracing::info;
/// Verify Content-Encoding header roundtrips through PUT, GET, and HEAD operations
#[tokio::test]
#[serial]
async fn test_content_encoding_roundtrip() {
init_logging();
info!("Starting Content-Encoding roundtrip test");
@@ -105,7 +103,6 @@ mod tests {
/// Issue #1857: Content-Encoding "aws-chunked" is used by SigV4 streaming clients and must
/// not be stored or returned. Upload with aws-chunked and verify GET/HEAD do not return it.
#[tokio::test]
#[serial]
async fn test_content_encoding_aws_chunked_not_returned_issue_1857() {
init_logging();
info!("Issue #1857: aws-chunked must not be persisted or returned");
@@ -161,7 +158,6 @@ mod tests {
/// Issue #2475 / Route A: when aws-chunked is combined with an effective object encoding,
/// only the effective encoding should roundtrip through GET/HEAD.
#[tokio::test]
#[serial]
async fn test_content_encoding_aws_chunked_with_effective_encoding_roundtrip() {
init_logging();
info!("aws-chunked,gzip should persist only gzip");
@@ -30,7 +30,6 @@ mod tests {
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
use rustfs_rio::{Checksum, ChecksumType as RioChecksumType};
use serial_test::serial;
use sha2::{Digest, Sha256};
use tracing::info;
@@ -114,7 +113,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_copy_supports_all_checksum_algorithms() {
init_logging();
@@ -196,7 +194,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_copy_without_algorithm_preserves_every_supported_source_checksum() {
init_logging();
@@ -262,7 +259,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_copy_without_algorithm_preserves_composite_checksum_type() {
init_logging();
@@ -352,7 +348,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_copy_rejects_unknown_algorithm_without_destination_mutation() {
init_logging();
@@ -453,7 +448,6 @@ mod tests {
/// bytes, return it in `CopyObjectResult.ChecksumSHA256`, and persist it so a checksum-mode
/// HEAD on the destination returns the identical value.
#[tokio::test]
#[serial]
async fn test_copy_with_checksum_algorithm_returns_and_persists_sha256() {
init_logging();
info!("Issue #4996: CopyObject with ChecksumAlgorithm=SHA256 must return and persist the checksum");
@@ -523,7 +517,6 @@ mod tests {
/// No algorithm requested: when the source object already carries a checksum, the copy must
/// preserve it on the destination (AWS default), visible via a checksum-mode HEAD.
#[tokio::test]
#[serial]
async fn test_copy_without_algorithm_preserves_source_checksum() {
init_logging();
info!("Issue #4996: CopyObject without ChecksumAlgorithm must preserve the source object's checksum");
@@ -603,7 +596,6 @@ mod tests {
/// checksum-not-inherited path, and exercises the CRC32 code path (a different branch of
/// ChecksumType::from_string than SHA256).
#[tokio::test]
#[serial]
async fn test_copy_requested_algorithm_overrides_source_checksum() {
init_logging();
info!("Issue #4996: a requested CopyObject checksum algorithm must override the source object's algorithm");
@@ -22,11 +22,9 @@ mod tests {
use aws_sdk_s3::types::{
BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, MetadataDirective, StorageClass, VersioningConfiguration,
};
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn copy_object_standard_metadata_copy_replace_and_clear() {
init_logging();
info!("Issue #2789: self-copy metadata replacement must preserve object data");
@@ -300,7 +298,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn copy_object_replace_accepts_each_standard_field_independently() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
@@ -416,7 +413,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn copy_object_replace_handles_versioned_multipart_source() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
@@ -530,7 +526,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn invalid_replacement_metadata_does_not_mutate_destination() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
@@ -21,7 +21,6 @@ mod tests {
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, MetadataDirective, TaggingDirective, VersioningConfiguration};
use serial_test::serial;
use std::collections::BTreeMap;
async fn object_tags(client: &Client, bucket: &str, key: &str) -> BTreeMap<String, String> {
@@ -39,7 +38,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn copy_object_applies_copy_replace_and_empty_tagging_directives() {
init_logging();
let mut env = RustFSTestEnvironment::new()
@@ -305,7 +303,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn copy_object_tag_replacement_honors_request_tag_policy_denial() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -21,11 +21,9 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn test_self_copy_of_historical_version_restores_data_and_metadata() {
init_logging();
info!("Issue #4238: self-copy of a historical version must be allowed and preserve metadata");
@@ -165,7 +163,6 @@ mod tests {
/// version copied via `x-amz-copy-source-version-id` (SDK `CopySourceVersionId`), kept distinct
/// from the newly created destination `x-amz-version-id`.
#[tokio::test]
#[serial]
async fn test_copy_of_non_latest_source_version_returns_copy_source_version_id() {
init_logging();
info!("Issue #4976: versioned CopyObject must return x-amz-copy-source-version-id for the exact source version");
@@ -47,7 +47,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
/// Signed raw `PUT` copy request with an explicit copy-source conditional
@@ -84,7 +83,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_copy_source_if_unmodified_since_valid_and_invalid() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -25,13 +25,11 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::types::{BucketLocationConstraint, CreateBucketConfiguration};
use serial_test::serial;
use std::error::Error;
/// `CreateBucket` with a `LocationConstraint` body must pass SigV4 validation
/// and create the bucket, mirroring `minio-go` `MakeBucket(bucket, "us-east-1")`.
#[tokio::test]
#[serial]
async fn test_create_bucket_with_us_east_1_location_constraint() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -62,7 +60,6 @@ mod tests {
/// A plain `CreateBucket` (no body) must also succeed; guards against a
/// regression where an empty body would be hashed incorrectly during SigV4.
#[tokio::test]
#[serial]
async fn test_create_bucket_without_location_constraint() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
-3
View File
@@ -15,7 +15,6 @@
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use rustfs_data_usage::DataUsageInfo;
use serial_test::serial;
use tokio::time::{Duration, sleep};
use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, RustFSTestEnvironment, TEST_BUCKET, awscurl_get, init_logging};
@@ -60,7 +59,6 @@ where
/// Regression test for data usage accuracy (issue #1012).
/// Launches rustfs, writes 1000 objects, then asserts admin data usage reports the full count.
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "Starts a rustfs server and requires awscurl; enable when running full E2E"]
async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -118,7 +116,6 @@ async fn data_usage_reports_all_objects() -> Result<(), Box<dyn std::error::Erro
/// Regression test for issue #3898.
/// Versioned buckets should expose versions and delete markers through admin data usage.
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "Starts a rustfs server and requires awscurl; enable when running full E2E"]
async fn data_usage_reports_versioned_objects_and_delete_markers() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -18,7 +18,6 @@ mod tests {
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use serial_test::serial;
async fn create_versioned_bucket(client: &Client, bucket: &str) {
client
@@ -72,7 +71,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_versioning_only_delete_marker_has_minio_compatible_visibility_for_migration_proof() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
@@ -113,7 +111,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_versioning_delete_marker_plus_history_remains_visible_for_migration_proof() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
@@ -24,7 +24,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::net::TcpStream;
@@ -92,7 +91,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_delete_object_version_without_content_length_succeeds() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
info!("🧪 TEST: signed DELETE Object?versionId succeeds without Content-Length");
@@ -29,7 +29,6 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use serial_test::serial;
use tracing::info;
fn create_s3_client(env: &RustFSTestEnvironment) -> Client {
@@ -42,7 +41,6 @@ mod tests {
/// a versioned bucket, calling `list_object_versions` **immediately** (with
/// no sleep) returns the newly-created DeleteMarker with `is_latest = true`.
#[tokio::test]
#[serial]
async fn test_delete_objects_delete_marker_immediately_visible() {
init_logging();
info!("🧪 TEST: DeleteMarker from delete_objects is immediately visible via list_object_versions");
@@ -190,7 +188,6 @@ mod tests {
/// a single `delete_objects` call all have their delete markers visible
/// immediately afterwards.
#[tokio::test]
#[serial]
async fn test_delete_objects_multiple_keys_delete_markers_immediately_visible() {
init_logging();
info!("🧪 TEST: Multiple delete markers from delete_objects are immediately visible");
@@ -33,7 +33,6 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, Delete, ObjectIdentifier, VersioningConfiguration};
use serial_test::serial;
use std::error::Error;
use tracing::info;
@@ -51,7 +50,6 @@ mod tests {
/// 4. Verify the object is NOT in LIST
/// 5. Verify HEAD returns 404
#[tokio::test]
#[serial]
async fn test_delete_removes_object_from_list() -> TestResult {
init_logging();
info!("RT-05: delete removes object from list");
@@ -132,7 +130,6 @@ mod tests {
/// Regression pattern: batch delete returns success but some objects
/// remain in LIST.
#[tokio::test]
#[serial]
async fn test_batch_delete_removes_all_objects() -> TestResult {
init_logging();
info!("RT-05c: batch delete removes all objects");
@@ -212,7 +209,6 @@ mod tests {
/// Covers the pattern where permanent deletion of a specific version
/// fails with FileAccessDenied (rustfs#4978).
#[tokio::test]
#[serial]
async fn test_versioned_permanent_delete() -> TestResult {
init_logging();
info!("RT-05d: versioned permanent delete");
@@ -283,7 +279,6 @@ mod tests {
/// Covers the pattern where creating a delete marker and then listing
/// versions shows incorrect state (rustfs#760).
#[tokio::test]
#[serial]
async fn test_versioned_delete_marker_and_list_consistency() -> TestResult {
init_logging();
info!("RT-05e: versioned delete marker and list consistency");
@@ -379,7 +374,6 @@ mod tests {
/// Regression pattern: after delete, the object data files remain on disk
/// (rustfs#5029: Node Does Not Remove Files After Reconnect).
#[tokio::test]
#[serial]
async fn test_delete_removes_object_head_returns_404() -> TestResult {
init_logging();
info!("RT-05f: delete → HEAD 404 consistency");
@@ -32,7 +32,6 @@
mod tests {
use crate::common::{RustFSTestClusterEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::error::Error;
use tokio::time::{Duration, sleep};
use tracing::info;
@@ -50,7 +49,6 @@ mod tests {
/// 3. Verify all nodes report healthy
/// 4. Verify S3 operations work through any node
#[tokio::test]
#[serial]
async fn test_four_node_cluster_startup_and_health() -> TestResult {
init_logging();
info!("RT-10: 4-node cluster startup and health");
@@ -103,7 +101,6 @@ mod tests {
/// Regression pattern: after a node restart, it cannot rejoin the cluster
/// or enters a faulty state (rustfs#2601).
#[tokio::test]
#[serial]
async fn test_cluster_survives_node_restart() -> TestResult {
init_logging();
info!("RT-10b: cluster survives node restart");
@@ -168,7 +165,6 @@ mod tests {
/// Regression pattern: bucket metadata is not replicated to all nodes,
/// causing NoSuchBucket errors on some nodes (rustfs#3191).
#[tokio::test]
#[serial]
async fn test_bucket_visible_from_all_nodes() -> TestResult {
init_logging();
info!("RT-10c: bucket visible from all nodes");
@@ -23,7 +23,6 @@ use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{Delete, ObjectIdentifier, Tag, Tagging};
use aws_sdk_s3::{Client, Config};
use serial_test::serial;
use tracing::info;
use uuid::Uuid;
@@ -174,7 +173,6 @@ async fn cleanup_bucket_and_object(admin: &Client, bucket: &str, key: &str) {
/// IAM identity policy: GetObject allowed only when `s3:ExistingObjectTag/security` == `public`.
#[tokio::test]
#[serial]
async fn test_e2e_iam_policy_existing_object_tag_get_object() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if !awscurl_available() {
@@ -233,7 +231,6 @@ async fn test_e2e_iam_policy_existing_object_tag_get_object() -> Result<(), Box<
/// Bucket policy: same `ExistingObjectTag` condition; user has no canned IAM policy attached.
#[tokio::test]
#[serial]
async fn test_e2e_bucket_policy_existing_object_tag_get_object() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if !awscurl_available() {
@@ -295,7 +292,6 @@ async fn test_e2e_bucket_policy_existing_object_tag_get_object() -> Result<(), B
/// STS `AssumeRole` with inline `Policy` (session policy): GetObject only when `ExistingObjectTag/security` is `public`.
#[tokio::test]
#[serial]
async fn test_e2e_sts_assume_role_session_policy_existing_object_tag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if !awscurl_available() {
@@ -372,7 +368,6 @@ async fn test_e2e_sts_assume_role_session_policy_existing_object_tag() -> Result
/// STS inline session policy: DeleteObjects must evaluate `s3:DeleteObject` per requested object key.
#[tokio::test]
#[serial]
async fn test_e2e_sts_session_policy_delete_objects_object_prefix_only() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if !awscurl_available() {
+2 -2
View File
@@ -4,8 +4,8 @@ This module is the shared failure-injection boundary for replication end-to-end
`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions. Each record also journals a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
+284 -5
View File
@@ -30,10 +30,12 @@ use s3s::access::{S3Access, S3AccessContext};
use s3s::auth::SimpleAuth;
use s3s::dto::{
AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput,
DeleteObjectTaggingInput, DeleteObjectTaggingOutput, ETag, GetBucketVersioningInput, GetBucketVersioningOutput,
GetObjectInput, GetObjectOutput, GetObjectTaggingInput, GetObjectTaggingOutput, HeadBucketInput, HeadBucketOutput,
HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
PutObjectOutput, PutObjectTaggingInput, PutObjectTaggingOutput, StreamingBlob, Tag, TagSet, Timestamp, TimestampFormat,
UploadPartInput, UploadPartOutput,
};
use s3s::service::{S3Service, S3ServiceBuilder};
use s3s::validation::{AwsNameValidation, NameValidation};
@@ -88,6 +90,13 @@ const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
/// Wire prefix of the SSE-C passthrough replication transport headers
/// (`X-Rustfs-Replication-*`). In the default mode the fake stores them like a
/// RustFS target and echoes SSE-C evidence back on HEAD/GET; with
/// [`FakeS3Target::drop_unlisted_replication_headers`] it models MinIO /
/// generic S3, which silently discard unknown x-* headers.
const REPLICATION_SSE_TRANSPORT_PREFIX: &str = "x-rustfs-replication-";
const REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER: &str = "x-rustfs-replication-ssec-algorithm";
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -103,6 +112,9 @@ pub enum Operation {
GetObject,
HeadObject,
DeleteObject,
GetObjectTagging,
PutObjectTagging,
DeleteObjectTagging,
ListObjectVersions,
CreateMultipartUpload,
UploadPart,
@@ -149,6 +161,42 @@ impl ReplicationTimestampHeaders {
}
}
/// Read-proxy related headers observed on a request, journaled so proxy
/// tests can assert the exact wire contract: the anti-loop marker present,
/// the replication-check exemption absent, and the client SSE-C key family
/// forwarded verbatim. The SSE-C key value itself is never retained — only
/// its presence.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ProxyHeaderSnapshot {
pub source_proxy_request: Option<String>,
pub replication_check: Option<String>,
pub ssec_algorithm: Option<String>,
pub ssec_key_present: bool,
pub ssec_key_md5: Option<String>,
/// Whether the request carried any `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header, so fail-closed tests can assert the
/// sender really shipped the material a dropping target discarded.
pub ssec_transport_present: bool,
}
impl ProxyHeaderSnapshot {
fn from_headers(headers: &HeaderMap) -> Self {
Self {
source_proxy_request: header_value(headers, &["x-rustfs-source-proxy-request", "x-minio-source-proxy-request"])
.map(bounded_journal_value),
replication_check: header_value(headers, &["x-rustfs-source-replication-check", "x-minio-source-replication-check"])
.map(bounded_journal_value),
ssec_algorithm: header_value(headers, &["x-amz-server-side-encryption-customer-algorithm"])
.map(bounded_journal_value),
ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"),
ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value),
ssec_transport_present: headers
.keys()
.any(|name| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX)),
}
}
}
/// Credential-free request metadata retained for deterministic assertions.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestRecord {
@@ -163,6 +211,7 @@ pub struct RequestRecord {
pub content_length: Option<u64>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub proxy_headers: ProxyHeaderSnapshot,
pub fault: Option<FaultAction>,
}
@@ -178,6 +227,10 @@ struct ControlState {
struct StoreState {
assign_own_version_ids: bool,
assign_own_multipart_version_ids: bool,
/// MinIO-like mode: silently discard non-whitelisted replication
/// transport headers instead of storing them (see
/// [`REPLICATION_SSE_TRANSPORT_PREFIX`]).
drop_unlisted_replication_headers: bool,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
total_bytes: usize,
@@ -199,6 +252,12 @@ struct ObjectVersion {
delete_marker: bool,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
/// Object tags as ordered key/value pairs (PutObjectTagging replaces the
/// whole set, DeleteObjectTagging clears it).
tags: Vec<(String, String)>,
/// SSE-C passthrough transport headers stored with the version (RustFS
/// target behavior); empty when the drop mode discarded them.
replication_sse_headers: Vec<(String, String)>,
}
#[derive(Clone)]
@@ -208,6 +267,7 @@ struct MultipartState {
version_id: String,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
replication_sse_headers: Vec<(String, String)>,
parts: BTreeMap<i32, MultipartPart>,
}
@@ -428,6 +488,15 @@ impl FakeS3Target {
/// Mint own version ids for the multipart path only — models a target
/// that adopts PutObject version ids but not CreateMultipartUpload ones.
/// MinIO-like mode: silently drop every `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header instead of storing it. The default (off)
/// models a RustFS target, which preserves the headers and echoes SSE-C
/// evidence (`x-amz-server-side-encryption-customer-algorithm`) on
/// HEAD/GET of the replica.
pub fn drop_unlisted_replication_headers(&self, enabled: bool) {
lock(&self.backend.store).drop_unlisted_replication_headers = enabled;
}
pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
}
@@ -569,6 +638,7 @@ impl S3Access for FaultAccess {
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok());
let replication_timestamps = ReplicationTimestampHeaders::from_headers(context.headers());
let proxy_headers = ProxyHeaderSnapshot::from_headers(context.headers());
let fault = record_request(
&self.control,
operation,
@@ -576,6 +646,7 @@ impl S3Access for FaultAccess {
parsed,
content_length,
replication_timestamps,
proxy_headers,
);
if let Some(RequestFault {
action: FaultAction::Status(status),
@@ -615,6 +686,9 @@ fn operation_from_s3_name(name: &str) -> Operation {
"GetObject" => Operation::GetObject,
"HeadObject" => Operation::HeadObject,
"DeleteObject" => Operation::DeleteObject,
"GetObjectTagging" => Operation::GetObjectTagging,
"PutObjectTagging" => Operation::PutObjectTagging,
"DeleteObjectTagging" => Operation::DeleteObjectTagging,
"CreateMultipartUpload" => Operation::CreateMultipartUpload,
"UploadPart" => Operation::UploadPart,
"CompleteMultipartUpload" => Operation::CompleteMultipartUpload,
@@ -630,6 +704,7 @@ fn record_request(
parsed: ParsedRequest,
content_length: Option<u64>,
replication_timestamps: ReplicationTimestampHeaders,
proxy_headers: ProxyHeaderSnapshot,
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
@@ -655,6 +730,7 @@ fn record_request(
content_length,
consumed_bytes: None,
replication_timestamps,
proxy_headers,
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
@@ -721,6 +797,15 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
(&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
(&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
(&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
(&Method::GET, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::GetObjectTagging
}
(&Method::PUT, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::PutObjectTagging
}
(&Method::DELETE, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::DeleteObjectTagging
}
// A replication PUT addresses the source version via `?versionId=`.
(&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
(&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
@@ -788,6 +873,29 @@ fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
Ok(version_id.to_string())
}
/// Capture the SSE-C passthrough transport headers a replication PUT carried.
/// Returns an empty set in the MinIO-like drop mode.
fn captured_replication_sse_headers(headers: &HeaderMap, drop_unlisted: bool) -> Vec<(String, String)> {
if drop_unlisted {
return Vec::new();
}
headers
.iter()
.filter(|(name, _)| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX))
.filter_map(|(name, value)| Some((name.as_str().to_string(), value.to_str().ok()?.to_string())))
.collect()
}
/// SSE-C evidence a RustFS-like target echoes for a stored passthrough
/// replica: the customer algorithm restored from the transport headers.
fn stored_sse_customer_algorithm(version: &ObjectVersion) -> Option<String> {
version
.replication_sse_headers
.iter()
.find(|(name, _)| name == REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER)
.map(|(_, value)| value.clone())
}
fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> {
header_value(headers, &SOURCE_ETAG_HEADERS)
.map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value)))
@@ -1135,6 +1243,33 @@ fn find_version(state: &StoreState, bucket: &str, key: &str, version_id: Option<
Ok(version.clone())
}
/// Replace (or clear, with an empty vec) the tag set of the addressed
/// version, returning its version id. Mirrors `find_version` addressing:
/// explicit version id or the latest version, delete markers rejected.
fn set_version_tags(
state: &mut StoreState,
bucket: &str,
key: &str,
version_id: Option<&str>,
tags: Vec<(String, String)>,
) -> S3Result<String> {
// Resolve first (immutable) so the error paths match find_version.
let resolved = find_version(state, bucket, key, version_id)?.version_id;
let versions = state
.buckets
.get_mut(bucket)
.expect("bucket existence checked by find_version")
.objects
.get_mut(key)
.expect("key existence checked by find_version");
let version = versions
.iter_mut()
.find(|version| version.version_id == resolved)
.expect("version existence checked by find_version");
version.tags = tags;
Ok(resolved)
}
#[async_trait]
impl S3 for FakeBackend {
async fn head_bucket(&self, req: S3Request<HeadBucketInput>) -> S3Result<S3Response<HeadBucketOutput>> {
@@ -1231,7 +1366,10 @@ impl S3 for FakeBackend {
let input = req.input;
let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let assign_own = lock(&self.store).assign_own_version_ids;
let (assign_own, drop_unlisted) = {
let state = lock(&self.store);
(state.assign_own_version_ids, state.drop_unlisted_replication_headers)
};
let version_id = new_version_id(&headers, assign_own)?;
let e_tag = match source_etag(&headers)? {
Some(value) => value,
@@ -1248,6 +1386,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: input.content_type,
metadata: input.metadata,
tags: Vec::new(),
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
};
upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?;
Ok(apply_response_fault(
@@ -1268,6 +1408,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(GetObjectOutput {
body: Some(StreamingBlob::new(Body::from(version.body.clone()))),
@@ -1277,6 +1418,7 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
@@ -1291,6 +1433,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(HeadObjectOutput {
content_length: Some(version.body.len() as i64),
@@ -1299,12 +1442,79 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
))
}
async fn get_object_tagging(&self, req: S3Request<GetObjectTaggingInput>) -> S3Result<S3Response<GetObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version = {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let tag_set: TagSet = version
.tags
.into_iter()
.map(|(key, value)| Tag {
key: Some(key),
value: Some(value),
})
.collect();
Ok(apply_response_fault(
S3Response::new(GetObjectTaggingOutput {
tag_set,
version_id: Some(ObjectVersionId::from(version.version_id)),
}),
fault.as_ref(),
))
}
async fn put_object_tagging(&self, req: S3Request<PutObjectTaggingInput>) -> S3Result<S3Response<PutObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let tags = input
.tagging
.tag_set
.into_iter()
.map(|tag| (tag.key.unwrap_or_default(), tag.value.unwrap_or_default()))
.collect();
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), tags)?
};
Ok(apply_response_fault(
S3Response::new(PutObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object_tagging(
&self,
req: S3Request<DeleteObjectTaggingInput>,
) -> S3Result<S3Response<DeleteObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), Vec::new())?
};
Ok(apply_response_fault(
S3Response::new(DeleteObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object(&self, req: S3Request<DeleteObjectInput>) -> S3Result<S3Response<DeleteObjectOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
@@ -1381,6 +1591,8 @@ impl S3 for FakeBackend {
delete_marker: true,
content_type: None,
metadata: None,
tags: Vec::new(),
replication_sse_headers: Vec::new(),
},
)?;
Ok(apply_response_fault(
@@ -1408,9 +1620,10 @@ impl S3 for FakeBackend {
ensure_upload_budget(&state)?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let upload_id = Uuid::new_v4().to_string();
// Read the flag before the mutable borrow of `state.uploads` below
// Read the flags before the mutable borrow of `state.uploads` below
// (and never re-lock the store: the mutex is not reentrant).
let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
let drop_unlisted = state.drop_unlisted_replication_headers;
let version_id = new_version_id(&headers, mint_own)?;
state.uploads.insert(
upload_id.clone(),
@@ -1420,6 +1633,7 @@ impl S3 for FakeBackend {
version_id,
content_type: input.content_type,
metadata: input.metadata,
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
parts: BTreeMap::new(),
},
);
@@ -1557,6 +1771,7 @@ impl S3 for FakeBackend {
version_id: upload.version_id.clone(),
content_type: upload.content_type.clone(),
metadata: upload.metadata.clone(),
replication_sse_headers: upload.replication_sse_headers.clone(),
parts: BTreeMap::new(),
},
selected,
@@ -1583,6 +1798,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: upload.content_type,
metadata: upload.metadata,
tags: Vec::new(),
replication_sse_headers: upload.replication_sse_headers,
};
let mut state = lock(&self.store);
let current = state
@@ -1787,6 +2004,65 @@ mod tests {
Ok(())
}
/// Default mode is RustFS-like: SSE-C passthrough transport headers are
/// stored and the customer algorithm is echoed on HEAD/GET. Drop mode is
/// MinIO-like: the headers are silently discarded, so no evidence comes
/// back — the exact difference the N2 fail-closed audit keys on. Both
/// modes journal that the sender shipped the transport headers.
#[tokio::test]
async fn ssec_passthrough_headers_echo_and_drop_modes() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
let put_with_transport_headers = |key: &'static str| {
client
.put_object()
.bucket("target-bucket")
.key(key)
.body(ByteStream::from_static(b"ciphertext"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-replication-ssec-algorithm", "AES256");
headers.insert("x-rustfs-replication-ssec-key-md5", "AAAAAAAAAAAAAAAAAAAAAA==");
Ok::<_, std::convert::Infallible>(request)
})
.send()
};
put_with_transport_headers("kept").await?;
let head = client.head_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(head.sse_customer_algorithm(), Some("AES256"));
let get = client.get_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(get.sse_customer_algorithm(), Some("AES256"));
target.drop_unlisted_replication_headers(true);
put_with_transport_headers("dropped").await?;
let head = client.head_object().bucket("target-bucket").key("dropped").send().await?;
assert_eq!(head.sse_customer_algorithm(), None, "drop mode must discard SSE-C evidence");
let requests = target.requests();
for key in ["kept", "dropped"] {
let record = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some(key))
.expect("PUT must be journaled");
assert!(
record.proxy_headers.ssec_transport_present,
"the journal must prove the sender shipped the transport headers for {key}"
);
}
let plain_head = requests
.iter()
.find(|record| record.operation == Operation::HeadObject)
.expect("HEAD must be journaled");
assert!(!plain_head.proxy_headers.ssec_transport_present);
target.shutdown().await;
Ok(())
}
macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{
let error = &$error;
@@ -3052,6 +3328,7 @@ mod tests {
version_id: index.to_string(),
content_type: None,
metadata: None,
replication_sse_headers: Vec::new(),
parts: BTreeMap::new(),
},
);
@@ -3074,6 +3351,7 @@ mod tests {
},
Some(0),
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
}
let records = lock(&control).requests.clone();
@@ -3096,6 +3374,7 @@ mod tests {
},
None,
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
{
let bounded_records = lock(&bounded_control);
@@ -66,7 +66,6 @@ mod tests {
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::collections::BTreeMap;
use std::error::Error;
@@ -277,7 +276,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn codec_streaming_matches_legacy_duplex_body_and_headers() -> TestResult {
init_logging();
-4
View File
@@ -17,7 +17,6 @@
use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_put, init_logging};
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::{Client, Config};
use serial_test::serial;
use tracing::info;
fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
@@ -84,7 +83,6 @@ async fn update_group_members_rejects_invalid_new_group_names() -> Result<(), Bo
/// Test that deleting a group with members fails, and deleting an empty group succeeds.
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "requires awscurl and spawns a real RustFS server"]
async fn test_delete_group_requires_empty_membership() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -144,7 +142,6 @@ async fn test_delete_group_requires_empty_membership() -> Result<(), Box<dyn std
/// Test that a user with only group membership (no explicit user policy) gets group policies
/// and can perform actions allowed by the group (regression test for #2028.1).
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "requires awscurl and spawns a real RustFS server"]
async fn test_user_with_only_group_gets_group_policies() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -212,7 +209,6 @@ async fn test_user_with_only_group_gets_group_policies() -> Result<(), Box<dyn s
/// Test that after deleting a user who was the only member of a group, the group can be deleted
/// (regression test for #2028.2: delete group uses backend membership, not stale cache).
#[tokio::test(flavor = "multi_thread")]
#[serial]
#[ignore = "requires awscurl and spawns a real RustFS server"]
async fn test_delete_group_after_deleting_user() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -16,7 +16,6 @@ use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::presigning::PresigningConfig;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
use serial_test::serial;
use std::time::Duration;
use tracing::info;
@@ -29,7 +28,6 @@ fn list_contains_key(output: &aws_sdk_s3::operation::list_objects_v2::ListObject
}
#[tokio::test]
#[serial]
async fn head_object_consistency_after_write_and_multipart_and_presigned_head()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1,6 +1,5 @@
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use tracing::info;
const RANGE_HEAD_BUCKET: &str = "range-head-test-bucket";
@@ -8,7 +7,6 @@ const RANGE_HEAD_KEY: &str = "range-head-object.bin";
const ACCEPT_RANGES_BYTES: &str = "bytes";
#[tokio::test]
#[serial]
async fn head_object_advertises_accept_ranges() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Starting HeadObject Accept-Ranges regression test");
@@ -19,7 +19,6 @@ mod tests {
use crate::chaos::signed_admin_post;
use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::collections::HashSet;
use std::error::Error;
use std::path::{Path, PathBuf};
@@ -63,7 +62,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_auto_heal_rebuilds_runtime_wiped_disk_without_restart() {
init_logging();
info!("Issue #1533: auto heal should rebuild a runtime-wiped disk in a 4-disk single-node erasure set without restart");
@@ -182,7 +180,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_admin_deep_heal_rebuilds_cleared_disk_in_single_node_erasure_set() {
init_logging();
info!("Discussion #2964: admin deep heal should rebuild a wiped disk in a 4-disk single-node erasure set");
@@ -332,7 +329,6 @@ mod tests {
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn test_cluster_root_heal_rebuilds_replaced_remote_disk() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
info!("Root recursive heal should rebuild data on a remote node after its disk is replaced and the node rejoins");
@@ -444,7 +440,6 @@ mod tests {
/// topology early-return or the merge hard-fail) turns the down-window
/// response into a 500 and fails this test.
#[tokio::test]
#[serial]
async fn test_background_heal_status_degrades_while_peer_down_and_recovers_after_rejoin()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -67,6 +67,9 @@ type MetricValues = Arc<Mutex<BTreeMap<String, MetricPointVersions>>>;
const KIB: usize = 1024;
const READER_PATH_COUNTER: &str = "rustfs_io_get_object_reader_path_by_size_total";
/// Physical bytes the erasure layer pulled from disk, emitted per shard read by
/// `crates/ecstore/src/erasure/coding/decode.rs`.
const SHARD_READ_BYTES_COUNTER: &str = "rustfs_io_get_object_shard_read_observed_bytes_total";
const MSGPACK_JSON_DECODE_COUNTER: &str = "rustfs_system_network_internode_msgpack_json_decode_total";
const MSGPACK_JSON_FALLBACK_COUNTER: &str = "rustfs_system_network_internode_msgpack_json_fallback_total";
const MSGPACK_JSON_DECODE_ERROR_COUNTER: &str = "rustfs_system_network_internode_msgpack_json_decode_error_total";
@@ -146,6 +149,7 @@ struct OtlpMetricCollector {
decode_values: MetricValues,
fallback_values: MetricValues,
decode_error_values: MetricValues,
shard_read_values: MetricValues,
task: JoinHandle<()>,
}
@@ -157,10 +161,12 @@ impl OtlpMetricCollector {
let decode_values = Arc::new(Mutex::new(BTreeMap::new()));
let fallback_values = Arc::new(Mutex::new(BTreeMap::new()));
let decode_error_values = Arc::new(Mutex::new(BTreeMap::new()));
let shard_read_values = Arc::new(Mutex::new(BTreeMap::new()));
let task_values = values.clone();
let task_decode_values = decode_values.clone();
let task_fallback_values = fallback_values.clone();
let task_decode_error_values = decode_error_values.clone();
let task_shard_read_values = shard_read_values.clone();
let task = tokio::spawn(async move {
loop {
let Ok((stream, _)) = listener.accept().await else {
@@ -170,6 +176,7 @@ impl OtlpMetricCollector {
let decode_values = task_decode_values.clone();
let fallback_values = task_fallback_values.clone();
let decode_error_values = task_decode_error_values.clone();
let shard_read_values = task_shard_read_values.clone();
tokio::spawn(async move {
let _ = hyper::server::conn::http1::Builder::new()
.serve_connection(
@@ -181,6 +188,7 @@ impl OtlpMetricCollector {
decode_values.clone(),
fallback_values.clone(),
decode_error_values.clone(),
shard_read_values.clone(),
)
}),
)
@@ -194,10 +202,48 @@ impl OtlpMetricCollector {
decode_values,
fallback_values,
decode_error_values,
shard_read_values,
task,
})
}
/// Total physical bytes read from disk across every shard-read label set.
async fn shard_read_bytes_total(&self) -> u64 {
self.shard_read_values
.lock()
.await
.values()
.map(|versions| versions.values().map(|(_, value)| *value).sum::<u64>())
.sum()
}
/// Waits until the shard-read counter stops advancing so a measurement window
/// is not polluted by exports still in flight.
///
/// Requires several consecutive equal samples spanning more than one export
/// interval (`RUSTFS_OBS_METER_INTERVAL=1`): a single unchanged sample only
/// proves the latest export has not landed yet, which silently reads as "no
/// disk reads happened" and makes any upper-bound assertion vacuous.
async fn wait_for_shard_read_bytes_to_settle(&self) -> TestResult<u64> {
const REQUIRED_STABLE_SAMPLES: usize = 5;
let mut last = self.shard_read_bytes_total().await;
let mut stable = 0;
for _ in 0..60 {
sleep(Duration::from_millis(500)).await;
let current = self.shard_read_bytes_total().await;
if current == last {
stable += 1;
if stable >= REQUIRED_STABLE_SAMPLES {
return Ok(current);
}
} else {
stable = 0;
last = current;
}
}
Err("timed out waiting for shard-read byte counter to settle".into())
}
async fn reader_path_total(&self, path: &str, object_class: &str, size_bucket: &str) -> u64 {
self.reader_path_values(path, object_class, size_bucket).await.values().sum()
}
@@ -321,6 +367,7 @@ async fn handle_metric_export(
decode_values: MetricValues,
fallback_values: MetricValues,
decode_error_values: MetricValues,
shard_read_values: MetricValues,
) -> Result<Response<Full<Bytes>>, Infallible> {
if request.uri().path() != "/v1/metrics" {
return Ok(response(StatusCode::NOT_FOUND));
@@ -354,7 +401,9 @@ async fn handle_metric_export(
let mut decode_values = decode_values.lock().await;
let mut fallback_values = fallback_values.lock().await;
let mut decode_error_values = decode_error_values.lock().await;
let mut shard_read_values = shard_read_values.lock().await;
record_reader_path_metrics(&export, &mut values);
record_shard_read_bytes_metrics(&export, &mut shard_read_values);
record_msgpack_decode_metrics(&export, &mut decode_values);
record_msgpack_fallback_metrics(&export, &mut fallback_values);
record_msgpack_decode_error_metrics(&export, &mut decode_error_values);
@@ -375,6 +424,50 @@ fn reader_path_metric_key(path: &str, object_class: &str, size_bucket: &str) ->
format!("{path}\u{1f}{object_class}\u{1f}{size_bucket}")
}
/// Accumulates `SHARD_READ_BYTES_COUNTER` across all label sets. Only the total
/// matters: it is the number of physical bytes the erasure layer actually pulled
/// from disk, which is what separates a bounded per-part read from a decode of
/// the whole object.
fn record_shard_read_bytes_metrics(export: &ExportMetricsServiceRequest, values: &mut BTreeMap<String, MetricPointVersions>) {
for resource_metrics in &export.resource_metrics {
for scope_metrics in &resource_metrics.scope_metrics {
for metric in &scope_metrics.metrics {
if metric.name != SHARD_READ_BYTES_COUNTER {
continue;
}
let Some(metric::Data::Sum(sum)) = &metric.data else {
continue;
};
for point in &sum.data_points {
let Some(number_data_point::Value::AsInt(value)) = point.value.as_ref() else {
continue;
};
let value = u64::try_from(*value).unwrap_or_default();
// Keyed by labels, not by position: point order within an export
// is not guaranteed stable, so an index key would alias distinct
// series across batches.
let key = format!(
"{}\u{1f}{}\u{1f}{}",
attribute_string(&point.attributes, "path").unwrap_or_default(),
attribute_string(&point.attributes, "role").unwrap_or_default(),
attribute_string(&point.attributes, "outcome").unwrap_or_default(),
);
values
.entry(key)
.or_default()
.entry(point.start_time_unix_nano)
.and_modify(|current| {
if point.time_unix_nano >= current.0 {
*current = (point.time_unix_nano, value);
}
})
.or_insert((point.time_unix_nano, value));
}
}
}
}
}
fn record_reader_path_metrics(export: &ExportMetricsServiceRequest, values: &mut BTreeMap<String, MetricPointVersions>) {
for resource_metrics in &export.resource_metrics {
for scope_metrics in &resource_metrics.scope_metrics {
@@ -1864,6 +1957,86 @@ async fn four_node_multipart_disk_compression_roundtrip() -> TestResult {
Ok(())
}
/// A tail range over a compressed multipart object must read only the physical
/// data it needs, not decode the object from byte zero.
///
/// The byte-exactness tests around this one stay green even if the seek path
/// regresses into decoding from the start of the object: the bytes returned are
/// still correct, only the read amplification explodes. This asserts the cost
/// side, using `SHARD_READ_BYTES_COUNTER` — already emitted per shard read by the
/// erasure layer, so no production code is instrumented for the test.
///
/// `get_compressed_offsets` skips whole preceding parts by their stored size and
/// then seeks inside the covering part via its compression index, so a bounded
/// read costs on the order of the covering part's block size against a ~5 MiB
/// object.
#[tokio::test]
#[serial]
async fn four_node_compressed_multipart_tail_range_reads_are_bounded() -> TestResult {
init_logging();
let collector = OtlpMetricCollector::start().await?;
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
configure_reader_metric_cluster(&mut cluster, &collector);
cluster.set_env("RUSTFS_COMPRESSION_ENABLED", "true");
cluster.set_env("RUSTFS_COMPRESSION_MULTIPART_ENABLED", "true");
cluster.start().await?;
let bucket = "inline-multipart-compression-tail-range";
cluster.create_test_bucket(bucket).await?;
let client = cluster.create_s3_client(0)?;
let key = "multipart/tail-range.txt";
let (body, _second_part, etag) = put_two_part_multipart(&client, bucket, key).await?;
// Establish that the object really took the compressed read path; otherwise a
// small delta below would only prove compression never happened.
assert_reader_path(
&collector,
&client,
ReaderPathExpectation::for_class(ReaderObject::new(bucket, key, &body, etag.as_deref(), None), LEGACY_DUPLEX, COMPRESSED),
)
.await?;
let baseline = collector.wait_for_shard_read_bytes_to_settle().await?;
let tail_len = 4 * KIB;
let start = body.len() - tail_len;
let end = body.len() - 1;
let range = client
.get_object()
.bucket(bucket)
.key(key)
.range(format!("bytes={start}-{end}"))
.send()
.await?;
let tail = range.body.collect().await?.into_bytes();
assert_eq!(tail.as_ref(), &body[start..], "tail range returned wrong bytes");
let after = collector.wait_for_shard_read_bytes_to_settle().await?;
let read_bytes = after.saturating_sub(baseline);
// A zero delta means the window caught nothing — an unexported counter, or a
// read served without touching the erasure layer — which would make the upper
// bound vacuously true. Fail instead of passing blind.
assert!(
read_bytes > 0,
"no shard reads observed for the tail range; the budget assertion below would be vacuous"
);
// Part 1 alone is MPU_PART_1_SIZE, so a whole-object decode cannot come in
// under it. Half the logical size leaves generous headroom for erasure padding
// and unrelated background reads while still failing loudly on a full decode.
let budget = (body.len() / 2) as u64;
assert!(
read_bytes < budget,
"tail range read {read_bytes} physical bytes for a {tail_len}-byte range (budget {budget}, object {} bytes): \
the read is not bounded to the covering part",
body.len()
);
Ok(())
}
#[tokio::test]
#[serial]
async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> TestResult {
@@ -101,7 +101,6 @@ use rustfs_config::{
};
use rustfs_protos::canonical_make_volume_request_body;
use rustfs_protos::proto_gen::node_service::{MakeVolumeRequest, MakeVolumeResponse, PingRequest, PingResponse};
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::error::Error;
use tonic::{Code, Request, Response, Status};
@@ -397,7 +396,6 @@ fn assert_rejected(result: Result<MakeVolumeResponse, Status>, expected: Code, e
/// Grouped into one server start because each case is independent and spawning
/// a `rustfs` process per assertion would dominate the runtime.
#[tokio::test]
#[serial]
async fn internode_rpc_signature_default_posture_e2e() -> TestResult {
init_logging();
align_rpc_secret_with_server();
@@ -424,7 +422,6 @@ async fn internode_rpc_signature_default_posture_e2e() -> TestResult {
/// epoch is learned from a real response, then the same server is restarted in place to prove its
/// replacement epoch rejects the captured request even though the nonce cache is necessarily new.
#[tokio::test]
#[serial]
async fn replay_scope_rejects_replay_path_transplant_and_stale_epoch_e2e() -> TestResult {
init_logging();
align_rpc_secret_with_server();
@@ -497,7 +494,6 @@ async fn replay_scope_rejects_replay_path_transplant_and_stale_epoch_e2e() -> Te
/// A mutating v2 request cannot use that lane; once the epoch proof is returned, the first v3
/// mutation succeeds. This protects a server restart without reopening a general downgrade path.
#[tokio::test]
#[serial]
async fn replay_scope_strict_requires_v3_after_ping_bootstrap_e2e() -> TestResult {
init_logging();
align_rpc_secret_with_server();
@@ -704,7 +700,6 @@ async fn legacy_only_signature_is_accepted_in_default_posture(url: &str) {
///
/// The paired v2 positive control rules out "strict simply breaks everything".
#[tokio::test]
#[serial]
async fn signature_strict_rejects_legacy_only_downgrade() -> TestResult {
init_logging();
align_rpc_secret_with_server();
@@ -741,7 +736,6 @@ async fn signature_strict_rejects_legacy_only_downgrade() -> TestResult {
/// takes the still-open legacy lane), which is what pins the rejection to the
/// handler's digest gate; the cited message confirms which check spoke.
#[tokio::test]
#[serial]
async fn body_digest_strict_rejects_digestless_mutation() -> TestResult {
init_logging();
align_rpc_secret_with_server();
@@ -27,12 +27,10 @@ use aws_sdk_s3::types::{
ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
};
use rustfs_rio::{Checksum, ChecksumType};
use serial_test::serial;
use tracing::{debug, info, warn};
/// Test 1: When bucket is configured with default SSE-S3 encryption, put_object should automatically apply encryption
#[tokio::test]
#[serial]
async fn test_bucket_default_sse_s3_put_object() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Testing bucket default SSE-S3 encryption impact on put_object");
@@ -155,7 +153,6 @@ async fn test_bucket_default_sse_s3_put_object() -> Result<(), Box<dyn std::erro
/// Test 2: When bucket is configured with default SSE-KMS encryption, put_object should automatically apply encryption and use the specified KMS key
#[tokio::test]
#[serial]
async fn test_bucket_default_sse_kms_put_object() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Testing bucket default SSE-KMS encryption impact on put_object");
@@ -275,7 +272,6 @@ async fn test_bucket_default_sse_kms_put_object() -> Result<(), Box<dyn std::err
/// Test 3: When bucket is configured with default encryption, create_multipart_upload should inherit the configuration
#[tokio::test]
#[serial]
async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Testing bucket default encryption impact on create_multipart_upload");
@@ -473,7 +469,6 @@ async fn test_bucket_default_sse_kms_multipart_crc32() -> Result<(), Box<dyn std
/// Test 4: Explicitly specified encryption parameters in requests should override bucket default configuration
#[tokio::test]
#[serial]
async fn test_explicit_encryption_overrides_bucket_default() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Testing explicitly specified encryption parameters override bucket default configuration");
@@ -569,7 +564,6 @@ async fn test_explicit_encryption_overrides_bucket_default() -> Result<(), Box<d
/// Test 5: Setting SSE-KMS without a specific key ID should auto-populate the
/// default KMS key ID so that GetBucketEncryption returns it (issue #3039).
#[tokio::test]
#[serial]
async fn test_sse_kms_without_key_id_populates_default() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Testing SSE-KMS without explicit key ID populates default key");
@@ -20,7 +20,6 @@ use super::common::{
};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, ServerSideEncryption, VersioningConfiguration};
use serial_test::serial;
use std::error::Error;
use uuid::Uuid;
@@ -386,7 +385,6 @@ async fn assert_versioned_sse_kms_roundtrip_and_cleanup(
}
#[tokio::test]
#[serial]
async fn test_configured_local_kms_admin_and_versioned_cleanup() -> TestResult {
let mut env = LocalKMSTestEnvironment::new().await?;
env.base_env.start_rustfs_server(Vec::new()).await?;
@@ -434,7 +432,6 @@ async fn test_configured_local_kms_admin_and_versioned_cleanup() -> TestResult {
}
#[tokio::test]
#[serial]
#[ignore = "requires a Vault binary"]
async fn test_configured_vault_kms_admin_and_versioned_cleanup() -> TestResult {
let mut env = VaultTestEnvironment::new().await?;
@@ -32,11 +32,9 @@ use aws_sdk_s3::types::{
MetadataDirective, ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration,
ServerSideEncryptionRule,
};
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn test_metadata_replace_self_copy_of_sse_object_stays_decryptable() {
init_logging();
info!("same-key CopyObject with REPLACE metadata must not re-key an SSE-S3 object");
@@ -136,7 +134,6 @@ async fn test_metadata_replace_self_copy_of_sse_object_stays_decryptable() {
}
#[tokio::test]
#[serial]
async fn test_metadata_replace_self_copy_dropping_sse_rewrites_plaintext() {
init_logging();
info!("same-key CopyObject that drops SSE must rewrite the data, not orphan the ciphertext");
@@ -233,7 +230,6 @@ async fn test_metadata_replace_self_copy_dropping_sse_rewrites_plaintext() {
}
#[tokio::test]
#[serial]
async fn test_metadata_replace_self_copy_under_bucket_default_sse_stays_decryptable() {
init_logging();
info!("bucket default encryption must also keep a same-key copy off the metadata-only path");
@@ -25,11 +25,9 @@ use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
use crate::common::init_logging;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, ServerSideEncryption, VersioningConfiguration};
use serial_test::serial;
use tracing::info;
#[tokio::test]
#[serial]
async fn test_self_copy_of_historical_sse_s3_version_is_readable() {
init_logging();
info!("Issue #4238 (SSE): restoring an encrypted historical version must stay decryptable");
@@ -22,7 +22,6 @@ use aws_sdk_s3::types::{
CompletedMultipartUpload, CompletedPart, ServerSideEncryption, ServerSideEncryptionByDefault,
ServerSideEncryptionConfiguration, ServerSideEncryptionRule,
};
use serial_test::serial;
use std::collections::{HashMap, VecDeque};
use tracing::info;
@@ -82,7 +81,6 @@ pub(super) fn assert_storage_encrypted(storage_root: &std::path::Path, bucket: &
}
#[tokio::test]
#[serial]
async fn test_head_reports_managed_metadata_for_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Validating SSE-S3 managed encryption metadata exposure");
@@ -143,7 +141,6 @@ async fn test_head_reports_managed_metadata_for_sse_s3() -> Result<(), Box<dyn s
}
#[tokio::test]
#[serial]
async fn test_head_reports_managed_metadata_for_sse_kms_and_copy() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Validating SSE-KMS managed encryption metadata (including copy)");
@@ -247,7 +244,6 @@ async fn test_head_reports_managed_metadata_for_sse_kms_and_copy() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_multipart_upload_writes_encrypted_data() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("Validating ciphertext persistence for multipart SSE-KMS uploads");
@@ -35,7 +35,6 @@ use aws_sdk_s3::config::{Config, Credentials, Region};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::ServerSideEncryption;
use serial_test::serial;
use std::time::Duration;
use tracing::info;
@@ -209,7 +208,6 @@ fn disable_body(key_id: &str) -> String {
/// Data-path matrix: SSE-KMS writes and reads are authorized against the resolved key.
#[tokio::test]
#[serial]
async fn sse_kms_per_key_authorization_negative_matrix() -> TestResult {
init_logging();
@@ -355,7 +353,6 @@ async fn sse_kms_per_key_authorization_negative_matrix() -> TestResult {
/// Runs without the SSE enforcement switch: admin scoping is unconditional, and
/// leaving the switch off proves the two planes are independent.
#[tokio::test]
#[serial]
async fn kms_admin_per_key_authorization_negative_matrix() -> TestResult {
init_logging();
@@ -24,13 +24,11 @@ use super::common::{
test_sse_kms_encryption, test_sse_s3_encryption,
};
use crate::common::{TEST_BUCKET, init_logging};
use serial_test::serial;
use tokio::time::{Duration, sleep};
use tracing::info;
/// Comprehensive test: Full KMS workflow with all encryption types
#[tokio::test]
#[serial]
async fn test_comprehensive_kms_full_workflow() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🏁 Start the KMS full-featured synthesis test");
@@ -99,7 +97,6 @@ async fn test_mixed_encryption_workload(
/// Comprehensive stress test: Large dataset with multiple encryption types
#[tokio::test]
#[serial]
async fn test_comprehensive_stress_test() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("💪 Start the KMS stress test");
@@ -134,7 +131,6 @@ async fn test_comprehensive_stress_test() -> Result<(), Box<dyn std::error::Erro
/// Test encryption key isolation and security
#[tokio::test]
#[serial]
async fn test_comprehensive_key_isolation() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🔐 Begin the comprehensive test of encryption key isolation");
@@ -206,7 +202,6 @@ async fn test_comprehensive_key_isolation() -> Result<(), Box<dyn std::error::Er
/// Test concurrent encryption operations
#[tokio::test]
#[serial]
async fn test_comprehensive_concurrent_operations() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("⚡ Started comprehensive testing of concurrent encryption operations");
@@ -252,7 +247,6 @@ async fn test_comprehensive_concurrent_operations() -> Result<(), Box<dyn std::e
/// Test encryption/decryption performance with different file sizes
#[tokio::test]
#[serial]
async fn test_comprehensive_performance_benchmark() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("📊 Start KMS performance benchmarking");
@@ -26,7 +26,6 @@ use crate::common::{TEST_BUCKET, init_logging};
use aws_sdk_s3::types::ServerSideEncryption;
use base64::Engine;
use md5::{Digest as Md5Digest, Md5};
use serial_test::serial;
use std::sync::Arc;
use tokio::sync::Semaphore;
use tracing::{info, warn};
@@ -39,7 +38,6 @@ fn md5_hex(input: impl AsRef<[u8]>) -> String {
/// Test encryption of zero-byte files (empty files)
#[tokio::test]
#[serial]
async fn test_kms_zero_byte_file_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS encryption with zero-byte files");
@@ -113,7 +111,6 @@ async fn test_kms_zero_byte_file_encryption() -> Result<(), Box<dyn std::error::
/// Test encryption of single-byte files
#[tokio::test]
#[serial]
async fn test_kms_single_byte_file_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS encryption with single-byte files");
@@ -206,7 +203,6 @@ async fn test_kms_single_byte_file_encryption() -> Result<(), Box<dyn std::error
/// Test multipart upload boundary conditions (minimum 5MB part size)
#[tokio::test]
#[serial]
async fn test_kms_multipart_boundary_conditions() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS multipart upload boundary conditions");
@@ -282,7 +278,6 @@ async fn test_kms_multipart_boundary_conditions() -> Result<(), Box<dyn std::err
/// Test invalid key scenarios and error handling
#[tokio::test]
#[serial]
async fn test_kms_invalid_key_scenarios() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS invalid key scenarios and error handling");
@@ -370,7 +365,6 @@ async fn test_kms_invalid_key_scenarios() -> Result<(), Box<dyn std::error::Erro
/// Test concurrent encryption operations
#[tokio::test]
#[serial]
async fn test_kms_concurrent_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS concurrent encryption operations");
@@ -478,7 +472,6 @@ async fn test_kms_concurrent_encryption() -> Result<(), Box<dyn std::error::Erro
/// Test key validation and security properties
#[tokio::test]
#[serial]
async fn test_kms_key_validation_security() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS key validation and security properties");
@@ -24,7 +24,6 @@
use super::common::LocalKMSTestEnvironment;
use crate::common::{TEST_BUCKET, init_logging};
use aws_sdk_s3::types::ServerSideEncryption;
use serial_test::serial;
use std::fs;
use std::time::Duration;
use tokio::time::sleep;
@@ -32,7 +31,6 @@ use tracing::{info, warn};
/// Test KMS behavior when key directory is temporarily unavailable
#[tokio::test]
#[serial]
async fn test_kms_key_directory_unavailable() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS behavior with unavailable key directory");
@@ -123,7 +121,6 @@ async fn test_kms_key_directory_unavailable() -> Result<(), Box<dyn std::error::
/// Test handling of corrupted key files
#[tokio::test]
#[serial]
async fn test_kms_corrupted_key_files() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS behavior with corrupted key files");
@@ -215,7 +212,6 @@ async fn test_kms_corrupted_key_files() -> Result<(), Box<dyn std::error::Error
/// Test multipart upload interruption and recovery
#[tokio::test]
#[serial]
async fn test_kms_multipart_upload_interruption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS multipart upload interruption and recovery");
@@ -399,7 +395,6 @@ async fn test_kms_multipart_upload_interruption() -> Result<(), Box<dyn std::err
/// Test KMS resilience to temporary resource constraints
#[tokio::test]
#[serial]
async fn test_kms_resource_constraints() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Testing KMS behavior under resource constraints");
@@ -51,7 +51,6 @@ use aws_sdk_s3::types::{
TransitionStorageClass,
};
use serde::Deserialize;
use serial_test::serial;
use std::time::{Duration as StdDuration, Instant};
use tracing::info;
@@ -424,7 +423,6 @@ async fn wait_for_restore_complete(client: &Client, bucket: &str, key: &str, dea
/// filter as the cause of the deletion and proves the encrypted bucket stays
/// readable end to end after the scanner has run.
#[tokio::test]
#[serial]
async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
init_logging();
@@ -485,7 +483,6 @@ async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
/// (the mechanism `reliant/tiering.rs` established), so the test does not
/// depend on scanner scheduling; the 1s scanner cycle stays on as a backstop.
#[tokio::test]
#[serial]
async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> TestResult {
init_logging();
@@ -24,11 +24,9 @@ use super::common::{
test_kms_key_management, test_sse_c_encryption,
};
use crate::common::{TEST_BUCKET, init_logging};
use serial_test::serial;
use tracing::{error, info};
#[tokio::test]
#[serial]
async fn test_local_kms_end_to_end() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_if_kms_admin_tool_unavailable("test_local_kms_end_to_end") {
@@ -114,7 +112,6 @@ async fn test_local_kms_end_to_end() -> Result<(), Box<dyn std::error::Error + S
}
#[tokio::test]
#[serial]
async fn test_local_kms_key_isolation() {
init_logging();
info!("Starting Local KMS Key Isolation Test");
@@ -215,7 +212,6 @@ async fn test_local_kms_key_isolation() {
}
#[tokio::test]
#[serial]
async fn test_local_kms_large_file() {
init_logging();
info!("Starting Local KMS Large File Test");
@@ -298,7 +294,6 @@ async fn test_local_kms_large_file() {
}
#[tokio::test]
#[serial]
async fn test_local_kms_multipart_upload() {
init_logging();
info!("Starting Local KMS Multipart Upload Test");
@@ -23,12 +23,10 @@
use super::common::{LocalKMSTestEnvironment, sse_customer_key_md5_base64};
use crate::common::{TEST_BUCKET, init_logging};
use serial_test::serial;
use tracing::{debug, info};
/// Step 1: Test the basic single-file encryption function (ensure that SSE-S3 works properly in non-sharded scenarios)
#[tokio::test]
#[serial]
async fn test_step1_basic_single_file_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Step 1: Test the basic single-file encryption function");
@@ -85,7 +83,6 @@ async fn test_step1_basic_single_file_encryption() -> Result<(), Box<dyn std::er
/// Step 2: Test the unencrypted shard upload (make sure the shard upload base is working properly)
#[tokio::test]
#[serial]
async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Step 2: Test unencrypted shard uploads");
@@ -184,7 +181,6 @@ async fn test_step2_basic_multipart_upload_without_encryption() -> Result<(), Bo
/// Step 3: Test Shard Upload + SSE-S3 Encryption (Focus Test)
#[tokio::test]
#[serial]
async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Step 3: Test Shard Upload + SSE-S3 Encryption");
@@ -308,7 +304,6 @@ async fn test_step3_multipart_upload_with_sse_s3() -> Result<(), Box<dyn std::er
/// Step 4: test larger multipart uploads (streaming encryption)
#[tokio::test]
#[serial]
async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Step 4: test large-file multipart encryption");
@@ -434,7 +429,6 @@ async fn test_step4_large_multipart_upload_with_encryption() -> Result<(), Box<d
/// Step 5: test multipart uploads for every encryption mode
#[tokio::test]
#[serial]
async fn test_step5_all_encryption_types_multipart() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
info!("🧪 Step 5: test multipart uploads for every encryption mode");
-3
View File
@@ -19,7 +19,6 @@
//! filtering, and comprehensive reporting capabilities.
use crate::common::init_logging;
use serial_test::serial;
use std::time::Instant;
use tokio::time::{Duration, sleep};
use tracing::{debug, error, info, warn};
@@ -458,7 +457,6 @@ impl KMSTestSuite {
/// Quick test suite for critical tests only
#[tokio::test]
#[serial]
async fn test_kms_critical_suite() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let config = TestSuiteConfig {
categories: vec![TestCategory::CoreFunctionality, TestCategory::MultipartEncryption],
@@ -481,7 +479,6 @@ async fn test_kms_critical_suite() -> Result<(), Box<dyn std::error::Error + Sen
/// Full comprehensive test suite
#[tokio::test]
#[serial]
async fn test_kms_full_suite() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let suite = KMSTestSuite::new();
let results = suite.run_test_suite().await;
@@ -24,7 +24,6 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::error::Error;
use tracing::info;
@@ -36,7 +35,6 @@ mod tests {
/// PUT with a leading-slash key must succeed and the object must be
/// readable under the normalized key (leading slash stripped).
#[tokio::test]
#[serial]
async fn test_put_object_with_leading_slash_key() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
info!("Starting test: PUT object with leading slash in key (Issue #2427)");
@@ -94,7 +92,6 @@ mod tests {
/// Duplicate and repeated slashes after a leading slash collapse MinIO-style.
#[tokio::test]
#[serial]
async fn test_put_object_with_duplicate_slashes_normalized() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
info!("Starting test: duplicate slash normalization (Issue #2427)");
@@ -36,7 +36,6 @@ mod tests {
BucketLifecycleConfiguration, BucketVersioningStatus, ExpirationStatus, LifecycleExpiration, LifecycleRule,
LifecycleRuleFilter, NoncurrentVersionExpiration, VersioningConfiguration,
};
use serial_test::serial;
use std::error::Error;
use tracing::info;
@@ -80,7 +79,6 @@ mod tests {
///
/// This tests the rule persistence path (rustfs#4963: 3 days → 0 days).
#[tokio::test]
#[serial]
async fn test_lifecycle_expiration_rule_persists_correctly() -> TestResult {
init_logging();
info!("RT-03: lifecycle expiration rule persists correctly");
@@ -148,7 +146,6 @@ mod tests {
/// Covers the pattern where noncurrent version expiration rules are
/// accepted but old versions are never cleaned up.
#[tokio::test]
#[serial]
async fn test_lifecycle_noncurrent_version_expiration_rule_persists() -> TestResult {
init_logging();
info!("RT-03b: noncurrent version expiration rule persists");
@@ -233,7 +230,6 @@ mod tests {
/// after restart. Transition rules require a configured remote tier
/// (tested in reliant/tiering.rs), so this test uses expiration only.
#[tokio::test]
#[serial]
async fn test_lifecycle_prefix_rule_persists() -> TestResult {
init_logging();
info!("RT-04: lifecycle prefix rule persists");
@@ -294,7 +290,6 @@ mod tests {
/// Regression pattern: DELETE on a versioned object fails or does not
/// create a delete marker, or the delete marker is not visible in LIST.
#[tokio::test]
#[serial]
async fn test_delete_marker_creation_and_visibility() -> TestResult {
init_logging();
info!("RT-05b: delete marker creation and visibility");
@@ -27,7 +27,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
/// Sends a SigV4-signed `GET` where the signature is computed over `sign_path`
@@ -67,7 +66,6 @@ mod tests {
/// `GET /` (path-style service call) returns `ListBuckets`.
#[tokio::test]
#[serial]
async fn test_list_buckets_single_slash() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -86,7 +84,6 @@ mod tests {
/// compat layer rewrites `//` to `/` before `s3s` parses/verifies the request,
/// so both routing and signature verification operate on `/`.
#[tokio::test]
#[serial]
async fn test_list_buckets_double_slash_browser_compat() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -105,7 +102,6 @@ mod tests {
/// (`GET //bucket`) must be left untouched by the compat layer — it is not a
/// `ListBuckets` request and s3s continues to reject the empty bucket name.
#[tokio::test]
#[serial]
async fn test_double_slash_rewrite_is_narrowly_scoped() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -15,7 +15,6 @@
use crate::common::{RustFSTestEnvironment, admin_ok, build_test_s3_config, build_test_sts_client, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::ProvideErrorMetadata;
use serial_test::serial;
use tokio::time::{Duration, Instant};
fn user_client(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str, session_token: Option<&str>) -> Client {
@@ -76,7 +75,6 @@ async fn create_service_account(
}
#[tokio::test]
#[serial]
async fn list_buckets_filters_with_iam_bucket_resources() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -24,7 +24,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use tracing::info;
@@ -54,7 +53,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_list_object_versions_metadata_extension_returns_metadata_tags_and_internal()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -21,7 +21,6 @@ mod tests {
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use serial_test::serial;
use tracing::info;
fn create_s3_client(env: &RustFSTestEnvironment) -> Client {
@@ -29,7 +28,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_list_object_versions_immediately_returns_latest_put_after_delete_marker() {
init_logging();
info!("🧪 TEST: ListObjectVersions returns the newest version immediately after put -> delete -> put");
@@ -182,7 +180,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_list_object_versions_prefix_with_marker_object_returns_children() {
init_logging();
info!("🧪 TEST: ListObjectVersions returns prefix children when a marker object also exists");
@@ -17,7 +17,6 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use tracing::info;
/// Helper function to create an S3 client for testing
@@ -60,7 +59,6 @@ mod tests {
/// The bug was that "folder/" (the object) and "folder/" (derived prefix) were both added to CommonPrefixes
/// when delimiter was "/" because the deduplication check was explicitly skipped for "/" delimiter.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_unique_common_prefixes() {
init_logging();
info!("Starting test: ListObjectsV2 should return unique CommonPrefixes");
@@ -140,7 +138,6 @@ mod tests {
/// When both "marker/subdir/" and "marker/subdir/file.txt" exist, listing with
/// Prefix="marker/" must not duplicate "marker/subdir/file.txt" in Contents.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_unique_contents_with_explicit_directory_markers() {
init_logging();
info!("Starting test: ListObjectsV2 should return unique keys with explicit directory markers");
@@ -208,7 +205,6 @@ mod tests {
/// and never produce the prefix entry `a/`. Delimiter="/" listings then
/// returned Contents `a` but silently dropped CommonPrefix `a/`.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_object_and_same_named_prefix_coexist() {
init_logging();
info!("Starting test: ListObjectsV2 should return both object `a` and CommonPrefix `a/`");
@@ -23,7 +23,6 @@ mod tests {
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::error::Error;
use tracing::info;
@@ -53,7 +52,6 @@ mod tests {
}
#[tokio::test]
#[serial]
async fn test_list_objects_v2_metadata_extension_returns_metadata_tags_and_internal()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -30,7 +30,6 @@ mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::collections::HashSet;
use tracing::info;
@@ -61,7 +60,6 @@ mod tests {
/// Test for Issue #2775: continuation forwarding must not
/// skip a child directory when the prefix component repeats in the key.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_repeated_prefix_continuation() {
init_logging();
info!("Starting test: ListObjectsV2 repeated-prefix continuation");
@@ -187,7 +185,6 @@ mod tests {
/// This is the core bug from issue #1596: the server was returning
/// IsTruncated=true even when all objects fit within the requested max_keys.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_not_truncated_when_all_objects_returned() {
init_logging();
info!("Starting test: ListObjectsV2 should not be truncated when all objects fit within max_keys");
@@ -252,7 +249,6 @@ mod tests {
/// 2. NextContinuationToken is returned (not NextMarker)
/// 3. Using ContinuationToken fetches the remaining objects
#[tokio::test]
#[serial]
async fn test_list_objects_v2_pagination_with_continuation_token() {
init_logging();
info!("Starting test: ListObjectsV2 pagination with continuation token");
@@ -394,7 +390,6 @@ mod tests {
/// Edge case: when max_keys exactly equals the number of objects,
/// IsTruncated should be false.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_max_keys_equals_object_count() {
init_logging();
info!("Starting test: ListObjectsV2 with max_keys equal to object count");
@@ -455,7 +450,6 @@ mod tests {
///
/// Edge case: IsTruncated should be false for empty bucket.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_empty_bucket() {
init_logging();
info!("Starting test: ListObjectsV2 with empty bucket");
@@ -495,7 +489,6 @@ mod tests {
/// Test ListObjectsV2 caps max_keys above the service limit and still paginates.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_max_keys_above_limit_returns_token() {
init_logging();
info!("Starting test: ListObjectsV2 with max_keys above limit");
@@ -563,7 +556,6 @@ mod tests {
/// S3 semantics: when max_keys is 0, the response should include no objects
/// and IsTruncated should be false.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_max_keys_zero() {
init_logging();
info!("Starting test: ListObjectsV2 with max_keys=0");
@@ -620,7 +612,6 @@ mod tests {
/// With max_keys=1000, all 5 visible results (3 prefixes + 2 objects) fit in one
/// page, so IsTruncated must be false even though raw entry count is much larger.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_delimiter_collapsed_prefix_no_false_truncation() {
init_logging();
info!("Starting test: ListObjectsV2 delimiter collapsed-prefix no false truncation");
@@ -744,7 +735,6 @@ mod tests {
/// Each page returns up to 50 CommonPrefixes. The server must correctly set
/// IsTruncated and provide a valid continuation token across all pages.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_delimiter_small_page_traverses_all() {
init_logging();
info!("Starting test: ListObjectsV2 delimiter small page traverses all keys");
@@ -867,7 +857,6 @@ mod tests {
/// but after delimiter collapse only 10 CommonPrefixes are visible (10 < 1000).
/// IsTruncated must be false since there are no additional visible results.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_raw_exceeds_maxkeys_but_visible_below() {
init_logging();
info!("Starting test: ListObjectsV2 raw > MaxKeys but visible < MaxKeys after collapse");
@@ -970,7 +959,6 @@ mod tests {
/// This complements test_list_objects_v2_max_keys_above_limit_returns_token which
/// tests the non-delimiter case.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_maxkeys_above_limit_with_delimiter() {
init_logging();
info!("Starting test: ListObjectsV2 MaxKeys above limit with delimiter");
@@ -1041,7 +1029,6 @@ mod tests {
/// the next page: with keys `a`, `a.txt`, `zz` and max_keys=1, page 2
/// returned `zz` and `a.txt` was never listed.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_continuation_keeps_keys_after_marker_stem() {
init_logging();
info!("Starting test: continuation must not skip keys sorting below the cursor tag");
@@ -31,7 +31,6 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::collections::HashSet;
use std::error::Error;
use tracing::info;
@@ -49,7 +48,6 @@ mod tests {
/// 3. Verify all 100 keys are returned exactly once
/// 4. Verify no duplicates or skipped keys
#[tokio::test]
#[serial]
async fn test_list_objects_v2_completeness_100_objects() -> TestResult {
init_logging();
info!("RT-06: listing completeness with 100 objects");
@@ -133,7 +131,6 @@ mod tests {
/// Regression pattern: prefix filter returns empty or includes wrong keys
/// (rustfs#5051: empty results for shallow prefixes).
#[tokio::test]
#[serial]
async fn test_list_objects_v2_prefix_filter_correctness() -> TestResult {
init_logging();
info!("RT-06b: prefix filter correctness");
@@ -233,7 +230,6 @@ mod tests {
/// Regression pattern: delimiter handling produces incorrect CommonPrefixes
/// or misses objects at the delimiter boundary.
#[tokio::test]
#[serial]
async fn test_list_objects_v2_delimiter_common_prefixes() -> TestResult {
init_logging();
info!("RT-06c: delimiter and CommonPrefixes");
@@ -290,7 +286,6 @@ mod tests {
/// Regression pattern: IsTruncated=false when there are more objects
/// (rustfs#4810: walk_dir timeout truncation with false IsTruncated).
#[tokio::test]
#[serial]
async fn test_list_objects_v2_is_truncated_correctness() -> TestResult {
init_logging();
info!("RT-06d: IsTruncated correctness");
@@ -13,7 +13,6 @@
// limitations under the License.
use crate::common::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, RustFSTestEnvironment};
use serial_test::serial;
use std::path::Path;
use std::process::Command;
use std::time::Duration;
@@ -73,7 +72,6 @@ fn count_files(root: &Path) -> usize {
}
#[tokio::test]
#[serial]
async fn test_mc_mirror_small_bucket_completes_without_list_timeout() -> TestResult {
crate::common::init_logging();
info!("Starting issue #3107 mc mirror regression test");
File diff suppressed because it is too large Load Diff
@@ -16,7 +16,6 @@ use crate::common::RustFSTestClusterEnvironment;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use bytes::Bytes;
use serial_test::serial;
use std::sync::Arc;
use tokio::sync::Barrier;
use tracing::{info, warn};
@@ -51,7 +50,6 @@ fn format_s3_error(err: SdkError<aws_sdk_s3::operation::put_object::PutObjectErr
}
#[tokio::test]
#[serial]
async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum() -> TestResult {
crate::common::init_logging();
info!("Starting namespace lock quorum regression test with auto cluster");
@@ -128,7 +126,6 @@ async fn test_concurrent_cluster_overwrites_do_not_fail_namespace_lock_quorum()
/// `StorageError::other(...)` → `StorageError::Io(...)`, which fell through to
/// `S3ErrorCode::InternalError` (500) in the error mapping.
#[tokio::test]
#[serial]
async fn test_concurrent_put_same_key_never_returns_500() -> TestResult {
crate::common::init_logging();
info!("Starting concurrent PUT 500 regression test");
@@ -37,7 +37,6 @@ use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::primitives::ByteStream;
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::request_signature_v4::{SIGN_V4_ALGORITHM, get_scope, get_signature, get_signing_key};
use serial_test::serial;
use std::fmt::Write as _;
use time::macros::format_description;
use time::{Duration, OffsetDateTime};
@@ -183,7 +182,6 @@ async fn setup(env: &mut RustFSTestEnvironment) -> Result<(), Box<dyn std::error
/// this, every negative assertion below could pass for the wrong reason (a
/// broken signer that never produces a valid signature).
#[tokio::test]
#[serial]
async fn valid_header_sigv4_request_succeeds() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -214,7 +212,6 @@ async fn valid_header_sigv4_request_succeeds() -> Result<(), Box<dyn std::error:
/// (a) Tampering the `Signature=` component must be rejected with
/// SignatureDoesNotMatch / 403.
#[tokio::test]
#[serial]
async fn tampered_signature_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -254,7 +251,6 @@ async fn tampered_signature_returns_signature_does_not_match() -> Result<(), Box
/// (b) A valid AccessKeyId paired with the wrong secret key must be rejected
/// with SignatureDoesNotMatch / 403.
#[tokio::test]
#[serial]
async fn wrong_secret_key_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -279,7 +275,6 @@ async fn wrong_secret_key_returns_signature_does_not_match() -> Result<(), Box<d
/// signature itself is valid (it covers the *declared* hash), so the server is
/// forced to detect the payload/hash mismatch while streaming the body.
#[tokio::test]
#[serial]
async fn tampered_payload_is_rejected() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -320,7 +315,6 @@ async fn tampered_payload_is_rejected() -> Result<(), Box<dyn std::error::Error
/// x-amz-date both derive from the same skewed timestamp, so skew — not a
/// signature mismatch — is the failure.
#[tokio::test]
#[serial]
async fn skewed_date_returns_request_time_too_skewed() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -344,7 +338,6 @@ async fn skewed_date_returns_request_time_too_skewed() -> Result<(), Box<dyn std
/// structurally invalid SigV4 header that must be rejected before any
/// credential/service handling.
#[tokio::test]
#[serial]
async fn malformed_authorization_header_returns_clean_4xx() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -32,7 +32,6 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use serial_test::serial;
use std::error::Error;
use tracing::info;
@@ -47,7 +46,6 @@ mod tests {
/// starts successfully with notification enabled and can serve S3 requests.
/// A full webhook delivery test is in notification_webhook_test.rs.
#[tokio::test]
#[serial]
async fn test_notification_enabled_server_starts_cleanly() -> TestResult {
init_logging();
info!("RT-01: notification enabled server starts cleanly");
@@ -92,7 +90,6 @@ mod tests {
/// 3. Restart server
/// 4. Verify notification config still exists
#[tokio::test]
#[serial]
async fn test_notification_config_survives_restart() -> TestResult {
init_logging();
info!("RT-02: notification config survives restart");
@@ -47,7 +47,6 @@ use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
use rustfs_utils::http::headers::{AMZ_REQUEST_ID, REQUEST_ID_HEADER};
use s3s::Body;
use serde_json::Value;
use serial_test::serial;
use std::error::Error;
use std::io::Cursor;
use std::path::Path;
@@ -625,7 +624,6 @@ fn assert_generated_request_id_correlation(record: &Value, request_id: &str) {
/// RUSTFS_NOTIFY_ENABLE, an HTTPS webhook using a configured CA must become
/// online and receive a real S3 event POST.
#[tokio::test]
#[serial]
async fn test_https_webhook_target_delivers_event_with_notify_env_enabled() -> TestResult {
init_logging();
@@ -680,7 +678,6 @@ async fn test_https_webhook_target_delivers_event_with_notify_env_enabled() -> T
/// PUT / multipart-complete / DELETE each deliver one event with correct fields,
/// and the prefix/suffix filter drops non-matching keys.
#[tokio::test]
#[serial]
async fn test_webhook_event_delivery_and_filtering() -> TestResult {
init_logging();
@@ -900,7 +897,6 @@ async fn test_webhook_event_delivery_and_filtering() -> TestResult {
/// An event queued while the target endpoint rejects delivery survives on the
/// durable store and is redelivered once the endpoint comes back.
#[tokio::test]
#[serial]
async fn test_webhook_redelivers_event_after_target_recovers() -> TestResult {
init_logging();
-17
View File
@@ -20,7 +20,6 @@ use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::{pre_sign_v4, sign_v4};
use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
use s3s::Body;
use serial_test::serial;
use std::collections::HashMap;
use std::error::Error;
use time::OffsetDateTime;
@@ -548,7 +547,6 @@ async fn read_listen_notification_event(
}
#[tokio::test]
#[serial]
async fn test_notification_target_persists_across_restart_and_delete() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -608,7 +606,6 @@ async fn test_notification_target_persists_across_restart_and_delete() -> Result
}
#[tokio::test]
#[serial]
async fn test_notification_target_with_path_is_online_via_transport_probe() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -641,7 +638,6 @@ async fn test_notification_target_with_path_is_online_via_transport_probe() -> R
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_accepts_presigned_requests() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -682,7 +678,6 @@ async fn test_get_object_lambda_accepts_presigned_requests() -> Result<(), Box<d
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_accepts_named_webhook_target_arn() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -722,7 +717,6 @@ async fn test_get_object_lambda_accepts_named_webhook_target_arn() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_invokes_runtime_webhook_target() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -790,7 +784,6 @@ async fn test_get_object_lambda_invokes_runtime_webhook_target() -> Result<(), B
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_passthroughs_non_success_webhook_response() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -850,7 +843,6 @@ async fn test_get_object_lambda_passthroughs_non_success_webhook_response() -> R
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_rejects_success_response_without_auth_headers() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -896,7 +888,6 @@ async fn test_get_object_lambda_rejects_success_response_without_auth_headers()
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_rejects_success_response_with_mismatched_auth_headers() -> Result<(), Box<dyn Error + Send + Sync>>
{
init_logging();
@@ -943,7 +934,6 @@ async fn test_get_object_lambda_rejects_success_response_with_mismatched_auth_he
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_rejects_unsupported_target_type() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -980,7 +970,6 @@ async fn test_get_object_lambda_rejects_unsupported_target_type() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_rejects_unconfigured_target() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -1017,7 +1006,6 @@ async fn test_get_object_lambda_rejects_unconfigured_target() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_get_object_lambda_rejects_disabled_target() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -1063,7 +1051,6 @@ async fn test_get_object_lambda_rejects_disabled_target() -> Result<(), Box<dyn
}
#[tokio::test]
#[serial]
async fn test_configure_object_lambda_target_rejects_invalid_endpoint() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -1106,7 +1093,6 @@ async fn test_configure_object_lambda_target_rejects_invalid_endpoint() -> Resul
}
#[tokio::test]
#[serial]
async fn test_configure_object_lambda_notify_webhook_rejects_response_header_timeout_key()
-> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -1140,7 +1126,6 @@ async fn test_configure_object_lambda_notify_webhook_rejects_response_header_tim
}
#[tokio::test]
#[serial]
async fn test_listen_notification_emits_after_put_object() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -1184,7 +1169,6 @@ async fn test_listen_notification_emits_after_put_object() -> Result<(), Box<dyn
}
#[tokio::test]
#[serial]
async fn test_listen_notification_emits_on_empty_bucket_when_notify_disabled() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -1219,7 +1203,6 @@ async fn test_listen_notification_emits_on_empty_bucket_when_notify_disabled() -
}
#[tokio::test]
#[serial]
async fn test_listen_notification_fans_in_remote_node_events() -> Result<(), Box<dyn Error + Send + Sync>> {
init_logging();
@@ -33,7 +33,6 @@ use aws_sdk_s3::types::{
ObjectLockMode, ObjectLockRetentionMode,
};
use chrono::{DateTime, Duration, Utc};
use serial_test::serial;
use tracing::info;
/// Initialize test logging
@@ -107,7 +106,6 @@ fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc>
// ============================================================================
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_compliance_retention() {
init_logging();
info!("🧪 Test: DeleteObject blocked by COMPLIANCE retention");
@@ -145,7 +143,6 @@ async fn test_delete_object_blocked_by_compliance_retention() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_governance_without_bypass() {
init_logging();
info!("🧪 Test: DeleteObject blocked by GOVERNANCE retention without bypass");
@@ -175,7 +172,6 @@ async fn test_delete_object_blocked_by_governance_without_bypass() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_allowed_by_governance_with_bypass() {
init_logging();
info!("🧪 Test: DeleteObject allowed by GOVERNANCE retention with bypass");
@@ -215,7 +211,6 @@ async fn test_delete_object_allowed_by_governance_with_bypass() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_creates_delete_marker_for_retained_current_version() {
init_logging();
info!("🧪 Test: DeleteObject creates delete marker for retained current version");
@@ -266,7 +261,6 @@ async fn test_delete_object_creates_delete_marker_for_retained_current_version()
}
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_legal_hold() {
init_logging();
info!("🧪 Test: DeleteObject blocked by Legal Hold");
@@ -299,7 +293,6 @@ async fn test_delete_object_blocked_by_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_allowed_with_legal_hold_off() {
init_logging();
info!("🧪 Test: DeleteObject allowed with Legal Hold OFF");
@@ -335,7 +328,6 @@ async fn test_delete_object_allowed_with_legal_hold_off() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_after_legal_hold_removed() {
init_logging();
info!("🧪 Test: DeleteObject succeeds after Legal Hold is removed");
@@ -369,7 +361,6 @@ async fn test_delete_object_after_legal_hold_removed() {
}
#[tokio::test]
#[serial]
async fn test_get_object_legal_hold_returns_updated_status() {
init_logging();
info!("🧪 Test: GetObjectLegalHold returns updated status");
@@ -425,7 +416,6 @@ async fn test_get_object_legal_hold_returns_updated_status() {
}
#[tokio::test]
#[serial]
async fn test_get_object_retention_returns_configured_values() {
init_logging();
info!("🧪 Test: GetObjectRetention returns configured values");
@@ -476,7 +466,6 @@ async fn test_get_object_retention_returns_configured_values() {
// creating a new current version. The lock protects the existing version
// from deletion; it never blocks new versions.
#[tokio::test]
#[serial]
async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: PutObject overwrite of a legal-hold version creates a new version");
@@ -561,7 +550,6 @@ async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_applies_requested_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject applies requested Legal Hold");
@@ -613,7 +601,6 @@ async fn test_copy_object_applies_requested_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_does_not_inherit_source_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject does not inherit source Legal Hold");
@@ -707,7 +694,6 @@ async fn test_copy_object_does_not_inherit_source_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject overwrite of a legal-hold destination creates a new version");
@@ -787,7 +773,6 @@ async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_create_multipart_upload_applies_requested_legal_hold() {
init_logging();
info!("🧪 Test: CreateMultipartUpload applies requested Legal Hold");
@@ -853,7 +838,6 @@ async fn test_create_multipart_upload_applies_requested_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_create_multipart_upload_creates_new_version_under_compliance_retention() {
init_logging();
info!("🧪 Test: CreateMultipartUpload over a COMPLIANCE-retained key creates a new version");
@@ -933,7 +917,6 @@ async fn test_create_multipart_upload_creates_new_version_under_compliance_reten
}
#[tokio::test]
#[serial]
async fn test_delete_completed_multipart_object_blocked_by_legal_hold() {
init_logging();
info!("🧪 Test: Delete completed multipart object blocked by Legal Hold");
@@ -993,7 +976,6 @@ async fn test_delete_completed_multipart_object_blocked_by_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_delete_completed_multipart_object_blocked_by_retention() {
init_logging();
info!("🧪 Test: Delete completed multipart object blocked by retention");
@@ -1055,7 +1037,6 @@ async fn test_delete_completed_multipart_object_blocked_by_retention() {
}
#[tokio::test]
#[serial]
async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under Legal Hold");
@@ -1135,7 +1116,6 @@ async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_complete_multipart_upload_creates_new_version_under_compliance_retention() {
init_logging();
info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under COMPLIANCE retention");
@@ -1209,7 +1189,6 @@ async fn test_complete_multipart_upload_creates_new_version_under_compliance_ret
}
#[tokio::test]
#[serial]
async fn test_write_paths_require_put_object_legal_hold_permission() {
init_logging();
info!("🧪 Test: write paths require PutObjectLegalHold permission");
@@ -1273,7 +1252,6 @@ async fn test_write_paths_require_put_object_legal_hold_permission() {
}
#[tokio::test]
#[serial]
async fn test_write_paths_require_put_object_retention_permission() {
init_logging();
info!("🧪 Test: write paths require PutObjectRetention permission");
@@ -1345,7 +1323,6 @@ async fn test_write_paths_require_put_object_retention_permission() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_delete_objects_mixed_locked_unlocked() {
init_logging();
info!("🧪 Test: DeleteObjects with mixed locked and unlocked objects");
@@ -1427,7 +1404,6 @@ async fn test_delete_objects_mixed_locked_unlocked() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_put_retention_compliance_cannot_shorten() {
init_logging();
info!("🧪 Test: PutObjectRetention cannot shorten COMPLIANCE retention");
@@ -1468,7 +1444,6 @@ async fn test_put_retention_compliance_cannot_shorten() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_compliance_can_extend() {
init_logging();
info!("🧪 Test: PutObjectRetention can extend COMPLIANCE retention");
@@ -1509,7 +1484,6 @@ async fn test_put_retention_compliance_can_extend() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_governance_extend_without_bypass() {
init_logging();
info!("🧪 Test: PutObjectRetention on GOVERNANCE can extend without bypass");
@@ -1553,7 +1527,6 @@ async fn test_put_retention_governance_extend_without_bypass() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_governance_shorten_requires_bypass() {
init_logging();
info!("🧪 Test: PutObjectRetention on GOVERNANCE requires bypass to shorten");
@@ -1615,7 +1588,6 @@ async fn test_put_retention_governance_shorten_requires_bypass() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_default_retention_applied_to_new_objects() {
init_logging();
info!("🧪 Test: Default retention is applied to new objects");
@@ -1685,7 +1657,6 @@ async fn test_default_retention_applied_to_new_objects() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_creates_delete_marker_for_default_retained_current_version() {
init_logging();
info!("🧪 Test: DeleteObject creates delete marker for default-retained current version");
@@ -1770,7 +1741,6 @@ async fn test_delete_object_creates_delete_marker_for_default_retained_current_v
}
#[tokio::test]
#[serial]
async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
init_logging();
info!("🧪 Test: write paths reject incomplete Object Lock retention headers");
@@ -1869,7 +1839,6 @@ async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_retention_uses_destination_policy() {
init_logging();
info!("🧪 Test: CopyObject retention follows destination policy");
@@ -2051,7 +2020,6 @@ async fn test_copy_object_retention_uses_destination_policy() {
}
#[tokio::test]
#[serial]
async fn test_multipart_default_retention_fixed_at_create() {
init_logging();
info!("🧪 Test: multipart default retention is fixed at CreateMultipartUpload");
@@ -2122,7 +2090,6 @@ async fn test_multipart_default_retention_fixed_at_create() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
init_logging();
info!("🧪 Test: Unretained Object Lock object delete and bucket cleanup (Issue #5339)");
@@ -2243,7 +2210,6 @@ async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
}
#[tokio::test]
#[serial]
async fn test_versioning_auto_enabled_with_object_lock() {
init_logging();
info!("🧪 Test: Versioning is auto-enabled when Object Lock is configured");
@@ -2302,7 +2268,6 @@ async fn test_versioning_auto_enabled_with_object_lock() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_error_message_distinguishes_legal_hold_from_retention() {
init_logging();
info!("🧪 Test: Error messages distinguish Legal Hold from Retention");
@@ -13,7 +13,6 @@
// limitations under the License.
use aws_sdk_s3::primitives::ByteStream;
use serial_test::serial;
use std::path::{Path, PathBuf};
use uuid::Uuid;
@@ -24,7 +23,6 @@ const TEST_OBJECT: &str = "large-object.bin";
const PAYLOAD_SIZE: usize = 512 * 1024;
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn unversioned_overwrite_removes_previous_physical_data_dir() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -43,7 +43,6 @@ use aws_sdk_s3::presigning::{PresignedRequest, PresigningConfig};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use serial_test::serial;
use std::time::{Duration, SystemTime};
use tracing::info;
@@ -157,7 +156,6 @@ async fn setup(env: &mut RustFSTestEnvironment) -> Result<(), Box<dyn std::error
/// stored bytes. Without this, every negative assertion could pass for the
/// wrong reason (a server that rejects all presigned URLs).
#[tokio::test]
#[serial]
async fn valid_presigned_get_succeeds() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -182,7 +180,6 @@ async fn valid_presigned_get_succeeds() -> Result<(), Box<dyn std::error::Error
/// Positive control (PUT): a valid presigned PUT must store the object, which we
/// verify with a follow-up authenticated HEAD.
#[tokio::test]
#[serial]
async fn valid_presigned_put_succeeds() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -211,7 +208,6 @@ async fn valid_presigned_put_succeeds() -> Result<(), Box<dyn std::error::Error
/// ("Request has expired"). s3s checks expiry BEFORE the signature, so the
/// signature here is otherwise valid — only the elapsed window is at fault.
#[tokio::test]
#[serial]
async fn expired_presigned_get_is_rejected() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -236,7 +232,6 @@ async fn expired_presigned_get_is_rejected() -> Result<(), Box<dyn std::error::E
/// (b) Tampering the `X-Amz-Signature` query value must be rejected with 403 /
/// SignatureDoesNotMatch.
#[tokio::test]
#[serial]
async fn tampered_signature_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -262,7 +257,6 @@ async fn tampered_signature_returns_signature_does_not_match() -> Result<(), Box
/// (c) A presigned URL generated with the WRONG secret (but the real access key
/// id) must be rejected with 403 / SignatureDoesNotMatch.
#[tokio::test]
#[serial]
async fn wrong_secret_key_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -290,7 +284,6 @@ async fn wrong_secret_key_returns_signature_does_not_match() -> Result<(), Box<d
/// check runs during auth, before any object lookup, so the swapped key need
/// not even exist.
#[tokio::test]
#[serial]
async fn tampered_target_key_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -325,7 +318,6 @@ async fn tampered_target_key_returns_signature_does_not_match() -> Result<(), Bo
/// (e / acceptance 4 negative half) Tampering the signature of a presigned PUT
/// must be rejected with 403 / SignatureDoesNotMatch — the write must not land.
#[tokio::test]
#[serial]
async fn tampered_presigned_put_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
@@ -22,7 +22,6 @@ use crate::protocols::sftp_compliance::{
};
use crate::protocols::sftp_core::{test_sftp_core_operations, test_sftp_idle_timeout_disconnects};
use crate::protocols::webdav_core::test_webdav_core_operations;
use serial_test::serial;
use std::time::Instant;
use tokio::time::{Duration, sleep};
use tracing::{error, info};
@@ -229,7 +228,6 @@ fn all_protocol_tests() -> Vec<TestDefinition> {
/// Test suite
#[tokio::test]
#[serial]
async fn test_protocol_core_suite() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let suite = ProtocolTestSuite::new();
let results = suite.run_test_suite().await;
-15
View File
@@ -15,7 +15,6 @@
use crate::common::{RustFSTestEnvironment, admin_request, awscurl_delete, awscurl_get, awscurl_post, awscurl_put, init_logging};
use aws_sdk_s3::Client;
use http::{Method, StatusCode};
use serial_test::serial;
use tokio::time::{Duration, sleep, timeout};
use tracing::{debug, info};
@@ -255,7 +254,6 @@ mod integration_tests {
use aws_sdk_s3::error::ProvideErrorMetadata;
#[tokio::test]
#[serial]
async fn test_quota_basic_operations() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -300,7 +298,6 @@ mod integration_tests {
/// with 400 UnexpectedContent, and an over-quota aws-chunked PUT must still get the quota
/// rejection.
#[tokio::test]
#[serial]
async fn test_quota_admission_aws_chunked_declared_encoding() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -352,7 +349,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_update_and_clear() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -388,7 +384,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_delete_operations() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -425,7 +420,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_usage_tracking() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -464,7 +458,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_statistics() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -498,7 +491,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_check_api() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -539,7 +531,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_multiple_buckets() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -580,7 +571,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_error_handling() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -616,7 +606,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_http_endpoints() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -682,7 +671,6 @@ mod integration_tests {
/// Test that a normal user with `readwrite` policy can read quota but cannot set/clear quota.
#[tokio::test]
#[serial]
async fn test_quota_normal_user_permissions() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -738,7 +726,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_copy_operations() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -784,7 +771,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_batch_delete() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {
@@ -843,7 +829,6 @@ mod integration_tests {
}
#[tokio::test]
#[serial]
async fn test_quota_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
if skip_without_awscurl() {

Some files were not shown because too many files have changed in this diff Show More